In most cases, applications are submitted electronically, where systems enforce validations such as page limits. In a small number of cases (in which applications are submitted on paper), NIH staff manually validate page limits. Applications that do not comply are either rejected during electronic submission or withdrawn from consideration. For application sections without page limits (such as the Protection of Human Subjects and the Vertebrate Animals section), applicants are advised not to include inappropriate content, and not to use the Appendix to circumvent page limits. NIH may withdraw applications from review and funding consideration in such cases. Notice NOT-OD-11-080 provides more details on the policy.
Only a single resubmission (A1) of an original application (A0) will be accepted.
Following an unsuccessful resubmission (A1) application, applicants may submit the same idea as a new (A0) application for the next appropriate new application due date (see NOT-OD-18-197 for exceptions).
Reviewers are instructed to evaluate the resubmission application as presented, taking into consideration the responses to comments from the previous scientific review group and changes made to the project. For resubmitted renewals, the committee will also consider the progress made in the last funding period.
For research grants and cooperative agreements, the five scored review criteria are Significance, Investigator(s), Innovation, Approach, and Environment. For a full overview of the scored review criteria, and additional review criteria and considerations for many funding mechanisms, visit the Review Criteria at a Glance document.
If it is the opinion of the reviewers that the project is not likely to answer the questions it poses, then overall impact is likely to be low. The degree of uncertainty about feasibility will likely determine whether this is rated as a minor, moderate or major weakness.
Significance is a stand-alone assessment of the project’s goals in the context of the relevant field, and to a large extent assumes that the investigator(s), approach and environment are adequate to allow for successful completion of the aims of the project even if later discussion of each of these review criteria will identify problems. When reviewers assess the Overall Impact of an application they are expected to take into account the scored review criteria (e.g., significance, investigator(s), innovation, approach and environment) and the additional review criteria to judge the potential of the project to exert a sustained, powerful influence on the field. For more information, visit the Overall Impact versus Significance document.
The Overall Impact score is a synthesis that takes into consideration all of the scored review criteria (i.e., for research applications: significance, investigator(s), innovation, approach and environment) as well as all of the applicable additional review criteria.
Not necessarily. The Overall Impact score considers all scored review criteria as well as all applicable additional review criteria. In addition, an application does not need to be strong in all scored review criteria to be judged likely to have a major scientific impact. Therefore, it is possible for one or more review criteria to overshadow the other review criteria, thus driving the Overall Impact score up or down. Please remember that there is no formula to derive the overall impact score from the individual criterion scores. Reviewers are instructed to weigh the different criteria as appropriate for each application in deriving the Overall Impact score.
One can envision such scenarios. For example, a talented investigator in a very strong environment proposes a highly innovative and very sound approach to address a generally important problem (e.g., breast cancer). However, the proposed project will be relevant to only a narrow area within the larger field of breast cancer research, thus reducing its Significance. Nevertheless, the Overall Impact score could still be strong since the strengths of the project in the other core review criteria give this work the potential to have a sustained, powerful influence on that part of this important field.
Yes. The Overall Impact score synthesizes all scored review criteria as well as all applicable additional review criteria. Thus, while the significance of the project is very strong, the investigator might lack key credentials, the innovation might be minimal, the approach might be problematic, and the environment might not offer adequate support for the project.
Yes. No single review criterion (e.g., Approach) alone determines the Overall Impact score. A project may have numerous minor weaknesses that affect the score for Approach, yet still have a very strong Overall Impact score if the application is exceptionally strong in the other review criteria and the quality of the team and environment lend confidence that the project will have a major overall impact on the field. “Minor weaknesses” are defined as “addressable weaknesses that do not substantially lessen overall impact.”
Not necessarily. The Significance score reflects whether a project addresses an important problem or critical barrier to progress within the field. For example, while a project may generally address a devastating disease with high prevalence, the specific problem addressed in the project may be only tangentially related to the disease, the problem may not be very important for patients with the disease, the proposed work may duplicate already published reports, or the expected results may be unlikely to substantially change knowledge, concepts and/or practice in the field.
No. Work on rare diseases, highly prevalent diseases with modest burden, and highly focused research questions can still be extremely important. Reviewers should judge whether the proposed goals and aims address an important problem or critical barrier to progress in the field, whether the proposed work will improve scientific knowledge, technical capability, and/or clinical practice in the field, or if the project will change the concepts, methods, technologies, treatments, services, or preventative interventions that drive the field.
Not necessarily. The verbal descriptors and additional guidance on Strengths/Weaknesses are offered only in an effort to provide continuity in scoring among study sections. However, reviewers must not just “count” strengths and weaknesses to arrive at the final Overall Impact score. Remember that the Overall Impact score represents a synthesis of all of the review criteria and what may be a moderate weakness to one reviewer may be only a very minor weakness to another reviewer. For example, a very strong investigator in a very strong environment is addressing an extremely important problem. The approach is very innovative and could revolutionize the field. However, because the approach is so new and has not been tried before, it is not guaranteed that the project’s goals will be met and some reviewers see this as a weakness. However, in the end, the study section concludes that the potential rewards far outweigh the risk(s) associated with the approach and that the application deserves a high Overall Impact score.
The mission of the NIH is to support research in pursuit of knowledge about the biology and behavior of living systems and to apply that knowledge to extend healthy life and reduce illness and disability. To accomplish this mission, the NIH supports biomedical and behavioral research representing a wide array of research fields as well as tool development, clinical trials and other projects in support of the biomedical research enterprise. In an effort to fairly evaluate scientific and technical merit through the peer review system of a broad range of applications (those that seek cures, not only for diabetes, heart disease, and autism, but also for the lesser recognized orphan diseases and those that ask basic biomedical questions), it is important that Significance and Overall Impact be evaluated within the context of the research field involved. NIH program staff and Institute leadership will evaluate each project’s relevance to their Institute mission in making funding decisions.
FOAs for IC-specific RFAs and targeted trans-NIH initiatives (e.g., Roadmap or Common Fund), including infrastructure and capacity building programs, may include other FOA-specific review criteria in addition to the scored review criteria and the standard NIH additional review criteria (human subjects, animal welfare, renewal, resubmission, biohazards). The Overall Impact score for applications submitted for these initiatives should reflect the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the scored review criteria as well as all additional review criteria (as applicable for the project proposed) and the likelihood that the project will advance the stated goals and objectives of the program as articulated in the FOA.
For fellowship applications (Fs), the overall impact score should reflect the reviewers’ assessment of the likelihood that the fellowship will enhance the candidate’s potential for, and commitment to, a productive independent scientific career in a health-related field, in consideration of the scored criteria (i.e., Fellowship Applicant, Sponsors/ Collaborators/Consultants, Research Training Plan, Training Potential and Institutional Environment & Commitment to Training) as well as all applicable additional review criteria.
For career development award applications (Ks) the overall impact score reflects the reviewers’ assessment of the likelihood for the candidate to maintain a strong research program, in consideration of the scored review criteria (i.e., Candidate, Career Development Plan/Career Goals & Objectives/Plan to Provide Mentoring, Research Plan, Mentor(s)/Consultant(s)/Collaborator(s), Environment and Institutional Commitment to the Candidate) as well as all applicable additional review criteria.
For institutional training grant applications (Ts), reviewers are asked to provide an overall impact score to reflect their assessment of the likelihood for the proposed training program to promote the training of pre- and postdoctoral fellows in biomedical, behavioral and clinical research, in consideration of the scored review criteria (i.e., Training Program and Environment, Training Program Director/Principal Investigator, Preceptors/Mentors, Trainees and Training Record) as well as all applicable additional review criteria.
For shared instrumentation applications (S10s), the overall impact/benefit score reflects the reviewers’ assessment of the potential benefit of the instrument requested for the overall research community and on NIH-funded research in consideration of the scored review criteria (i.e., Justification of Need, Technical Expertise, Research Projects, Administration, Institutional Commitment) as well as all applicable additional review criteria.
The Overall Impact paragraph provides the reviewer with the opportunity of explaining how the Overall Impact score was derived (i.e., those factors that contributed to the score). If a project has a strong/weak overall impact score then the reviewer should highlight those scored criteria that contributed to the favorable/poor score. For example, if the potential significance of a study was so great as to overshadow a number of methodological weaknesses then this should be clearly stated. Likewise, if the design of the study is so flawed as to negate any potential significance and/or innovation of the study then this should be clearly stated. Importantly, the Overall Impact paragraph should provide a clear view into the reviewer’s thought process that led to his/her Overall Impact score. It is not intended to simply summarize and/or restate the strengths and weakness detailed in the critique.
The current scoring scale for individual reviewers’ scores uses a 9-point scale, with a score of 1 indicating an exceptionally strong application with essentially no weaknesses. A score of 9 indicates an application with serious and substantive weaknesses with very few strengths; 5 is considered an average score. Ratings are in whole numbers only (no decimal ratings). This scale is used by all eligible (without conflict of interest) Scientific Review Group members to provide an overall impact score and for assigned reviewers to score review criteria (e.g., Significance, Investigator(s), Innovation, Approach, and Environment for research applications). For more information, visit the Scoring System and Procedure document.
Assigned reviewers provide a preliminary overall impact score for each application. They also provide criterion scores for each of the scored review criteria. At the review meeting, each eligible review group member provides a final overall impact score for each discussed application. The overall impact score that appears in the summary statement for each discussed application is calculated by averaging the final overall impact scores given by all eligible members, multiplying by ten and rounding to the nearest whole number (i.e., 10 to 10.49 = 10; 10.5 to 11.49 = 11).
The scores are rounded to prevent the appearance of a precision that does not exist. Tie scores indicate that the tied applications cannot be reliably distinguished from each other based solely on scientific and technical merit.
Only the percentiles are rounded up. The overall impact scores are rounded traditionally, making the range 10 to 90. Getting a score of 10 would not necessarily require total unanimity. For example, if 24 reviewers scored 1 and one reviewer scored 2, the average would be 1.04 (26/25). Multiplication by 10 yields 10.4, which would be rounded to 10 rather than 11.
Percentile bases are determined for standing study sections, but no separate percentile base exists for ESIs, NIs, or established investigators. A CSR-all base is often used for percentiling applications reviewed by Special Emphasis Panels (SEPs), or in some instances, Institutes/Centers may have their own base for percentiling SEPs reviewed in their respective Institute/Center.
The 1 - 9 scale is used to assign the individual criterion scores, the preliminary impact/priority score and the final overall impact score. It is not used for criteria that are only rated "acceptable" or "unacceptable" such as protection of human subjects.
No. Stanines impose a specific distribution of scores based on pre-determined standard deviations from the mean. The NIH scoring scale does not impose such a distribution. Percentiling is used to achieve comparability of scores across review groups.
All applications – whether they have been discussed in the review meeting or not – receive criterion scores. Reviewers use the criterion scores to help them determine the overall impact score, but the criterion scores are not expected to be weighted equally. It is up to each reviewer to determine an overall impact score that best describes the likely overall impact that each application will have. Such determinations reflect each reviewer’s best estimate of how much impact the application will have, given the importance of the questions being asked and the likelihood that the project will succeed, given the combination of investigators, approach and environment that are described in the application.
The review panel does NOT vote on each criterion. However, during the discussion of an application, review committee members could choose to discuss individual criterion scores to make sure they understand the points raised by an assigned reviewer. It is important to understand that criterion scores are intended to be pieces of information to help understand each reviewer’s evaluation, but are not intended to be the sole basis of how to decide on an overall impact score.
In some cases, an SRO may wish to bring in a reviewer who is a content expert on a particular part of an application. For example, a statistician might be used as a reviewer for an application that relies heavily on the use of statistics to analyze a biomedical issue. In this case, it would be inappropriate for the statistician to score parts of the application for which he/she does not have expertise.
For this reason, reviewers are free to leave sections of the critique template empty of comments if they do not apply for his/her evaluation. In addition, entering criterion scores into eRA systems is not mandatory. However, if a criterion score is entered, a critique must be uploaded.
The scores that are in the table in the summary statement are accepted as final. Reviewers are instructed NOT to enter scores with their critiques and that errant scores in the critiques may be removed in finalizing the summary statement.
No, reviewers download a Word file that contains the critique template for the mechanism they are reviewing. Many standard review critique templates are available on the Guidelines and Fill-able Templates for Reviewers page. Once reviewers fill out the template in Word, they upload the entire file into IAR.
The hyperlinks that are associated with each review criterion only work with Microsoft Word 2007 or later versions. These hyperlinks all point to the same Web site, but bring the user to different “anchor” points on the page that correspond to a particular review criterion or consideration. For users with Microsoft 2003, the hyperlinks associated with each review criterion do not work, but an accessible hyperlink is provided at the top of the template that links to the same Web site, so that the same information is available.
Each resubmission, renewal or revision is evaluated usingstandard criteria, plus additional criteria that go beyond that of a new grant application. See the enhanced review criteria for specifics on each of these criteria.
IAR accepts scores for only 5 criteria. Any additional scored criteria have to be added into the text of the critique and added manually on the summary statement. Please contact your SRO (Scientific Review Officer) for details.
The post-submission materials policy states “the only post-submission grant application materials that the NIH will accept are those resulting from unforeseen administrative issues”, and lists the following acceptable post-submission materials:
Revised budget page(s) (e.g., change in budget request due to new funding or institutional acquisition of equipment)
Biographical sketches (e.g., change in senior/key personnel due to the hiring, replacement, or loss of an investigator)
Letters of support or collaboration resulting from a change in senior/key personnel due to the hiring, replacement, or loss of an investigator
Adjustments resulting from natural disasters (e.g., loss of an animal colony)
Adjustments resulting from change of institution (e.g., PI moves to another university)
News of an article accepted for publication (a copy of the article should not be sent)
An investigator may submit any number of such items, but must follow the page limits specified in the policy. That is, for post-submission materials that are not required on a form page, and are not covered by one of the exceptions such as an RFA or institutional training mechanism, each explanation or letter is limited to one page. Therefore, if a research team lost a member after the application was submitted, and the PD/PI wanted to replace that individual with two substitute personnel, he could submit a one-page explanation and biographical sketch for each new person, plus a revised budget page(s).
NIH will accept news of all articles that were accepted for publication after the application was submitted and are relevant to the proposed project. News of an article accepted for publication is considered another category of acceptable post-submission materials, separate from information being submitted from unforeseen administrative issues.) For each article accepted, you may submit only the following: Authors, institutional affiliations, title of the article, the journal that accepted it, and the expected time of publication.
The purpose of the policy is to provide an even playing field across the agency, types of applications, and study sections in terms of when the reviewers have access to information and in terms of the types of information they receive. The thirty-day window before a review meeting is time when reviewers are concentrating on the applications and their critiques, and SROs are making preparations for the meetings. In most cases, reviewers are asked to submit their preliminary critiques, criterion scores, and impact scores a week before the meeting.
In most cases NIH expects that news of an article accepted for publication will be in reference to an article authored by the investigators submitting the grant application, but news of an article authored by other investigators could be accepted if the SRO determines that it directly affects the work proposed in the application.
In most cases, the time from submission to the review meeting date is two-three months, and reviewers gain access to the applications five-six weeks before the review meeting date. Therefore, only a few weeks are left in that window in which new data could be gathered, analyzed, and submitted. For those mechanisms in which the time from submission to the review meeting date may be longer, such as applications submitted in response to Requests for Applications, or institutional training grant applications, the policy provides exceptions to accommodate this additional time.
No. The status of IND and IDE exemptions will be assessed at the time of award, if an application is considered for funding. Therefore, such updates should be sent to the Program Officer assigned to the application.
No. Budget considerations are not typically included in assessing the scientific and technical merit of the proposed work, but can be negotiated at the time of award. Therefore, an applicant who learns of local underwriting efforts should notify the Program Officer for the application, but not the SRO.
No, the PHS398 instructions for applicants state that PDF images of material such as electron micrographs or gels may be included in the Appendix; however, a photocopy of each must also be included within the page limits of the Research Strategy. Therefore, applicants should include in the Appendix original copies of figures that may not reproduce well if scanned.
An applicant getting ready to submit a fellowship or career development application should check the eRA Commons listing for the application a week or so in advance, and check with the individuals who have agreed to serve as referees. If they are unable to meet the deadline, or cannot be reached, the applicant should consider asking another individual to submit a Reference Form (F) or letter (K) instead.
Yes. The mentor/sponsor may notify the SRO that the notice of grant award was received, and the SRO may accept that information, but it must adhere to the one-page limit for transmitting that information.
Yes. The applicant/candidate may notify the SRO that the notice of grant award was received, and the SRO may accept that information, but it must adhere to the one-page limit for transmitting that information. However, note that for certain mechanisms, receipt of another NIH award, e.g., as a PD/PI on an R03, renders the applicant/candidate no longer eligible for the fellowship or award, and the application would be withdrawn.
The Project Director(s)/Principal Investigator(s) [PD(s)/PI(s)] and the members of the organizing committee for the conference or meeting are the Senior/Key Personnel for a conference or meeting grant application.
The only post-submission grant application materials that the NIH will accept are those resulting from unforeseen administrative issues. Post-submission grant application materials are those submitted after submission of the grant application but prior to the initial peer review. This option is to be used when an unexpected event such as the departure of a Senior/Key Person, natural disaster, etc. has occurred, not to correct oversights/errors discovered after submission of the application. For conference or meeting grant applications, acceptable post-submission materials include:
Revised budget page(s) (e.g., change in budget request due to new funding or sources of support, changes in rental or supply charges, or speaker fees)
Biographical sketches due to hiring, replacement, or loss of Senior/Key Personnel
Letters of agreement resulting from the hiring, replacement, or loss of Senior/Key Personnel
Adjustments to logistical arrangements resulting from natural disasters (e.g., loss of meeting facility or family care resources)
Adjustments resulting from change of institution (e.g., PD/PI moves to another organization)
News of key participants accepting or declining invitations to participate in the conference or meeting. The PD/PI may submit a one-page list of invitations declined or accepted by key participants, or a one-page explanation of changes needed to the meeting agenda due to those declined or accepted invitations.