Peer Review Process

Photo of Obama

"To maintain our edge . . . we've got to protect our rigorous peer review system and ensure that we only fund proposals that promise the biggest bang for taxpayer dollars . . . that's what's going to maintain our standards of scientific excellence for years to come."

Remarks by President Barack Obama on the 150th Anniversary of the National Academy of Sciences, April 29, 2013
On This Page:
 

 

Overview

The core values of peer review drive the NIH to seek the highest level of ethical standards, and form the foundation for the laws, regulations, and policies that govern the NIH peer review process. The NIH dual peer review system is mandated by statute in accordance with section 492 of the Public Health Service Act and federal regulations governing "Scientific Peer Review of Research Grant Applications and Research and Development Contract Projects" (42 CFR Part 52h). NIH policy is intended to promote a process whereby grant applications submitted to the NIH are evaluated on the basis of a process that is fair, equitable, timely, and free of bias.

The first level of review is carried out by a Scientific Review Group (SRG) composed primarily of non-federal scientists who have expertise in relevant scientific disciplines and current research areas. The second level of review is performed by Institute and Center (IC) National Advisory Councils or Boards. Councils are composed of both scientific and public representatives chosen for their expertise, interest, or activity in matters related to health and disease. Only applications that are recommended for approval by both the SRG and the Advisory Council may be recommended for funding.  Final funding decisions are made by the IC Directors.

For more information see NIH Peer Review (PDF)
 

Back to Top

 
First Level of Review

Initial peer review meetings are administered by either the Center for Scientific Review (CSR) or another NIH IC. The focus of review is specified in the Funding Opportunity Announcement. Peer review meetings are announced in the Federal Register. The meetings are closed to the public, although some meetings may have an open session; the Federal Register provides the details of each meeting.

A.  Peer Review Roles and Meeting Overview

Scientific Review Officer:

Each SRG is led by a Scientific Review Officer (SRO). The SRO is an extramural staff scientist and the Designated Federal Official responsible for ensuring that each application receives an objective and fair initial peer review, and that all applicable laws, regulations, and policies are followed.

SROs:

  • Analyze the content of each application, and check for completeness.
  • Document and manage conflicts of interest.
  • Recruit qualified reviewers based on scientific and technical qualifications and other considerations, including:
    1. Authority in their scientific field (42 CFR 52h.4)
    2. Dedication to high quality, fair, and objective reviews
    3. Ability to work collegially in a group setting
    4. Experience in research grant review
    5. Balanced representation
  • Assign applications to reviewers for critique preparation and assignment of individual criterion scores.
  • Attend and oversee administrative and regulatory aspects of peer review meetings.
  • Prepare summary statements for all applications reviewed.

SRG Members

Chair:

  • Serves as moderator of the discussion of scientific and technical merit of the applications under review.
  • Is also a peer reviewer for the meeting.

Reviewers:

Other NIH Staff

  • Federal officials who have need-to-know or pertinent related responsibilities are permitted to attend closed review meetings.
  • NIH IC or other federal staff members wishing to attend an SRG meeting must have advance approval from the responsible SRO. These individuals may provide programmatic or grants management input at the SRO's discretion.

  Peer Review Meeting Procedures

  • Applications are reviewed based on established review criteria (see Review Criteria at a Glance).
  • Assigned reviewers summarize their prepared critiques for the group.
  • An open discussion follows.
  • Final scoring of overall impact scores is conducted by private ballot.

B. Peer Review Criteria and Considerations

Review Criteria for Research Grants and Cooperative Agreements

The mission of the NIH is to support science in pursuit of knowledge about the biology and behavior of living systems and to apply that knowledge to extend healthy life and reduce illness and disability. As part of this mission, applications submitted to the NIH for grants or cooperative agreements to support biomedical and behavioral research are evaluated for scientific and technical merit through the NIH peer review system.

Overall Impact. Reviewers will provide an overall impact/priority score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following review criteria, and additional review criteria (as applicable for the project proposed).

Scored Review Criteria. Reviewers will consider each of the review criteria below in the determination of scientific and technical merit, and give a separate score for each. An application does not need to be strong in all categories to be judged likely to have major scientific impact. For example, a project that by its nature is not innovative may be essential to advance a field.

Significance. Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

Investigator(s). Are the PD/PIs, collaborators, and other researchers well suited to the project? If Early Stage Investigators or New Investigators, or in the early stages of independent careers, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-PD/PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project?

Innovation. Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?

Approach. Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed? If the project involves clinical research, are the plans for 1) protection of human subjects from research risks, and 2) inclusion of minorities and members of both sexes/genders, as well as the inclusion of children, justified in terms of the scientific goals and research strategy proposed?

Environment. Will the scientific environment in which the work will be done contribute to the probability of success? Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?

Additional Review Criteria. As applicable for the project proposed, reviewers will evaluate the following additional items while determining scientific and technical merit and in providing an overall impact/priority score, but will not give separate scores for these items.

Protections for Human Subjects
Inclusion of Women, Minorities, and Children
Vertebrate Animals
Biohazards
Resubmission
Renewal
Revision

Additional Review Considerations. As applicable for the project proposed, reviewers will consider each of the following items, but will not give scores for these items and should not consider them in providing an overall impact/priority score.

Applications from Foreign Organizations
Select Agent
Resource Sharing Plans
Budget and Period Support

C.  Scoring

The NIH utilizes a 9-point rating scale (1 = exceptional; 9 = poor) for all applications; the same scale is used for overall impact scores and for criterion scores. (NOT-OD-09-024)

Before the SRG meeting, each reviewer and discussant assigned to an application gives a separate score for each of (at least) five review criteria (i.e., Significance, Investigator(s), Innovation, Approach, and Environment for research grants and cooperative agreements; see above). For all applications the individual scores of the assigned reviewers and discussant(s) for these criteria are reported to the applicant.

In addition, each reviewer and discussant assigned to an application gives a preliminary overall impact score for that application. The preliminary scores are used to determine which applications will be discussed in full at the meeting. For each application that is discussed at the meeting, a final impact score is given by each eligible committee member (without conflicts of interest) including the assigned reviewers. Each member's score reflects his/her evaluation of the overall impact that the project is likely to have on the research field(s) involved, rather than being a calculation of the reviewer's scores for each criterion.

The final overall impact score for each discussed application is determined by calculating the mean score from all the eligible members' impact scores, and multiplying the average by 10; the final overall impact score is reported on the summary statement. Thus, the final overall impact scores range from 10 (high impact) through 90 (low impact). Numerical impact scores are not reported for applications that are not discussed (ND), which may be reported as ++ on the face page of the summary statement and typically rank in the bottom half of the applications.

Applicants just receiving their scores or summary statements, should consult our What's Next page for detailed guidance.  Applicants seeking advice beyond that available online may want to contact the NIH Program Officer listed at the top of the summary statement.

An application may be designated Not Recommended for Further Consideration (NRFC) by the Scientific Review Group if it lacks significant and substantial merit; presents serious ethical problems in the protection of human subjects from research risks; or presents serious ethical problems in the use of vertebrate animals, biohazards, and/or select agents. Applications designated as NRFC do not proceed to the second level of peer review (National Advisory Council/Board) because they cannot be funded.

 


 

D.  Summary Statement

Applications that are not discussed at the meeting will be given the designation "ND" as an overall impact score, but the applicant, as well as NIH staff, will see the scores from the assigned reviewers and discussants for each of the scored review criteria as additional feedback on their summary statement.

Understanding the Percentile

  • A percentile is the approximate percentage of applications that received a better overall impact/priority score from the study section during the past year.
  • For applications reviewed in ad hoc study sections, a different base may be used to calculate percentiles.
  • All percentiles are reported as whole numbers
  • Only a subset of all applications receive percentiles. The types of applications that are percentiled vary across different NIH Institutes and Centers.
  • The summary statement will identify the base that was used to determine the percentile.

E.  Appeals

NIH established a peer review appeal system (see NOT-OD-11-064) to provide investigators and applicant organizations the opportunity to seek reconsideration of the initial review results if, after consideration of the summary statement, they believe the review process was flawed as outlined below.  This policy does not apply to appeals of the technical evaluation of R&D contract projects through the NIH peer review process, appeals of NIH funding decisions, or appeals of decisions concerning extensions of MERIT award. 

An appeal is a written communication from a Project Director/Principal Investigator (PD/PI) and/or official of the applicant institution that is received after issuance of the summary statement and up to 30 calendar days after the second level of peer review, and describes a flaw in the review process for a particular application. It must display concurrence of the Authorized Organization Representative (AOR). An appeal letter will be accepted only if the letter 1) describes a flaw(s) or perceived flaw(s) in the review process for the application in question, 2) explains the reasons for the appeal, and 3) is based on one or more of the following issues related to the process of the initial peer review:

  • Evidence of bias on the part of one or more peer reviewers
  • Conflict of interest, as specified in regulation at 42 CFR 52h  “Scientific Peer Review of Research Grant Applications and Research and Development Contract Projects”, on the part of one or more non-federal peer reviewers
  • Lack of appropriate expertise within the SRG
  • Factual error(s) made by one or more reviewers that could have altered the outcome of review substantially.

Appeal letters based solely on differences of scientific opinion will not be accepted.  A letter that does not meet these criteria and/or does not include the concurrence of the AOR will not be considered an appeal, but rather a grievance.  The IC will handle grievances according to IC-specific procedures.

The IC cannot deny the PD/PI and/or the applicant institution the opportunity to have an appeal letter made available to Council, but the IC may determine which appeal letters warrant discussion by the Council members, and Council members may raise certain ones for discussion if they so choose.  The Council may concur:

  • with the appeal, and recommend that the application be re-reviewed.
  • with the SRG’s recommendation and deny the appeal. 

The recommendation of Council concerning resolution of an appeal is final and will not be considered again by the NIH through this or another process.

Back to Top

 
Second Level of Review - Advisory Council or Board

Who Reviews the Application?

The Advisory Council/Board of the potential awarding IC performs the second level of review. Advisory Councils/Boards are composed of scientists from the extramural research community and public representatives ( NIH Federal Advisory Committee Information ). Members are chosen by the respective IC and are approved by the Department of Health and Human Services. For certain committees, members are appointed by the President of the United States.

Recommendation Process

  • NIH program staff members examine applications, their overall impact scores, percentile rankings (if applicable) and their summary statements and consider these against the IC's needs.
  • Program staff provide a grant-funding plan to the Advisory Board/Council.
  • Beginning in September 2012, Council members receive a list of competing applications that will be considered for funding from PD/PIs that meet the threshold for Special Council Review. These are investigators who currently receive $1 million or more in direct costs of NIH funding to support Research Project Grants (see NOT-OD-12-140). Council members will be asked to recommend consideration of funding for applications that afford a unique opportunity to advance research which is both highly promising and distinct from the other funded projects from the PD/PI. This does not represent a cap to NIH funding.
  • The Advisory Board/Council also considers the IC’s goals and needs and advises the IC director.
  • The IC director makes final funding decisions based on staff and Advisory Council/Board advice.

Back to Top

 
Post-Review

Not Funded - Next Steps?

The NIH receives thousands of applications for each application receipt round. Funding on the first attempt is difficult, but not impossible. If an application does not result in funding, there may be an opportunity to respond to the reviewers’ comments and resubmit the application.  Applicants just receiving their summary statements, should consult our Next Steps page for detailed guidance.  Applicants seeking advice beyond that available online may want to contact the NIH Program Officer listed at the top of the summary statement. 

Fundable Score - Next Steps?

Some of the ICs publish paylines as part of their funding strategies to guide applicants on their likelihood of receiving funding.  Application scores can only be compared against the payline for the fiscal year when the application will be considered for funding, which is  not necessarily the year when it was submitted.  There may be a delay of several months to determine paylines at the beginning of fiscal years.   If the application is assigned to an IC that does not announce a payline, the Program Officer listed at the top of the summary statement may be able to provide guidance on the likelihood of funding. After the Advisory Council meeting, if an application results in an award, the applicant will be working closely with the program officer  of the funding Institute or Center on scientific and programmatic matters and a Grants Management Officer on budgetary or administrative issues. The Grants Management Specialist will contact the applicant to collect information needed to prepare the award. 
 

Back to Top

 
More Details

For more details about Peer Review, visit Peer Review Policies & Practices.

Related Resources
 

Guides, Tips, and Tutorials:

 

<--- Previous Topic | Next Topic --->