A Proposed Simplified Framework for NIH Peer Review Criteria

An overview of a proposal intended to help focus reviewers on the review of scientific merit and reduce bias.

Synopsis of Issue

NIH is proposing a revised, simplified framework for peer review that will reorganize the major regulatory criteria (42 C.F.R. Part 52h.8) under three factors and reduce the number of non-score driving review considerations that peer reviewers evaluate in research project grant (RPG) applications during the first stage of review. NIH believes that these changes will allow peer reviewers to refocus on the critical task of assessing scientific merit and will improve those assessments by reducing bias.

The Current Process

The first stage of NIH peer review serves to provide expert advice to NIH on the scientific and technical merit of grant applications. Applications are evaluated based on five scored criteria: Significance, Investigators, Innovation, Approach, and Environment (see complete definitions). Assessments of all five criteria contribute to the overall impact score and each of the five receive a criterion score from assigned reviewers.

Proposed Simplified Review Framework

NIH proposes to reorganize the five regulatory review criteria into three factors.

The Overall Impact Score (scored 1-9) reflects the overall scientific and technical merit of the application; all three factors will be considered in arriving at the Overall Impact Score.

Factor 1: Importance of the Research (scored 1-9)

Factor 1 is based on the criteria Significance and Innovation.

Factor 2: Feasibility and Rigor (scored 1-9)

Factor 2 is based on the criterion Approach.

Factor 3: Expertise and Resources (evaluated, but not individually scored)

Factor 3 is based on the criteria Investigator and Environment.

Reviewers would rate these as “fully capable” (Investigator), or “appropriate” (Environment) or, if gaps are identified, as “additional expertise/capability needed” (Investigator) or “additional resources needed” (Environment). If either of the latter two are selected, a narrative explanation of the gaps would be required.

The 3-factor structure is intended to focus the evaluation of scientific merit on key questions—how important the research is and whether it is both rigorous and feasible. Contributions of the investigator/institution to scientific merit are framed in the context of the research proposed.

Additional Criteria (not scored, but considered in the Overall Impact Score):

Human Subject Protections
Inclusion of Women, Minorities, and Individuals Across the Lifespan
Vertebrate Animals
Biohazards
Resubmission/Renewal/Revisions

Each of the Additional Criteria except the last will be rated as “Appropriate”, with no comments required, or as “Concerns”, which must be briefly justified. Resubmission/Renewal/Revisions will be given brief written evaluations, where applicable.

Additional Review Considerations (not scored; not considered in Overall Impact Score):

Authentication of Key Biological and/or Chemical Resources

Rated as “Appropriate” with no comments required, or as “Concerns”, which would require a brief account of concerns.
Budget and Period of Support

Rated as “Appropriate”, “Excessive”, or “Inadequate”; the latter two ratings would require a brief account of concerns.

Under the proposed framework, the additional review considerations Foreign Organizations, Select Agents, and Resource Sharing Plans, will no longer be evaluated by reviewers.

CURRENT

PROPOSED

Main Review Criteria (will affect Overall Impact Score)

Individually scored:

Significance
Investigator(s)
Innovation
Approach
Environment

Factor 1: Importance of Research (individually scored)

Significance, Innovation
Factor 2: Rigor and Feasibility (individually scored)

Approach
Factor 3: Expertise and Resources (not individually scored; affects Overall Impact Score)

Investigators, Environment

Additional Review Criteria (can affect Overall Impact Score)

Human Subject Protections; Inclusion of Women; Minorities, and Children; Vertebrate Animal: Biohazards; Resubmission/Renewal/Revisions - some modifications expected for review of clinical trials RPGs

Additional Review Considerations (no effect on Overall Impact Score)

Application from Foreign Organizations
Select Agent Research
Resource Sharing Plans
Authentication of Key Biological and/or Chemical Resources
Budget and Period of Support

Authentication of Key Biological and/or Chemical Resources
Budget and Period of Support

View detailed language for the three factors, additional review criteria, and additional review considerations.

Principal Drivers of the Change

The first stage of NIH peer review serves to provide expert advice to NIH on the scientific and technical merit of grant applications. Applications are evaluated based on five scored criteria: Significance, Investigators, Innovation, Approach, and Environment (see complete definitions). Assessments of all five criteria contribute to the overall impact score and each of the five receive a criterion score from assigned reviewers.

Expansion of reviewer duties beyond the goal of first-level scientific peer review

Restructuring the categorization and scoring of criteria as proposed would reduce the number of scores reviewers need to provide, and policy considerations reviewers need to take into account when evaluating scientific merit in the first stage of review. The simplified framework would reduce reviewer burden and better focus reviewers on the two most important judgements about a proposed research project: how important the research is, and how rigorous and feasible the approach is.

Reducing reputational bias in the peer review process

Reputational bias interferes with the identification of research with the highest potential impact. An example of reputational bias is an application that is scored well despite serious deficiencies because the investigator is an acclaimed scientist. “Halo effects” can also adversely influence scores for otherwise strong applications from lesser-known scientists or schools. NIH has heard concerns about this kind of bias for years, and a findings from a large study done by CSR produced findings consistent with the idea that halo effects bias review (Nakamura et al., 2021, eLife 10:e71368. https://doi.org/10.7554/eLife.71368).

Under Factor 3, Expertise and Resources, the criteria Investigator and Environment, will be rated as appropriate or gaps identified. If the reviewer identifies a need for additional expertise or resources, justification must be provided. Evaluation of expertise and resources will be made in context of the proposed research rather than evaluation of the general reputation of the investigator or institution.

Restructuring review criteria is one of several concurrent CSR initiatives to reduce bias and increase fairness to facilitate the identification of the strongest, highest-impact research: https://public.csr.nih.gov/AboutCSR/Address-Bias-in-Peer-Review.

Seeking External Input

NIH gathered input from many sources in forming this proposal. Unsolicited comments over a period of years, reflecting sustained concerns from reviewers and applicants, led CSR to form two working groups to the CSR Advisory Council―one of which focused on recommendations to simplify non-clinical trials R01s and R21s, with the second focused on clinical trials applications. Each working group included members of the CSR Advisory Council, ad-hoc participants with appropriate expertise, and NIH staff. To inform those groups, CSR published a Review Matters blog which was cross-posted on the Office of Extramural Research blog, Open Mike. The blog received more than 9,000 views by unique individuals and over 400 comments. Interim recommendations were presented in a public forum, at CSR Advisory Council meetings (March 2020 video, slides; March 2021 video, slides) and adopted by the council. Final recommendations from the CSR Advisory Council (report) were considered by the major extramural committees of the NIH that included leadership from across NIH institutes and centers. This process produced some modifications, resulting in the proposed framework.

Detailed language for the three factors, additional review criteria, and additional review considerations can be found here.

Through the request for information (RFI), NIH sought additional public input on the proposed revised review framework. The RFI closed March 10, 2023; a content analysis will be published and linked here when available.

Summary of Feedback from the RFI