An overview of a proposal intended to help focus reviewers on the review of scientific merit and reduce bias.
Synopsis of Issue
NIH is proposing a revised, simplified framework for peer review that will reorganize the major regulatory criteria (42 C.F.R. Part 52h.8) under three factors and reduce the number of non-score driving review considerations that peer reviewers evaluate in research project grant (RPG) applications during the first stage of review. NIH believes that these changes will allow peer reviewers to refocus on the critical task of assessing scientific merit and will improve those assessments by reducing bias.
The Current Process
The first stage of NIH peer review serves to provide expert advice to NIH on the scientific and technical merit of grant applications. Applications are evaluated based on five scored criteria: Significance, Investigators, Innovation, Approach, and Environment (see complete definitions). Assessments of all five criteria contribute to the overall impact score and each of the five receive a criterion score from assigned reviewers.
Proposed Simplified Review Framework
NIH proposes to reorganize the five regulatory review criteria into three factors.
The Overall Impact Score (scored 1-9) reflects the overall scientific and technical merit of the application; all three factors will be considered in arriving at the Overall Impact Score.
The 3-factor structure is intended to focus the evaluation of scientific merit on key questions—how important the research is and whether it is both rigorous and feasible. Contributions of the investigator/institution to scientific merit are framed in the context of the research proposed.
Additional Criteria (not scored, but considered in the Overall Impact Score):
- Human Subject Protections
- Inclusion of Women, Minorities, and Individuals Across the Lifespan
- Vertebrate Animals
- Biohazards
- Resubmission/Renewal/Revisions
Each of the Additional Criteria except the last will be rated as “Appropriate”, with no comments required, or as “Concerns”, which must be briefly justified. Resubmission/Renewal/Revisions will be given brief written evaluations, where applicable.
Additional Review Considerations (not scored; not considered in Overall Impact Score):
-
Authentication of Key Biological and/or Chemical ResourcesRated as “Appropriate” with no comments required, or as “Concerns”, which would require a brief account of concerns.
-
Budget and Period of SupportRated as “Appropriate”, “Excessive”, or “Inadequate”; the latter two ratings would require a brief account of concerns.
Under the proposed framework, the additional review considerations Foreign Organizations, Select Agents, and Resource Sharing Plans, will no longer be evaluated by reviewers.
Individually scored:
- Significance
- Investigator(s)
- Innovation
- Approach
- Environment
-
Factor 1: Importance of Research (individually scored)Significance, Innovation
-
Factor 2: Rigor and Feasibility (individually scored)Approach
-
Factor 3: Expertise and Resources (not individually scored; affects Overall Impact Score)Investigators, Environment
- Application from Foreign Organizations
- Select Agent Research
- Resource Sharing Plans
- Authentication of Key Biological and/or Chemical Resources
- Budget and Period of Support
- Authentication of Key Biological and/or Chemical Resources
- Budget and Period of Support
Principal Drivers of the Change
The first stage of NIH peer review serves to provide expert advice to NIH on the scientific and technical merit of grant applications. Applications are evaluated based on five scored criteria: Significance, Investigators, Innovation, Approach, and Environment (see complete definitions). Assessments of all five criteria contribute to the overall impact score and each of the five receive a criterion score from assigned reviewers.
Restructuring the categorization and scoring of criteria as proposed would reduce the number of scores reviewers need to provide, and policy considerations reviewers need to take into account when evaluating scientific merit in the first stage of review. The simplified framework would reduce reviewer burden and better focus reviewers on the two most important judgements about a proposed research project: how important the research is, and how rigorous and feasible the approach is.
Reputational bias interferes with the identification of research with the highest potential impact. An example of reputational bias is an application that is scored well despite serious deficiencies because the investigator is an acclaimed scientist. “Halo effects” can also adversely influence scores for otherwise strong applications from lesser-known scientists or schools. NIH has heard concerns about this kind of bias for years, and a findings from a large study done by CSR produced findings consistent with the idea that halo effects bias review (Nakamura et al., 2021, eLife 10:e71368. https://doi.org/10.7554/eLife.71368).
Under Factor 3, Expertise and Resources, the criteria Investigator and Environment, will be rated as appropriate or gaps identified. If the reviewer identifies a need for additional expertise or resources, justification must be provided. Evaluation of expertise and resources will be made in context of the proposed research rather than evaluation of the general reputation of the investigator or institution.
Restructuring review criteria is one of several concurrent CSR initiatives to reduce bias and increase fairness to facilitate the identification of the strongest, highest-impact research: https://public.csr.nih.gov/AboutCSR/Address-Bias-in-Peer-Review.
Seeking External Input
NIH gathered input from many sources in forming this proposal. Unsolicited comments over a period of years, reflecting sustained concerns from reviewers and applicants, led CSR to form two working groups to the CSR Advisory Council―one of which focused on recommendations to simplify non-clinical trials R01s and R21s, with the second focused on clinical trials applications. Each working group included members of the CSR Advisory Council, ad-hoc participants with appropriate expertise, and NIH staff. To inform those groups, CSR published a Review Matters blog which was cross-posted on the Office of Extramural Research blog, Open Mike. The blog received more than 9,000 views by unique individuals and over 400 comments. Interim recommendations were presented in a public forum, at CSR Advisory Council meetings (March 2020 video, slides; March 2021 video, slides) and adopted by the council. Final recommendations from the CSR Advisory Council (report) were considered by the major extramural committees of the NIH that included leadership from across NIH institutes and centers. This process produced some modifications, resulting in the proposed framework.
Detailed language for the three factors, additional review criteria, and additional review considerations can be found here.
Through the request for information (RFI), NIH sought additional public input on the proposed revised review framework. The RFI closed March 10, 2023; a content analysis will be published and linked here when available.