Synopsis of Issue
NIH is recommending changes to the peer review of Ruth L. Kirschstein National Research Service Award (NRSA) fellowship applications by restructuring the review criteria and modifying some sections of the PHS Fellowship Supplemental Form that are specific to NRSAs. The goal of this effort is to facilitate the mission of NRSA fellowship peer review – to identify the most promising trainees and the excellent, individualized training programs that will help them become the outstanding scientists of the next generation. The proposed changes will 1) allow peer reviewers to better evaluate the applicant’s potential and the quality of the scientific training plan without undue influence of the sponsor’s or institution’s reputation; and 2) ensure that the information provided in the application is aligned with the restructured criteria and targeted to the fellowship candidate’s specific training needs.
The first stage of NIH peer review serves to provide expert advice to NIH on assessment of the likelihood that the fellowship will enhance the candidate’s potential for, and commitment to, a productive independent scientific research career in a health-related field. The criteria for the review of NRSA fellowship applications derive from the NRSA regulation 42 CFR 66.106, with four pertinent factors: (1) the scientific, technical, or educational merit of the particular proposal; (2) the availability of resources and facilities to carry it out; (3) the qualifications and experience of the applicant; and (4) the need for personnel in the subject area of the proposed research or training. NIH currently organizes these criteria into the following categorical labels: Applicant, Sponsors and Collaborators, Research Training Plan, Training Potential, and Institutional Environment and Commitment. By NIH policy, peer reviewers are currently also required to evaluate Training in the Responsible Conduct of Research, Biohazards, Resubmissions, Applications from Foreign Organizations, Select Agents, Resource Sharing Plans, Budget and Period of Support, and Authentication of Key Biological and/or Chemical Resources.
Reasons for Change
The goal of the change is to improve the chances that the most promising applicants, no matter who they are or where they are based, will be consistently identified by peer review. The proposed changes are intended to:
- Better focus reviewer attention on three key assessments: potential of the applicant, strength of the science, and quality of the training plan.
- Define criteria to give less advantaged applicants a better chance, without disadvantaging others. The fellowship application instructions will be modified to align with the 3 core criteria and emphasize substantive statements pertinent to the individual student. Reviewers will evaluate accomplishments and trajectory in the context of their opportunities, and personal characteristics that contribute to success.
- Reduce bias in review by reducing inappropriate consideration of sponsor and institutional reputation. Reviewers will evaluate sponsor and institution with respect to the quality of the science and quality of the training plan.
Improving fellowship review criteria is one of several concurrent NIH initiatives to reduce bias and increase fairness to facilitate the identification of the strongest, highest-impact research.
Recommendations for Improving NRSA Fellowship Review
Recommendation 1: Revise the Criteria Used to Evaluate NRSA Fellowship Applications
An Overall Impact Score (scored 1-9) will reflect the scientific and
educational merit of the proposal and an assessment of the likelihood
that the fellowship will enhance the applicant’s potential for, and
commitment to, an independent, productive research career in a
health-related field. Reviewers will take into account their assessments
of the three criteria in determining an Overall Impact Score. Each of
the three criteria will receive an individual score. The additional
review criteria below will not receive individual scores but will be
considered in arriving at the Overall Impact Score. Two review
considerations will be evaluated but have no effect on the Overall
Scientific Potential, Fellowship Goals, and Preparedness of the
- Criterion emphasizes their accomplishments in the context of their stage of training and the scientific opportunities they have had, as well as other factors that bear on their potential to succeed, such as determination, persistence, and creativity. (scored 1-9)
Science and Scientific Resources
- Criterion emphasizes the extent to which needed technical, scientific, and clinical resources are specified and are realistically available to the applicant, and whether the scientific expertise of the mentorship team is appropriate for the proposed science and that the role of each mentor is clearly defined. Peer reviewers would not assess financial support for the proposed research; that would be done by NIH staff. (scored 1-9)
Training Plan and Training Resources
- Criterion emphasizes whether the necessary training resources are well-specified and available, specifically the practical availability of resources, and to include an evaluation of the training philosophy of the sponsor, their approach to training, time commitments, and their accessibility. (scored 1-9)
Additional Criteria (not scored, but affecting Overall Impact; no changes proposed, see current language):
- Protections for Human Subjects
- Inclusion of Women, Minorities, and Individuals Across the Lifespan
- Vertebrate Animals
Each of the Additional Criteria except the last will be rated as “Appropriate”, with no comments required, or as “Concerns”, which must be briefly justified. When applicable, Resubmission will be given brief written evaluations.
Additional Review Considerations (not scored and having no effect on Overall Impact) no changes proposed, see current language):
Training in the Responsible Conduct of Research
- Five subsections (format, subject matter, faculty participation, duration, frequency), each rated as “Appropriate” with no comments required, or as “Concerns”, which must be briefly described.
Authentication of Key Biological and/or Chemical Resources
- Rated as “Appropriate” with no comments required, or as “Concerns”, which must be briefly described.
Budget and Period of Support
- Rated as “Appropriate”, “Excessive”, or “Inadequate”; the latter two ratings requiring a brief account of concerns.
Applications from Foreign Organizations
- Rated as “Acceptable”, “Unacceptable”, or “Not Applicable”.
- Rated as “Acceptable”, “Unacceptable”, or “Not Applicable (No Select Agents)”.
Resource Sharing Plans
- Rated as “Acceptable”, “Unacceptable”, or “Not Applicable (No Relevant Resources)”.
Recommendation 2: Revise the PHS Fellowship Supplemental Form
Changes to the application instructions are needed to better align the information applicants provide with the revised review criteria. The PHS Fellowship Supplemental Form currently includes the following four sections: Fellowship Applicant; Research Training Plan; Sponsor(s), Collaborator(s), and Consultant(s); Institutional Environment and Commitment to Training. The NIH proposes to revise the Fellowship Applicant section and the Sponsor(s), Collaborator(s) and Consultant(s) section. NIH also proposes to change the instructions for reference letters. An additional proposed change would allow an optional Statement of Special Circumstances from the fellowship applicant. Changes to each section are summarized below. The full text of the proposed changes to the application can be found here.
Summary of Revisions to Applicant Section
Grades would no longer be required or allowed; however, the titles of courses completed would be required.
Applicants would be asked to submit five statements: (1) a statement of professional and fellowship goals, (2) fellowship qualifications, (3) a self-assessment, (4) a statement of scientific perspective, and (5) a detailed account of activities planned under this award. Additionally, grades would no longer be required or allowed; however, the titles of courses completed would be required.
Summary of Revisions to Applicant Sponsor and Co-sponsor Section
Sponsors and Co-sponsors would be asked to submit three statements: (1) an account of the training plan, environment, and research facilities; (2) the number of fellows/trainees to be supervised during the fellowship; and (3) the applicant’s qualifications and potential for a research career.
Summary of Revisions to Applicant Instructions for Reference Letters
NIH proposes to update the instructions for reference letters with more structure so that the resulting letters better assist reviewers in understanding the applicant’s strengths, weaknesses, and potential to pursue a productive career in biomedical science. Writers would be instructed to respond to four prompts: (1) identify the 2-4 most important characteristics of this applicant that will contribute to this applicant becoming a successful scientist and explain why; (2) identify 2-4 areas for growth that the applicant should address to improve their prospects of becoming a productive biomedical scientist; (3) what scientific and intellectual contributions has the applicant made during their training/research experience with you?; (4) give an overall assessment of the applicant’s readiness for this fellowship taking into account their background, potential, current skills or abilities, and career goals.
Allow an Optional Statement of Special Circumstance
NIH recommends allowing fellowship applicants to submit an optional Statement of Special Circumstance to address situations that may have hindered the trainee’s progress, such as harassment, the COVID-19 pandemic, or other personal or professional circumstances.
- Fellowship Applicant
- Sponsors, Collaborators, & Consultants
- Research Training Plan
- Training Potential
- Institutional Environment & Commitment to Training
- Scientific potential, fellowship goals, and preparedness of the applicant
- Science and scientific resources
- Training plan and training resources
Seeking External Input
NIH gathered input from many sources in forming this proposal. Unsolicited comments over a period of years, reflecting persistent concerns that the NRSA fellowship review process disadvantages some highly-qualified, promising applicants led the Center for Scientific Review (CSR) to form a working group to the CSR Advisory Council. To inform that group, CSR published a Review Matters blog which was cross-posted on the Office of Extramural Research blog, Open Mike. The blog received more than 1,500 views by unique individuals and numerous comments. The working group presented interim report to the CSR Advisory Council, which adopted the recommendations, at public CSR Advisory Council meetings (March 2022 video, slides; September 2022 video, slides). Final recommendations from the CSR Advisory Council (report) were considered by the CSR Director, as well as major NIH extramural program, review, and policy committees that included leadership from across NIH. Recommendations were presented to the NIH Advisory Committee to the Director in December 2022 (video, slides).
Through the request for information (RFI), NIH sought additional public input on the proposed revised review framework. The RFI closed June 23, 2023.
Questions may be directed to: firstname.lastname@example.org