Sally Amero: Hello, and welcome to the peer review portion of the NIH virtual seminar, the 2020 version. I'm Sally Amero, the NIH review policy officer. We're going to spend the next half hour together looking at key aspects of the NIH peer review process. But first, I want to stop and pose the following problem for you to consider. Currently, the NIH receives over 80,000 grant applications a year. And as we know, they're not all funded. And so, what we're going to talk about today is what happens in between those two stages of the extramural program. So, we're going to fill in hopefully, a number of those question marks. And as we go through the talk, I want you to think about this problem.
If you could design a peer review system to evaluate those 80,000 plus applications and make decisions about which ones could be funded, what would be important in that process? Would you want a system of checks and balances to make sure that everything was done correctly? Would you want to manage inappropriate influences to make sure that the decisions were based on sound scientific considerations? Would you want to protect the ideas so they're not stolen? How would you want to evaluate the applications? Remember when you were in grade school, you would always ask, “What's going to be on the test?So how are they going to be graded, on the curve? Are you going to use a percentile system? " You would probably want to know who makes the final decision. And above all, you would want the system to be fair. So, let's see what the NIH does to accomplish these important considerations.
Let's start with the checks and balances. It's really important that as an applicant, you understand the process. So, the first thing you need to know is that an application must go through two levels of peer review in order for an award to be made. There are a few notable exceptions, but this is the general rule and in fact, this is a law. So, I'm going to walk you through the process here. Once you submit your application over on the left side of the slide, the first stop at NIH is in the Center for Scientific Review, CSR, Division of Receipt and Referral. And a number of decisions are made there about your application, such as where it's going to undergo peer review. The first level of peer review we call initial peer review. That's in the middle of the slide. And those are groups of outside scientists that are called Scientific Review Groups, SRGs, or study sections. That process is managed by NIH Scientific Review Officers. We call them SROs.
Once the initial peer review has been completed, your application then progresses to the second level of peer review, which is the National Advisory Council. That occurs in the funding institute that is going to consider your application for funding. And that's where the final funding decisions are made. I want to point out that right now special exceptions are in place for certain initiatives under the COVID pandemic emergency. So some of these steps may be bypassed in an emergency.
All right, we also thought we ought to think about managing influences. An important part of the process is managing Conflicts Of Interest -- COI. And in general terms, we identify two types of conflicts of interest. One is the Out of the Study Section or SRG conflict, which means that the reviewer with a conflict of interest cannot serve on that panel for a meeting or the application has to go to another study section. The other type of conflict is out of the room, which means that the reviewer with the conflict can go to the meeting, but has to leave the room when the application that's causing the conflict comes up for discussion and evaluation. There are different types, such as direct financial benefit, if one of the reviewer’s personal relatives, close relatives is involved in the application or if the conflict involves a member of a study section that occurs regularly. Those are all out of the study section conflicts, other types of conflicts such as employment, if the reviewer is an employee of the applicant organization, they have to leave the room for that discussion. There are all kinds of professional associations that cause conflicts such as mentor-mentee relationships, collaborations, co-authorship, letters of support and involvement in the project. And then this catch all category, the appearance of a conflict of interest can create an out of the room conflict.
It's really important that you understand the following point, reviewers must declare conflicts of interest and sign conflict of interest certifications. One certification is before the meeting and one is after the meeting. These are legally binding documents and if a reviewer fails to declare a conflict that was known to the reviewer at the time of the review, we can pursue some legal avenues as a result of that breach of integrity. Applicants may indicate conflicts of interest in their cover letter or in the assignment request form the ARF. I urge you to listen to the on demand masterclass in review integrity where we explore some of these elements in greater detail. I also need to point out to you that these rules for conflict of interest apply to reviews of grant applications. A different set of rules is in place for reviews of R&D contract proposals.
We also mentioned that you might want to protect the ideas that are coming in through the grant applications and we do this by maintaining confidentiality. All materials discussions, individual reviewer scores and assignments are confidential. The review meetings are closed to the public. And all questions must be referred to the scientific review officer running the meeting until release of the summary statement, at which time the point of contact becomes the NIH program official. It is very important that you pay attention to these next two questions. Should you contact a reviewer about your application? No. Should you ask a friend to contact a reviewer about your application? No. Doing so will get you into a world of trouble, reviewers are instructed to contact us when they are contacted by applicants and we will pursue this as a breach of integrity. Again, I urge you to listen to the on-demand recording.
Okay, what will be evaluated? What will be on the test? For every funding opportunity announcement that we issue, and we issue thousands of them a year, the overall assessment is listed as "overall impact." So reviewers are asked to aggregate -- to consider all of the questions that we ask of them into the overall assessment of overall impact. Now this is defined differently for each type of application. For research applications, overall impact is defined as the likelihood for the project to exert a sustained, powerful influence on the research fields involved. For a training grant application, however, the definition of overall impact would be different.
In every funding opportunity announcement, the review criteria are stated in section five. These cannot change from the time the FOA’s issued until the review. We cannot introduce review criteria after the funding opportunity announcement has been issued, so everyone knows how the applications will be evaluated. And moreover, all of the applications submitted to that announcement must be evaluated according to those criteria.
Now we have different types of review criteria and you need to understand this as well. We have three categories over on the left hand side of the slide, scored review criteria, additional review criteria and review considerations. Scored criteria, typically there are five scored criteria for research. These are significance, investigators ,innovation, approach and environment. And we call them scored because each of those receives a numerical criterion score unless the funding opportunity announcement indicates otherwise. And yes, these can affect the overall impact score. The second category, additional review criteria, also can affect the overall impact score, but these are invoked based on the work proposed in the application. So study timeline is applied to clinical trial applications, protections of human subjects, vertebrate animals, inclusion across the lifespan and biohazards are included in the review if the application involves human subjects or vertebrate animals or bio hazards. The final category does not affect the score -- the additional review considerations. But these are considered at the time of award, such as whether the applicant organization is a foreign institution, if the work involves select agents of the resource sharing plan and authentication of key resources. I'll point out to you that the protections of human subjects, vertebrate animals and inclusion would stop an award if the study section determined that those were unacceptable and the applicant organization was not able to resolve the problems before the award.
Okay, how will the applications be graded? So this is our scoring system. Numerical scores are used for both the criterion scores and the final impact scores if the application is discussed, and we will get into that process in just a moment. The scores offered by the reviewers range from one through nine in whole integers. One, two, three indicate high impact. Four, five, six, moderate impact. And seven, eight and nine, low impact.
Each member of the study section who does not have a conflict of interest votes after the application has been discussed. These are not just the assigned reviewers, and I'll explain who those people are in just a minute. Every member votes by private ballot at the meeting. Those votes are averaged and multiplied by ten, so the final impact scores range from 10 through 90. For some mechanisms, we have a process to percentile the scores to normalize them across different study sections. So ten is highest impact 90 is lowest. Now I mentioned that the numerical score would result if the application had been discussed. So this refers to the streamlining procedure which allows discussion of the more meritorious applications at the study section. Less meritorious applications, those with the likelihood of low impact, are tabled and not discussed further. They will be designated ND for not discussed on your summary statement. ND requires full concurrence of the entire study section. So if even one member does not agree with the ND outcome, they can rescue the application and put it into discussion of the entire panel. Your summary statements for a Not Discussed application will include the reviewer critiques and criterion scores from the assigned reviewers. So receipt and referral, remember, was the first step in the process once your application comes to us. And receipt and referral makes a number of very key decisions regarding the future of your application. They look at the application for compliance with policy, such as appropriate format, was it received on time, does it have the required components and attachments and so forth. They also decide where the application will go for initial peer review and they decide which NIH Institutes and Centers will consider the application for funding. In the old days when paper applications came in, we actually sent one copy to the study section and one to the Funding Institute. Of course, now it's all done electronically. And by the time of the Council, the second level of review, all of this information comes together. I urge you to look at the presentation by Dr. Cathy Cooper in the on demand section of the virtual seminar. She is the Director of the Division of Receipt and Referral and will certainly give you more information in that presentation. Okay, so your application has come in it has gone through receipt and referral and it is now in the hands of a Scientific Review Officer to manage initial peer review. It's important to keep in mind that the standard due dates repeat three times a year, and we call those council rounds. So the National Advisory Councils for nearly all the institutes meet three times a year. I believe for one or two of the institutes they meet four times a year, and those are the council rounds. And each step in the process is managed by a different NIH official. So for the next few minutes, we're going to talk about initial peer review, which is in the hands of the Scientific Review Officer or SRO. So what are the scientific review groups and who are they? Each study section or SRG is managed by a Scientific Review Officer or an SRO. These are PhD level officials in the NIH Center for Scientific Review or in an NIH Institute or Center. They ensure compliance with the government rules for peer review. They manage conflicts of interest, they arrange the meeting, they recruit the reviewers and assign them to applications. They preside at the meeting and prepare and issue the summary statements. It's important to note SROs do not engage in the evaluations of the applications. They are neutral. They are there to manage the process according to the laws and regulations. Now the reviewers are not government employees and they are recruited on the basis of scientific and technical expertise, stature in the field, their ability to display mature judgment, impartiality, the ability to work well in a group -- it can be a long day otherwise. They also have to manage conflicts of interest. We balance representation to the best of our ability. And of course the reviewer has to be available in order to serve. Another key point: SRGs do not make funding decisions. We will get into that in a little bit. Remember, in previous slides I talked about the assigned reviewers? Well, who are these people? Well, they are members of the study section and for each application, at least three qualified reviewers are assigned for an in depth assessment. They are asked to read the application. They are asked to prepare written critiques and provide those criterion scores that we mentioned before for the scored review criteria. They are assigned on the basis of scientific expertise and the science that's proposed in the application. We try to balance workload and, of course, we have to avoid conflicts of interest in assigning the reviewers. In addition, after the review meeting is over, the assigned reviewers are expected to adjust their criterion scores and written critiques to reflect the discussion at the meeting. Do not try to find out who the assigned reviewers were. This is highly confidential information and we will not release that information to you. And if you ask an NIH official, they will tell you that they’re not at liberty to reveal that. Okay, so your application’s come through receipt and referral. The SRO has put together a review panel of signed reviewers; the reviewers are now collected at a meeting. Right now we're meeting virtually, but typically these are in person meetings. So at the meeting the SRO would make introductory comments, remind the reviewers of key policy points that they need to remember in evaluating the application. Then for each application that is not streamlined, members with conflicts would leave the room. So what might happen would be the chair of the study section would announce the first application for discussion and might say, “Oh, Doctor so and so has a conflict let's wait while Doctor, so and so leaves the room.” And then the assigned reviewers would discuss that application in turn, the primary reviewer first, secondary and tertiary reviewer. They typically do not explain the entire application. They would highlight the goals and the approaches proposed and then highlight the weaknesses or strengths that they found in the application. After the assigned reviewers, then the eligible members those who do not have conflicts join the discussion and then the chair summarizes the discussion. The assigned reviewers provide their final impact scores, and this sets a range of scores based on the discussion of strengths and weaknesses. Then all the members provide their final scores privately and other issues are discussed, such as the budget, the data sharing plan, foreign applications, and so forth -- the non-scored considerations. So I urge you to attend to the mock study section, which will be a live presentation, where you will see all of this happen in real time. So your application made it through the meeting. When do you get your results? Your results will appear in the ERA Commons. The final impact score should be available within three days after the meeting ends. The summary statement should be available within four to eight weeks. And that is provided to the funding Institute program officer, the NIH program officer, the applicant organization signing official, the program director and or principal investigator, other NIH officials, and advisory council members. Remember, a favorable score does not guarantee funding, And so don't start celebrating quite yet. So we mentioned that the summary statement would be available within four to eight weeks. And here's just a brief explanation of what would be in that summary statement. This is the final outcome of the initial peer review process. This is the official document that reports the outcome of initial peer review. On the first page, up in the left hand corner, you will find the name of your NIH program official and that person's contact information. So remember, after we release of the summary statement, that person is your point of contact at the NIH. It will give you the final impact score or whether it was discussed or not discussed. If it were percentile that would appear. You will see the codes for human subjects, vertebrate animals and inclusion plans. Whether they were acceptable or not acceptable. We typically report your budget request. On subsequent pages you will find a resume and summary of discussion if the application had been discussed. This is prepared by the Scientific Review Officer to reflect the score driving elements that came up in the review. We grab the description from your application. We list the criterion scores from the three or more assigned reviewers. We present the reviewer critiques from the assigned reviewers largely unedited. If there were any administrative notes that came up from the study section as well as the roster of the study section members. Now, you might also instead of a roster of names, you might see a link there that takes you to an aggregate roster. The aggregate roster would have the names of several study sections put together into one list. And this typically is done if the individual meetings had a very small number of reviewers, and this is in order to protect the reviewers. Whom do you contact after the review? As I mentioned, this now is in the hands of your NIH program official. I know, the first thing you want to do is to pick up the phone and call that person. But I urge you to read and consider the summary statement, think carefully and try to assimilate the information in the summary statement before you call the program official. Based on the outcome, you may need to submit additional information just in time. You may have to consider your options at that point. If it's not likely to be funded, you may submit a new application, depending on the funding opportunity announcements available. You may revise and resubmit your application again, depending on what's being accepted for that funding opportunity announcement. Or you could appeal the outcome of review. This is something you need to discuss with your program official. Every applicant has the option unless the application was submitted to a request for applications. You can appeal the outcome of review. However, I must caution you that this is often a lengthy process and it can set you back a funding cycle. Whereas if you need to get your funding decisions early, you might consider the other two options. Okay. So now, we're through initial peer review and we're moving on to the National Advisory Council group, the second level of review. So who are these people? Well, this process is managed by NIH executive secretaries, or Exec Secs, as we call them. These are also senior PhD level officials in the NIH funding institute, such as the National Cancer Institute, the National Institute of General Medical Sciences, and so forth. These people ensure compliance with the government rules for council review, and they manage the conflicts of interest because these people are -- I'll get to that in a minute. Exec Secs do not engage in counsel recommendations. They also remain neutral. They are there to manage the process according to federal and NIH rules and regulations. Who are the council members? National Advisory Council members tend to represent a broad and diverse membership. These can be basic or research scientists, clinician scientists and often you will see public members on a National Advisory Council. Sometimes a disease advocates will appear or a patient advocates will appear. And a key distinction between these reviewers and reviewers in initial peer review is the council members are appointed as special government employees. And the chair of the Advisory Council is the director of the NIH Institute or Center where the application could be funded. The council makes recommendations for funding particular applications. They consider concepts for new initiatives and they adjudicate appeals of initial peer review. So if an appeal comes in and the program official and the Scientific Review Officer cannot resolve the appeal of peer review, then it is given to the council to adjudicate. Again, National Advisory Councils do not make funding decisions. They make recommendations. So who does make the final decision? That is the director of the NIH funding Institute, who makes the final funding decision. Now, the outcome of initial peer review is a very important component in that decision but it's not the only one. The director must also assess the fit with the mission of the Institute program priorities or congressional mandates. They sometimes consider additional outside expertise if needed. They listen to the recommendations of their counsel and their own program staff. And of course, they have to consider how much money they have available to spend. I want to take just a moment to highlight some special considerations for the process now under the COVID pandemic emergency. You may be aware that for certain initiatives right now, for the January and May 2021 counsels there could be due date extensions. We have extended a special exception for preliminary data as post submission materials. We have given guidance to all of our reviewers to assume the temporary emergency situations due to the pandemic will be assessed by NIH officials before an award is made. And all of our meetings right now are virtual. You can certainly find more information at this website here. And I would like to remind you that every participant in the NIH review process is responsible for upholding the core values of peer review. And I'd like to spend a minute here on these so you understand and don't get yourselves into trouble. We talked a little bit about confidentiality. So any attempt to gain information about the review by contacting reviewers, if reviewers reveal information that they're not allowed to reveal, this is a breach of confidentiality. Integrity refers to the fact that every NIH reviewer is instructed to bring any allegations of research misconduct to the attention of the scientific review officer or Exec Sec directly and immediately. What is research misconduct? Well, according to DHHS policy, that is fabrication falsification, or plagiarism. And we do not want reviewers to share that allegation with other reviewers or council members, they need to go directly to their Scientific Review Officer. The SRO will decide what to do with the application. In most cases, the application would be deferred until we can work out what's going on and see if the application can get a fair review or not. Fairness, so from the point of view of the NIH fairness means what I’ve mentioned before: that all of the applications that goes through review, that were submitted to a particular funding opportunity are reviewed according to the same review criteria and the same scoring scale. So we don't evaluate those applications differently. Each application is reviewed according to the same process. Security has become very important recently. And this refers to maintaining the security of our computer systems. Each reviewer is issued a password to gain access into our secure peer review site. And we need to maintain those firewalls and keep those passwords confidential, and we are working on that. Impartiality. So any attempt to bias the review, to bribe, threaten, game the system, all of these are falling under the heading of impartiality and they are forbidden. Expert assessment means that the NIH does our very best to recruit scientific expertise to review the applications. And this is one basis for an appeal if you believe that the expertise is not adequate. But keep in mind that expertise is in the eye of the beholder and it is up to the NIH to provide that expertise. Efficiency. We do our darndest to get the use of reviews done in the most expedient manner so you get your results as soon as possible. But we also have to balance that against cutting corners. So we have to make sure that we get the right reviewers, we have to make sure that we consider conflicts of interest, we have to make sure that we get the reviews done, that there's sufficient time for discussion and all of those things. And you'll see that efficiency is caddy cornered on this slide from integrity. So that's a tension in the system that we have to balance. And transparency. Remember that I told you all of the review criteria are announced in the funding opportunity announcement so you know how your application will be assessed. And note here that transparency is caddy cornered from confidentiality so that's another tension in the system. We are glad to explain the process, and the process is transparent. But we cannot tell you what goes on in the actual review meeting of a particular application. So here I urge you to read the core values document that is posted in the on-demand section, and listen to the masterclass and review integrity. Okay, so you have some responsibilities here. You need to keep informed. You can do this in a couple of ways. You can read the Open Mike blog. This often has some provocative stories in it about peer review integrity, as well as policy announcements and upcoming initiatives. You can join the Guide Table of Contents, the TOC, which is issued weekly. This is where new funding opportunity announcements will appear. This is where guide notices and new policies are announced formally. So I asked you again to think about this. If you could design a peer review system, what would be important to you? Checks and balances, managing inappropriate influences, protecting ideas, how applications will be evaluated and graded, getting the results out quickly, deciding who decides and maintaining fairness and integrity. And I hope if you can think about how NIH manages each of these considerations, you will know a lot about peer review. And you can find additional information at these sites. You can also contact me at any time. I urge you though to do so through email. If you contact any NIH official right now, I urge you to do it through email. We're often in meetings. We're not in their offices most of the time, so email is much more efficient. Thank you for your time.