Announcer: From the National Institutes of Health in Bethesda, Maryland, this is All About Grants.
Megan Columbus: Here we are for another addition of All About Grants. I’m your host Megan Columbus from the Communications Division of NIH’s Office of Extramural Research. Today we’ll be talking to Dr. Alan Willard, chief of the scientific review branch at the National Institute on Neurological Diseases and Stroke, here at the NIH. Welcome.
Alan Willard: Thank you.
Megan: We’re going to be talking to you today about what actually happens inside an NIH study section meeting. Remember that the study section meeting is the first stage of NIH’s two-stage peer review process. But before we get too far down that road, can we hear a little bit about your role here at NIH and a little bit about your background?
Alan: Well, NIH has played an important role in my entire scientific career. As a faculty member first at Harvard Medical School and then at the University of North Carolina in Chapel Hill, my research was supported by a variety of NIH research grants. I served as a member of a study section in the late 80s and early 90s and chaired a study section in the mid 90s. I enjoyed NIH review so much that in 1998 I came to NINDS as then a scientific review administrator. We change our titles from time to time, although we don’t have epaulettes, we are now scientific review officers. And I currently serve as the chief of the scientific review branch at NINDS.
Megan: NIH has a whole variety of resources that people can find useful as they try to learn more about NIH’s peer review process. NIH’s Center of Scientific Review recently put out a video that provides the perspective of some very senior reviewers that people might find useful. And that resource, along with others, can be found at csr.nih.gov, and there’s additional resources that we’ll point to later in this podcast. Your role as a scientific review officer, when you think about it, in the meeting itself, what is that role?
Alan: Well there are really two roles. One of them is official, in that the SRO is indeed the designated federal official who is responsible for making sure that all relevant rules and regulations are complied with so that the meeting is within the scope of the law and that all processes for regulation of conflict of interest and those kinds of things are followed. Equally important a scientific review officer is responsible for making sure that all applications get a very appropriate and thorough and fair review. This is generally done in partnership with the chair of the study section, making sure that when a particular category of grant is discussed that people are discussing the review factors that are appropriate for that grant, and they’re not talking about factors that are irrelevant to that grant or inappropriate, also making sure that people only talk about factors that are relevant to review and not just things that are perhaps of interest to a scientific field, but have nothing whatsoever to do with a particular grant. Scientists will sometimes veer off into sidebar conversations. We all tend to do that, and it’s the role of the chair and the SRO to make sure people stay on time and on target.
Megan: You’ve mentioned the chair a couple of times. Can you tell a little bit about who a chair is and how you select a chair and what their role is in the meeting?
Alan: The requirements for the chair are, first of all, that they be someone who is recognized as a scientific authority in that field. And they really need to be someone who works with the SRO to make sure that the meeting is conducted in a very fair and thorough manner. They need to make sure that every application that is discussed gets a fair discussion but also making sure that people don’t perseverate. Sometimes if there are differences in opinion, people may just start to repeat those opinions, and the chair needs to remind people that discussion consists of raising new points, not just of repeating the same point over and over. And so one of the most important things that a chair does is to make sure that all of the relevant factors have been discussed, but once he or she has figured out that really there’s been enough discussion so that all the people around the table know how they’re going to vote that that discussion can terminate and they can stay on good schedule.
Megan: How are the other reviewers in the room selected and how many might there be in a room?
Alan: For the vast majority of standing committees, the typical numbers are going to be somewhere between 16 and 30 depending on the workloads that those committees have. In any kind of review meeting the first requirement is that people have some form of expertise that’s relevant to the applications being reviewed. So it could be scientific expertise, it could be technological expertise, depends a bit on what category of grant. For study sections that are reviewing training grants or fellowships, you might want people who have a lot of experience in training students or in helping the career development of junior faculty, and that might be at least as important as their specific scientific expertise. In addition to the scientific factors, the other important thing, especially for people who are going to be regular members of a committee, are more human factors, so their ability to communicate clearly. They may know the material very well, but if they can’t explain to the rest of the panel in a succinct and clear and understandable fashion why they came to the conclusions they came, it’s not very helpful. Likewise people who don’t listen to others, who are unwilling to change their opinion based on what they hear, are not very useful members of committees. So it’s not just the scientific expertise; they have to be people who are good members of a team.
Megan: There are select other NIH staff that might attend. Who might they be?
Alan: The most common ones who would be attending would be the program officers who will be administering the applications if they are awarded, and who will be communicating with the applicants after the review meeting. So typically those people are there as observers. Sometimes there will be grants management officers present, especially if there are complicated budget issues that may come up. I guess those are the most common other people who are there.
Megan: As an SRO you normally would either kick off the meeting, giving those reviewers an orientation, or potentially have done the orientation before they got to the meeting. But what does that orientation entail?
Alan: So there are several elements to the orientation. Some of them are essential for every review meeting, and that is to go over the rules of conflict of interest. Some conflicts are obvious. Reviewers can’t review applications from their own institution. They can’t review their own applications. They can’t review applications of active collaborators. We also during the orientation remind people of confidentiality -- that the reviewers need to be able to have a very frank and open discussion, and they need to do so with the knowledge that what they say in the room stays in the room, even though I will point out that very few of the meetings are held in Las Vegas. In a meeting where a number of different types of grants are being discussed, the reviewers will be reminded that there are several different types of grants and that they have different review requirements and that everyone needs to be careful that they’re using appropriate criteria for the specific grant that they’re discussing.
Megan: Let’s move in to the actual process of the meeting itself. For someone who’s never been in a review meeting how does it unfold and how does that happen?
Alan: I think for most people when they come to their first review meeting, and it was certainly true for me, the thing that strikes you is it’s an extremely human process. It is not just a bunch of robots, reading applications and discussing review criteria in some very formulaic way. The discussion of any particular application begins with the so-called primary reviewer. And I should back up and say that at a typical meeting, there are three reviewers assigned to an application: primary reviewer, and then 2 other reviewers. The primary reviewer is always the person who is responsible for describing what the application is about to the rest of the panel. And one of the things that is very important for people to understand about NIH grant review is at the majority of the meetings most of the reviewers sitting around the table, who will be voting on a final score for the application, have not read the application in great detail and have not prepared written critiques. So in a typical meeting with let’s say 20 or 25 reviewers and there’s only 3 assigned to each application, and at that meeting 50-100 applications are being discussed, the role of the primary reviewer is extremely important because they will describe what are the basic goals of the application. And then after that, that reviewer will describe what they had identified as the major strengths and weaknesses of that application, and they will have a suggested rating for the application. After the primary reviewer has presented their evaluation the additional reviewers will present any additional points that they have. If they really are in complete agreement with the primary reviewer and have no other strengths or weaknesses to add they can essentially just grunt in agreement and say I have nothing to add. Often the most important discussions are the ones where they read the same application, and they came to a rather different conclusion from the other reviewers. So those kinds of discussions go on. After each of the assigned reviewers has presented their views, then it’s opened up to the rest of the panel, and people may ask questions of clarification or other people may have additional perspectives, other points they want to raise. Once the discussion is complete or terminated, if it’s starting to perseverate then the chair may intervene and say “OK, I’m starting to hear people repeat themselves, let’s stop and let’s have everyone vote.” Typically, the assigned reviewers will say what score they are now going to vote. This is very important. The scores that they end up with could be dramatically different from the scores that they initially recommended. So the assigned reviewers say what scores they’re going to vote. The rest of the panel is asked if they’re prepared to vote within that range. If there are people who have listened to the discussion and don’t think the scores match it, they will speak up and say it. After that scoring then there’s a discussion of other issue such as budget and other administrative issues that don’t impact the priority score but that are important for reviewers to evaluate.
Megan: One of the things that your description reminds me is how important it is for people to put together applications that are well constructed and easy for people to flip through and find things during the discussion. We will be doing additional podcasts in the future that get to some of those grantsmanship issues. There will be a lot of people in the room who will be flipping through the application who may not have read it cover to cover who will be scoring that application.
Alan: That’s very important. People will be skimming through an application, so applications that have nice use of headings and good formatting with space between paragraphs and things like that make it much easier for people to find things.
Megan: We haven’t talked here about streamlining and those applications that are not discussed. What’s that process?
Alan: Typically at meetings that are reviewing a large number of applications, in order to make most effective use of the reviewer’s time only those applications that are considered to be in the top half are actually discussed. And the most common process that is used, particularly at the Center for Scientific Review to determine which applications to discuss, are based on the preliminary scores. At most meetings that are run by the Center for Scientific Review, the applications are discussed more or less in descending order of preliminary scores. The purposes of doing it that way are to try to insure relatively consistent scoring and so applications that have similar preliminary ratings have a higher probability of being relatively similar quality and so reviewers are not jumping around from a really good to a really bad. I should point out that if members of the panel want to discuss other applications they can. The decision not to discuss an application is made by the entire panel. It’s not the case that one reviewer says “This is bad. We can’t discuss it.”
Megan: And the applications that are not discussed, they still get summary statements, so they still get the comments and the scores?
Alan: Right. So you’ve reminded me of one of the most important roles of the scientific review officer, which is to be taking notes on the discussion, and in particular, taking notes of those factors that were mentioned during the discussion that the majority of people around the table heard and based their votes on. I remind you that most people around the table have not read the full application, have not prepared critiques, so what they’re voting on is what they’ve heard. And so the paragraph that the SRO writes called “Resume and Summary of Discussion” is very important and should capture those points that were raised in the discussion that really influenced how people voted.
Megan: That paragraph is included in the summary statement itself?
Alan: Of the discussed applications, right.
Megan: After the meeting, how long does it take for SROs to generally get summary statements out?
Alan: The majority of SROs will try to have the majority of summary statements done within a month. CSR strongly encourages them to get the summary statements for new investigators out within a week or 10 days. The scores are generally released within 48 working hours of the meeting. They can see that in their eRA commons account.
Megan: And so then the applicant when they get the summary statement, they shouldn’t call a program official before they actually get the summary statement?
Alan: Right. There’s nothing useful that can be discussed between the program officer and the applicant until the summary statement has been released.
Megan: I think this has been a very thorough overview of what goes on in a meeting, and I appreciate your being here. For NIH and OER, this is Megan Columbus.
Announcer: To view the Peer Review related video, please visit the Center for Scientific Review’s website at csr.nih.gov. And to learn more about the NIH peer review process, visit the Office of Extramural Research’s website at grants.nih.gov. That’s G-R-A-N-T-S dot N-I-H dot G-O-V.