NIH All About Grants Podcast

Using Reviewer Critiques and Program Input on Applications

Announcer: From the National Institutes of Health in Bethesda, Maryland, this is All About Grants.

David Kosub: Hello, and welcome to another edition of NIH's All About Grants podcast. I'm your host, David Kosub, with the NIH's Office of Extramural Research. And today, we're going to be talking about an important topic, something that probably many of y'all have experienced out there if you've ever applied for an NIH grant. And that's how to use the reviewer critiques and your program input when you actually get them on your application, kind of what you should be thinking about and what- you know, as your next steps and all that good stuff. I'm glad to say we have with us two guests, we have Dr. Tracy Rankin. She is a Program Director within the National Institute of Diabetes and Digestive and Kidney Diseases. And we also have Dr. Brian Hosha

w, who is a Review Chief within the National Eye Institute. And I welcome you both to the show.

Brian Hoshaw: Hey, thanks, David.

Tracy Rankin: Yep. Great to be here.

David Kosub: Great. Glad y'all are both here. All right, Brian, I'll start with you. What should someone expect to see on a review or critique?

Brian Hoshaw: Great. Sure. So when you get the critique on the front page, you'll see information about the grant, you know, the first place, you look right in the middle, you'll see the impact score, and then the percentile for some applications. And then at the top left, you'll have the program contact, and that's your program officer, and their name and phone number, how you can get a hold of them. And then the second page, if it was discussed, you'll see a resume and summary of discussion. And this is the scientific review officer taking notes during discussion whilst trying to summarize the main score driving issues into one paragraph. And this is really a good resource because they kind of highlight what drove the reviewer scores during the discussion. And then beyond that, you'll have the full critiques from the assigned reviewers. So all of the assigned reviewers, whether it's discussed or not discussed, they submit individual criterion scores for the five criteria. And then they have their full written critique with strengths and weaknesses. And then at the very end, it'll have the roster of the review meeting.

David Kosub: Cool. Thank you for that, Brian. So you're on the review side, and Tracy's on the program side. What exactly, I guess, in brief terms is the difference between the role of the scientific review officer or the program officer in this process, and maybe even as it relates to who one might contact if they had questions?

Brian Hoshaw: Sure, if I can address the SRO side. So before the meeting, when you're meeting, when you get assigned a study section, you'll see the name of the scientific review officer, that's your point of contact before the meeting, as far as logistics, the date, when to expect your scores and critiques, information on how you submit supplemental material, if that's applicable for your application. So they're your point of contact up until the meeting. Then I'll hand it over to Tracy.

Tracy Rankin: Sure. Once basically scores are released, and scores generally tend to be released after a review meeting within 48 hours, you'll get a score. And that's when you should be considering to reach out to the program officer. In program, we are in the institutes. So we are taking the information produced in the summary statement that is forthcoming, of course, six weeks or so after the meeting, and using that to formulate a funding strategy. So we are your point of contact once you have that score. And certainly once you have that summary statement, as Brian mentioned, program officers are right there on the front page, upper left, email phone number. Please reach out to us to discuss the next steps after review. You know, what is this information going to be used for in terms of making a funding strategy and talk about various timelines for funding if your score is really great, or strategies, next steps for a resubmission if the score is not as competitive as you would have liked.

David Kosub: Let's actually jump on that just a bit more, Tracy. So, like, you mentioned a lot of good things right there. So what would be like the first thing someone should do when they get that critique?

Tracy Rankin: Well, certainly read it. I mean, the score should be of no surprise to you, because you've seen that six weeks prior when the scores have been released. And depending on where you're sitting, most institutes have a funding policy that they publish. So you might be in a position where you know what the pay line is for a particular institute. And if your score's within pay line, you know, you can read the critique and be very happy and excited about the potential for funding. On the converse, if the scoring isn't that great, you should still read it, but perhaps take a few deep breaths, maybe depending on the intensity of your reaction on the score, take some days before you decide to reach out to a program officer. That way, everybody's in a good emotional space to have a constructive conversation about the science you proposed in your application and the critiques you're receiving. We're all human. If, you know, the score comes back and you're not particularly happy with it, totally understandable. But then let's have a great conversation about how to make that application stronger.

David Kosub: I always appreciate it when people are reminded that program officers and scientific review officers are human too. And it's also, you know, I love hearing, you know, take a breath. I think that's critically important, you know, when you're doing this, and this kind of leads me into, like, what I want to ask next is, you know, is do you provide the same sort of advice, you know, if this was someone who's well-established in the research career, or different than maybe say someone who's earlier in their career just kind of starting off and maybe only has one grant rather? You know, do you provide the same sort of advice?

Tracy Rankin: Well, every program officer has sort of a different approach to these conversations. From my standpoint, I like to have these conversations investigator-driven. Established investigators, they've been to this rodeo before, they've seen many summary statements over their career, they may just want to be getting straight to the point about funding, right? Is this salvageable? Is this project something you guys are still interested in? Et cetera. Whereas an early stage investigator will have a lot more questions about process and timelines, and, you know, how to really work through a summary statement so that they can get the most out of it for their resubmission. That said, I always, my approach is always to have the principal investigators start the conversation, and I ask them, what did you take away from the summary statement? And how can I help you with some additional context? Or really helping, you know, for me, my standpoint, focus on those, as Brian said, score driving issues, if it was discussed, that are present in that resume, and ensuring that they touch on those in the resubmission.

David Kosub: It's all about that resubmission. If someone is thinking about it, we do have a podcast that we recorded on resubmission. So I definitely encourage folks to listen to that. This next question kind of aligns with that, and kind of along the same line that what Tracy was also mentioning, it was just like, you know, how could someone kind of address this feedback? You know, some of it may be positive, some of it may be negative, or be viewed that way anyways. You know, how would you recommend someone kind of addressing this feedback, you know, if they're thinking about doing the resubmission?

Tracy Rankin: Sure, I'm happy to start off and Brian can certainly jump in with some reviewer reactions to various strategies for resubmission. Certainly, please keep the mindset that the reviewers are actually trying to help you. They are not there to attack you and to bring you down, they are really trying to help you improve the science proposed in the application. It's always good to not annoy them, certainly, with an introduction to a resubmitted application. And be very respectful, even if you don't particularly agree with some of those critiques you're reading in the summary statement. Start off positive, thanking them for their time, because it's a lot of time to review all of these applications, and they are not being compensated really for that time that they're spending with your proposal. And then, very much focused on the resume. If it's discussed, the resume will outline what those strengths and weaknesses are. Even for high scoring applications that you've just missed the pay line, there might not be very many points to address, but you should make sure you address them. I have rarely read a summary statement where there are no critiques, even if it's highly scoring application. So there's always a wiggle room for and always room for improvement. But again, you know, be respectful. You don't want to put the reviewers in a negative mindset right from the get go because that's the first thing they're going to review and that revised application when it comes back in again.

Brian Hoshaw: Yeah, in all of the-- When you have a resubmission, you have a one-page introduction to address the comments in the last review, and the ones I've seen that are very good and that are very effective, as Tracy said, you know, address- try and address the score-driving issues and especially things that you can change. You have one page to address, you know, there'd be a lot of comments in the resume and all the critiques and some are major, some are minor, you're not going to get to all of them. So I would say score-driving one, and the ones you can change. So if someone says, one reviewer says, for example, you know, the application didn't have enough expertise in this area, and you were able to recruit someone in that area, then saying, "We noted the comment, we're able to recruit so and so, you know, the bio sketch is listed in the amended application." So address what you can, I don't know how helpful it is to sort of- if it's a difference of opinion on something, to try and counter it unless you have, like, data or proof to show to back up your argument, because if it's an issue of a difference of opinion, I'm not sure if it's worthwhile just using your time effectively there. And some of the ones that are really good will take the first sentence or two to say, oh, we thank the review, they note that it was high significance, you know, squeezing in a couple of strengths that they've noted from the last critique, because the reviewers for the amended application will have access to the last summary statement, but I'm not going to read the whole thing, so kind of pointing out some of the highlights can also be helpful. And then finally, one thing that will annoy the reviewer is if you try and cram as many words and text into one page as you can. So it's just sort of, you know, pushing the envelope of font size and margins, like, you know, the page, they're reading a lot. So just be respectful of their time as well.

David Kosub: Don't overstuff those applications.

Brian Hoshaw: Exactly.

Tracy Rankin: Someone pointed out in a review meeting way long time ago when I started, "Don't give the reviewers a headache." Right? You don't want your review of your proposal starting out with "This application gave me a headache because there's so much font, because there's so much text." Give them a break, give them an eye break.

Brian Hoshaw: Right. And remember, the reviewers have to follow the same rules for their application. So they know when PIs are breaking those rules or pushing it, and it doesn't help the reviewer, that's for sure.

David Kosub: Well, Brian and Tracy, this has been great, a lot of great advice. Before we go, I always like to give folks an opportunity to say any final thoughts, you know, you might want to, if there's any point you want to reiterate before folks go or bring up something new, I turn the floor to y'all.

Brian Hoshaw: So I'll just say, I think Tracy mentioned this earlier, but don't take the comment in there personally. It's review, a lot of it is opinion, you might disagree with it, or something you can address, or if you can take those comments and improve your application. But it's not- reviewers are also scientists, they are not just there enjoying giving out, you know, pointing out weaknesses and that sort of thing. A lot of it is- you know, take it as feedback and the best you can use that to improve your grant.

Tracy Rankin: Absolutely. And even if your application is not discussed, we've mentioned the resume a lot of times, a lot during the last few minutes. If you don't have a resume, it is still really worthwhile to contact the program officer, talk about your proposal, and talk about potential resubmission strategies and ways you can approach the critiques that you've received. Even though it doesn't have a resume, and we didn't hear a discussion during the review meeting, we have read a lot of summary statements during the course of our time here at NIH as program officials and listened to a lot of review meetings, and we can help you do a qualitative coding exercise on that summary statement if you will to really point out the high level issues that you need to think about in your resubmission.

David Kosub: Well, Tracy And Brian, I appreciate your time. This has been wonderful in sharing your thoughts on how our applicant community could use reviewer critiques and program input as they, you know, consider moving forward on a potential grant application. If anyone wants any more information, I welcome you to visit the NIH's grants website. There's a wealth of information on the application process and for folks to learn more about. This has been David Kosub with NIH's All About Grants. Thank you.