Christi Keene: Hi, everyone. We'll go ahead and get started today. My name is Christi Keene. I'm the Senior Director of Research Finance at the University of Chicago and the Co-Chair of the Finance, Audit and Costing Committee with FDP. I am joined today by several of my colleagues from FDP and NIH, and we are very happy that you are spending this next hour with us, hour and a half. So we'll go ahead and get started. You have the Q&A feature available where you can ask questions. We will be monitoring that and will go into the next slide and talk about the agenda today. All right. So we'll give you a background on the pilot. If you haven't already heard some of this, we'll fill you in on what you've missed. We'll give you some updates on phase one of the pilot. We'll talk about the round table discussions and the feedback we received at those. We'll also talk about the DMPTool. I'm sure many are excited to hear about that. And then we'll make a call for additional participants. All the while, if you have questions, again, use the Q&A feature, and we'll be monitoring that. Next slide. Okay, so a bit of background, if you haven't heard about the FDP pilot, under the new NIH Policy for Data Management and Sharing, each Institute and Center and Office has the flexibility to develop more specific requirements to meet the needs of the particular field. However, significant variation in these IC-specific requirements can place substantial administrative burden on researchers to navigate the requirements before they can begin developing their plans. In other words, each IC can have their own template. So the background, the aim of the pilot is to streamline those templates. So this pilot is a collaboration of FDP and NIH, and we've engaged with the ICs, Office of Extramural Research, subject matter experts and the Office of Science Policy. Additionally, OPERA and Compliance has been very much front and center. And we aim to generate greater consistency in the plan requirements across all ICs and programs, and we want to mitigate the administrative burden for researchers associated with the plan development and implementation. Next slide. So we have decided to break this pilot into two phases. The first phase, underway right now, is focusing on the templates. So we're testing the effectiveness and usability of two DMS Plan Templates. These were developed in collaboration with representatives from participating ICs. We have lovingly named them Alpha and Bravo. Alpha is a prescriptive template designed to limit the need for free text entry. So it's a little bit more smart form. If you answer this question "yes," you get another question to answer, whereas Bravo is kind of a hybrid between a smart form, prescriptive text and more free text. We will gather the data from the researchers and those who are preparing these plans. Find out what worked for them. What information did the plan request that they anticipated? What didn't it request that they anticipated? And then we'll also get the NIH Program Perspective, very important to understand what works, and what did the template provide to the program officers? And how did it help them review the plans? So again, that's phase one. It's well underway, started in March. And then we will move to phase two, which is the cost policies. And there, we aim to establish common cost principles, identify types of costs required and determine how to identify additional or unforeseen costs that may be required to meet the spirit of the data-sharing policy. So we plan to kick that phase off in December of 2023, which I think will be here before we know it. And so as we get closer to that phase, we'll definitely give more information around that. But we are very much focused on phase one at this time. Next slide. And as I said, phase one kicked off March 2023. We have 20 institutions who have formally agreed to participate. Thank you so much to those institutions that are participating and providing their data. We definitely appreciate all that you've shared with us thus far. And additional institutions can share the templates with their faculty and use them, but you don't have to fulfill all the obligations of participation and we recognize that there is a slow start in the first quarter. But June 5th is here very, very soon and July 5th for those big proposal deadlines, so we anticipate that we will see a lot of data roll in after those two deadlines. And now I will hand it off to Melissa Korf.
Melissa Korf: Thanks so much, Christi. Hi, everyone. I'm Melissa Korf. I'm Senior Director for Research Contracts, Data and Security at Harvard Medical School, and I also co-chair their Research Compliance Committee with FDP. So here with my colleagues Christi and Jim representing the FDP Administrative Member Organization contingent. So we ... In addition to some of the more traditional data collection, quarterly reporting, Qualtrics questionnaire, feedback collection, components of the pilot, we wanted to make sure that we also had opportunities to be sharing a lot more qualitative feedback with each other. So we decided on a series of round tables and town halls like we're doing right now. The round tables are intended to be smaller group discussions where we can have really focused conversations on the experiences of organizations participating in the pilot. These will be by invitation only so that we can make sure that the participating group is small enough to really foster that candid conversation that will help us move the dial forward. And it's limited to organizations participating in the pilot. Those of you that have agreed to help us out and be our much-beloved guinea pigs with the templates and sharing your feedback, this is an opportunity to share that feedback directly with NIH and NIH program staff. All organizations that are participating in the pilot will be invited to participate in one. These sessions won't be recorded because, again, we're really hoping to foster really candid conversation that helps us get the feedback that we need to kind of make this policy implementation work for everyone. We already had a session on May 16th, and we'll have two additional round tables upcoming on June 14th and June 29th. So if you're participating in a pilot, be on the lookout for an invitation to one of those opportunities if you haven't already. And there's this thing coming up at the end of September. Here, it's the end of the government fiscal year or something like that, so we may take a hiatus from these round tables in order to allow everyone to do what they need to do for fiscal year close and then look to potentially schedule some more opportunities after September. During our first round table, we did hear some really excellent feedback. We particularly invited those institutions that have already had researchers give the pilot templates a try and complete a Qualtrics questionnaire. So they'd already gone that extra mile of trying the templates, filling out a questionnaire, and we invited both the administrative compliance library reps from those institutions as well as the faculty members that had tried out the templates. And asked them a series of questions to try to gauge what was working, what wasn't working, what were their suggestions. And we did hear a couple things. There were some technical challenges using the Alpha template, so those of you who have looked at the templates, Alpha is currently published in a very smart PDF format. It's wonderfully programmed, and it's a beautiful form. However, it is very technical in its nature. There's a lot of coding that went into creating that PDF, which means that it can't be opened in a web browser, unlike some other PDF forms. You actually have to select to download it, which I think you can right-click on the link or set your browser only to download documents, but you do need to make sure that you open it in the desktop application, and further, you can't open it using the free version of Adobe. You do need to have a license for Adobe in order to open the form. We're working on making sure that we're posting some of these instructions to the website so that everyone hopefully knows what we know about avoiding some of those technical issues and working on whether or not there is a more elegant solution. But if you do go take a look at that template and want to give it a try, please make sure to download it rather than trying to open it in a web browser. One of the other pieces of feedback that we heard was general discomfort, perhaps, with the very specific, concrete nature of the information that's requested. So the narrative sample format, the more narrative-style sample format page that's published right now is very paragraph in nature, and it's a little bit easier to provide the information that you have that you think is answering the questions, whereas what we're hearing from our NIH program colleagues is that they need to be able to quickly identify certain information and that certain information has been provided in order to review and approve the plans. And so the templates are, you know, asking for very specific information about the repository that is intending to be used, the data that is being collected. And at some level, that can feel a little uncomfortable, trying to make some of those decisions 9 months before a project initiates. So in talking with some of our NIH colleagues, and we need to get into this a little bit more in the Q&A portion of this session, as well, you know, there is recognition that there may be a need, just like with any other component of a research project, to make updates as the project moves forward. And so something like a repository, changing that may require prior approval if an update is necessary. But there's an expectation that researchers will be making a good-faith attempt to provide the information that's requested and that's necessary and that similarly, if in making that good-faith attempt, there is a new repository that's identified later down the road or the research needs change, a new data type needs to be generated, these are not insurmountable things. It may require a prior approval request to make those updates or to make those updates during the RPPR phase. But there's an understanding that the data management sharing plan needs to be a living document and may require updates. So even though we're being asked to provide a lot of concrete information at what feels like an early stage, there is an understanding that changes may need to be made and prior approval requests submitted. The other type of more dynamic opportunity that we're using for sharing feedback with each other is town halls like this one. These listening sessions are intended to be for a much broader audience. It's not necessarily just limited to organizations that are participating in the pilot. We're inviting everyone with an interest in participating and an interest in the pilot. This session, as any of other of the town halls that we have, will be recorded, and we'll be sure to share the recording on our pilot web page. There's a link here in the slide, which I'm sure we'll share as well afterwards. But if you go to www.thefdp.org, right up at the top of the header of the website there is a link to the NIH DMS pilot web page, as well. Our next session of these will be on Monday, July 17th, and so you know, through similar channels like you received the invitation to attend today, look for registration information forthcoming shortly for that session, as well. And similar to the round tables, after September we may look to schedule additional of these opportunities. So just a little plug for the web page I just mentioned on the other slide, we do have a baseline website for the pilot that is available, and there is the link in full text. We've got a wealth of resources there already and are working on posting more. Bear with us, those of you who are able to join us at the FDP meeting last week, might remember a really exciting update from the communications committee about a beautiful new website they're working on building for us. But unfortunately, that means you know that our platform for a previous website isn't being updated because we're really putting a lot of energy into the new website. So it is a little bit more of a technical challenge for us to make updates to the existing website, and so we're working on adding some additional materials. But it is a little bit slower going because of those technical hurdles. But there are slides and video from presentations and meetings. We'll post slides from the FDP session last week, slides in the recording from this session and other opportunities to share these types of material. The both Alpha and Bravo DMS Plan Templates are available. For both, there are two versions, one version that is just the template, so it's nice and clean for researchers, a great point for them to start from; as well as the second version that has a little bit more material associated with it, maybe the instructions or a glossary, things like that. There is the quarterly report template that is posted. We're working on posting a sample invitation to faculty that you might want to reach out to and ask them if they're willing to participate in a pilot and give the pilot templates a try. As we're able to start posting preliminary results, we'll post those reports there and other resources as they're identified and we're able to produce them. And now I am very excited to be able to hand-off, so I'm going to stop sharing, and my colleague, Maria Praetzellis from DMPTool is going to start sharing and share with us some of our exciting progress and a little bit of a demo.
Maria Praetzellis: All right. All righty. Hi, everyone. Thanks for having me. So I am the Product Manager for the DMPTool. And before I jump into the FDP Pilot, for those who aren't familiar with the DMPTool, we are a free tool. I always like to be really clear. We're not a vendor. We are run out of the University of California. We're community-supported. We've been around for over 10 years, so we're very established in this space. We've been working with data management plans and supporting the community of data librarians for quite a long time now. Really, the goal of the DMP tool is to support communication between librarians and researchers and to give the ability for this communication to happen at scale. So I know a lot of data librarians have been sort of tasked with a lot of extra work with these new NIH requirements, so DMPTool can be a way to help facilitate giving information and guidance to researchers at a scalable method. So we have funder-specific templates for all of the big federal funders, also private foundations, NIH, of course. That's what I'll talk about today. And really, our focus for the DMPTool is to make sure that we have best practice guidance to make sure that data management plans are structured and optimized so that they reflect best practice and that we have good metadata about the plan itself. So DMPTool is pretty large. We've got 384 participating organizations at this point, so many larger academic centers in the US. We have a growing number of medical centers that have joined us, as well. And we are supported by a wonderful group of data librarians who help keep everything updated. So we have an editorial board. Some of these folks are on the call today, and we also had a wonderful NIH working group that was chaired by Nina Exner, so I saw on the participant list for this call. So they really helped me produce some great guidance and sample language for an NIH template that we have in the DMPTool now. So for the past few months, we've been working on adding Alpha and Bravo to the DMPTool. It took a little while because we needed to build a few features to accommodate some of the types of questions that were included in those two templates. So that's why we didn't release it right away. So we got some extra development time, and we were able to put those templates into the DMPTool. So right now, we have three templates in the DMPTool for NIH. There's the NIH GEN, and this was done by the working group that I just mentioned. And then we have the FDP Pilot Alpha and Bravo. So when a researcher goes to the DMPTool and they select NIH, they will be presented with those three choices of templates. So we do have all of the customization, all of the normal features that are available for templates within the DMPTool. So if you have a managed account, and you have an administrator for DMPTool, they'll be able to customize those templates as normal. And so by customize, I mean they can provide local guidance if they have ... Maybe your institution has specific resources or information you want to share pertaining to sensitive information or selecting a repository or you know, research IT. There may be certain areas that you want to draw your users to get some local assistance with their research. And so customizing templates is a good way to do that. Another feature that we have that will be ... is available for these templates is the ability to provide feedback on things. So this is a feature that is something that an institution has to enable. It is off by default. Some institutions don't have the resources to really handle providing one-on-one feedback. But if you have enabled it, researchers will see a button when they're writing their plan to request feedback. And when they click that, they receive an automated e-mail notification that also goes to the ... you know, whoever the administrator is on the institution side. And then you can have a back-and-forth about whatever the questions are pertaining to that data management plan. We have full documentation on these, and I have a link up there on the corner for more information on those features. So just a few tips on things I think would be useful, particularly thinking about the FDP pilot. Administrators have an admin dashboard. So this can be a nice way to get a quick view into who's filled out these templates. It will tell you the creator's name, and it will give you their e-mail address. So if you wanted to follow up, you will have an easy way to access everybody who has used that template. There is also customized links to templates, so if you want to share a link to the template so that people don't have to go through multiple clicks to get to the right place, you can send it directly to the Alpha or directly to Bravo template. And so I think that those would be good sort of shortcuts. If you have the staffing, enabling the feedback feature can be a good way to work closely with researchers and answer questions about the plans. And then, of course, adding guidance or links to local resources is kind of what DMPTool is all about. So you can customize those templates. So I just want to show real quick ... I'll make sure you can still see my screen. So this is where ... I'm kind of doing a shortcut here. These are all the NIH Templates that I mentioned. I think this is the quickest way to enter a plan. From here, I can create a new plan. This little clipboard is where I can copy the direct link to that template. One thing to know about that is, if I update the template, say we need to add a new data type or some of the things Melissa was talking about. If we need to iterate on the template a little bit and I change the template, the link changes. So you should check it every once in a while and make sure that any links that you share are still working. So here we are. I went into Alpha. It looks like normal DMPTool. This right plan is where we basically took the ... This was the PDF version, I think. We took that, those questions, and then just put them into the normal DMPTool format, and that's all in the right plan. They can, of course, set their plan visibility. It's going to be private by default. We encourage folks to register for a DMP ID. This is a persistent identifier for your plan, so it's a good thing to encourage folks to do. And of course, they can download it. This is where you go, and you can get your PDF that you could use to submit the plan. And I'll just click over because I want to show you Bravo. Bravo was a little bit different. So Bravo, here, we put much of the plan here in the sort of traditional narrative structure of a data management plan. But we also instruct researchers to go to the Research Outputs tab, and this is where we're collecting information in a more structured way about the specific outputs that are going to be generated through the course of research. So it's the same data types that you'll see in that Bravo template. You select a repository. If the repository is not listed, you can add your own custom repository and same with metadata standards. So we took all of the sort of fields that we had in that original Bravo template and made sure that they were all reflected here in research outputs. And the reason this is important to do it this way is because we get good, structured metadata about the project, which is important for down-the-road tracking. It helps us with machine-actionable data management plans, just something we're all working towards. So getting this good structured information about research outputs is really important. So I think that's the basics on the templates. Let me just see if I had more. Again, we've got documentation on our GitHub. We've got some tutorials on how to customize templates. If you're not a participating organization, and you'd like to be, you can e-mail at [email protected]. It's free to join, and we're happy to help onboard people if they have questions. I can also answer those, but that is what I have to share, and I don't know, Melissa, if you want to save questions for the end.
Melissa Korf: If it's all right with you, there are a few questions in the Q&A ahead, I think might be helpful to answer now, so ...
Maria Praetzellis: Okay, sure.
Melissa Korf: I can show you some of them . So ...
Maria Praetzellis: Yeah.
Melissa Korf: You answered this sort of in your recent comments, but is DMPTool free? So it's free to join, but then also you don't have to join in order to be able to use it. Is that right?
Maria Praetzellis: Yes, so DMPTool is completely free. We are not a vendor. We're run out of the University of California, so it's a completely free service. But you're ... You don't need to be a participating organization to log in and create a plan. What you need, if you want to customize a template, if you want to add local guidance, if you want to implement that feedback feature, that's the instances where you would need to sign up as a participating organization.
Melissa Korf: Thanks, and so one question sort of related to some of those benefits, if an institution created institution-specific guidance for the NIH GEN Template, the one that existed previously, is it possible to copy that over into the new templates?
Maria Praetzellis: That's a great question. It is not possible to do it in an automated way. You would need to just go in and copy and paste it over because the fields aren't one-to-one. There's no way to map that in a way that would work. So it needs to be a manual thing where you can go in and copy what you had pertaining to a specific element, and then you can paste it into the corresponding question in the Alpha and Bravo template.
Melissa Korf: And are the templates live now? Or ...
Maria Praetzellis: Yes. Yes.
Melissa Korf: They are.
Maria Praetzellis: I published them yesterday, so they should be up. I haven't looked to see if anyone has used them yet. But, yes, they are up and live now.
Melissa Korf: And then Zach has asked a very nice question about some of the fancy functionality. I'm going to ask one more tactical question and then get to that one just so that Zach doesn't wonder why I'm posing other questions first. But can we explain why one would use DMPTool versus just directly using the FDP Template as a pilot? And I know how I would answer from my own institution, but, Maria, what do you think?
Maria Praetzellis: Yeah. There's lots of reasons, but I think for one, we're giving people options, right? It's not, we're saying that there's one way to do it. If they prefer to use the PDF, that's fine. The benefit of using DMPTool is institutions can add their own guidance. You can have that feedback feature. You can easily see everybody who has filled it out so that you can get back to them and ask them about surveying. It also provides structured metadata about your project so that you can get a DMP ID, so you can get a persistent identifier for your plan, which can help with tracking research. It also connects to your ORCID record, so researchers are able to demonstrate that they're good stewards of their data and then transparency in their research by being open about their data management plan. So there's a whole list of kind of reasons why you'd want to be participating in that sort of open system. But I don't think we're trying to be prescriptive, right? If a researcher wants to use the PDF, they have that option, and that's fine.
Melissa Korf: I will say, at my own institution, from the perspective of a group that uses DMPTool, our data librarian really likes some of the collaboration functionality because if someone is asking for her help reviewing the DMPTool, and they share the DMPTool with her, then she's able to see all of the dynamic changes that they might make while she's also working to review. So she's had some cases that they're just sending her a static PDF document as an attachment to an e-mail. They've made changes by the time she needs them to go over things, and so it's just a really nice, dynamic way to work with others and ask others to help review different sections. We also love the ability to kind of have side-by-side with the template some of the institution-specific guidance, right? Right there in the column, links to our own policies and everything. So lover of DMPTool in our institution. So in terms of, you mentioned in that answer the ability to create a DOI, so a question, that DMPTool can make DOIs, is a superpower. And is that part of the workflow still opt-in? Or is it possible to automatically assign a DOI to plans in the future and link to ORCIDs?
Maria Praetzellis: It is still an opt-in. I hadn't really thought about making it automatic. I think I would be a little nervous because people use DMPTool in different ways. So sometimes we don't know when they actually consider something final, right? It might be something they're doing for a class, or maybe they've worked on it a bit in DMPTool. Then, they downloaded it to their laptop and finish it. So it sort of requires a researcher to say, "This has my blessing, and I'm good to go, and I want to get ... I want a persistent identifier." So I feel like it always needs to be something that is a researcher's choice like, yes, I want to mint this DOI rather than just doing it because I wouldn't know when it's done if they haven't said ... if they haven't told us.
Melissa Korf: All right. I think one more question that seems really two questions that I think are really related to DMPTool, and then I'll let you get out of the hot seat, Maria, is ... but I think that you're able to stay with us in case additional questions come up, right?
Maria Praetzellis: Yep.
Melissa Korf: So can you say more about the DMPTool Outcomes feature? Is this intended to align with the reporting of products in the RPPR?
Maria Praetzellis: Yeah. I like that question. Yes, we are trying to create the ability to ... Well, we've already built it, so we have the ability to connect data management plans with eventual outputs. Right now, it's manual, so you're just inputting the the DOI or whatever persistent identifier or URL points to the output. And then we connect that back to the persistent identifier for the plan as a way to track research over time. And we're also building out now the ability to connect to external aggregators of research outputs so that we can automatically update data management plans and associate them with outputs. I hope that answers the question.
Melissa Korf: And then the one last question before we move on, Maria, that it looks like DMPTool allows the insertion of hyperlinks in plans.
Maria Praetzellis: Mm-hmm.
Melissa Korf: However, NIH has reminded us that hyperlinks are not permitted in DMS plan attachments. So is it possible to disable the feature, or is there other guidance you can give on that?
Maria Praetzellis: That's a tough one because we support data management plans for many, many, many institutions and agencies, and some funders have no problem with URLs in data management plans. So I can't strip them out automatically, so it is up to the researcher, the creator, to not include URLs in there in their text. We don't generate ... If you're connecting to ... Say you've connected your output to a repository. We don't include the repository URL in the identifier, so we strip out URLs and any of the persistent identifiers that you've inputted into your plan. But if somebody is writing a narrative, and they put a link in, it's still going to be there. I can't strip them out automatically because I'd be stripping out ones that people intentionally wanted to be there.
Melissa Korf: All right. Thanks so much, Maria. All right. I'm going to finish up a couple slides that we have left. And we had heard from a number of institutions when we were initially kicking off phase one of the pilot that their institution had done such a big push with the DMPTool and a lot of training and were either strongly encouraging or, in some cases, requiring use of DMPTool for creation of the DMS plans that their availability in DMPTool was a critical factor towards their ability to be able to participate. We also have refined our understanding of the templates and have been able to share more information. And so if your organization is willing and able to participate now that these, the templates, are available in DMPTool or more research is available, less of a ... For any reason, we are accepting new participating organizations in quarter two of the pilot, which would begin June 1st, tomorrow. If you are interested in being a formal participant, send us an e-mail at [email protected]. And we'll send you a copy of the MOU for your review and signature. Also happy to provide any information that may be of use to you in determining whether your organization can participate. And then just a note that we're not limiting participation only to FDP member organizations. So if your organization is a friend of FDP and not formally a full member yet, or you received information about this town hall otherwise, please let us know if you're interested in participating. And so now is the really good part of today's session where we really want to hear from you. So just a reminder, our e-mail address, [email protected], if you have questions that you think of later, but we really want to be able to hear your questions. Use that Q&A to let us know what you want to hear more about. Share your feedback with us, and I'm going to stop sharing my screen now so that we can all get to some of the other Q&A.
Christi Keene: There's some questions in the Q&A about using the templates and providing feedback. Yes, we ask that if you do use the templates, even if you're not a participating pilot member, please share your feedback. That feedback is very important. We will share a link where you can provide that feedback in a Qualtrics survey. So stand by for that link.
Melissa Korf: There is also a question in the Q&A, and I think Maria is working on typing an answer. But just a note that the pilot templates can be used to actually submit. We recognize that if we were asking researchers to complete two DMS plans, one for their actual submission and one for the pilot, that would perhaps limit interest in participation. Also, we really want to be able to receive program officer feedback, as well, regarding whether or not the templates result in a DMS plan. They feel like they can review and approve quickly, has all the information they need. So a lot of benefits to being able to actually use the pilot templates in the actual submission. There was one question that received a lot of upvotes. Is NIH going to share the specific pieces of information they are looking for? And I see that Michelle has provided some information regarding the guide notices that includes the information that's required to be in the plans. And I would also just offer the observation that the pilot Templates were created, I want to say, in very close collaboration, but created by NIH program colleagues. So if you're interested in information on what Program is looking for in these plans, I would strongly recommend looking at the pilot Templates. That is what we're hearing from Program that they think they're going to need. So I think that they're actually a great rubric for the information that is being sought. I see that there is a comment that there are a lot of comments in the chat. If you have a question, if you add it to the Q&A, it's a little bit easier for us to track that we have responded to it. So we'll take a look at the chat, as well. I think a lot of questions, again, in the Q&A about whether or not institutions have to be formal participants in the pilot in order to give the Templates a try and provide feedback. And the answer is no. The Templates are out there, and so once they're published in DMPTool as they are now, they're accessible to anyone who logs into DMPTool and selects that they're creating a DMP to submit to NIH. I will just add, though, that participating fully in the pilot is a much more helpful way to get us the full feedback. And I think there is some suggestions in the Q&A about perhaps NIH could be providing reports. The information that we're looking for in the quarterly reports is really information that the institution is best positioned to provide, and it demonstrates a level of commitment to helping us obtain the researcher feedback, keep track of who is given a pilot template to try, keep track of who has submitted the questionnaire, help us get those folks to be completing the questionnaire. So if you are able to fully participate, our strong preference is that you do that. Next best is, if you have a faculty member who is interested in giving the template a try, they'll let them do that but make sure that they fill out that questionnaire for us.
Jim Luther: Melissa, the only thing I would add to that is, I think early on, we had somewhere in the neighborhood of 80 to 90 institutions that expressed interest. Is that right, Melissa and Christi, somewhere around there, 80 or so? And largely I think 50 or 60 dropped out because there was a very large contingent that really ... that institutionally they were pushing the DMPTool. And if the DMPTool wasn't a solution, then being part of the pilot was very difficult. As Melissa just shared, that changes in ... changes tomorrow, right? I think the second thing was concern about burden, in interacting with the faculty to collect this data, to have them do the survey and so forth. I would suggest that for those of you that have concern in that space, I don't think that we've found that it's going to be as burdensome as you think, number one, and number two, I think there's some significant value in working with your faculty, with your organization just in the process of communicating the pilot and so forth. And then I'd say the third thing to consider is the value proposition in being part of the pilot. That was shared in the opening discussion. Being part of this town hall is wonderful, but being in a small group, again, I think in our first round table, we had four institutions, and we had 15 or 16 different federal representatives from OPERA, from the ICs and so forth. That's direct accessibility to discuss your concerns, your faculty members' concerns and so forth. And then I guess number four would be, we're moving towards phase two, as Christi shared, about the costing aspects of it. Again, this is going to be another great opportunity to have the detailed conversations about costing, about Just-In-Time, about the program officer feedback and the peer review process and so forth. So I just wanted to mention those couple of things, and again, Melissa put the link out there, but if you have questions, reach out to us because we're happy to talk. Thanks.
Melissa Korf: Thanks, Jim. There was a question in the Q&A, as well, about if an institution formally participates as able and submits feedback. Would there be an opportunity to join the round table discussions? And unfortunately the round table discussions is one of the benefits of full participation in the pilot. So if your institution is interested in having a seat at one of those round tables, full participation is the way, to quote "Mandalorian." No love for "Star Wars" in my household. And there is also a question in the chat about an institution that considered joining, but it wasn't practical to try to get all the faculty on board with using the Template. So I just wanted to note that we recognize that it's hard to get faculty all using the same form and that we're asking for a little bit something special. We've tried to limit the Qualtrics questionnaire to maybe 5, 10 minutes to complete. But still, it's something a little bit extra, and not all faculty are able to take the time to do that. So we've really only asked participating organizations to commit to trying to get about 25 percent of submissions to use one of the pilot Templates. So we recognize that it isn't practical for most institutions to get 100-percent adoption to participate in the pilot. But then ideally those faculty that give the pilot templates a try, we're looking for hopefully 75% return of the feedback questionnaires. So if that helps at all, I think we're not looking for perfect. We're just looking for a representative sample of feedback.
Christi Keene: There's a question in the Q&A around phase two of the pilot. That's the costing phase. We're looking at December to kick that off. We're still very much in the early planning phases of that. We feel that by using the templates and capturing the information that is needed to create a good data management plan, it will be to our benefit to use those as artifacts to determine, what are those costs associated with what you've said you're doing? So using phase one to inform phase two and definitely more to come on that. And again, early phases, so I think we'll be making a call for participating institutions as that gets closer.
Melissa Korf: So there is one question that I can make an effort at. But I'm hoping my NIH colleagues will chime in, that the NIH SF 424 encourages that the DMS plan be limited to two pages. DMPTool does not restrict investigators from exceeding this page limit for the NIH template. And the question is, are the FDP templates within the two-page limit encouraged by NIH? And I believe the answer on that is no. If you just ... If you look at some of the templates without adding any information or potentially adding lines or tables, they exceed the two-page limit. I will say that some of that is a result of the formatting, and the formatting also facilitates ease of finding the information that you're looking for versus trying to find information in a much longer DMS plan. And it's not a page limit. It's encouraging ...
Michelle Bulls: There you go. That's the key. Yes.
Melissa Korf: Yeah. I don't know, Michelle, if you want to add anything on that.
Michelle Bulls: No, I don't. I was just waiting for you to say it's encouraged. That's the key. And we have answered that question, and I think it's actually out on the FAQs, as well. Did you want us to talk a little bit ... The question came in about the phase two. Christi answered it, but I wasn't sure if you wanted us to just give a little bit more about what we would be thinking about phase two, or how would you like to do that?
Christi Keene: Yes, let's do that.
Michelle Bulls: Okay, so one of the things that we were thinking was that we know and recognize that there are some challenges with several different areas within the phase two and the budget area. We understand that there are DMS implementation costs for programs that already have budget caps at NIH. We also recognize that there may be additional administrative burdens on the applicants and our recipients. We understand also that there is the time where you're projecting cost, and you really don't know what those costs may be, especially post-award or post the project. There are fees and all kinds of things that are being looked at. And so looking at the phase two, we want to be able to collect that kind of data. Look at, is it going to be a lot for repositories? How can we help our recipients budget for that? How can we help our recipients stay out of audit risks? How can we help our recipients with these challenges for the projections? The archiving of data, it's so many different areas that our recipients at our FDP, our colleagues have raised, and I think it's just important for us to take the time to unpack each of those areas and take a true look at what's actually needed, what the costs will be, and whether or not those costs will impact both NIH and the recipients, and we know that that will happen. We've seen COGR studies. We've heard from you guys firsthand. And so those are the kinds of things we're going to be talking about. But the main thing that I think we've heard out of the gate really is about the budget and the single line item budget. That has been a tremendous source of concern for our FDP colleagues, our recipients as well as for OPERA and NIH, as well. And what we are looking at is walking the single line item back and requiring our recipients to place those costs in the appropriate categories so that they map to our cost principles and make sure that we don't do anything that would put ourselves as well as our colleagues at risk. So with that being said, we're looking at trying to do that as we go forward in the next fiscal year because we don't want to co-mingle two different policies. Christi and I will put our heads together and work with leadership to make sure that that guide notice gets out. And we'll look at all of the other implications of that, changing the budget instructions and that sort of thing. But we do recognize that asking the recipients and applicants to place costs in categories where they don't fit is probably something we need to be rethinking. So with that being said, I think we're really excited about seeing where those costs come up in the various categories and identifying whether or not we need to be thinking about working with OMB and other people to identify how those costs are reflected going forward. But many of the costs are in categories that we already have, so having our colleagues do that would be the best way forward. But with that being said, I don't think I have anything else to add except for, I'm really excited about what we gather in the first phase because that's really going to catapult us and inform our second phase, so that's it for me. Kristin ...
Kristin Ta: No, I think you covered it well, and we are going to be working to collect feedback internally, as well, on what folks are seeing, whether that's meeting our needs so that we can make sure that we're getting what we need while keeping the burden as minimal as we can on you all because we know it is a challenge.
Michelle Bulls: Yeah. Thanks, Kristin. Thanks, Melissa.
Melissa Korf: So there is one question in the Q&A just sort of asking for clarification on that. So this would be a planned elimination of the additional line where we would assign costs to the standard categories.
Michelle Bulls: Right.
Melissa Korf: It's all of the other costs, right?
Michelle Bulls: Right, right, because right now, we're saying put all of the costs in the single category. But there are personnel costs. There are supplies. There is equipment. And those really impact indirect cost rates. So we really have to be careful about that, which is why we are looking at just having those costs being placed in the normal cost categories as we move forward with the second phase of the pilot.
Kristin Ta: And we would still ... You would still identify in your budget justification how ...
Michelle Bulls: That's exactly right.
Kristin Ta: Right, so it's not that it's completely disappearing, but it just wouldn't be in the single line.
Michelle Bulls: Right. You do need to tag it as DMS or GDS. Yeah. Did we forget anything, Jim or Melissa or Christi?
Melissa Korf: Just in terms of phase two, Jim, I don't know if you want to share any highlights from the thought exchange as part of our planning for other topics. We did a thought exchange leading up to the FDP meeting last week.
Jim Luther: Sure. Yeah, for those of you not familiar with it, a thought exchange is a way that individuals get a prompting question, and in this case, it was about phase two and costing and questions or concerns. And then individuals put thoughts there, and then as other individuals come in to add thoughts, they also get the opportunity to put one to five stars on level of importance for the individual items. So I think we had 80-plus individuals respond with 250 different thoughts. Based on what we looked at last week, the top three or four were related to single line, which Michelle and Kristin addressed. I think also high on the list was just general questions about, hey, I have a faculty member and a postdoc and a grad student and these other costs. How do I know how much of what they're doing really should be classified as related to data management and sharing? And I think there's lots of things on various NIH websites, and I think this will be continuing to be a topic of discussion and supported by NIH as we go through this. And, Melissa, we've had recent conversations about the importance of culture change within organizations, and that is a new way of thinking. So this is clearly going to be an issue we can't solve today, and it's clearly a consistent issue that when individuals are looking at cost in a grant, breaking them out and putting them into that category is difficult. So a lot more to come on that. We think that the pilot will support that. And then I think the other most significant topic was related to budgeting expenditures that may be related to storage costs after the end of the period of performance. And I think those were the three highlights, all of which will be certainly clarified throughout the next couple of months with the Just-In-Time process feedback from the program staff as well as when we move into phase two. Michelle, Melissa, Christi, Kristen, any additional thoughts, anything on this?
Michelle Bulls: No, I think you really captured it nicely, and I do think that a large part of what we collect here is going to truly inform that thought exchange. I think we're going to see much more of it, which is why we really want folks to get involved with the pilot because that's the only way we're going to know how to make some changes or look at what we're doing in that second phase. It's critical. And folks consistently have talked about the fact that the second phase is an area of concern that they really want to hear, see and understand how this is going to make a difference. We really need to have the up-front, here-and-now participation so that we can really drive some different thoughts and really come up with other ways that we might be able to capture these costs and what recipients are facing and what we are facing as a federal agency, as well.
Melissa Korf: So there's one more question in the Q&A kind of related a little bit to the single line item. Would we still have to split the effort of personnel for DMS activities and the budget justification? And I think if I understand your comments correctly, Michelle, it's maybe not necessarily split, but in the budget justification description of DMS costs, some indication of how much effort. But also in the thought exchange, I think we heard several comments that researchers are - especially those that haven't had to do a DMS plan previously - a little uncertain exactly how much effort to estimate. So in phase two, one of the things that we might be able to take a look at would be tools or resources to help them estimate that effort. We do still need to think about what that's going to take, even if it's not separately budgeted in a single line item.
Michelle Bulls: And I think what we also discussed, which is really important, is that we do know that there is effort. There is levels of effort going into this, right? But people are saying zero. And that's a little concerning, as well, because then that forces us into another category of other things, cost matching, cost sharing, that kind of thing. So we've got to be careful with that, too. So especially for just understanding the costing policies and understanding the cost principles and making sure that the effort that you're putting in is accounted for, it's just going to be important. And if it's zero, it's zero, but our instructions and the understanding and the tools shouldn't force that to be zero. That needs to be accurate, so ...
Melissa Korf: We do have a couple additional questions about phase one in the Q&A, and so one of them is, if we want to encourage faculty to choose Alpha or Bravo in the DMPTool, what would you suggest we advise them? And I'm not 100 percent sure what that's getting at. Is it asking if we would recommend one template over the other? But I think I would encourage my faculty to take a look at both and select the one that they think is going to work best for their proposed project. I might, knowing the templates the way that I do, if I have an idea of what their project is, I might steer them in one direction or the other. But both of them are great templates. If the person that submitted that question was looking for something else, if you can give us a little bit more clarification, or if ... I don't know if anyone else has comments on that question.
Christi Keene: I'm not sure I'm going to get at the question, either, but what I would say, at least at my own institution, is ... and again, this varies based on institution. I'm at a very decentralized institution where we provided guidance to researchers on how best to address the six elements of the plan. And we use DMPTool quite heavily, so I think my tactic is to meet with researchers early on and share with them the options and, again, like Melissa said, what works best for their needs and is ideal for what they're doing. And then, again, I think because we do use DMPTool pretty heavily, it's kind of the natural place people go. Their plans are saved. But I think whether it's the templates, the PDF, the Word document, DMPTool, it's really working with researchers and making sure the supports are in place at your institution. And I think that looks different for everyone. I know at my own institution the libraries were very much involved, the research computing centers. It was a very collaborative approach to implementing the policy, but again, it looks different everywhere. I think we have kind of a template letter that we've shared that you can send out to your faculty. I'm happy to share that again. But I think it's ...
Melissa Korf: So it's ...
Christi Keene: ... getting the word out.
Melissa Korf: I think we just got a little bit more context, and it says, versus using the Gen Template ...
Christi Keene: Ah, yeah.
Melissa Korf: People want to know why they should use Alpha or Bravo ...
Christi Keene: Yeah.
Melissa Korf: ... rather than the template that's already there. And for some of events, like some faculty, when I show them the template or the templates versus the more narrative format, they're like, "Sweet. I can type one-word answers. This will be so much easier. It just walks me through what I have to put. I put the answer there, and it's done. I don't have to worry about writing paragraphs and sentences. I'm just providing information. So in some level, depending on the faculty member, they may like that a lot better. It's easier than trying to string together paragraphs. The other benefit is, as we mentioned before, these templates were designed by our program colleagues at NIH to provide the information they're looking for. So I am not at NIH, and I cannot promise anything. However, if program officers are saying, "This is information that we're going to need to review and improve a plan using the template that clearly and easily provides that information," it can only mean good things. I don't know if you said anything ...
Jim Luther: The only thing I could add to that, Melissa ...
Melissa Korf: Yeah.
Jim Luther: And you said this many times in our small groups, but you said it publicly, and that is, there is likely to be one or two or more templates settled on at some point. We don't know if they're going to look at these, but they're not going to be free-form. And that's, again, what the value of being in the pilot is, is to have the opportunity to inform and be a part of what the future templates are going to be because it's not likely to be, "Here are the six areas that you need to fill in." It's going to be one or two or more templates. We don't know how many, but this is the opportunity for institutions and your faculty specifically to weigh in on this.
Melissa Korf: And then I think a great wrap-up question, if folks have other questions, please go ahead and pop those in the Q&A. We do have ... We have 18 more minutes. But someone just added one. Oh, okay. When are results expected to get released after phase one, and what is the intended format output? Will this be public or reserved for participants? So I think we're hoping to write a whole bunch of wonderful white paper reports, summaries of the feedback that we received. Those types of work product will likely be made publicly available on the FDP website, summarized data, no raw, individual faculty member or institution-level data. The pilot is a full year from March 1st, so we wouldn't anticipate that any kind of final reports would be available until probably a couple months, at least, after the end of the year to the extent that we're able to release some preliminary results in the format of reports. We would do that. Hopefully we'll be able to provide some summarized remarks from this session or from the roundtables and be sharing those. And I think, as Jim just mentioned, also a likely output is a refined template that then hopefully goes through the process and is adopted by NIH, but that would need to follow ... that would need to follow a process that Michelle and Kristin are far more familiar with and eventually hopefully be benefiting everyone, not just pilot participants. One question about, if an institution doesn't participate in phase one, would they still be able to participate in phase two? And I think the answer of that is yes. We're not making them wholly dependent on each other, right?
Christi Keene: Correct.
Michelle Bulls: Yeah. I think it would be helpful if they were in both phases just because it informs the other, but I don't think we can make it wholly dependent. We would just need to understand how those costs were captured, what tool was used, what format was. You know what I mean? So we just need to make sure that we understand how, if there are vast differences, I would think. So, Melissa, it's very interesting because I'm going to throw out a question to the recipient community, that it's very interesting, though we have instructions about the budget and about the single cost categories and the fact that recipients should be including them in the single cost categories, some of our program officials have received budgets where the recipient or the applicant has not complied with that requirement. Do you have any idea of whether or not folks understand that the existing policy is still in place? So y'all going to leave me hanging, right? Nobody is saying anything.
Melissa Korf: Can you say that again, Michelle?
Michelle Bulls: There are budgets that are coming in where the costs are not in the single cost category, which could be a challenge because it's under the current policy. Is that something that ... Are folks confused by that, or is ... Are folks just saying, "Look, we've got to be able to put these costs in the categories that they belong in so that we are not at risk"? Help me help you.
Melissa Korf: All right. I'm going to own it, Michelle.
Michelle Bulls: Okay.
Melissa Korf: The compliance risk for institutions like mine, for example, of breaking PI and key personnel effort out of the personnel category is so significantly high.
Michelle Bulls: Yep.
Melissa Korf: And it's so hard to do. It's nontrivial to ask a researcher to try to split apart just the piece of their research activity that's related to data management and sharing because as we all know, following best data management and sharing practices is a part of doing good research that many of our PI and key personnel are just not able to break it out. That said, if there are other costs such as repository deposit fees or server purchases or probably hopefully better than server purchases, cloud costs or something like that. Those, we do still suggest would go into the single line item.
Michelle Bulls: Right.
Melissa Korf: But I think a lot of folks are targeting repositories that are free right now, and so the most significant cost is probably that personnel piece that people are struggling with.
Michelle Bulls: Okay. Thank you very much for that. I really appreciate that.
Jim Luther: There are a handful of comments, Michelle, in the chat that are popping up, or in the Q&A. But again, to emphasize what Melissa said, there could be a zero in that single line because people don't understand, right? That's one scenario. The other scenario is they chose to budget it in the cost category that they felt was more important, whether that's key personnel or equipment or something like that. And I think ... or the third scenario is it's not in the single line as a number, but it is in the budget narrative somewhere that explains that there is a dollar amount. So again, it's not a budgeted number, but there is some reference to it somewhere. I think and I see some others - someone commented that I know some investigators are very confused about what falls into an F&A and what we can charge separately as direct costs that they're choosing not to budget at all.
Michelle Bulls: Yeah.
Jim Luther: I think it's all over the place when we're talking about IT costs. Some institutions provide a base level to every faculty member. If you exceed that, they might charge. How they charge that, whether it's part of F&A, whether it's a direct charge, whether it's a departmental charge, it varies so much. So I think it's ... Now, you're getting lots of comments, Michelle, if you're looking in the Q and A.
Melissa Korf: But I think, Michelle, that this is a lot of support for some resources to help us estimate that effort so that we can be providing better estimates to you all and that if/once we are able to comfortably, confidently leave those expenses in the personnel budget and have more confidence in the estimates that we're providing, I think you'll see more accurate figures.
Michelle Bulls: Absolutely. Absolutely, and I think that's a large part of why I raised it, because I think ... We can all just kind of sit and be quiet about it, but it's happening. And we have to make some decisions about that, and we also need to provide that compliance coverage on our side to protect our grants managers and program officials because there is a policy out there that requires otherwise. So I ... and I called on you because I know that you know that there's a great deal of love there for you and that you would tell the group what you've experienced. And I think having this open discussion and having these town halls to really unpack and unleash some of the challenges that we are seeing is a good thing because that's the purpose of the pilot. So thank you very much for your being vulnerable and being collaborative.
Melissa Korf: Any time, Michelle.
Michelle Bulls: You're just such a good colleague.
Melissa Korf: There's one more question about the single line item in the chat that I think may be a policy clarification, too, Michelle. Will NIH allow institutions to rebudget single line item costs currently proposed until NIH makes a policy change? And I think that's asking if we have to request approval to rebudget out of this cost that we're in the single line item.
Michelle Bulls: I think that is exactly what it's asking, which is what I like to talk ... I need to look at policy and make some decisions just to provide the compliance coverage on your end and on our end, as well, because if we're accepting it, we need to document that file and ... Yeah.
Melissa Korf: And there is a question in the chat. Does the NIH mandate costs for the DM line item? I thought PIs have the option to allocate funds. And I believe there is a requirement for the line item to exist in a detailed budget, but it won't bounce if it's $0, but the line item has to be there.
Jim Luther: I think we mentioned several times the value of the pilot, but I do think we neglected to introduce Michelle and Kristin and the team who we meet with every week. And I'm not sure who else is on here, Michelle, from the program side, but I saw names popping up that I recognize. So again, this conversation that we've had, the last 10 minutes, 20 minutes, 30 minutes is exactly the value of a pilot, right? This is ... Michelle and Kristin and the program staff are in the positions to influence how this is going to be executed. And again, our input and the faculty members' input is really critical to that. But I did want to make sure that we introduced, Michelle, you and Kristin. Is there anybody else that we ... I know we had you on the front slide.
Michelle Bulls: We have ...
Jim Luther: It was your names. Was there anybody else we should introduce?
Michelle Bulls: Now, we have several of our program officials that we meet with weekly as well as the larger group. I tried ... We sent it out to as many folks, and so we're really excited. We do plan, though, to engage more of our program officials on the panel going forward as we meet with some of the program officials within NIH internally just to see kind of what they're starting to see and sharing some of those perspectives, as well, because I think that's going to be great. Thank you so much, Jim and Christi and Melissa for introducing us. We really appreciate your partnership. Jim is going to loop us in.
Christi Keene: We appreciate, as does the entire recipient community, appreciate the opportunity to be a part of the pilot. I think this would look significantly different right now if it were not for the pilot, so certainly grateful for NIH and for those that are participating. It's very valuable feedback, so please keep it coming.
Melissa Korf: Just to echo that, it's been just a really special experience to see how everyone is really coming together at the table to work on this really important policy. Nothing is maybe as important as making sure that the critical research data comes out of our federally funded research and our NIH-funded research or being managed and shared to maximize the benefit. And it's just really amazing to see how everyone is coming together to work together on this. So it's been great. We have slowed down on the Q and A, so ...
Michelle Bulls: I would like to just give a shout out to our Division of Communications office, Cynthia Dwyer and Deron Turner, for helping us with this webinar. They have been amazing, as always, and so I just want to thank them as they work for Megan Columbus as the director. So thank you very much, guys, for helping us and jumping in, one big happy family.
Jim Luther: Hey, Michelle. there is one question. Jumping back to the previous comment, the roll-out of transitioning back from the single line item is at TBD at the moment, question mark. And that is correct, right?
Michelle Bulls: That is correct.
Jim Luther: You want to clarify that? You want to clarify that? Thank you.
Michelle Bulls: Want me to clarify it? I'm good? Oh, okay, yeah, so it is to be determined because we need to be thinking about what we can do in the interim, if there's anything that we can do, right? And then thinking about the guide notice that needs to go out prior to October 1, long before October 1 so that we can start anew and going forward. But we need to look at the instructions. We need to look at the eRA systems to make sure that whatever we do as we roll it back, we don't break it any further for you or ourselves. So we will definitely be in touch, and we'll be talking through it and keeping you guys in touch. I just need to make sure that our boss is good with the timeline and all that good stuff.
Melissa Korf: And the stellar communications team that you mentioned, Michelle, has reminded us that the recording and the transcript will be available in about 5 to 7 business days, so on that time frame, folks can look for all of these materials to be posted on the FDP website, and we'll send out a communication. Michelle, do you know if this is going to be posted anywhere on NIH, as well, or we're primarily posting on the FDP pilot page?
Michelle Bulls: I can talk to Cynthia about that. It might be good to post it on ours, as well. And, Kristin, shout out to KT, who really worked with Cynthia and Deron to get this going. So I'll have them come together, figure that out, and then I'll let you guys know.
Christi Keene: Wonderful. Thank you, everyone, for your attendance today. We really appreciate your time, and we look forward to further conversations.