Key Dates

Related Announcements

• February 19, 2016 - AHRQ Announces New Policy for Public Access to AHRQ-Funded Scientific Publications. See Notice NOT-HS-16-008.
• May 26, 2020 - The Agency for Healthcare Research and Quality Data Management Plan Policy. See Notice NOT-HS-20-011.

Issued by

AGENCY FOR HEALTHCARE RESEARCH AND QUALITY (AHRQ)

Purpose

 AHRQ seeks public input on the AHRQ plan for updating the agency policy for public access to scientific publications and scientific data resulting from AHRQ funding

(AHRQ Public Access Plan) (https://www.ahrq.gov/funding/policies/publicaccess/index.html).

Background

On February 22, 2013, the White House Office of Science and Technology Policy (OSTP) released the memorandum entitled "Increasing Access to the Results of Federally Funded Scientific Research." This memorandum requires federal agencies to make the results of federally funded scientific research available to and useful for the public, industry, and the scientific community. In response, AHRQ published a plan for establishing a policy for public access to scientific publications and scientific data resulting from AHRQ funding (https://www.ahrq.gov/funding/policies/publicaccess/index.html) in February 2015, a Policy for Public Access to AHRQ funded Scientific Publications (NOT-HS-16-008) in February 2016,) and in May 2020, a Data Management Plan Policy (NOT-HS-20-011).

On August 25, 2022, the White House OSTP released a memorandum on “Ensuring Free, Immediate, and Equitable Access to Federally Funded Research” (2022 Memo). The 2022 Memo establishes new guidance for improving public access to scholarly publications and data resulting from federally supported research. Accordingly, AHRQ updated its Public Access Plan that outlines the proposed approach AHRQ will take to implement the new guidance, consistent with its longstanding commitment to public access.

Request for Information

AHRQ seeks public information regarding the updated AHRQ Public Access Plan. Comments are welcome on all elements of the AHRQ Public Access Plan.

How to Submit a Response:

Comments can be submitted to [email protected] with subject line as Response to AHRQ Updated Public Access Plan. Responses will be accepted through May 31, 2023.

Responses to this RFI are voluntary and may be submitted anonymously. You may also voluntarily include your name and contact information with your response. Other than your name and contact information, please do not include in the response any personally identifiable information or any information that you do not wish to make public. Proprietary, classified, confidential, or sensitive information should not be included in your response.

AHRQ Updated Public Access Plan:

This document is the Agency for Healthcare Research and Quality (AHRQ) plan for establishing an updated policy for public access to scientific publications and scientific data in digital format resulting from AHRQ funding.

1. Background & Purpose

2. Scope

3. Requirements

4. Definitions

5. Applicability

6. Legal Authorities

7. Roles and Responsibilities

8. Implementation

9. Metrics, Compliance, and Evaluation

10. Timeline

11. Resources

12. Public Consultation

13. Interagency Coordination

14. Public Notice

15. Update and Re-Evaluation of the Policy

16. Additional Information

17. Material Superseded

1. Background & Purpose

On February 22, 2013, the White House Office of Science and Technology Policy (OSTP) released the memorandum entitled "Increasing Access to the Results of Federally Funded Scientific Research." This memorandum requires federal agencies to make the results of federally funded scientific research available to and useful for the public, industry, and the scientific community. In response, in February 2015, AHRQ published a plan for establishing a policy for public access to scientific publications and scientific data in digital format resulting from AHRQ funding, in February 2016 published a Policy for Public Access to AHRQ funded Scientific Publications (NOT-HS-16-008), and in May 2020, published a Data Management Plan Policy (NOT-HS-20-011). 

On August 25, 2022, the White House Office of Science and Technology Policy (OSTP) released the memorandum entitled “Ensuring Free, Immediate, and Equitable Access to Federally Funded Research” (2022 Memo).  The 2022 Memo provides policy guidance to federal agencies with research and development expenditures on updating their public access policies. In accordance with this memorandum, OSTP recommends that federal agencies, to the extent consistent with applicable law:

1. Update their public access policies as soon as possible, and no later than December 31, 2025, to make publications and their supporting data resulting from federally funded research publicly accessible without an embargo on their free and public release;

2. Establish transparent procedures that ensure scientific and research integrity is maintained in public access policies; and,

3. Coordinate with OSTP to ensure equitable delivery of federally funded research results and data.

This document is the AHRQ updated plan for updating policies for public access to scientific publications and scientific data in digital format resulting from AHRQ funding pursuant to recommendations from the 2022 Memo. AHRQ intends, to the fullest extent possible and contingent on the availability of funds, to make available to the public all scientific publications and data arising from unclassified research and programs funded wholly or in part by AHRQ.

The AHRQ mission is to produce evidence that makes health care safer, higher quality, more accessible, equitable and affordable, and to work within the U.S. Department of Health and Human Services (HHS) and with other partners to make sure the evidence is understood and used.

The AHRQ Public Access Policy promotes the following objectives:

• Establish AHRQ's commitment to public access to scientific research results.
• Ensure that the public can read, download, and analyze in digital format final published documents.
• Facilitate easy public search, analysis of and access to peer-reviewed scholarly publications directly arising from research funded by AHRQ.
• Ensure full public access to publications' metadata without charge.
• Ensure the attributes to authors, journals, and original publishers are maintained.
• Ensure that publications and metadata are in an archival solution.
• Ensure that all extramural researchers receiving AHRQ grants, cooperative agreements, and contracts for scientific research, and intramural researchers develop data management plans, as appropriate, describing how they will provide for long-term preservation of and access to scientific data in digital format.
• Ensure the expedited translation of research results into knowledge, products, and procedures to improve health.
• Facilitate the education of new researchers and enable the exploration of topics not envisioned by the initial investigators.
• Permit the creation of new datasets when data from multiple sources are combined.
• Establish transparent procedures that ensure scientific and research integrity is maintained in public access policies; and
• Coordinate with OSTP to ensure equitable delivery of federally funded research results and data.

2. Scope

The updated AHRQ Public Access Policy will be fully effective by December 31, 2025, with the policy implementing Scientific and Research Integrity in Agency Public Access Policies by December 31, 2026. The Public Access Policy will apply to all scientific publications and data in digital format. Implementation will be prospective and will not apply to any publication or digital data set arising from an AHRQ-sponsored grant, cooperative agreement, contract, or intramural research project funded prior to publication of the final updated AHRQ Public Access Policy.

The updated AHRQ Public Access Policy will apply to all research funded by AHRQ, except where such research is administered or performed by a partner Agency with a comparable Public Access Policy, in which case AHRQ will defer to the partner Agency's policies on the management of scholarly publications and digital data sets.

Digital scientific data that are covered by this policy include:

• Field data.
• Lab data.
• Other data (e.g., quality control samples, sample ID data, and instrument calibration data).
• Digital scientific, clinical, or institutional data that can be in scope at the discretion of the  program or in an appropriate context:
  • Models and model-related content, including parameters and outputs, including models of public health emergencies.
  • Command files, such as SAS or STATA files created from secondary sources (typically referred to as secondary or outside data).
  • De-identified Electronic Health Record (EHR) patient treatment records.
• Non-proprietary records and data collected as part of the National Hospital Preparedness Program.
• Digital scientific data that are not in scope for this policy include:
  • Personally identifiable data, however, AHRQ will attempt to make all research data available to the public by creating de-identified public use data files.
  • Proprietary trade data.
  • Data related to protecting critical infrastructure.
  • Other data whose release is limited by law, regulation, security requirements, or policy.

Per OMB Circular A-110, the following are not research data and therefore not subject to this policy: preliminary analyses, drafts of scientific papers, plans for future research, peer reviews, communications with colleagues, or physical objects, such as laboratory specimens. However, investigators are encouraged to include relevant content from laboratory or field notebooks in their published data sets if that information provides documentation that would help future users successfully re-use the data. (That is, lab and field notebooks are not themselves scientific data, but may contain relevant metadata to be included in the published data set.).

3. Requirements

Publications

For scholarly publications, the AHRQ Public Access Policy has required and will continue requiring authors to submit the final peer-reviewed accepted journal manuscripts to PubMed Central (PMC). In lieu of the final peer-reviewed manuscript, AHRQ has and will continue accepting the final published article, provided the authors can ensure AHRQ has the rights to make the published version public. AHRQ's Public Access Policy is subject to law; Agency mission; resource constraints; U.S. national, homeland, and economic security; and the objectives listed in the OSTP directive.

Scientific data

Scientific data underlying peer-reviewed scholarly publications resulting from federally funded research should be made freely available and publicly accessible by default at the time of publication, unless subject to limitations as described in Section 3(c)(i) of the August 25, 2022 OSTP memo and should be subject to federal agency guidelines for researcher responsibilities regarding data management and sharing plans, consistent with Section 3(c) of the memorandum.

Scientific Data include the recorded factual material commonly accepted in the scientific community as of sufficient quality to validate and replicate research findings. Such scientific data do not include laboratory notebooks, preliminary analyses, case report forms, drafts of scientific papers, plans for future research, peer-reviews, communications with colleagues, or physical objects and materials, such as laboratory specimens, artifacts, or field notes. The definition of “scientific data” is similar to but broader than the term “research data” defined by 2 CFR 200.315 (e) and 45 CFR 75.322 (e).

AHRQ funded researchers must follow federal laws and OMB policies that govern federal agencies’ information management practices and protect certain types of data, to the extent that the scientific data created by, collected by, under the control or direction of, or maintained by the federal researchers is subject to those laws and policies.

To the extent feasible and consistent with applicable law and policy; agency mission; resource constraints; U.S. national, homeland, and economic security; and the objectives listed below, digitally formatted scientific data resulting from unclassified research supported wholly or in part by federal funding should be stored and publicly accessible to search, retrieve, and analyze. For sharing of data in digital format, all AHRQ-funded researchers have been and will continue to be required to include a data management plan for sharing final research data in digital format, or state why data sharing is not possible. Data management plans will be evaluated by AHRQ staff and peer review committees for merit while considering the values of long-term preservation, access, and the associated cost, and administrative burden. AHRQ expects the timely release and sharing of data to be no later than the acceptance for publication of the main findings from the final dataset. The specific time will be influenced by the nature of the data collected. Researchers will be directed to work with AHRQ staff and commercial repositories to deposit data upon publication. It may not be feasible to share all data; the costs and benefits of sharing data should be considered in data management planning. AHRQ intramural research is conducted using public use data. These data are also available to the public. AHRQ data that include personally identifiable data are available to the public in the AHRQ data center. AHRQ will attempt to make all intramural research data available to the public by creating de-identified public use data files.

If an AHRQ-funded grant, cooperative agreement, contract or intramural research project transforms or links datasets, rather than producing a new set of data, the researchers will be required to submit a data management plan. If there are limitations associated with the data sharing agreements for the original data that preclude subsequent sharing, the research applicants must explain these limitations.

Data from small studies can be analyzed and submitted for publication relatively quickly. If data from large epidemiologic or longitudinal studies are collected over several discrete time periods or waves, it is reasonable to expect that the data would be released in waves as data become available or main findings from waves of the data are published. AHRQ-funded extramural and intramural researchers will be expected to include updates on public access to publications and data in digital format resulting from their research in quarterly reports, where applicable, annual progress reports, final reports, and via NLM's Bibliography tool, My NCBI. All new grant and cooperative agreement applications, and contract proposals will include a section detailing the applicants' progress to-date with sharing of publication and data in digital format arising from previous AHRQ funding.

Scientific and Research Integrity in Agency Public Access Policies

AHRQ will take actions to ensure that scientific and research integrity policies are in place in order to strengthen public trust in federally funded science. By December 31, 2024, AHRQ plans to submit to OSTP and OMB a second update to their public access plans specifying approaches taken to implement the provisions in this Section 4 of the 2022 memo. AHRQ plans to complete and publish full policy development for plans implementing these provisions by December 31, 2026, with an effective date no later than one year after the publication of the agency plan.

In the plan to be submitted to OSTP and OMB by December 31, 2024, AHRQ will discuss and explore how AHRQ plans to act to achieve the goals specified and approaches recommended in the OSTP memo.

White House Office of Management and Budget Open Data Policy Memorandum (M-13-13)

AHRQ will meet the requirements of M-13-13 as specified in the accompanying memorandum, titled HHS Guiding Principles and Common Approach for Enhancing Public Access. AHRQ will develop and maintain an AHRQ Enterprise Data Inventory for the data sets related to their major Information Technology investments. AHRQ will develop and maintain an AHRQ Enterprise Data Inventory, cataloguing the various sets of data acquired or managed. The inventory will catalog both data sets that are made publicly available and data sets that are private to the Agency.

4. Definitions

Scientific Data: “scientific data” include the recorded factual material commonly accepted in the scientific community as of sufficient quality to validate and replicate research findings. Such scientific data do not include laboratory notebooks, preliminary analyses, case report forms, drafts of scientific papers, plans for future research, peer-reviews, communications with colleagues, or physical objects and materials, such as laboratory specimens, artifacts, or field notes. The definition of “scientific data” is similar to but broader than the term “research data” defined by 2 CFR 200.315 (e) and 45 CFR 75.322 (e).

Research Data: Research data is defined in OMB Circular A-110 as the recorded factual material commonly accepted in the scientific community as necessary to validate research findings, but not any of the following: preliminary analyses, drafts of scientific papers, plans for future research, peer reviews, or communications with colleagues. This 'recorded' material excludes physical objects (e.g., laboratory samples).

Research data do not include:

• Trade secrets, commercial information, materials necessary to be held confidential by a researcher until they are published, or similar information which is protected under law.
• Personnel and medical information and similar information, the disclosure of which would constitute a clearly unwarranted invasion of personal privacy, such as information that could be used to identify a particular person in a research study.

Digital Scientific Data: For the purpose of this plan and consistent with OMB Circular A-110, digital scientific data is defined as "the digital recorded factual material commonly accepted in the scientific community as necessary to validate research findings including data sets used to support scholarly publications, but does not include laboratory notebooks, preliminary analyses, drafts of scientific papers, plans for future research, peer reviews, communications with colleagues, or physical objects, such as laboratory specimens."

For the purpose of this plan, the definition of digital scientific data does not include software. However, AHRQ recognizes that in some cases, software and other tools such as interview protocols, measures, coding guides, or manuals may be necessary to interpret data. In such cases, the data management plan will be expected to include a description of these tools.

A published data set consists of at least one formal metadata document, the digital scientific data described by that metadata, and supplemental information provided to assist the data user. The metadata for scientific data will include, at a minimum, the common core metadata schema in use by the Federal Government, found at https://project-open-data.cio.gov/.

A published data set can be cited in the scientific literature and, following the practice used by most publishers of scholarly journals, has a persistent and unique identifier associated with it. A published data set is expected to persist over time, just like the scholarly articles that are based on the digital scientific data.

Data Sharing Plan: A data sharing plan outlines whether and how data will be made available to others. The plan must also explicitly describe how the data that underlies scientific publications will be available for discovery, retrieval, and analysis. It may include the expected timeline for when the data will be available, the format of the final dataset, the documentation and any analytic tools that will be provided, and the mode of data sharing (e.g., under their own auspices by mailing a disk or posting data on their institutional or personal website, through a data archive or enclave). A plan might also specify whether or not a data sharing agreement will be required and, if so, a brief description of such an agreement (including the criteria for deciding who can receive the data and whether or not any conditions will be placed on their use). If data sharing is not possible, the plan would provide an explanation.

Data Management Plan: Data management plans are more comprehensive than data sharing plans in that they include additional elements such as descriptions of the data to be produced in the proposed study, any standards to be used for collected data and metadata, mechanisms for providing access to and sharing of the data (including provisions for protection of privacy, confidentiality, security, intellectual property, or other rights), provisions for reuse and redistribution, and plans for archiving and long-term preservation of the data, or explaining why long-term preservation and access cannot be justified.

Final peer-reviewed manuscript: A final peer-reviewed manuscript is defined as an author's final manuscript of a peer-reviewed paper accepted for journal publication, including all modifications from the peer-review process.

Final published article: A final published article is defined as a publisher's authoritative copy of the paper, including all modifications from the publishing peer-review process, copy editing, stylistic edits, and formatting changes.

5. Applicability

The AHRQ Public Access Policy will apply to the following groups:

• All AHRQ intramural researchers.
• Award recipients of AHRQ grants and cooperative agreements.
• All AHRQ contractors conducting research.

6. Legal Authorities

• Section 902(a) of the Public Health Service Act, 42 U.S.C. 299a(a).
• Section 944(a) of the Public Health Service Act, 42 U.S.C. 299c -3(a).
• Section 465 of the Public Health Service Act, 42 U.S.C. 286.
• 42 C.F.R. Part 67.
• 45 C.F.R. Part 74.
• Section 103 of The America COMPETES Reauthorization Act of 2010 (Pub. L. 111-358).

7. Roles and Responsibilities

The Director of AHRQ:

• Assigns overall responsibility for establishing, managing, implementing, and evaluating public access to scholarly publications and data in digital format funded by AHRQ.
• Assigns overall responsibility for establishing, managing, implementing, and evaluating the AHRQ public access policy.

AHRQ Director of the Office of Extramural Research, Education and Priority Populations (OEREP):

• Assumes AHRQ-wide authority, control, and management of the Public Access Policy.
• Manages the implementation of this policy.
• Coordinates with AHRQ's Grants and Contracts offices to ensure the requirements of this policy are included as terms and conditions of funding for grants, contracts, and cooperative greements.
• Develops practical funding strategies to ensure the successful implementation of this policy.
• Coordinates the collaboration and cooperation of public access and data sharing with other   federal agencies.
• Ensures compliance with this Public Access Policy.

Grant, Cooperative Agreement, and Contract Awardees and their institutions:

• Ensure that authors are aware of and comply with the AHRQ Public Access Policy.
• Ensure all terms and conditions of awards are met.
• Submit data management plan with grant, cooperative agreement, and contract applications.
• Include reasonable cost for public access to publications and data in digital format as a component of grant, cooperative agreements, and research contract applications.

Authors:

Work with the publisher before any publication rights are transferred to ensure that all conditions of the AHRQ Public Access Policy can be met.

8. Implementation

This Public Access Plan includes the following implementation steps.

Planning

AHRQ intends to establish an internal working groups/task forces with expertise in data access, data management and publications to assist with development of the update public access policy.  The working groups/task forces will evaluate and determine the feasibility of implementing each of the recommendations as well as strategies of implementations.

AHRQ will continue to collaborate and cooperate with HHS and Federal Government-wide agencies to use a transparent process for soliciting views from stakeholders, including federally funded researchers, universities, libraries, publishers, users of federally funded researcher results, and civil society groups, and take such views into account.

Peer-reviewed publications

AHRQ has contracted with National Library of Medicine (NLM) to use PubMed Central (PMC) since 2016 to make peer-reviewed scholarly publications publicly accessible. The current AHRQ Policy for Public Access to AHRQ funded Scientific Publications requires that AHRQ-funded authors submit an electronic version of the author’s final peer-reviewed accepted manuscript to the NLM's PubMed Central (PMC) upon acceptance by the journal, and to be made publicly available within 12 months of the publisher’s date of publication.

PMC allows AHRQ to engage in public-private partnerships, since PMC is a public-private partnership. AHRQ's Public Access Policy is beneficial to businesses or will otherwise enhance innovation and competitiveness to satisfy a requirement in section 3 of the 2013 OSTP memo.

AHRQ plans to update its Policy for Public Access to AHRQ funded Scientific Publications (NOT-HS-16-008: https://grants.nih.gov/grants/guide/notice-files/not-hs-16-008.html ) to ensure, as appropriate and consistent with applicable law, that all peer-reviewed scholarly publications authored or co-authored by individuals or institutions resulting from federally funded research are made freely available and publicly accessible by default in agency-designated repositories without any embargo or delay after publication. In doing that, AHRQ will explore in close collaboration with NLM/PMC how to maximize equitable reach of public access to peer-reviewed scholarly publications, including by providing free online access to peer-reviewed scholarly publications in formats that allow for machine-readability and enabling broad accessibility through assistive devices, as appropriate and consistent with applicable law; and, the circumstances or prerequisites needed to make the publications freely and publicly available by default, including any use and re-use rights, and which restrictions, including attribution, may apply.

PMC is a public-private partnership to preserve and make public full-text journal articles in a nonproprietary and widely-distributed archival format. PMC makes the abstract and citation information available to NLM's abstract and metadata service, PubMed, where it can be accessed without charge. PubMed makes metadata available and combines them with any indexing terms or key works after the embargo period. Metadata of text are also stored in nonproprietary or widely distributed archival formats.

AHRQ will continue to maintain attribution to authors, journals, and original publishers through the use of PMC and PubMed. PubMed abstracts include a publisher-supplied link to the paper on a publisher site and a link to the PMC version if available. PubMed metadata, like the full text of PMC, are permanently archived. Full text displays of papers on PMC always include publisher-supplied links to the publisher site, authors, journal name and the other information required for a complete academic citation.

AHRQ will establish a mechanism for compliance with the public access policy.

AHRQ will continue to ensure the public can search, read, and download the final peer-reviewed manuscripts or published articles without charge and without embargo once established.

AHRQ will post the final updated Public Access Policy on the AHRQ website. AHRQ will also publish the final updated Public Access Policy in the NIH Guide for Grants and Contracts. In order to ensure full integration of the Public Access Policy with the AHRQ research and award management systems,

AHRQ will continue its partnership with the NIH to use PMC as the repository of publications resulting from AHRQ-funded grants, contracts, cooperative agreements, and intramural research.

Scientific Data

AHRQ will explore and develop approaches and timelines for sharing other federally funded scientific data that are not associated with peer-reviewed scholarly publications wherever feasible.

AHRQ will explore the feasible approaches to accept and manage data submitted by extramural, intramural, and contract researchers, which may include contracting with a commercial repository.

AHRQ staff will coordinate with the researcher upon notification of funding, prior to the start of the research project, and throughout its life cycle to ensure the current usability, long-run preservation and access to the data. Data will be made available in a digital format free of charge to the public via the selected repository. AHRQ will publicize the location of the data to the public on its website and provide a link to the data. Information on how to access all AHRQ's funded data will be provided on AHRQ's website with links to the data.

AHRQ will continue working with OSTP and other federal agencies in developing guidance to researchers that ensures the digital repositories used align, to the extent practicable, with the National Science and Technology Council document entitled “Desirable Characteristics of Data Repositories for Federally Funded Research.”

The AHRQ Public Access Policy update and development process will include the following:

Continue and refine the iterative process of policy design, planning, implementation, evaluation/impact assessment, and policy redesign.

Work in full and open consultation with all stakeholders, including other Federal organizing bodies, to develop, maintain, and improve the Public Access Policy.

Explore new approaches and partnerships with publishers and other stakeholders to obtain final peer-reviewed manuscripts or published articles.

Maintain attribution to authors, journals, and original publishers through the use of PMC and PubMed. PubMed abstracts include a publisher-supplied link to the paper on a publisher site and a link to the PMC version if available. PubMed metadata, like the full text of PMC, are permanently archived. Full text displays of papers on PMC always include publisher-supplied links to the publisher site, authors, journal name and the other information required for a complete academic citation.

Establish a mechanism for compliance with the policy.

Ensure the public can search, read, and download the final peer-reviewed manuscripts or published articles without charge or embargo period.

AHRQ will post the final updated Public Access Policy on the AHRQ website. AHRQ will also publish the final updated Public Access Policy in the NIH Guide for Grants and Contracts.

AHRQ has included requirements to submit publications in accordance with the Public Access Policy in contract solicitations and grant Notices of Funding Opportunity and will update such requirements according to the updated policies. AHRQ will continue using PMC as an approach for optimizing search, archival, and dissemination features that encourage innovation in accessibility and interoperability, while ensuring long-term stewardship of the results of federally funded research. It will also continue encouraging public-private collaboration to maximize the impact of the federal research investment through the use of PMC. The significant usage of PMC underscores the value of public access to the peer-reviewed scientific literature.

The NLM databases, with their XML data accessible through a variety of Web services, can also support additional economic activity. NLM has found that more than 300 private sector firms obtain data from Medline for a variety of applications: to supplement literature database products and search tools; to build literature analysis tools for end users of research; to conduct research in a variety of biomedical areas; and to gain intelligence on experts and trends. Many others that we do not have the means to track use NLM's other resources.

Scientific and Research Integrity in Agency Public Access Policies

Public access policies that deliver transparent, open, secure, and free communication of federally funded research and activities in an expeditious manner are an important tool to uphold scientific and research integrity.

AHRQ will take actions to ensure that these elements of scientific and research integrity are in place in order to strengthen public trust in federally funded science. By December 31, 2024, AHRQ plans to submit to OSTP and OMB a second update to their public access plans specifying approaches taken to implement the provisions in Section 4 of the OSTP memo. AHRQ plans to complete and publish full policy development for plans implementing these provisions by December 31, 2026, with an effective date no later than one year after the publication of the agency plan.

In the plan to be submitted to OSTP and OMB by December 31, 2024, AHRQ will discuss how AHRQ plans to act to achieve the goals specified and approaches recommended in OSTP 2022 Memo.

Submission of Manuscripts/Publications

AHRQ has utilized PMC to enable the submission of final peer-reviewed manuscripts or final published articles that includes the following functionalities:

• Allow users to submit and manage manuscripts directly.
• Allow submission by the author, the publisher, or the manager of the funding agreement.
• Accept manuscripts in a range of common electronic formats.
• Accept any additional files of figures, tables, or supplementary information included with the manuscript.
• Provide flexible and multiple approaches to manuscript submission.
• Authors will regularly report on the status of their deposit of materials using quarterly reports, where applicable; annual progress reports; and final progress reports.

Management

• AHRQ has utilized PMC as a repository of full-text scholarly publications.
• The system stores, organizes, and manages the publications collected or submitted under this policy.
• PMC uses an open architecture and follows industry standards to facilitate open government, enable integration, and promote interoperability.
• AHRQ currently has interoperability through its use of the NIH's grant management system eRA Commons, used by several federal agencies and HHS operating divisions. NIH has integrated eRA Commons with public access compliance information using My NCBI.
• PMC makes access to AHRQ-supported papers, as well as over two million other papers, easier to access while taking appropriate steps to prevent unauthorized mass redistribution. NLM has established sophisticated monitoring systems to detect and prevent potential misuse that are very similar to the processes used by publishers' websites. NLM systems detect and prevent bulk downloading and will immediately cut off any sites, foreign or domestic, that appear to be abusing copyrighted property. Unless publishers have chosen PMC as the sole distributor of the electronic contents of their journals, all articles that are accessible on PMC are also accessible on publisher websites.
• PMC enable AHRQ to monitor compliance with its Public Access Policy.

Access and Discoverability

The AHRQ has established and will continue an interagency agreement with NLM to use PMC for its primary repository for AHRQ publications. The use of PMC has ensured easy search, analysis, and download of peer-reviewed scholarly publications arising from research funded by AHRQ without charge no later than 12 months following publication. The public's ability to locate and access digital data resulting from AHRQ's federally funded scientific research as well as for publications will be improved through the use of PMC. The full text of PMC papers can be searched on PMC. In addition, the metadata (abstract, citation, and any index terms) can be searched in PubMed.

AHRQ will continue to ensure that the system is accessible to people with disabilities and compliant with Section 508a of the Rehabilitation Act (29 USC 794d). AHRQ will provide an accommodation for final publisher PDF versions that are not 508 compliant but cannot be remediated due to copyright constraints.

AHRQ's current maximum embargo period is 12 months. AHRQ will explore approaches to ensure all peer-reviewed scholarly publications authored or co-authored by individuals or institutions resulting from federally funded research are made freely available and publicly accessible by default in agency-designated repositories without any embargo or delay after publication, wherever feasible considering all applicable legal and regulatory constraints, in close collaborations with NIH PubMed.

Preservation

Preservation is one of the Public Access Policy's primary objectives. As discussed in NIH's Public Access Policy, NLM maintains multiple backup copies of the PMC database, both onsite and at a remote, secure location.

Further, content on PMC is actively curated in that all article views are generated dynamically from the XML record of an article. In this way, every use of an article validates that the archival record is still usable and viable.

Digital Data

Planning

AHRQ will continue exploring feasible approaches to support a data repository to allow public access to data in digital format, which may include allowing data repositories identified and/or utilized by AHRQ funded researchers or recommending data repositories widely used by federally funded researchers. AHRQ will require all primary data collected by grant, contract, or intramural research to be submitted to a data repository, unless confidentiality restrictions prevent the data from being made public. For research using secondary data that is publicly available, such as survey or claims data, AHRQ is considering allowing researchers to submit command files, such as SAS or STATA files, to the AHRQ data repository. Researchers using public use data, such as the Medical Expenditure Panel Survey (MEPS) - a set of large-scale surveys that collect data on the cost and use of health care and health insurance coverage of families and individuals, their medical providers, and employers across the United States—or the Healthcare Cost and Utilization Project (HCUP), the largest collection of nationwide and State-specific longitudinal hospital care data in the United States, will be exempt from submitting these data to the data repository. The rationale for this exemption is that if the data are already publicly available, there is no value in making additional copies of the data available; what the public and other researchers need is the SAS or STATA code used to analyze the data. AHRQ-funded research that involves data use agreements or licensing agreements that prohibit data sharing will be exempt from submitting these data to the data repository.

Management

AHRQ will maximize access, by the general public and without charge, to digitally formatted scientific data created with Federal funds by designing, implementing and maintaining a sustainable network of data management capabilities to enable discovery, appropriate use, and long-term management of digitally formatted scientific data.

This includes but is not limited to:

• Ensuring the digital repositories to be used align, to the extent practicable, with the National Science and Technology Council document entitled “Desirable Characteristics of Data Repositories for Federally Funded Research,” to enable discovery and identification.
• Development of policies to ensure the appropriate cataloguing of data sets resulting from AHRQ-funded research to make them findable, accessible, and citable.
• AHRQ will ensure that data generated by AHRQ-funded research can be cited and attribution can be provided in a consistent manner.  AHRQ will coordinate with AHRQ funded researchers to ensure that data repositories used to store data include a data discovery index that will use unique identifiers for data sets generated by AHRQ-funded research, analogous to the PMC identification number (PMCID) that is assigned to all submitted publications resulting from AHRQ-funded research. The identifier would provide a means of linking the data with the associated PubMed records. Such indexing of data sets would also facilitate generation of full citations to allow for differential recognition of scientific contributions (authorship on the scientific paper) and data-related contributions (authorship on the AHRQ data index entry). Additional efforts will be made to work with OSTP, federal agencies, HHS operating divisions and other public and private stakeholders to develop consistent practices for citation of data sets across scientific communities and other data set attribution systems and to develop methods for federated searching across multiple indices. AHRQ will also consult with AHRQ funded researchers to determine the best method for identifying and providing appropriate attribution to scientific data sets that are made available under the public access plan.
• In collaboration with other Departments and Agencies through the Interagency Open Science Working Groups under the leadership of OSTP, AHRQ will continue to explore the development of a research data common, a shared space for research output including data, software and narrative associated with AHRQ-funded research. Such a commons would adhere to the FAIR principles of Find, Access, Interoperate and Reuse for data resulting from AHRQ-funded research. The goal will be to make digital research data accessible to the public in a manner that optimizes search, archival, and dissemination features that encourage innovation in accessibility and interoperability, while ensuring long-term stewardship of research data results. A particular focus of the effort will be on making the data underlying the conclusions of peer-reviewed scientific publications resulting from federally funded scientific research available for free at the time of publication.
• Development of procedures to ensure data integrity over time.

AHRQ will provide guidance to awardees and training, education, and workforce development for its employees about the new requirements related to scientific data management, analysis, storage, preservation, and stewardship in advance of the implementation date. This guidance and training will describe any required changes to the conditions of awards and a publicly available timeline for the implementation of the policy. AHRQ will also use public websites and internal mechanisms to disseminate information and solicit feedback before and after the implementation date. The capability for AHRQ and the science communities it supports to help one another through ongoing dialogue is expected to be a key part of a successful policy implementation.
 AHRQ will modify existing policies or create new policies as necessary to incorporate the recommendations from the OSTP 2022 memo. AHRQ will include updated requirements to comply with the AHRQ Public Access Policy in contract solicitations and grant Notices of Funding Opportunity.

AHRQ will allow the inclusion of appropriate costs for data management and access in applications and proposals for AHRQ funding. The appropriateness of requested funding levels for data management plans will be assessed on a case-by-case basis through technical evaluation, and by program and grants management staff. Comparable assessments will need to be explored for intramural researchers. AHRQ will confer with interagency partners to estimate the costs of current data management activities in order to determine how to support future data management.

In order to ensure public access to data in digital format resulting from AHRQ-funded grants,  contracts, and cooperative agreements AHRQ has required since 2020 all research applicants to include a data management plan as a component of their grant applications or contract proposals. Intramural researchers are also required to submit a data management plan, where applicable. The AHRQ Data Management Plan Policy will be updated to reflect the changes in response to the OSTP 2022 memo.

AHRQ will require periodic reporting on compliance with the approved data management plans as part of the terms and conditions of award and will develop mechanisms to monitor such compliance. The data management plan currently undergoes objective review and approval by AHRQ's grant and technical peer review groups to assess its merits. The data management plan has become part of the terms and conditions of award for all appropriate grants and contracts. AHRQ will continue to require all data management plans to be consistent with applicable laws and regulations governing the privacy and confidentiality of individual human data. AHRQ's Public Access Policy recognizes proprietary interests, business confidential information, and intellectual property rights and avoids significant negative impact on intellectual property rights, innovation, and U.S. competitiveness of research grants and cooperative agreements and appropriate contracts. 
    

For requirements under AHRQ Data management plans, refer to AHRQ Data Management Plan Policy at https://grants.nih.gov/grants/guide/notice-files/NOT-HS-20-011.html.

AHRQ encourages grantees and contractors to register all clinical trials with ClinicalTrials.gov (whether or not they are subject to Title VIII of Food and Drug Administration Amendments Act (FDAAA)) and is in the process of developing a policy for intramural researchers to extend the registration requirement to all clinical trials. AHRQ will give further consideration to taking steps to expand submission of results to encompass all AHRQ-funded clinical trials, not only those subject to FDAAA.

Preservation

In order to ensure long-term preservation and full access to the public, AHRQ will continue exploring feasible approaches to support a data repository to allow public access to data in digital format, which may include allowing use of data repositories identified and utilized by AHRQ funded researchers or providing guidance on recommended data repositories widely used by federally funded researchers. AHRQ staff will coordinate with the researcher upon notification of funding, prior to the start of the research project, and throughout its life cycle to ensure the current usability, long-run preservation and access to the data. Data will be made available in a digital format free of charge to the public.

Long-term preservation and sustainability are part of the current requirements of data management plans. Development of criteria for and conduct of periodic reviews to identify gaps in preservation coverage and respond to changing needs arising from new data types may become responsibilities of the Director of OEREP. In order to determine how best to develop and sustain repositories for digital scientific data, AHRQ intends to continue to collaborate with HHS operating divisions and other agencies that support research in related areas.

9. Metrics, Compliance, and Evaluation

Oversight methods

AHRQ will continue to include the public access requirement in all grant and contract solicitations. Standard requirements are included in all research contracts and grant awards. All research proposals are required to include a Data Management Plan (DMP) as part of the overall research proposal. Approved DMPs are included in terms and conditions of grant notices of award and contract awards. Accordingly, failure to comply with the terms and conditions of the funding agreement could lead to enforcement actions, including the withholding of funding, consistent with 45 CFR 74.62. Researchers will be expected to follow the approved DMP, to the best of their ability. Failure to do so will negatively influence future funding opportunities. AHRQ will monitor all non-competing continuation grant awards using the electronic Research Performance Progress Reports (RPPR) to verify compliance with the public access policy. Current oversight of grants allows for withholding or adjustments of funds at the end of each performance period. Grant final reports will be used to confirm compliance with the Public Access Policy. Contract progress reports will also be monitored for compliance.

Prior to funding new grant and contract awards AHRQ will confirm awardees are in compliance with the AHRQ Public Access Policy. AHRQ will not make grant or contract awards in cases of non-compliance. AHRQ may delay processing of non-competing continuation awards that are not in compliance with the Public Access Policy.

Intramural research proposals are also required to include a DMP. AHRQ leadership will ensure intramural research compliance with the Public Access Policy.

AHRQ staff will periodically manually check applications, proposals, or reports for compliance with the Public Access Policy. They will confirm that a citation includes the appropriate reference number that indicates compliance, such as the PubMed Central Identifier (PMCID).

AHRQ will also use the NLM's Bibliography tool My NCBI.  My NCBI tracks the public access compliance status of every paper associated with an award, generates a citation with the current identifier and clearly indicates its compliance status. It also allows awardees to collaborate with their colleagues to associate publications with AHRQ awards.

To simplify reporting and tracking, awardees will use My NCBI to report public access compliance on these awards. This process will be fully electronic for our Research Performance Progress Reports (RPPR).

AHRQ will report to OSTP, when requested, on the status of the public access plans and policy implementation, including relevant statistics collected by the agency, such as number of publications resulting from agency-supported research grants, contracts, and cooperative agreements.

Measures

Papers will be compliant when they are posted to PMC. This will form the numerator of an Agency-wide compliance measure. AHRQ's denominator for overall compliance will be determined by the number of papers AHRQ estimates fall under the policy. AHRQ will generate this number from papers linked to specific AHRQ awards by:

• Authors in the AHRQ Manuscript Submission system.
• Authors and principal investigators in the My NCBI bibliography management system.
• Principal investigators and institutions in electronic progress reports.
• Authors and publishers in the acknowledgement section of the text of papers, as indexed in Medline.

AHRQ's expectations for data sharing are and will continue to be communicated to intramural and extramural researchers through guidance documents and outreach activities, including through the notice of grant award or the contract award. AHRQ staff will be responsible for monitoring and taking steps to ensure compliance with the terms and conditions of the award. Data management plans that are approved and made a term and condition in the notice of grant award or the contract award can be enforced. Accordingly, failure to comply with the terms and conditions of the funding agreement could lead to enforcement actions, including the withholding of funding, consistent with 45 CFR 74.62.

Program staff overseeing extramural grants and contracts will also review progress reports carefully for adherence with data management plans. Under the intramural program, the AHRQ's Center Directors will be responsible for ensuring intramural researchers are in compliance with data management plans. AHRQ will conduct an annual audit of its intramural researchers to monitor overall compliance with data management.

10. Timeline

Since 2016, AHRQ has required all intramural projects and new grant and contract research awards to submit manuscripts to PMC. Since 2020, AHRQ has required all intramural projects and new grant and contract research applications to submit data management plans. AHRQ will update its public access polices according to the deadlines recommended in sections 3 and 4 the OSTP 2022 memo.   

11. Resources

AHRQ has used and will continue to use annual appropriation funds for the implementation of PMC and for the implementation of the data management plan policy.

12. Public Consultation

AHRQ will publish the revised draft public access plan for public input on the agency website and in the NIH Guide for Grants and Contracts. AHRQ will participate in any public consultations organized by OSTP and HHS, and will be responsive to feedbacks from such sessions and other channels.

13. Interagency Coordination

AHRQ will continue interagency coordination by the following:

• Full participation on the OSTP organized interagency public access work group.
• Full participation on the HHS interagency public access work group.
• Full participation on the government-wide public access publications and data work groups facilitated by the Office of Science and Technology Policy.
• Coordinating with NIH in the use of PMC and adherence to all established PMC requirements.

14. Public Notice

The current AHRQ public access policies are posted on the AHRQ website and in the NIH Guide for Grants and Contracts (https://www.ahrq.gov/funding/policies/publicaccess/policy.html).

AHRQ will post the final updated Public Access Policy on the AHRQ website and in the NIH Guide for Grants and Contracts.

15. Update and Re-Evaluation of the Policy

AHRQ will conduct periodic updates on the final published Public Access Policy, which will be informed by feedback from all stakeholders. Feedback will be sought periodically through Requests for Information published in the NIH Guide for Grants and Contracts.

16. Additional Information

Additional public access policies and guidance include the following:

• Memorandum for the Heads of Executive Departments and Agencies—Open Government Directive, Executive Office of the President, Office of Management and Budget, August 25, 2022: https://www.whitehouse.gov/wp-content/uploads/2022/08/08-2022-OSTP-Public-access-Memo.pdf
• Memorandum for the Heads of Executive Departments and Agencies—Open Government Directive, Executive Office of the President, Office of Management and Budget, December 8, 2009: http://www.whitehouse.gov/omb/assets/memoranda_2010/m10-06.pdf (80.52 KB).
• National Digital Information Infrastructure & Preservation Program: A Collaborative Initiative of the Library of Congress: http://www.digitalpreservation.gov/.
• NIH Public Access: http://publicaccess.nih.gov.
• Open Government Directive, Memorandum for the Heads of Executive Departments and Agencies, M-10-06 (Dec 8, 2009):         http://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10-06.pdf (80.52 KB).
• Planets—Preservation and Long-Term Access through Networked Services. Open Planets Foundation: http://www.planets-project.eu External Link Disclaimer.
• Public Access to Scholarly Publications Policy Objectives—Interagency Working Group (March, 2012): http://www.whitehouse.gov/sites/default/files/microsites/ostp/public_access-final.pdf (411.75 KB).
• Scholarly Publishing Roundtable, Report and Recommendations from the Scholarly Publishing Roundtable (2009): https://www.aau.edu/key-issues/report-and-recommendations-scholarly- publishing-roundtable  External Link Disclaimer.

17. Material Superseded

Initial release.

Inquiries

Please direct all inquiries to:

Francis D. Chesley, Jr., M.D.
Director, Office of Extramural Research, Education, and Priority Populations
Email: [email protected]