Key Dates

Related Announcements

None

Issued by

National Cancer Institute (NCI)

Purpose

The purpose of this Request for Information (RFI) is to solicit broad community input on priority artificial intelligence (AI) benchmarks and associated benchmark datasets to advance the development, evaluation, and validation of AI applications in cancer research  and care.

Background

The rapid progress in artificial intelligence (AI) has the potential to revolutionize cancer research and care. These advancements span the entire spectrum of cancer-related efforts, including understanding the fundamental biology of cancer, enhancing early detection and diagnosis, accelerating drug development, improving treatment strategies, and predicting patient outcomes. To fully harness AI's potential in these areas, it is essential for models to demonstrate accuracy, reproducibility, and generalizability.

Benchmarks play a key role in advancing AI by enabling the assessment and comparison of model performance. A benchmark typically includes a dataset, a problem specification, and a defined score. Benchmark datasets consist of carefully curated, expert-labeled data that accurately represent the condition of interest while capturing the diversity of the target population and variations in data collection systems and methods. Benchmarks are essential for validating and comparing AI models, helping researchers assess their reliability and generalizability across different applications. They are critical for ensuring the accuracy and trustworthiness of AI models, thereby increasing confidence in their ability to perform effectively in real-world scenarios. While it is important for benchmark datasets to be available for model validation, it is also important that these datasets remain sequestered so that they are not used in the initial model training.

In cancer research, and biomedical research more broadly, AI benchmarks have been critical for advancing AI development in several domains, most notably protein structure prediction, medical image analysis and natural language processing. Often AI benchmarks are defined in the context of a challenge competitions such Critical Assessment of Structure Prediction (CASP) as well as those as those organized through Dialogue for Reverse Engineering Assessment and Methods (DREAM), and Medical Image Computing and Computer Assisted Intervention (MICCAI), among many others.  However, these benchmarks vary in their broader adoption, maintenance, availability, and usability. Establishing high quality, widely available AI benchmarks can play an important role in advancing key opportunities for AI in cancer research and care  

Information Requested

The National Cancer Institute (NCI) seeks input on priorities for AI benchmark development and the key characteristics that will ensure they meet the needs of the cancer research and AI communities. Feedback is welcome from a diverse set of professionals, including researchers, scientists, administrators, and healthcare professionals. Respondents can be members of academia, government, or industry.

NCI aims to understand the creation of benchmarks that facilitate robust and transparent AI applications to improve cancer prevention, detection, treatment, and survivorship outcomes.

Questions for Response:

1. What are AI-relevant use cases or tasks in cancer research and care that could be advanced through the availability of high-quality benchmarks? Please be as specific as possible; e.g, brain tumor image segmentation; cancer treatment extraction from EHR data; somatic variant calling from long read sequencing data. Of particular interest are 1) use cases and tasks with broad interest; and 2) use cases and tasks domains where benchmarks are currently scarce.
2. What are the desired characteristics of benchmarks for these use cases, including but not limited to considerations of quality, utility, and availability?
3. What datasets currently exist that could contribute to or be adapted for benchmarking? Please include information about their size, annotation, availability, as well as AI use cases they could support.
4. What are the biggest barriers to creating and/or using benchmarks in cancer research and care?
5. Please provide any additional information you would like to share on this topic.

How to Submit a Response

Responses to this RFI will be accepted at https://rfi.grants.nih.gov/?s=6802a1ad769f629767052184

Responders are free to address any or all the questions listed above. The webform is the preferred mode of input, but a file with associated answers may also be sent to [email protected].

Responses to this RFI are entirely voluntary and responders are free to address any or all the categories listed above. Please do not include any proprietary, classified, confidential, or sensitive information in your response.

Inquiries

Please direct all inquiries to:

Juli Klemm, PhD

National Cancer Institute (NCI)

Telephone: 202-853-7889

Email: [email protected]