Dataset Information

Accuracy of an Artificial Intelligence System for Cancer Clinical Trial Eligibility Screening: Retrospective Pilot Study.

ABSTRACT:

Background

Screening patients for eligibility for clinical trials is labor intensive. It requires abstraction of data elements from multiple components of the longitudinal health record and matching them to inclusion and exclusion criteria for each trial. Artificial intelligence (AI) systems have been developed to improve the efficiency and accuracy of this process.

Objective

This study aims to evaluate the ability of an AI clinical decision support system (CDSS) to identify eligible patients for a set of clinical trials.

Methods

This study included the deidentified data from a cohort of patients with breast cancer seen at the medical oncology clinic of an academic medical center between May and July 2017 and assessed patient eligibility for 4 breast cancer clinical trials. CDSS eligibility screening performance was validated against manual screening. Accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for eligibility determinations were calculated. Disagreements between manual screeners and the CDSS were examined to identify sources of discrepancies. Interrater reliability between manual reviewers was analyzed using Cohen (pairwise) and Fleiss (three-way) κ, and the significance of differences was determined by Wilcoxon signed-rank test.

Results

In total, 318 patients with breast cancer were included. Interrater reliability for manual screening ranged from 0.60-0.77, indicating substantial agreement. The overall accuracy of breast cancer trial eligibility determinations by the CDSS was 87.6%. CDSS sensitivity was 81.1% and specificity was 89%.

Conclusions

The AI CDSS in this study demonstrated accuracy, sensitivity, and specificity of greater than 80% in determining the eligibility of patients for breast cancer clinical trials. CDSSs can accurately exclude ineligible patients for clinical trials and offer the potential to increase screening efficiency and accuracy. Additional research is needed to explore whether increased efficiency in screening and trial matching translates to improvements in trial enrollment, accruals, feasibility assessments, and cost.

SUBMITTER: Haddad T

PROVIDER: S-EPMC8088869 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:To advance new therapies into clinical care, clinical trials must recruit enough participants. Yet, many trials fail to do so, leading to delays, early trial termination, and wasted resources. Under-enrolling trials make it impossible to draw conclusions about the efficacy of new therapies. An oft-cited reason for insufficient enrollment is lack of study team and provider awareness about patient eligibility. Automating clinical trial eligibility surveillance and study team and provider notification could offer a solution. To address this need for an automated solution, we conducted an observational pilot study of our TAES (TriAl Eligibility Surveillance) system. We tested the hypothesis that an automated system based on natural language processing and machine learning algorithms could detect patients eligible for specific clinical trials by linking the information extracted from trial descriptions to the corresponding clinical information in the electronic health record (EHR). To evaluate the TAES information extraction and matching prototype (i.e., TAES prototype), we selected five open cardiovascular and cancer trials at the Medical University of South Carolina and created a new reference standard of 21,974 clinical text notes from a random selection of 400 patients (including at least 100 enrolled in the selected trials), with a small subset of 20 notes annotated in detail. We also developed a simple web interface for a new database that stores all trial eligibility criteria, corresponding clinical information, and trial-patient match characteristics using the Observational Medical Outcomes Partnership (OMOP) common data model. Finally, we investigated options for integrating an automated clinical trial eligibility system into the EHR and for notifying health care providers promptly of potential patient eligibility without interrupting their clinical workflow. Although the rapidly implemented TAES prototype achieved only moderate accuracy (recall up to 0.778; precision up to 1.000), it enabled us to assess options for integrating an automated system successfully into the clinical workflow at a healthcare system. Once optimized, the TAES system could exponentially enhance identification of patients potentially eligible for clinical trials, while simultaneously decreasing the burden on research teams of manual EHR review. Through timely notifications, it could also raise physician awareness of patient eligibility for clinical trials.

Project description:Background Although artificial intelligence (AI) shows promise across many aspects of radiology, the use of AI to create differential diagnoses for rare and common diseases at brain MRI has not been demonstrated. Purpose To evaluate an AI system for generation of differential diagnoses at brain MRI compared with radiologists. Materials and Methods This retrospective study tested performance of an AI system for probabilistic diagnosis in patients with 19 common and rare diagnoses at brain MRI acquired between January 2008 and January 2018. The AI system combines data-driven and domain-expertise methodologies, including deep learning and Bayesian networks. First, lesions were detected by using deep learning. Then, 18 quantitative imaging features were extracted by using atlas-based coregistration and segmentation. Third, these image features were combined with five clinical features by using Bayesian inference to develop probability-ranked differential diagnoses. Quantitative feature extraction algorithms and conditional probabilities were fine-tuned on a training set of 86 patients (mean age, 49 years ± 16 [standard deviation]; 53 women). Accuracy was compared with radiology residents, general radiologists, neuroradiology fellows, and academic neuroradiologists by using accuracy of top one, top two, and top three differential diagnoses in 92 independent test set patients (mean age, 47 years ± 18; 52 women). Results For accuracy of top three differential diagnoses, the AI system (91% correct) performed similarly to academic neuroradiologists (86% correct; P = .20), and better than radiology residents (56%; P < .001), general radiologists (57%; P < .001), and neuroradiology fellows (77%; P = .003). The performance of the AI system was not affected by disease prevalence (93% accuracy for common vs 85% for rare diseases; P = .26). Radiologists were more accurate at diagnosing common versus rare diagnoses (78% vs 47% across all radiologists; P < .001). Conclusion An artificial intelligence system for brain MRI approached overall top one, top two, and top three differential diagnoses accuracy of neuroradiologists and exceeded that of less-specialized radiologists. © RSNA, 2020 Online supplemental material is available for this article. See also the editorial by Zaharchuk in this issue.

Project description:Introduction: Oncologists have traditionally administered the maximum tolerated doses of drugs in chemotherapy. However, these toxicity-guided doses may lead to suboptimal efficacy. CURATE.AI is an indication-agnostic, mechanism-independent and efficacy-driven personalised dosing platform that may offer a more optimal solution. While CURATE.AI has already been applied in a variety of clinical settings, there are no prior randomised controlled trials (RCTs) on CURATE.AI-guided chemotherapy dosing for solid tumours. Therefore, we aim to assess the technical and logistical feasibility of a future RCT for CURATE.AI-guided solid tumour chemotherapy dosing. We will also collect exploratory data on efficacy and toxicity, which will inform RCT power calculations. Methods and analysis: This is an open-label, single-arm, two-centre, prospective pilot clinical trial, recruiting adults with metastatic solid tumours and raised baseline tumour marker levels who are planned for palliative-intent, capecitabine-based chemotherapy. As CURATE.AI is a small data platform, it will guide drug dosing for each participant based only on their own tumour marker levels and drug doses as input data. The primary outcome is the proportion of participants in whom CURATE.AI is successfully applied to provide efficacy-driven personalised dosing, as judged based on predefined considerations. Secondary outcomes include the timeliness of dose recommendations, participant and physician adherence to CURATE.AI-recommended doses, and the proportion of clinically significant dose changes. We aim to initially enrol 10 participants from two hospitals in Singapore, perform an interim analysis, and consider either cohort expansion or an RCT. Recruitment began in August 2020. This pilot clinical trial will provide key data for a future RCT of CURATE.AI-guided personalised dosing for precision oncology. Ethics and dissemination: The National Healthcare Group (NHG) Domain Specific Review Board has granted ethical approval for this study (DSRB 2020/00334). We will distribute our findings at scientific conferences and publish them in peer-reviewed journals. Trial registration number: NCT04522284.

Project description:Summary Background Anthropomorphic phantoms are used in surgical planning and intervention. Ideal accuracy and high efficiency are prerequisites for its clinical application. We aimed to develop a fully automated artificial intelligence-based three-dimensional (3D) reconstruction system (AI system) to assist thoracic surgery and to determine its accuracy, efficiency, and safety for clinical use. Methods This AI system was developed based on a 3D convolutional neural network (CNN) and optimized by gradient descent after training with 500 cases, achieving a Dice coefficient of 89.2%. Accuracy was verified by comparing virtual structures predicted by the AI system with anatomical structures of patients in retrospective (n = 113) and prospective cohorts (n = 139) who underwent lobectomy or segmentectomy at the Peking University Cancer Hospital. Operation time and blood loss were compared between the retrospective cohort (without AI assistance) and prospective cohort (with AI assistance) for safety evaluation. The time consumption for reconstruction and the quality score were compared between the AI system and manual reconstruction software (Mimics®) for efficiency validation. This study was registered at https://www.chictr.org.cn as ChiCTR2100050985. Findings The AI system reconstructed 13,608 pulmonary segmental branches from retrospective and prospective cohorts, and 1573 branches of interest corresponding to phantoms were detectable during the operation for verification, achieving 100% and 97% accuracy for segmental bronchi, 97.2% and 99.1% for segmental arteries, and 93.2% and 98.8% for segmental veins, respectively. With the assistance of the AI system, the operation time was shortened by 24.5 min for lobectomy (p < 0.001) and 20 min for segmentectomy (p = 0.007). Compared to Mimics®, the AI system reduced the model reconstruction time by 14.2 min (p < 0.001), and it also outperformed Mimics® in model quality scores (p < 0.001). Interpretation The AI system can accurately predict thoracic anatomical structures with higher efficiency than manual reconstruction software. Constant optimization and larger population validation are required. Funding This study was funded by the 10.13039/501100004826Beijing Natural Science Foundation (No. L222020) and other sources.

Project description:IntroductionLung cancer (LC) is the most common cause of cancer-related deaths worldwide. Its early detection can be achieved with a CT scan. Two large randomised trials proved the efficacy of low-dose CT (LDCT)-based lung cancer screening (LCS) in high-risk populations. The decrease in specific mortality is 20%-25%.Nonetheless, implementing LCS on a large scale faces obstacles due to the low number of thoracic radiologists and CT scans available for the eligible population and the high frequency of false-positive screening results and the long period of indeterminacy of nodules that can reach up to 24 months, which is a source of prolonged anxiety and multiple costly examinations with possible side effects.Deep learning, an artificial intelligence solution has shown promising results in retrospective trials detecting lung nodules and characterising them. However, until now no prospective studies have demonstrated their importance in a real-life setting.Methods and analysisThis open-label randomised controlled study focuses on LCS for patients aged 50-80 years, who smoked more than 20 pack-years, whether active or quit smoking less than 15 years ago. Its objective is to determine whether assisting a multidisciplinary team (MDT) with a 3D convolutional network-based analysis of screening chest CT scans accelerates the definitive classification of nodules into malignant or benign. 2722 patients will be included with the aim to demonstrate a 3-month reduction in the delay between lung nodule detection and its definitive classification into benign or malignant.Ethics and disseminationThe sponsor of this study is the University Hospital of Nice. The study was approved for France by the ethical committee CPP (Comités de Protection des Personnes) Sud-Ouest et outre-mer III (No. 2022-A01543-40) and the Agence Nationale du Medicament et des produits de Santé (Ministry of Health) in December 2023. The findings of the trial will be disseminated through peer-reviewed journals and national and international conference presentations.Trial registration numberNCT05704920.

Project description:Prostate cancer is a leading cause of morbidity and mortality for adult males in the US. The diagnosis of prostate carcinoma is usually made on prostate core needle biopsies obtained through a transrectal approach. These biopsies may account for a significant portion of the pathologists' workload, yet variability in the experience and expertise, as well as fatigue of the pathologist may adversely affect the reliability of cancer detection. Machine-learning algorithms are increasingly being developed as tools to aid and improve diagnostic accuracy in anatomic pathology. The Paige Prostate AI-based digital diagnostic is one such tool trained on the digital slide archive of New York's Memorial Sloan Kettering Cancer Center (MSKCC) that categorizes a prostate biopsy whole-slide image as either "Suspicious" or "Not Suspicious" for prostatic adenocarcinoma. To evaluate the performance of this program on prostate biopsies secured, processed, and independently diagnosed at an unrelated institution, we used Paige Prostate to review 1876 prostate core biopsy whole-slide images (WSIs) from our practice at Yale Medicine. Paige Prostate categorizations were compared to the pathology diagnosis originally rendered on the glass slides for each core biopsy. Discrepancies between the rendered diagnosis and categorization by Paige Prostate were each manually reviewed by pathologists with specialized genitourinary pathology expertise. Paige Prostate showed a sensitivity of 97.7% and positive predictive value of 97.9%, and a specificity of 99.3% and negative predictive value of 99.2% in identifying core biopsies with cancer in a data set derived from an independent institution. Areas for improvement were identified in Paige Prostate's handling of poor quality scans. Overall, these results demonstrate the feasibility of porting a machine-learning algorithm to an institution remote from its training set, and highlight the potential of such algorithms as a powerful workflow tool for the evaluation of prostate core biopsies in surgical pathology practices.

Project description:ObjectiveTo examine the accuracy of artificial intelligence (AI) for the detection of breast cancer in mammography screening practice.DesignSystematic review of test accuracy studies.Data sourcesMedline, Embase, Web of Science, and Cochrane Database of Systematic Reviews from 1 January 2010 to 17 May 2021.Eligibility criteriaStudies reporting test accuracy of AI algorithms, alone or in combination with radiologists, to detect cancer in women's digital mammograms in screening practice, or in test sets. Reference standard was biopsy with histology or follow-up (for screen negative women). Outcomes included test accuracy and cancer type detected.Study selection and synthesisTwo reviewers independently assessed articles for inclusion and assessed the methodological quality of included studies using the QUality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool. A single reviewer extracted data, which were checked by a second reviewer. Narrative data synthesis was performed.ResultsTwelve studies totalling 131 822 screened women were included. No prospective studies measuring test accuracy of AI in screening practice were found. Studies were of poor methodological quality. Three retrospective studies compared AI systems with the clinical decisions of the original radiologist, including 79 910 women, of whom 1878 had screen detected cancer or interval cancer within 12 months of screening. Thirty four (94%) of 36 AI systems evaluated in these studies were less accurate than a single radiologist, and all were less accurate than consensus of two or more radiologists. Five smaller studies (1086 women, 520 cancers) at high risk of bias and low generalisability to the clinical context reported that all five evaluated AI systems (as standalone to replace radiologist or as a reader aid) were more accurate than a single radiologist reading a test set in the laboratory. In three studies, AI used for triage screened out 53%, 45%, and 50% of women at low risk but also 10%, 4%, and 0% of cancers detected by radiologists.ConclusionsCurrent evidence for AI does not yet allow judgement of its accuracy in breast cancer screening programmes, and it is unclear where on the clinical pathway AI might be of most benefit. AI systems are not sufficiently specific to replace radiologist double reading in screening programmes. Promising results in smaller studies are not replicated in larger studies. Prospective studies are required to measure the effect of AI in clinical practice. Such studies will require clear stopping rules to ensure that AI does not reduce programme specificity.Study registrationProtocol registered as PROSPERO CRD42020213590.