Development and Validation of a High-Quality Composite Real-World Mortality Endpoint.
Ontology highlight
ABSTRACT:
SUBMITTER: Curtis MD
PROVIDER: S-EPMC6232402 | biostudies-literature | 2018 Dec
REPOSITORIES: biostudies-literature
Publications
Development and Validation of a High-Quality Composite Real-World Mortality Endpoint.
Curtis Melissa D MD Griffith Sandra D SD Tucker Melisa M Taylor Michael D MD Capra William B WB Carrigan Gillis G Holzman Ben B Torres Aracelis Z AZ You Paul P Arnieri Brandon B Abernethy Amy P AP
Health services research 20180514 6
<h4>Objective</h4>To create a high-quality electronic health record (EHR)-derived mortality dataset for retrospective and prospective real-world evidence generation.<h4>Data sources/study setting</h4>Oncology EHR data, supplemented with external commercial and US Social Security Death Index data, benchmarked to the National Death Index (NDI).<h4>Study design</h4>We developed a recent, linkable, high-quality mortality variable amalgamated from multiple data sources to supplement EHR data, benchma ...[more]
Similar Datasets
Project description:ObjectiveWe expanded the previous assessment of a mortality variable suited for real-world evidence-focused oncology research.Data sourceWe used a nationwide electronic health record (EHR)-derived de-identified database.Data collectionWe included patients with at least 1 of 18 cancer types between January 1, 2011 and December 31, 2017. Patient-level structured data (EHRs, obituaries, and Social Security Death Index) and unstructured EHR data (abstracted) were linked to generate a composite mortality variable.Study designWe benchmarked sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and ±15-day agreement against the National Death Index (NDI). Real-world overall survival (rwOS) was estimated using the Kaplan-Meier method. We performed sensitivity analyses using a smaller patient cohort that underwent next-generation sequencing testing.Principal findingsCompared with the NDI across 18 cancer types (overall N = 160 436): sensitivity, 83.9%-91.5% (17/18 cancer types had sensitivity ≥85.0%); specificity, 93.5%-99.7%; PPV, 96.3%-98.3%; NPV, 75.0%-98.7%; ±15-day agreement, 95.6%-97.6%; and median rwOS estimates ranging from 2.8% to 12.7% greater. Sensitivity analysis results (n = 17 540) were consistent with the main analysis.ConclusionsAcross all cancer types analyzed, this composite mortality variable showed high sensitivity, specificity, PPV, NPV, and ±15-day agreement, and yielded median rwOS values modestly overestimated when compared to NDI-based results.
| S-EPMC8586476 | biostudies-literature
Project description:ImportanceTo our knowledge, this multicenter analysis is the first to test and validate (1) the prognostic impact of comorbidities on 1-year mortality after initial therapy of acute myeloid leukemia (AML) and (2) a novel, risk-stratifying composite model incorporating comorbidities, age, and cytogenetic and molecular risks.ObjectiveTo accurately estimate risks of mortality by developing and validating a composite model that combines the most significant patient-specific and AML-specific features.Design, setting, and participantsThis is a retrospective cohort study. A series of comorbidities, including those already incorporated into the hematopoietic cell transplantation–comorbidity index (HCT-CI), were evaluated. Patients were randomly divided into a training set (n = 733) and a validation set (n = 367). In the training set, covariates associated with 1-year overall mortality at a significance level of P < .10 constructed a multivariate Cox proportional hazards model in which the impact of each covariate was adjusted for that of all others. Then, the adjusted hazard ratios were used as weights. Performances of models were compared using C statistics for continuous outcomes and area under the curve (AUC) for binary outcomes.ExposuresInitial therapy for AML.Main outcomes and measuresDeath within 1 year after initial therapy for AML.ResultsA total of 1100 patients, ages 20 to 89 years, were treated for AML between January 1, 2008, and December 31, 2012, at 5 academic institutions specialized in treating AML; 605 (55%) were male, and 495 (45%) were female. In the validation set, the original HCT-CI had better C statistic and AUC estimates compared with the AML comorbidity index for prediction of 1-year mortality. Augmenting the original HCT-CI with 3 independently significant comorbidities, hypoalbuminemia, thrombocytopenia, and high lactate dehydrogenase level, yielded a better C statistic of 0.66 and AUC of 0.69 for 1-year mortality. A composite model comprising augmented HCT-CI, age, and cytogenetic/molecular risks had even better predictive estimates of 0.72 and 0.76, respectively.Conclusions and relevanceIn this cohort study, comorbidities influenced 1-year survival of patients with AML, and comorbidities are best captured by an augmented HCT-CI. The augmented HCT-CI, age, and cytogenetic/molecular risks could be combined into an AML composite model that could guide treatment decision-making and trial design in AML. Studying physical, cognitive, and social health might further clarify the prognostic role of aging. Targeting comorbidities with interventions alongside specific AML therapy might improve survival.
| S-EPMC5824273 | biostudies-literature
Project description:The adoption of high-sensitivity flow cytometry (HSFC) in routine laboratory settings has been slow owing to concerns regarding the reliability and reproducibility of results. Validation is an essential prerequisite for conducting assays, and implementing the CLSI guidelines has been confusing, primarily because many aspects are not yet established. We aimed to validate an HSFC protocol for detecting follicular helper T (Tfh) cells in a real-world laboratory environment. The analytical validity of the Tfh cell panel was ensured through rigorous testing, including evaluations of precision, stability, carryover, and sensitivity, following the CLSI H62 guidelines. We found that Tfh cells, present in very small numbers in the blood, could be sufficiently detected through HSFC, and concerns about the reliability and reproducibility of the results in real-world laboratories could be solved through systematic validation. Establishing the lower limit of quantification (LLOQ) is a critical step in HSFC evaluations. By selecting an appropriate sample, for example, collecting residual cells from CD4 isolation in our experiment and using them as low-level samples, the LLOQ could be accurately established. The strategic validation of flow cytometry panels can facilitate the adoption of HSFC in clinical laboratories, even with limited resources.
| S-EPMC10345173 | biostudies-literature
Project description:BackgroundThe availability of high fidelity electronic health record (EHR) data is a hallmark of the learning health care system. Washington State's Surgical Care Outcomes and Assessment Program (SCOAP) is a network of hospitals participating in quality improvement (QI) registries wherein data are manually abstracted from EHRs. To create the Comparative Effectiveness Research and Translation Network (CERTAIN), we semi-automated SCOAP data abstraction using a centralized federated data model, created a central data repository (CDR), and assessed whether these data could be used as real world evidence for QI and research.ObjectivesDescribe the validation processes and complexities involved and lessons learned.MethodsInvestigators installed a commercial CDR to retrieve and store data from disparate EHRs. Manual and automated abstraction systems were conducted in parallel (10/2012-7/2013) and validated in three phases using the EHR as the gold standard: 1) ingestion, 2) standardization, and 3) concordance of automated versus manually abstracted cases. Information retrieval statistics were calculated.ResultsFour unaffiliated health systems provided data. Between 6 and 15 percent of data elements were abstracted: 51 to 86 percent from structured data; the remainder using natural language processing (NLP). In phase 1, data ingestion from 12 out of 20 feeds reached 95 percent accuracy. In phase 2, 55 percent of structured data elements performed with 96 to 100 percent accuracy; NLP with 89 to 91 percent accuracy. In phase 3, concordance ranged from 69 to 89 percent. Information retrieval statistics were consistently above 90 percent.ConclusionsSemi-automated data abstraction may be useful, although raw data collected as a byproduct of health care delivery is not immediately available for use as real world evidence. New approaches to gathering and analyzing extant data are required.
| S-EPMC5983060 | biostudies-literature
Project description:BackgroundClinical trials are vital for developing new therapies but can also delay drug development. Efficient trial data management, optimized trial protocol, and accurate patient identification are critical for reducing trial timelines. Natural language processing (NLP) has the potential to achieve these objectives.ObjectiveThis study aims to assess the feasibility of using data-driven approaches to optimize clinical trial protocol design and identify eligible patients. This involves creating a comprehensive eligibility criteria knowledge base integrated within electronic health records using deep learning-based NLP techniques.MethodsWe obtained data of 3281 industry-sponsored phase 2 or 3 interventional clinical trials recruiting patients with non-small cell lung cancer, prostate cancer, breast cancer, multiple myeloma, ulcerative colitis, and Crohn disease from ClinicalTrials.gov, spanning the period between 2013 and 2020. A customized bidirectional long short-term memory- and conditional random field-based NLP pipeline was used to extract all eligibility criteria attributes and convert hypernym concepts into computable hyponyms along with their corresponding values. To illustrate the simulation of clinical trial design for optimization purposes, we selected a subset of patients with non-small cell lung cancer (n=2775), curated from the Mount Sinai Health System, as a pilot study.ResultsWe manually annotated the clinical trial eligibility corpus (485/3281, 14.78% trials) and constructed an eligibility criteria-specific ontology. Our customized NLP pipeline, developed based on the eligibility criteria-specific ontology that we created through manual annotation, achieved high precision (0.91, range 0.67-1.00) and recall (0.79, range 0.50-1) scores, as well as a high F1-score (0.83, range 0.67-1), enabling the efficient extraction of granular criteria entities and relevant attributes from 3281 clinical trials. A standardized eligibility criteria knowledge base, compatible with electronic health records, was developed by transforming hypernym concepts into machine-interpretable hyponyms along with their corresponding values. In addition, an interface prototype demonstrated the practicality of leveraging real-world data for optimizing clinical trial protocols and identifying eligible patients.ConclusionsOur customized NLP pipeline successfully generated a standardized eligibility criteria knowledge base by transforming hypernym criteria into machine-readable hyponyms along with their corresponding values. A prototype interface integrating real-world patient information allows us to assess the impact of each eligibility criterion on the number of patients eligible for the trial. Leveraging NLP and real-world data in a data-driven approach holds promise for streamlining the overall clinical trial process, optimizing processes, and improving efficiency in patient identification.
| S-EPMC11319878 | biostudies-literature
Project description:BackgroundDigital technologies are transforming the health care system. A large part of information is generated as real-world data (RWD). Data from electronic health records and digital biomarkers have the potential to reveal associations between the benefits and adverse events of medicines, establish new patient-stratification principles, expose unknown disease correlations, and inform on preventive measures. The impact for health care payers and providers, the biopharmaceutical industry, and governments is massive in terms of health outcomes, quality of care, and cost. However, a framework to assess the preliminary quality of RWD is missing, thus hindering the conduct of population-based observational studies to support regulatory decision-making and real-world evidence.ObjectiveTo address the need to qualify RWD, we aimed to build a web application as a tool to translate characterization of some quality parameters of RWD into a metric and propose a standard framework for evaluating the quality of the RWD.MethodsThe RWD-Cockpit systematically scores data sets based on proposed quality metrics and customizable variables chosen by the user. Sleep RWD generated de novo and publicly available data sets were used to validate the usability and applicability of the web application. The RWD quality score is based on the evaluation of 7 variables: manageability specifies access and publication status; complexity defines univariate, multivariate, and longitudinal data; sample size indicates the size of the sample or samples; privacy and liability stipulates privacy rules; accessibility specifies how the data set can be accessed and to what granularity; periodicity specifies how often the data set is updated; and standardization specifies whether the data set adheres to any specific technical or metadata standard. These variables are associated with several descriptors that define specific characteristics of the data set.ResultsTo address the need to qualify RWD, we built the RWD-Cockpit web application, which proposes a framework and applies a common standard for a preliminary evaluation of RWD quality across data sets-molecular, phenotypical, and social-and proposes a standard that can be further personalized by the community retaining an internal standard. Applied to 2 different case studies-de novo-generated sleep data and publicly available data sets-the RWD-Cockpit could identify and provide researchers with variables that might increase quality.ConclusionsThe results from the application of the framework of RWD metrics implemented in the RWD-Cockpit application suggests that multiple data sets can be preliminarily evaluated in terms of quality using the proposed metrics. The output scores-quality identifiers-provide a first quality assessment for the use of RWD. Although extensive challenges remain to be addressed to set RWD quality standards, our proposal can serve as an initial blueprint for community efforts in the characterization of RWD quality for regulated settings.
| S-EPMC9627468 | biostudies-literature
Project description:BackgroundPrevious cardiovascular risk prediction models in Japan have utilized prospective cohort studies with concise data. As the health information including health check-up records and administrative claims becomes digitalized and publicly available, application of large datasets based on such real-world data can achieve prediction accuracy and support social implementation of cardiovascular disease risk prediction models in preventive and clinical practice. In this study, classical regression and machine learning methods were explored to develop ischemic heart disease (IHD) and stroke prognostic models using real-world data.MethodsIQVIA Japan Claims Database was searched to include 691,160 individuals (predominantly corporate employees and their families working in secondary and tertiary industries) with at least one annual health check-up record during the identification period (April 2013-December 2018). The primary outcome of the study was the first recorded IHD or stroke event. Predictors were annual health check-up records at the index year-month, comprising demographic characteristics, laboratory tests, and questionnaire features. Four prediction models (Cox, Elnet-Cox, XGBoost, and Ensemble) were assessed in the present study to develop a cardiovascular disease risk prediction model for Japan.ResultsThe analysis cohort consisted of 572,971 invididuals. All prediction models showed similarly good performance. The Harrell's C-index was close to 0.9 for all IHD models, and above 0.7 for stroke models. In IHD models, age, sex, high-density lipoprotein, low-density lipoprotein, cholesterol, and systolic blood pressure had higher importance, while in stroke models systolic blood pressure and age had higher importance.ConclusionOur study analyzed classical regression and machine learning algorithms to develop cardiovascular disease risk prediction models for IHD and stroke in Japan that can be applied to practical use in a large population with predictive accuracy.
| S-EPMC9989775 | biostudies-literature
Project description:Complex disorders usually affect multiple symptom domains measured by several outcomes. The importance of these outcomes is often different among patients. Current approaches integrate multiple outcomes without considering patient preferences at the individual level. In this paper, we propose a new composite Desirability of Outcome Ranking (DOOR) that integrates individual level ranking of outcome importance and define a winning probability measuring the overall treatment effect. Stratified randomization can be performed based on the participants' baseline outcome rankings. A Wilcoxon-Mann-Whitney U-statistic is used to average the pairwise DOOR between one treated and one control patient, considering the difference in these patients' ranking of outcome importance. We use both theoretical and empirical methods to examine the statistical properties of our method and to compare with conventional approaches. We conclude that the proposed composite DOOR properly reflects patient-level preferences and can be used in pivotal trials or comparative effectiveness trials for a patient-centered evaluation of overall treatment benefits.
| S-EPMC10655937 | biostudies-literature
Project description:ObjectivesThe objectives of this study are to construct the high definition phenotype (HDP), a novel time-series data structure composed of both primary and derived parameters, using heterogeneous clinical sources and to determine whether different predictive models can utilize the HDP in the neonatal intensive care unit (NICU) to improve neonatal mortality prediction in clinical settings.Materials and methodsA total of 49 primary data parameters were collected from July 2018 to May 2020 from eight level-III NICUs. From a total of 1546 patients, 757 patients were found to contain sufficient fixed, intermittent, and continuous data to create HDPs. Two different predictive models utilizing the HDP, one a logistic regression model (LRM) and the other a deep learning long-short-term memory (LSTM) model, were constructed to predict neonatal mortality at multiple time points during the patient hospitalization. The results were compared with previous illness severity scores, including SNAPPE, SNAPPE-II, CRIB, and CRIB-II.ResultsA HDP matrix, including 12 221 536 minutes of patient stay in NICU, was constructed. The LRM model and the LSTM model performed better than existing neonatal illness severity scores in predicting mortality using the area under the receiver operating characteristic curve (AUC) metric. An ablation study showed that utilizing continuous parameters alone results in an AUC score of >80% for both LRM and LSTM, but combining fixed, intermittent, and continuous parameters in the HDP results in scores >85%. The probability of mortality predictive score has recall and precision of 0.88 and 0.77 for the LRM and 0.97 and 0.85 for the LSTM.Conclusions and relevanceThe HDP data structure supports multiple analytic techniques, including the statistical LRM approach and the machine learning LSTM approach used in this study. LRM and LSTM predictive models of neonatal mortality utilizing the HDP performed better than existing neonatal illness severity scores. Further research is necessary to create HDP-based clinical decision tools to detect the early onset of neonatal morbidities.
| S-EPMC7991779 | biostudies-literature
Project description:Social dysfunction is a hallmark of schizophrenia that is associated with emotional disturbances. Researchers have employed ecological momentary assessment (EMA) to measure social and emotional functioning in people with schizophrenia. Yet, few studies have evaluated quality of real-world social interactions, and it is unclear how interactions impact emotional experiences in this population. Using novel EMA that passively collects audio data, we examined daily social behavior and emotion in schizophrenia (n = 38) and control (n = 36) groups. Contrary to hypotheses, both groups interacted with others at the same rate and exhibited similar levels of positive emotion. However, as expected, the schizophrenia group exhibited significantly less high-quality interactions and reported more negative emotion than controls. Social versus non-social context did not influence experienced emotion in either group. This is the first real-world study to passively assess quality of social interactions in schizophrenia. Although those with schizophrenia did not differ in their number of interactions, they were less likely to engage in substantive, personal conversations. Because high-quality interactions are linked with better social outcomes, this finding has important potential treatment implications. Future research should investigate quality of interactions across different types of social activities to gain a more nuanced understanding of social dysfunction in schizophrenia.
| S-EPMC8206017 | biostudies-literature