Dataset Information

Manual review of electronic medical records as a reference standard for case definition development: a validation study.

ABSTRACT: BACKGROUND:The Canadian Primary Care Sentinel Surveillance Network (CPCSSN) previously carried out a validation study of case definitions for 8 chronic diseases (diabetes mellitus, hypertension, osteoarthritis, depression, dementia, chronic obstructive pulmonary disease, parkinsonism and epilepsy) using direct review of "raw" electronic medical record data. Although effective, this method is time-consuming and can present methodological and organizational challenges. We aimed to determine whether the processed and standardized data contained with the CPCSSN database might function as a reference standard for case definition validation. METHODS:Using a traditional validation study design, we compared the case identification results of the chart reviews for the 8 chronic diseases with the results of a manual review of the CPCSSN processed data for the same conditions in the same patient sample. Patients were randomly sampled from the June 30, 2012 CPCSSN database, with oversampling of patients with rare conditions. RESULTS:We analyzed data for 1906 patients. Manual review of the CPCSSN records for case ascertainment yielded sensitivity ranging from 77.5% (95% confidence interval [CI] 73.3%-81.6%) for depression to 97.2% (95% CI 95.4%-99.0%) for diabetes. Specificity was high for all definitions (range 93.1% [95% CI 91.4%-94.7%] to 99.4% [95% CI 99.0%-99.8%]). Positive predictive values and negative predictive values also showed high accuracy of the manual CPCSSN record review relative to review of the raw chart data. INTERPRETATION:The use of CPCSSN records as the reference standard to validate case definitions substantially reduces the burden on sentinel physicians and clinic managers as well as on researchers while offering a reference standard that is a reasonable substitution for chart review.

SUBMITTER: Williamson T

PROVIDER: S-EPMC5741416 | biostudies-literature | 2017 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Manual review of electronic medical records as a reference standard for case definition development: a validation study.

Williamson Tyler T Miyagishima Rebecca C RC Derochie Janeen D JD Drummond Neil N

CMAJ open 20171201 4

<h4>Background</h4>The Canadian Primary Care Sentinel Surveillance Network (CPCSSN) previously carried out a validation study of case definitions for 8 chronic diseases (diabetes mellitus, hypertension, osteoarthritis, depression, dementia, chronic obstructive pulmonary disease, parkinsonism and epilepsy) using direct review of "raw" electronic medical record data. Although effective, this method is time-consuming and can present methodological and organizational challenges. We aimed to determin ...[more]

PMID: 29242256

Similar Datasets

Project description:BackgroundThe secondary use of structured electronic medical record (sEMR) data has become a challenge due to the diversity, sparsity, and high dimensionality of the data representation. Constructing an effective representation for sEMR data is becoming more and more crucial for subsequent data applications.ObjectiveWe aimed to apply the embedding technique used in the natural language processing domain for the sEMR data representation and to explore the feasibility and superiority of the embedding-based feature and patient representations in clinical application.MethodsThe entire training corpus consisted of records of 104,752 hospitalized patients with 13,757 medical concepts of disease diagnoses, physical examinations and procedures, laboratory tests, medications, etc. Each medical concept was embedded into a 200-dimensional real number vector using the Skip-gram algorithm with some adaptive changes from shuffling the medical concepts in a record 20 times. The average of vectors for all medical concepts in a patient record represented the patient. For embedding-based feature representation evaluation, we used the cosine similarities among the medical concept vectors to capture the latent clinical associations among the medical concepts. We further conducted a clustering analysis on stroke patients to evaluate and compare the embedding-based patient representations. The Hopkins statistic, Silhouette index (SI), and Davies-Bouldin index were used for the unsupervised evaluation, and the precision, recall, and F1 score were used for the supervised evaluation.ResultsThe dimension of patient representation was reduced from 13,757 to 200 using the embedding-based representation. The average cosine similarity of the selected disease (subarachnoid hemorrhage) and its 15 clinically relevant medical concepts was 0.973. Stroke patients were clustered into two clusters with the highest SI (0.852). Clustering analyses conducted on patients with the embedding representations showed higher applicability (Hopkins statistic 0.931), higher aggregation (SI 0.862), and lower dispersion (Davies-Bouldin index 0.551) than those conducted on patients with reference representation methods. The clustering solutions for patients with the embedding-based representation achieved the highest F1 scores of 0.944 and 0.717 for two clusters.ConclusionsThe feature-level embedding-based representations can reflect the potential clinical associations among medical concepts effectively. The patient-level embedding-based representation is easy to use as continuous input to standard machine learning algorithms and can bring performance improvements. It is expected that the embedding-based representation will be helpful in a wide range of secondary uses of sEMR data.

Project description:BackgroundTo date, there has been no validated method to identify cases of pelvic floor disorders in primary care electronic medical record (EMR) data. We aimed to develop and validate symptom-based case definitions for urinary incontinence, fecal incontinence and pelvic organ prolapse in women, for use in primary care epidemiologic or clinical research.MethodsOur retrospective study used EMR data from the Southern Alberta Primary Care Research Network (SAPCReN) and the Canadian Primary Care Sentinel Surveillance Network (CPCSSN) in southern Alberta. Trained researchers remotely reviewed a random sample of EMR charts of women aged 18 years or older from 6 rural and urban clinics to validate case definitions for urinary incontinence, fecal incontinence and pelvic organ prolapse. We calculated sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV), and estimated SAPCReN prevalence as appropriate.ResultsCharts of 900 women were included. Sensitivity was 81.9% (95% confidence interval [CI] 75.1-87.2) for urinary incontinence, 61.2% (95% CI 46.2-74.5) for fecal incontinence, and 51.8% (95% CI 40.6-62.8) for pelvic organ prolapse. Corresponding specificity values were 71.9% (95% CI 68.4-75.1), 99.2% (95% CI 98.2-99.6) and 98.8% (95% CI 97.7-99.4), PPVs 40.6% (95% CI 35.4-46.0), 81.1% (95% CI 64.3-91.4) and 81.1% (95% CI 67.6-90.1), and NPVs 94.4% (95% CI 92.1-96.1), 97.8% (95% CI 96.5-98.6) and 95.3% (95% CI 93.6-96.6). The SAPCReN-observed prevalence for urinary incontinence was 29.7% (95% CI 29.3-30.0), but the adjusted prevalence was 2.97%.InterpretationThe case definition for urinary incontinence met our standard for validity (sensitivity and specificity > 70%), and the case definitions for fecal incontinence and pelvic organ prolapse had PPVs greater than 80%. The urinary incontinence definition may be used in epidemiologic research, and those for fecal incontinence and pelvic organ prolapse may be used in quality-improvement studies or creation of disease registries. Our symptom-based case definitions could also be adapted for research in other EMR settings.

Project description:BackgroundEvaluation is a challenging but necessary part of the development cycle of clinical information systems like the electronic medical records (EMR) system. It is believed that such evaluations should include multiple perspectives, be comparative and employ both qualitative and quantitative methods. Self-administered questionnaires are frequently used as a quantitative evaluation method in medical informatics, but very few validated questionnaires address clinical use of EMR systems.MethodsWe have developed a task-oriented questionnaire for evaluating EMR systems from the clinician's perspective. The key feature of the questionnaire is a list of 24 general clinical tasks. It is applicable to physicians of most specialties and covers essential parts of their information-oriented work. The task list appears in two separate sections, about EMR use and task performance using the EMR, respectively. By combining these sections, the evaluator may estimate the potential impact of the EMR system on health care delivery. The results may also be compared across time, site or vendor. This paper describes the development, performance and validation of the questionnaire. Its performance is shown in two demonstration studies (n = 219 and 80). Its content is validated in an interview study (n = 10), and its reliability is investigated in a test-retest study (n = 37) and a scaling study (n = 31).ResultsIn the interviews, the physicians found the general clinical tasks in the questionnaire relevant and comprehensible. The tasks were interpreted concordant to their definitions. However, the physicians found questions about tasks not explicitly or only partially supported by the EMR systems difficult to answer. The two demonstration studies provided unambiguous results and low percentages of missing responses. In addition, criterion validity was demonstrated for a majority of task-oriented questions. Their test-retest reliability was generally high, and the non-standard scale was found symmetric and ordinal.ConclusionThis questionnaire is relevant for clinical work and EMR systems, provides reliable and interpretable results, and may be used as part of any evaluation effort involving the clinician's perspective of an EMR system.

Project description:BackgroundElectronic health record (EHR)-based computable phenotype algorithms allow researchers to efficiently identify a large virtual cohort of Human Immunodeficiency Virus (HIV) patients. Built upon existing algorithms, we refined, improved, and validated an HIV phenotype algorithm using data from the OneFlorida Data Trust, a repository of linked claims data and EHRs from its clinical partners, which provide care to over 15 million patients across all 67 counties in Florida.MethodsOur computable phenotype examined information from multiple EHR domains, including clinical encounters with diagnoses, prescription medications, and laboratory tests. To identify an HIV case, the algorithm requires the patient to have at least one diagnostic code for HIV and meet one of the following criteria: have 1+ positive HIV laboratory, have been prescribed with HIV medications, or have 3+ visits with HIV diagnostic codes. The computable phenotype was validated against a subset of clinical notes.ResultsAmong the 15+ million patients from OneFlorida, we identified 61,313 patients with confirmed HIV diagnosis. Among them, 8.05% met all four inclusion criteria, 69.7% met the 3+ HIV encounters criteria in addition to having HIV diagnostic code, and 8.1% met all criteria except for having positive laboratories. Our algorithm achieved higher sensitivity (98.9%) and comparable specificity (97.6%) relative to existing algorithms (77-83% sensitivity, 86-100% specificity). The mean age of the sample was 42.7 years, 58% male, and about half were Black African American. Patients' average follow-up period (the time between the first and last encounter in the EHRs) was approximately 4.6 years. The median number of all encounters and HIV-related encounters were 79 and 21, respectively.ConclusionBy leveraging EHR data from multiple clinical partners and domains, with a considerably diverse population, our algorithm allows more flexible criteria for identifying patients with incomplete laboratory test results and medication prescribing history compared with prior studies.

Project description:ObjectiveTo provide focused evaluation of predictive modeling of electronic medical record (EMR) data to predict 30 day hospital readmission.DesignSystematic review.Data sourceOvid Medline, Ovid Embase, CINAHL, Web of Science, and Scopus from January 2015 to January 2019.Eligibility criteria for selecting studiesAll studies of predictive models for 28 day or 30 day hospital readmission that used EMR data.Outcome measuresCharacteristics of included studies, methods of prediction, predictive features, and performance of predictive models.ResultsOf 4442 citations reviewed, 41 studies met the inclusion criteria. Seventeen models predicted risk of readmission for all patients and 24 developed predictions for patient specific populations, with 13 of those being developed for patients with heart conditions. Except for two studies from the UK and Israel, all were from the US. The total sample size for each model ranged between 349 and 1 195 640. Twenty five models used a split sample validation technique. Seventeen of 41 studies reported C statistics of 0.75 or greater. Fifteen models used calibration techniques to further refine the model. Using EMR data enabled final predictive models to use a wide variety of clinical measures such as laboratory results and vital signs; however, use of socioeconomic features or functional status was rare. Using natural language processing, three models were able to extract relevant psychosocial features, which substantially improved their predictions. Twenty six studies used logistic or Cox regression models, and the rest used machine learning methods. No statistically significant difference (difference 0.03, 95% confidence interval -0.0 to 0.07) was found between average C statistics of models developed using regression methods (0.71, 0.68 to 0.73) and machine learning (0.74, 0.71 to 0.77).ConclusionsOn average, prediction models using EMR data have better predictive performance than those using administrative data. However, this improvement remains modest. Most of the studies examined lacked inclusion of socioeconomic features, failed to calibrate the models, neglected to conduct rigorous diagnostic testing, and did not discuss clinical impact.

Project description:BACKGROUND:Electronic medical records (EMRs) contain a wealth of information that can support data-driven decision making in health care policy design and service planning. Although research using EMRs has become increasingly prevalent, challenges such as coding inconsistency, data validity, and lack of suitable measures in important domains still hinder the progress. OBJECTIVE:The objective of this study was to design a structured way to process records in administrative EMR systems for health services research and assess validity in selected areas. METHODS:On the basis of a local hospital EMR system in Singapore, we developed a structured framework for EMR data processing, including standardization and phenotyping of diagnosis codes, construction of cohort with multilevel views, and generation of variables and proxy measures to supplement primary data. Disease complexity was estimated by Charlson Comorbidity Index (CCI) and Polypharmacy Score (PPS), whereas socioeconomic status (SES) was estimated by housing type. Validity of modified diagnosis codes and derived measures were investigated. RESULTS:Visit-level (N=7,778,761) and patient-level records (n=549,109) were generated. The International Classification of Diseases, Tenth Revision, Australian Modification (ICD-10-AM) codes were standardized to the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) with a mapping rate of 87.1%. In all, 97.4% of the ICD-9-CM codes were phenotyped successfully using Clinical Classification Software by Agency for Healthcare Research and Quality. Diagnosis codes that underwent modification (truncation or zero addition) in standardization and phenotyping procedures had the modification validated by physicians, with validity rates of more than 90%. Disease complexity measures (CCI and PPS) and SES were found to be valid and robust after a correlation analysis and a multivariate regression analysis. CCI and PPS were correlated with each other and positively correlated with health care utilization measures. Larger housing type was associated with lower government subsidies received, suggesting association with higher SES. Profile of constructed cohorts showed differences in disease prevalence, disease complexity, and health care utilization in those aged above 65 years and those aged 65 years or younger. CONCLUSIONS:The framework proposed in this study would be useful for other researchers working with EMR data for health services research. Further analyses would be needed to better understand differences observed in the cohorts.

Dataset Information

Manual review of electronic medical records as a reference standard for case definition development: a validation study.

Publications

Manual review of electronic medical records as a reference standard for case definition development: a validation study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets