Dataset Information

A Human-Algorithm Integration System for Hip Fracture Detection on Plain Radiography: System Development and Validation Study.

ABSTRACT:

Background

Hip fracture is the most common type of fracture in elderly individuals. Numerous deep learning (DL) algorithms for plain pelvic radiographs (PXRs) have been applied to improve the accuracy of hip fracture diagnosis. However, their efficacy is still undetermined.

Objective

The objective of this study is to develop and validate a human-algorithm integration (HAI) system to improve the accuracy of hip fracture diagnosis in a real clinical environment.

Methods

The HAI system with hip fracture detection ability was developed using a deep learning algorithm trained on trauma registry data and 3605 PXRs from August 2008 to December 2016. To compare their diagnostic performance before and after HAI system assistance using an independent testing dataset, 34 physicians were recruited. We analyzed the physicians' accuracy, sensitivity, specificity, and agreement with the algorithm; we also performed subgroup analyses according to physician specialty and experience. Furthermore, we applied the HAI system in the emergency departments of different hospitals to validate its value in the real world.

Results

With the support of the algorithm, which achieved 91% accuracy, the diagnostic performance of physicians was significantly improved in the independent testing dataset, as was revealed by the sensitivity (physician alone, median 95%; HAI, median 99%; P<.001), specificity (physician alone, median 90%; HAI, median 95%; P<.001), accuracy (physician alone, median 90%; HAI, median 96%; P<.001), and human-algorithm agreement [physician alone ?, median 0.69 (IQR 0.63-0.74); HAI ?, median 0.80 (IQR 0.76-0.82); P<.001. With the help of the HAI system, the primary physicians showed significant improvement in their diagnostic performance to levels comparable to those of consulting physicians, and both the experienced and less-experienced physicians benefited from the HAI system. After the HAI system had been applied in 3 departments for 5 months, 587 images were examined. The sensitivity, specificity, and accuracy of the HAI system for detecting hip fractures were 97%, 95.7%, and 96.08%, respectively.

Conclusions

HAI currently impacts health care, and integrating this technology into emergency departments is feasible. The developed HAI system can enhance physicians' hip fracture diagnostic performance.

SUBMITTER: Cheng CT

PROVIDER: S-EPMC7732715 | biostudies-literature | 2020 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A Human-Algorithm Integration System for Hip Fracture Detection on Plain Radiography: System Development and Validation Study.

Cheng Chi-Tung CT Chen Chih-Chi CC Cheng Fu-Jen FJ Chen Huan-Wu HW Su Yi-Siang YS Yeh Chun-Nan CN Chung I-Fang IF Liao Chien-Hung CH

JMIR medical informatics 20201127 11

<h4>Background</h4>Hip fracture is the most common type of fracture in elderly individuals. Numerous deep learning (DL) algorithms for plain pelvic radiographs (PXRs) have been applied to improve the accuracy of hip fracture diagnosis. However, their efficacy is still undetermined.<h4>Objective</h4>The objective of this study is to develop and validate a human-algorithm integration (HAI) system to improve the accuracy of hip fracture diagnosis in a real clinical environment.<h4>Methods</h4>The H ...[more]

PMID: 33245279

Similar Datasets

Project description:BackgroundNatural language processing (NLP) methods are powerful tools for extracting and analyzing critical information from free-text data. MedTaggerIE, an open-source NLP pipeline for information extraction based on text patterns, has been widely used in the annotation of clinical notes. A rule-based system, MedTagger-total hip arthroplasty (THA), developed based on MedTaggerIE, was previously shown to correctly identify the surgical approach, fixation, and bearing surface from the THA operative notes at Mayo Clinic.ObjectiveThis study aimed to assess the implementability, usability, and portability of MedTagger-THA at two external institutions, Michigan Medicine and the University of Iowa, and provide lessons learned for best practices.MethodsWe conducted iterative test-apply-refinement processes with three involved sites-the development site (Mayo Clinic) and two deployment sites (Michigan Medicine and the University of Iowa). Mayo Clinic was the primary NLP development site, with the THA registry as the gold standard. The activities at the two deployment sites included the extraction of the operative notes, gold standard development (Michigan: registry data; Iowa: manual chart review), the refinement of NLP algorithms on training data, and the evaluation of test data. Error analyses were conducted to understand language variations across sites. To further assess the model specificity for approach and fixation, we applied the refined MedTagger-THA to arthroscopic hip procedures and periacetabular osteotomy cases, as neither of these operative notes should contain any approach or fixation keywords.ResultsMedTagger-THA algorithms were implemented and refined independently for both sites. At Michigan, the study comprised THA-related notes for 2569 patient-date pairs. Before model refinement, MedTagger-THA algorithms demonstrated excellent accuracy for approach (96.6%, 95% CI 94.6%-97.9%) and fixation (95.7%, 95% CI 92.4%-97.6%). These results were comparable with internal accuracy at the development site (99.2% for approach and 90.7% for fixation). Model refinement improved accuracies slightly for both approach (99%, 95% CI 97.6%-99.6%) and fixation (98%, 95% CI 95.3%-99.3%). The specificity of approach identification was 88.9% for arthroscopy cases, and the specificity of fixation identification was 100% for both periacetabular osteotomy and arthroscopy cases. At the Iowa site, the study comprised an overall data set of 100 operative notes (50 training notes and 50 test notes). MedTagger-THA algorithms achieved moderate-high performance on the training data. After model refinement, the model achieved high performance for approach (100%, 95% CI 91.3%-100%), fixation (98%, 95% CI 88.3%-100%), and bearing surface (92%, 95% CI 80.5%-97.3%).ConclusionsHigh performance across centers was achieved for the MedTagger-THA algorithms, demonstrating that they were sufficiently implementable, usable, and portable to different deployment sites. This study provided important lessons learned during the model deployment and validation processes, and it can serve as a reference for transferring rule-based electronic health record models.

Project description:Background The ability to take care of oneself after hip fracture surgery is important for older adults. Various scales have been developed for evaluation of this ability, but a scale specifically focusing on hip fracture has not been developed. The aim of this study was to develop and validate a scale (Hip Fracture Surgery Self-Care Scale, HFS-SC) to evaluate self-care for older adults undergoing hip fracture surgery. Methods The scale was developed according to the guidelines by DeVellis. Initial items were derived from a literature review and individual interviews with 11 older adults who underwent hip fracture surgery. To confirm the suitability of the questions, a preliminary survey was conducted on 25 older adults. Psychometric testing was performed on 300 older adults 65 years old or over living at home after surgery for hip fracture. Psychometric properties of the scale were examined by content validity, construct validity, concurrent validity, internal consistency reliability, and test-retest reliability. Results Exploratory factor analysis and confirmatory factor analysis demonstrated that the 18-item scale comprised five factors (functional independence, symptom recognition and management, positive mental health, participation and support in social activities, and a safe environment). The results of EFA showed that the factor loadings ranging from 0.51 to 0.87. The results of CFA were χ2=375.83, χ2/df=2.14, RMSEA 0.07, SRMR 0.05, GFI 0.88, TLI 0.91, and CFI 0.92 for the 18-item scale. The reliabilities of the scale were 0.91 for Cronbach’s alpha and 0.82 for test-retest reliability. Conclusions The HFS-SC has acceptable validity and reliability and is expected to be useful for evaluating the levels of self-care for older adults undergoing hip fracture surgery and adjusting at the post-fracture period at the community or out-patient department. Supplementary Information The online version contains supplementary material available at 10.1186/s12912-022-00982-3.

Dataset Information

A Human-Algorithm Integration System for Hip Fracture Detection on Plain Radiography: System Development and Validation Study.

Background

Objective

Methods

Results

Conclusions

Publications

A Human-Algorithm Integration System for Hip Fracture Detection on Plain Radiography: System Development and Validation Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets