Dataset Information

Neural Machine Translation-Based Automated Current Procedural Terminology Classification System Using Procedure Text: Development and Validation Study.

ABSTRACT:

Background

Administrative costs for billing and insurance-related activities in the United States are substantial. One critical cause of the high overhead of administrative costs is medical billing errors. With advanced deep learning techniques, developing advanced models to predict hospital and professional billing codes has become feasible. These models can be used for administrative cost reduction and billing process improvements.

Objective

In this study, we aim to develop an automated anesthesiology current procedural terminology (CPT) prediction system that translates manually entered surgical procedure text into standard forms using neural machine translation (NMT) techniques. The standard forms are calculated using similarity scores to predict the most appropriate CPT codes. Although this system aims to enhance medical billing coding accuracy to reduce administrative costs, we compare its performance with that of previously developed machine learning algorithms.

Methods

We collected and analyzed all operative procedures performed at Michigan Medicine between January 2017 and June 2019 (2.5 years). The first 2 years of data were used to train and validate the existing models and compare the results from the NMT-based model. Data from 2019 (6-month follow-up period) were then used to measure the accuracy of the CPT code prediction. Three experimental settings were designed with different data types to evaluate the models. Experiment 1 used the surgical procedure text entered manually in the electronic health record. Experiment 2 used preprocessing of the procedure text. Experiment 3 used preprocessing of the combined procedure text and preoperative diagnoses. The NMT-based model was compared with the support vector machine (SVM) and long short-term memory (LSTM) models.

Results

The NMT model yielded the highest top-1 accuracy in experiments 1 and 2 at 81.64% and 81.71% compared with the SVM model (81.19% and 81.27%, respectively) and the LSTM model (80.96% and 81.07%, respectively). The SVM model yielded the highest top-1 accuracy of 84.30% in experiment 3, followed by the LSTM model (83.70%) and the NMT model (82.80%). In experiment 3, the addition of preoperative diagnoses showed 3.7%, 3.2%, and 1.3% increases in the SVM, LSTM, and NMT models in top-1 accuracy over those in experiment 2, respectively. For top-3 accuracy, the SVM, LSTM, and NMT models achieved 95.64%, 95.72%, and 95.60% for experiment 1, 95.75%, 95.67%, and 95.69% for experiment 2, and 95.88%, 95.93%, and 95.06% for experiment 3, respectively.

Conclusions

This study demonstrates the feasibility of creating an automated anesthesiology CPT classification system based on NMT techniques using surgical procedure text and preoperative diagnosis. Our results show that the performance of the NMT-based CPT prediction system is equivalent to that of the SVM and LSTM prediction models. Importantly, we found that including preoperative diagnoses improved the accuracy of using the procedure text alone.

SUBMITTER: Joo H

PROVIDER: S-EPMC8190648 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BACKGROUND:Accurate anesthesiology procedure code data are essential to quality improvement, research, and reimbursement tasks within anesthesiology practices. Advanced data science techniques, including machine learning and natural language processing, offer opportunities to develop classification tools for Current Procedural Terminology codes across anesthesia procedures. METHODS:Models were created using a Train/Test dataset including 1,164,343 procedures from 16 academic and private hospitals. Five supervised machine learning models were created to classify anesthesiology Current Procedural Terminology codes, with accuracy defined as first choice classification matching the institutional-assigned code existing in the perioperative database. The two best performing models were further refined and tested on a Holdout dataset from a single institution distinct from Train/Test. A tunable confidence parameter was created to identify cases for which models were highly accurate, with the goal of at least 95% accuracy, above the reported 2018 Centers for Medicare and Medicaid Services (Baltimore, Maryland) fee-for-service accuracy. Actual submitted claim data from billing specialists were used as a reference standard. RESULTS:Support vector machine and neural network label-embedding attentive models were the best performing models, respectively, demonstrating overall accuracies of 87.9% and 84.2% (single best code), and 96.8% and 94.0% (within top three). Classification accuracy was 96.4% in 47.0% of cases using support vector machine and 94.4% in 62.2% of cases using label-embedding attentive model within the Train/Test dataset. In the Holdout dataset, respective classification accuracies were 93.1% in 58.0% of cases and 95.0% among 62.0%. The most important feature in model training was procedure text. CONCLUSIONS:Through application of machine learning and natural language processing techniques, highly accurate real-time models were created for anesthesiology Current Procedural Terminology code classification. The increased processing speed and a priori targeted accuracy of this classification approach may provide performance optimization and cost reduction for quality improvement, research, and reimbursement tasks reliant on anesthesiology procedure codes.

Project description:IntroductionUnlike antibiotic and perfusion support, guidelines for sepsis source control lack high-quality evidence and are ungraded. Internally valid administrative data methods are needed to identify cases representing source control procedures to evaluate outcomes.MethodsOver five modified Delphi rounds, two independent reviewers identified Current Procedural Terminology (CPT) codes pertinent to source control. In each round, codes with perfect agreement were retained or excluded, whereas disagreements were reviewed by the panelists. Manual review of 400 patient records meeting Sepsis-3 criteria (2010-2017) clinically adjudicated which encounters included source control procedures (gold standard). The performance of consensus codes was compared with the gold standard to assess sensitivity, specificity, predictive values, and likelihood ratios.ResultsOf 5752 CPT codes, 609 consensus codes represented source control procedures. Of 400 hospitalizations for sepsis, 39 (9.8%; 95% confidence interval [CI] 7.0%-13.1%) underwent gold standard source control procedures and 29 (7.3%; 95% CI 4.9-10.3%) consensus code-defined source control procedures. Thirty consensus codes were identified (20.0% gastrointestinal/intraabdominal, 10.0% genitourinary, 13.3% hepatopancreatobiliary, 23.3% orthopedic/cranial, 23.3% soft tissue, and 10.0% intrathoracic), which had 61.5% (95% CI 44.6%-76.6%) sensitivity, 98.6% (95% CI 96.8%-99.6%) specificity, 83.2% (95% CI 66.6%-92.4%) positive, and 95.9% (95% CI 93.9%-97.2%) negative predictive values. With pretest probability at sample prevalence, an identified consensus code had a posttest probability of 83.0% (95% CI 66.0%-92.0%), whereas consensus code absence had a probability of 4.0% (95% CI 3.0-6.0) for undergoing a source control procedure.ConclusionsUsing modified Delphi methodology, we created and validated CPT codes identifying source control procedures, providing a framework for evaluation of the surgical care of patients with sepsis.