Dataset Information

Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose Risk Among Medicare Beneficiaries With Opioid Prescriptions.

ABSTRACT:

Importance

Current approaches to identifying individuals at high risk for opioid overdose target many patients who are not truly at high risk.

Objective

To develop and validate a machine-learning algorithm to predict opioid overdose risk among Medicare beneficiaries with at least 1 opioid prescription.

Design, setting, and participants

A prognostic study was conducted between September 1, 2017, and December 31, 2018. Participants (n = 560 057) included fee-for-service Medicare beneficiaries without cancer who filled 1 or more opioid prescriptions from January 1, 2011, to December 31, 2015. Beneficiaries were randomly and equally divided into training, testing, and validation samples.

Exposures

Potential predictors (n = 268), including sociodemographics, health status, patterns of opioid use, and practitioner-level and regional-level factors, were measured in 3-month windows, starting 3 months before initiating opioids until loss of follow-up or the end of observation.

Main outcomes and measures

Opioid overdose episodes from inpatient and emergency department claims were identified. Multivariate logistic regression (MLR), least absolute shrinkage and selection operator-type regression (LASSO), random forest (RF), gradient boosting machine (GBM), and deep neural network (DNN) were applied to predict overdose risk in the subsequent 3 months after initiation of treatment with prescription opioids. Prediction performance was assessed using the C statistic and other metrics (eg, sensitivity, specificity, and number needed to evaluate [NNE] to identify one overdose). The Youden index was used to identify the optimized threshold of predicted score that balanced sensitivity and specificity.

Results

Beneficiaries in the training (n = 186 686), testing (n = 186 685), and validation (n = 186 686) samples had similar characteristics (mean [SD] age of 68.0 [14.5] years, and approximately 63% were female, 82% were white, 35% had disabilities, 41% were dual eligible, and 0.60% had at least 1 overdose episode). In the validation sample, the DNN (C statistic = 0.91; 95% CI, 0.88-0.93) and GBM (C statistic = 0.90; 95% CI, 0.87-0.94) algorithms outperformed the LASSO (C statistic = 0.84; 95% CI, 0.80-0.89), RF (C statistic = 0.80; 95% CI, 0.75-0.84), and MLR (C statistic = 0.75; 95% CI, 0.69-0.80) methods for predicting opioid overdose. At the optimized sensitivity and specificity, DNN had a sensitivity of 92.3%, specificity of 75.7%, NNE of 542, positive predictive value of 0.18%, and negative predictive value of 99.9%. The DNN classified patients into low-risk (76.2% [142 180] of the cohort), medium-risk (18.6% [34 579] of the cohort), and high-risk (5.2% [9747] of the cohort) subgroups, with only 1 in 10 000 in the low-risk subgroup having an overdose episode. More than 90% of overdose episodes occurred in the high-risk and medium-risk subgroups, although positive predictive values were low, given the rare overdose outcome.

Conclusions and relevance

Machine-learning algorithms appear to perform well for risk prediction and stratification of opioid overdose, especially in identifying low-risk subgroups that have minimal risk of overdose.

SUBMITTER: Lo-Ciganic WH

PROVIDER: S-EPMC6583312 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ImportanceNonadherence to buprenorphine may increase patient risk of opioid overdose and increase health care spending. Quantifying the impacts of nonadherence can help inform clinician practice and policy.ObjectiveTo estimate the association between buprenorphine treatment gaps, opioid overdose, and health care spending.Design, setting, and participantsThis longitudinal case-control study compared patient opioid overdose and health care spending in buprenorphine-treated months with treatment gap months. Individuals who were US Medicare fee-for-service beneficiaries diagnosed with opioid use disorder who received at least 1 two-week period of continuous buprenorphine treatment between 2010 and 2017 were included. Analysis took place between January 2010 and December 2017.InterventionsA gap in buprenorphine treatment in a month lasting more than 15 consecutive days.Main outcomes and measuresOpioid overdose and total, medical, and drug spending (combined patient out-of-pocket and Medicare spending).ResultsOf 34 505 Medicare beneficiaries (17 927 [520%] male; 16 578 [48.1%] female; mean [SD] age, 49.5 [12.7] years; 168 [0.5%] Asian; 2949 [8.5%] Black; 2089 [6.0%] Hispanic; 266 [0.8%] Native American; 28 525 [82.7%] White; 508 [1.5%] other race), 11 524 beneficiaries (33.4%) experienced 1 or more buprenorphine treatment gaps. Treatment gap beneficiaries, compared with nontreatment gap beneficiaries, were more likely to be younger, be male, have a disability, and be Medicaid dual-eligible while less likely to be White, close to a buprenorphine prescriber, and treated with buprenorphine monotherapy (ie, buprenorphine hydrochloride). Beneficiaries were 2.89 (95% CI, 2.20-3.79) times more likely to experience an opioid overdose during buprenorphine treatment gap months compared with treated months. During treatment gap months, spending was $196.41 (95% CI, $110.53-$282.30) more than in treated months. Patients who continued to take buprenorphine dosages of greater than 8 mg/d and 16 mg/d were 2.61 and 2.84 times more likely to overdose in a treatment gap month, respectively, while patients taking buprenorphine dosages of 8 mg/d or less were 3.62 times more likely to overdose in a treatment gap month (maintenance of >16 mg/d: hazard ratio (HR), 2.64 [95% CI, 1.80-3.87]; maintenance of >8 mg/d: HR, 2.84 [95% CI, 2.13-3.78]; maintenance of ≤8 mg/d: HR, 3.62 [95% CI, 1.54-8.50]). Buprenorphine monotherapy was associated with greater risk of overdose and higher spending during treatment gaps months than buprenorphine/naloxone.Conclusions and relevanceMedicare patients treated with buprenorphine between 2010 and 2017 had a lower associated opioid overdose risk and spending during treatment months than treatment gap months.

Project description:ImportanceConcurrent benzodiazepine use is associated with an increased risk of opioid-related overdose; however, it remains unknown how the overdose risk varies with the days of exposure to both medications.ObjectiveTo evaluate the exposure-response association between the days with concurrent prescription opioid and benzodiazepine use and the risk of overdose.Design, setting, and participantsRetrospective cohort study of Medicare Part D claims data from January 1, 2013, to December 31, 2014. Analyses were conducted in fall 2017 and spring 2018. Participants were Medicare Part D beneficiaries who filled at least 1 prescription for an opioid in 2014.ExposuresPatients were divided into 2 groups based on whether they had opioid and benzodiazepine supplies on the day before overdose or censoring. The first group only had a supply of opioids (n?=?50?583) and the second group had supplies of opioids and benzodiazepines (n?=?20?665). The second group was further categorized into 4 subgroups based on the cumulative number of days with overlapping supplies of opioids and benzodiazepines: 1 to 90 days (n?=?3603), 91 to 180 days (n?=?2930), 181 to 270 days (n?=?4082), and 271 days or more (n?=?10?050).Main outcomes and measuresCox proportional hazard models were constructed to compare opioid-related overdose (including fatal and nonfatal overdoses) across time-dependent treatment groups, controlling for demographic characteristics, insurance factors, clinical characteristics, and number of unique opioid and benzodiazepine prescribers.ResultsOf 71?248 total participants, 25?600 (35.9%) were male and 59?532 (83.6%) were white. Mean (SD) age was 66.5 (14.8) years. On the day before overdose or censoring event, 20?665 of 71?248 patients with an opioid prescription (29.0%) were concurrently using benzodiazepines and 14?132 of 20?665 concurrent users (68.4%) had more than 180 days of overlapping supplies of both medications. The risk of overdose was highest on the first days of concurrent opioid and benzodiazepine use and decreased over time; compared with opioid use alone, the hazard ratio for overdose was 5.05 (95% CI, 3.68-6.93) during the first 90 days of concurrent opioid and benzodiazepine use and 1.87 (95% CI, 1.25-2.80) for days 91 to 180 among those who did not have an event before 90 days.Conclusions and relevanceDuring the first 90 days, concurrent benzodiazepine use is associated with a 5-fold increase in the risk of opioid-related overdose. The implementation of policies deterring concurrent opioid and benzodiazepine use is warranted. Patients using both medications should be closely monitored, particularly during the first days of concurrent use.

Project description:ImportancePrescription opioid misuse is a public health problem that leads to overdose. Although existing interventions focus on limiting prescribing to patients at high risk, individuals may still access prescription opioids dispensed to family members.ObjectiveTo determine whether opioid prescriptions to family members were associated with overdose for individuals who themselves did not have an opioid prescription.Design, setting, and participantsWe conducted a 1:4 matched case-control study using health care utilization data from 2004 through 2015 from a large US commercial insurance company. Eligible individuals were required to have at least 12 months of continuous enrollment and 1 or more family members in the database. Individuals who experienced overdose were identified by their first opioid overdose after the baseline period and matched to control participants by time in the database, calendar time, age, sex, and number of individuals in the family unit. Both groups were restricted to individuals with no prior opioid dispensing of their own. Data analysis was conducted from January 2018 to August 2018.ExposuresAny prior opioid dispensing to a family member, total morphine milligram equivalents dispensed to family members, and the type of opioid product dispensed.Main outcomes and measuresIndividual odds of opioid overdose resulting in an emergency department visit or hospitalization were the primary end point. The primary analysis evaluated the odds of overdose among individuals whose family members had been dispensed an opioid. Sensitivity analyses examined the odds stratified by age and timing relative to the dispensing of opioids to family members.ResultsA total of 2303 individuals who experienced opioid overdose and 9212 matched control individuals were identified. The mean (SD) age was 23.2 (18.1) years; 1158 affected individuals and 4632 control individuals (50.3%) were female. The mean (SD) time in the database before an overdose case was 3.2 (3.3) years. Prior opioid dispensing to family members was associated with individual overdose (odds ratio [OR], 2.89 [95% CI, 2.59-3.23]). There was a significant dose-response association between increasing amounts of opioids dispensed to family members and odds of overdose (>0-<50 morphine milligram equivalents per day: OR, 2.71 [95% CI, 2.42-3.03]; 50-<90 morphine milligram equivalents per day: OR, 7.80 [95% CI, 3.63-16.78]; ≥90 morphine milligram equivalents per day: OR, 15.08 [95% CI, 8.66-26.27]).Conclusions and relevanceIn this analysis, opioid prescriptions to family members were associated with overdose among individuals who do not receive opioid prescriptions. Interventions may focus on expanding access to opioid antagonists, locking prescription opioids in the home, and providing greater patient education to limit fatal overdose among family members.

Project description:ObjectiveTo develop and validate a machine-learning algorithm to improve prediction of incident OUD diagnosis among Medicare beneficiaries with ≥1 opioid prescriptions.MethodsThis prognostic study included 361,527 fee-for-service Medicare beneficiaries, without cancer, filling ≥1 opioid prescriptions from 2011-2016. We randomly divided beneficiaries into training, testing, and validation samples. We measured 269 potential predictors including socio-demographics, health status, patterns of opioid use, and provider-level and regional-level factors in 3-month periods, starting from three months before initiating opioids until development of OUD, loss of follow-up or end of 2016. The primary outcome was a recorded OUD diagnosis or initiating methadone or buprenorphine for OUD as proxy of incident OUD. We applied elastic net, random forests, gradient boosting machine, and deep neural network to predict OUD in the subsequent three months. We assessed prediction performance using C-statistics and other metrics (e.g., number needed to evaluate to identify an individual with OUD [NNE]). Beneficiaries were stratified into subgroups by risk-score decile.ResultsThe training (n = 120,474), testing (n = 120,556), and validation (n = 120,497) samples had similar characteristics (age ≥65 years = 81.1%; female = 61.3%; white = 83.5%; with disability eligibility = 25.5%; 1.5% had incident OUD). In the validation sample, the four approaches had similar prediction performances (C-statistic ranged from 0.874 to 0.882); elastic net required the fewest predictors (n = 48). Using the elastic net algorithm, individuals in the top decile of risk (15.8% [n = 19,047] of validation cohort) had a positive predictive value of 0.96%, negative predictive value of 99.7%, and NNE of 104. Nearly 70% of individuals with incident OUD were in the top two deciles (n = 37,078), having highest incident OUD (36 to 301 per 10,000 beneficiaries). Individuals in the bottom eight deciles (n = 83,419) had minimal incident OUD (3 to 28 per 10,000).ConclusionsMachine-learning algorithms improve risk prediction and risk stratification of incident OUD in Medicare beneficiaries.

Project description:BackgroundLittle is known about whether machine-learning algorithms developed to predict opioid overdose using earlier years and from a single state will perform as well when applied to other populations. We aimed to develop a machine-learning algorithm to predict 3-month risk of opioid overdose using Pennsylvania Medicaid data and externally validated it in two data sources (ie, later years of Pennsylvania Medicaid data and data from a different state).MethodsThis prognostic modelling study developed and validated a machine-learning algorithm to predict overdose in Medicaid beneficiaries with one or more opioid prescription in Pennsylvania and Arizona, USA. To predict risk of hospital or emergency department visits for overdose in the subsequent 3 months, we measured 284 potential predictors from pharmaceutical and health-care encounter claims data in 3-month periods, starting 3 months before the first opioid prescription and continuing until loss to follow-up or study end. We developed and internally validated a gradient-boosting machine algorithm to predict overdose using 2013-16 Pennsylvania Medicaid data (n=639 693). We externally validated the model using (1) 2017-18 Pennsylvania Medicaid data (n=318 585) and (2) 2015-17 Arizona Medicaid data (n=391 959). We reported several prediction performance metrics (eg, C-statistic, positive predictive value). Beneficiaries were stratified into risk-score subgroups to support clinical use.FindingsA total of 8641 (1·35%) 2013-16 Pennsylvania Medicaid beneficiaries, 2705 (0·85%) 2017-18 Pennsylvania Medicaid beneficiaries, and 2410 (0·61%) 2015-17 Arizona beneficiaries had one or more overdose during the study period. C-statistics for the algorithm predicting 3-month overdoses developed from the 2013-16 Pennsylvania training dataset and validated on the 2013-16 Pennsylvania internal validation dataset, 2017-18 Pennsylvania external validation dataset, and 2015-17 Arizona external validation dataset were 0·841 (95% CI 0·835-0·847), 0·828 (0·822-0·834), and 0·817 (0·807-0·826), respectively. In external validation datasets, 71 361 (22·4%) of 318 585 2017-18 Pennsylvania beneficiaries were in high-risk subgroups (positive predictive value of 0·38-4·08%; capturing 73% of overdoses in the subsequent 3 months) and 40 041 (10%) of 391 959 2015-17 Arizona beneficiaries were in high-risk subgroups (positive predictive value of 0·19-1·97%; capturing 55% of overdoses). Lower risk subgroups in both validation datasets had few individuals (≤0·2%) with an overdose.InterpretationA machine-learning algorithm predicting opioid overdose derived from Pennsylvania Medicaid data performed well in external validation with more recent Pennsylvania data and with Arizona Medicaid data. The algorithm might be valuable for overdose risk prediction and stratification in Medicaid beneficiaries.FundingNational Institute of Health, National Institute on Drug Abuse, National Institute on Aging.