Performance evaluation of a prescription medication image classification model: an observational cohort.
Ontology highlight
ABSTRACT:
SUBMITTER: Lester CA
PROVIDER: S-EPMC8316316 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:ObjectiveTo develop key performance indicators that evaluate the effectiveness of a prescription medication system.MethodsA modified RAND/UCLA appropriateness method was used to develop key performance indicators (KPIs) for a prescription medication system. A broad list of potential KPIs was compiled. A multidisciplinary group composed of 21 experts rated the potential KPIs. A face-to-face meeting was held following the first rating exercise to discuss each potential KPI individually. The expert panel undertook a final rating of KPIs. The final set of KPIs were those indicators where at least 80 percent of experts rated the indicator highly i.e. rating of ? 7 on a scale from 1 to 9.Results292 KPIs were identified from the published literature. After removing duplicates and combining similar indicators 71 KPIs were included. The final ranking resulted in six indicators being ranked 7 or higher by 80% of the respondents and an additional seven indicators being ranked 7 or higher by ?70 but ?80% of respondents. The six selected indicators include four specific disease areas, measure structural and process aspects of health service delivery, and assessed three of the domains of healthcare quality: efficiency, effectiveness, and safety.ConclusionsThese indicators are recommended as a starting point to assess the current performance of prescription medication systems. Consideration should be given to developing indicators in additional disease areas as well as indicators that measure the domains of timeliness and patient-centeredness. Future work should focus on the feasibility of measuring these indicators.
| S-EPMC6333341 | biostudies-literature
Project description:In medical image classification tasks, it is common to find that the number of normal samples far exceeds the number of abnormal samples. In such class-imbalanced situations, reliable training of deep neural networks continues to be a major challenge, therefore biasing the predicted class probabilities toward the majority class. Calibration has been proposed to alleviate some of these effects. However, there is insufficient analysis explaining whether and when calibrating a model would be beneficial. In this study, we perform a systematic analysis of the effect of model calibration on its performance on two medical image modalities, namely, chest X-rays and fundus images, using various deep learning classifier backbones. For this, we study the following variations: (i) the degree of imbalances in the dataset used for training; (ii) calibration methods; and (iii) two classification thresholds, namely, default threshold of 0.5, and optimal threshold from precision-recall (PR) curves. Our results indicate that at the default classification threshold of 0.5, the performance achieved through calibration is significantly superior (p < 0.05) to using uncalibrated probabilities. However, at the PR-guided threshold, these gains are not significantly different (p > 0.05). This observation holds for both image modalities and at varying degrees of imbalance. The code is available at https://github.com/sivaramakrishnan-rajaraman/Model_calibration.
| S-EPMC8794113 | biostudies-literature
Project description:The aim of this study was to examine the prescribing patterns of medications quantified by the performance measures for acute myocardial infarction (AMI).Current performance measures for AMI are designed to improve quality by quantifying the use of evidence-based treatments. However, these measures only assess medication prescription. Whether patients receive optimal dosing of secondary prevention medications at the time of and after discharge after AMI is unknown.We assessed treatment doses of beta-blockers, statins, and angiotensin-converting enzyme inhibitors (ACEI)/angiotensin II receptor blockers (ARBs) at discharge and 12 months after AMI among 6,748 patients from 31 hospitals enrolled in 2 U.S. registries (2003 to 2008). Prescribed doses were categorized as none, low (<50% target [defined from seminal clinical trials]), moderate (50% to 74% target), or goal (? 75% target). Patients with contraindications were excluded from analyses for that medication.Most eligible patients (>87%) were prescribed some dose of each medication at discharge, although only 1 in 3 patients were prescribed these medications at goal doses. Of patients not discharged on goal doses, up-titration during follow-up occurred infrequently (approximately 25% of patients for each medication). At 12 months, goal doses of beta-blockers, statins, and ACEI/ARBs were achieved in only 12%, 26%, and 32% of eligible patients, respectively. After multivariable adjustment, prescription of goal dose at discharge was strongly associated with being at goal dose at follow-up: beta-blockers, adjusted odds ratio (OR): 6.08 (95% confidence interval [CI]: 3.70 to 10.01); statins, adjusted OR: 8.22 (95% CI: 6.20 to 10.90); ACEI/ARBs, adjusted OR: 5.80 (95% CI: 2.56 to 13.16); p < 0.001 for each.Although nearly all patients after an AMI are discharged on appropriate secondary prevention medications, dose increases occur infrequently, and most patients are prescribed doses below those with proven efficacy in clinical trials. Integration of dose intensity into performance measures might help improve the use of optimal medical therapy after AMI.
| S-EPMC3819453 | biostudies-literature
Project description:Raster image correlation spectroscopy (RICS) is a fluorescence image analysis method for extracting the mobility, concentration, and stoichiometry of diffusing fluorescent molecules from confocal image stacks. The method works by calculating a spatial correlation function for each image and analyzing the average of those by model fitting. Rules of thumb exist for RICS image acquisitioning, yet a rigorous theoretical approach to predict the accuracy and precision of the recovered parameters has been lacking. We outline explicit expressions to reveal the dependence of RICS results on experimental parameters. In terms of imaging settings, we observed that a twofold decrease of the pixel size, e.g., from 100 to 50 nm, decreases the error on the translational diffusion constant (D) between three- and fivefold. For D = 1 ?m2 s-1, a typical value for intracellular measurements, ?25-fold lower mean-squared relative error was obtained when the optimal scan speed was used, although more drastic improvements were observed for other values of D. We proposed a slightly modified RICS calculation that allows correcting for the significant bias of the autocorrelation function at small (?50 × 50 pixels) sizes of the region of interest. In terms of sample properties, at molecular brightness E = 100 kHz and higher, RICS data quality was sufficient using as little as 20 images, whereas the optimal number of frames for lower E scaled pro rata. RICS data quality was constant over the nM-?M concentration range. We developed a bootstrap-based confidence interval of D that outperformed the classical least-squares approach in terms of coverage probability of the true value of D. We validated the theory via in vitro experiments of enhanced green fluorescent protein at different buffer viscosities. Finally, we outline robust practical guidelines and provide free software to simulate the parameter effects on recovery of the diffusion coefficient.
| S-EPMC7018992 | biostudies-literature
Project description:ObjectiveDelirium is a common condition associated with increased morbidity and mortality. Medication side effects are a possible source of modifiable delirium risk and provide an opportunity to improve delirium predictive models. This study characterized the risk for delirium diagnosis by applying a previously validated algorithm for calculating central nervous system adverse effect burden arising from a full medication list.MethodUsing a cohort of hospitalized adult (age 18-65) patients from the Massachusetts All-Payers Claims Database, we calculated medication burden following hospital discharge and characterized risk of new coded delirium diagnosis over the following 90 days. We applied the resulting model to a held-out test cohort.ResultsThe cohort included 62,180 individuals of whom 1.6% (1019) went on to have a coded delirium diagnosis. In the training cohort (43,527 individuals), the medication burden feature was positively associated with delirium diagnosis (OR = 5.75, 95% CI 4.34-7.63) and this association persisted (aOR = 1.95; 1.31-2.92) after adjusting for demographics, clinical features, prescribed medications, and anticholinergic risk score. In the test cohort, the trained model produced an area under the curve of 0.80 (0.78-0.82). This performance was similar across subgroups of age and gender.ConclusionAggregating brain-related medication adverse effects facilitates identification of individuals at high risk of subsequent delirium diagnosis.
| S-EPMC8249339 | biostudies-literature
Project description:BackgroundPrescription medication (PM) misuse/abuse has emerged as a national crisis in the United States, and social media has been suggested as a potential resource for performing active monitoring. However, automating a social media-based monitoring system is challenging-requiring advanced natural language processing (NLP) and machine learning methods. In this paper, we describe the development and evaluation of automatic text classification models for detecting self-reports of PM abuse from Twitter.MethodsWe experimented with state-of-the-art bi-directional transformer-based language models, which utilize tweet-level representations that enable transfer learning (e.g., BERT, RoBERTa, XLNet, AlBERT, and DistilBERT), proposed fusion-based approaches, and compared the developed models with several traditional machine learning, including deep learning, approaches. Using a public dataset, we evaluated the performances of the classifiers on their abilities to classify the non-majority "abuse/misuse" class.ResultsOur proposed fusion-based model performs significantly better than the best traditional model (F1-score [95% CI]: 0.67 [0.64-0.69] vs. 0.45 [0.42-0.48]). We illustrate, via experimentation using varying training set sizes, that the transformer-based models are more stable and require less annotated data compared to the other models. The significant improvements achieved by our best-performing classification model over past approaches makes it suitable for automated continuous monitoring of nonmedical PM use from Twitter.ConclusionsBERT, BERT-like and fusion-based models outperform traditional machine learning and deep learning models, achieving substantial improvements over many years of past research on the topic of prescription medication misuse/abuse classification from social media, which had been shown to be a complex task due to the unique ways in which information about nonmedical use is presented. Several challenges associated with the lack of context and the nature of social media language need to be overcome to further improve BERT and BERT-like models. These experimental driven challenges are represented as potential future research directions.
| S-EPMC7835447 | biostudies-literature
Project description:Classification methods used in microarray studies for gene expression are diverse in the way they deal with the underlying complexity of the data, as well as in the technique used to build the classification model. The MAQC II study on cancer classification problems has found that performance was affected by factors such as the classification algorithm, cross validation method, number of genes, and gene selection method. In this paper, we study the hypothesis that the disease under study significantly determines which method is optimal, and that additionally sample size, class imbalance, type of medical question (diagnostic, prognostic or treatment response), and microarray platform are potentially influential. A systematic literature review was used to extract the information from 48 published articles on non-cancer microarray classification studies. The impact of the various factors on the reported classification accuracy was analyzed through random-intercept logistic regression. The type of medical question and method of cross validation dominated the explained variation in accuracy among studies, followed by disease category and microarray platform. In total, 42% of the between study variation was explained by all the study specific and problem specific factors that we studied together.
| S-EPMC4000205 | biostudies-literature
Project description:Fingerprint recognition plays an important role in many commercial applications and is used by millions of people every day, e.g. for unlocking mobile phones. Fingerprint image segmentation is typically the first processing step of most fingerprint algorithms and it divides an image into foreground, the region of interest, and background. Two types of error can occur during this step which both have a negative impact on the recognition performance: 'true' foreground can be labeled as background and features like minutiae can be lost, or conversely 'true' background can be misclassified as foreground and spurious features can be introduced. The contribution of this paper is threefold: firstly, we propose a novel factorized directional bandpass (FDB) segmentation method for texture extraction based on the directional Hilbert transform of a Butterworth bandpass (DHBB) filter interwoven with soft-thresholding. Secondly, we provide a manually marked ground truth segmentation for 10560 images as an evaluation benchmark. Thirdly, we conduct a systematic performance comparison between the FDB method and four of the most often cited fingerprint segmentation algorithms showing that the FDB segmentation method clearly outperforms these four widely used methods. The benchmark and the implementation of the FDB method are made publicly available.
| S-EPMC4865205 | biostudies-literature
Project description:Prior image constrained compressed sensing (PICCS) is an image reconstruction framework which incorporates an often available prior image into the compressed sensing objective function. The images are reconstructed using an optimization procedure. In this paper, several alternative unconstrained minimization methods are used to implement PICCS. The purpose is to study and compare the performance of each implementation, as well as to evaluate the performance of the PICCS objective function with respect to image quality.Six different minimization methods are investigated with respect to convergence speed and reconstruction accuracy. These minimization methods include the steepest descent (SD) method and the conjugate gradient (CG) method. These algorithms require a line search to be performed. Thus, for each minimization algorithm, two line searching algorithms are evaluated: a backtracking (BT) line search and a fast Newton-Raphson (NR) line search. The relative root mean square error is used to evaluate the reconstruction accuracy. The algorithm that offers the best convergence speed is used to study the performance of PICCS with respect to the prior image parameter α and the data consistency parameter λ. PICCS is studied in terms of reconstruction accuracy, low-contrast spatial resolution, and noise characteristics. A numerical phantom was simulated and an animal model was scanned using a multirow detector computed tomography (CT) scanner to yield the projection datasets used in this study.For λ within a broad range, the CG method with Fletcher-Reeves formula and NR line search offers the fastest convergence for an equal level of reconstruction accuracy. Using this minimization method, the reconstruction accuracy of PICCS was studied with respect to variations in α and λ. When the number of view angles is varied between 107, 80, 64, 40, 20, and 16, the relative root mean square error reaches a minimum value for α ≈ 0.5. For values of α near the optimal value, the spatial resolution of the reconstructed image remains relatively constant and the noise texture is very similar to that of the prior image, which was reconstructed using the filtered backprojection (FBP) algorithm.Regarding the performance of the minimization methods, the nonlinear CG method with NR line search yields the best convergence speed. Regarding the performance of the PICCS image reconstruction, three main conclusions can be reached. (1) The performance of PICCS is optimal when the weighting parameter of the prior image parameter is selected to be near α = 0.5. (2) The spatial resolution measured for static objects in images reconstructed using PICCS from undersampled datasets is not degraded with respect to the fully-sampled reconstruction for α near its optimal value. (3) The noise texture of PICCS reconstructions is similar to that of the prior image, which was reconstructed using the conventional FBP method.
| S-EPMC3257752 | biostudies-other
Project description:Maximum likelihood estimation of parameters involving mixture model is known to have significant and specific patterns of errors. Population pharmacokinetic (PopPK) modeling using NONMEM is no exception. A few relevant studies on estimation and classification performance were done, but a comprehensive study was not yet available. The current study aims to evaluate performance and likelihood ratio test (LRT)-based true covariate detection rate when fitting a bimodal mixture of drug clearance (CL) in NONMEM. A large number of PopPK datasets with various settings were simulated and then estimated. The estimates were compared to the simulated values and summarized. The separation between the CL distributions of the two subpopulations is systematically overestimated. The major factor associated with the performance is the change in the minimum objective function value after removing the mixture component (dOFV). Other significant factors include estimated disparity index (DI), estimated mixing proportion, and number of subjects in the dataset. Small dOFV and large estimated DI are associated with the worst performance. Omitting a true mixture resulted in reduced true covariate detection rates. It is recommended that on top of routinely generated standard errors and model diagnostics, dOFV, and other factors when necessary, should be taken into account for the evaluation of performance when fitting mixture model using NONMEM. In addition, when fitting mixture model for CL is intended, the mixture component should be introduced prior to LRT-based covariate model development for CL.
| S-EPMC8674007 | biostudies-literature