Preventing dataset shift from breaking machine-learning biomarkers.
Ontology highlight
ABSTRACT:
SUBMITTER: Dockes J
PROVIDER: S-EPMC8478611 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:ObjectiveThe change in performance of machine learning models over time as a result of temporal dataset shift is a barrier to machine learning-derived models facilitating decision-making in clinical practice. Our aim was to describe technical procedures used to preserve the performance of machine learning models in the presence of temporal dataset shifts.MethodsStudies were included if they were fully published articles that used machine learning and implemented a procedure to mitigate the effects of temporal dataset shift in a clinical setting. We described how dataset shift was measured, the procedures used to preserve model performance, and their effects.ResultsOf 4,457 potentially relevant publications identified, 15 were included. The impact of temporal dataset shift was primarily quantified using changes, usually deterioration, in calibration or discrimination. Calibration deterioration was more common (n = 11) than discrimination deterioration (n = 3). Mitigation strategies were categorized as model level or feature level. Model-level approaches (n = 15) were more common than feature-level approaches (n = 2), with the most common approaches being model refitting (n = 12), probability calibration (n = 7), model updating (n = 6), and model selection (n = 6). In general, all mitigation strategies were successful at preserving calibration but not uniformly successful in preserving discrimination.ConclusionThere was limited research in preserving the performance of machine learning models in the presence of temporal dataset shift in clinical medicine. Future research could focus on the impact of dataset shift on clinical decision making, benchmark the mitigation strategies on a wider range of datasets and tasks, and identify optimal strategies for specific settings.
| S-EPMC8410238 | biostudies-literature
Project description:The use of machine learning for predicting ecotoxicological outcomes is promising, but underutilized. The curation of data with informative features requires both expertise in machine learning as well as a strong biological and ecotoxicological background, which we consider a barrier of entry for this kind of research. Additionally, model performances can only be compared across studies when the same dataset, cleaning, and splittings were used. Therefore, we provide ADORE, an extensive and well-described dataset on acute aquatic toxicity in three relevant taxonomic groups (fish, crustaceans, and algae). The core dataset describes ecotoxicological experiments and is expanded with phylogenetic and species-specific data on the species as well as chemical properties and molecular representations. Apart from challenging other researchers to try and achieve the best model performances across the whole dataset, we propose specific relevant challenges on subsets of the data and include datasets and splittings corresponding to each of these challenge as well as in-depth characterization and discussion of train-test splitting approaches.
| S-EPMC10584858 | biostudies-literature
Project description:The recent changes in policies in several countries regarding cannabis use has increased cannabis usage and research [1,2]. Cannabis is the second most used psychoactive substance word-wide [3]. Cannabis remains the subject of many research works. The cannabis can be classified into different classes according to their external features like colour, shape, and size using some computer vision and machine learning techniques. Precise classification or recognition is the unmet need of the agriculture business. This attracts many researchers to produce solutions with machine learning and deep learning techniques. Neat and clean dataset is the primary requirements to build accurate and robust machine learning model and minimize misclassification for the real-time environment. To achieve this objective, we have created an image dataset of cannabis seed. Accordingly, we have considered seventeen cannabis seeds to create dataset. The dataset contains 17 subfolders of cannabis seeds and folder is named with the category of seed. We strongly believe the cannabis seeds dataset will be very helpful for training, testing, and validation of cannabis classification or recognition with machine learning models.
| S-EPMC9925849 | biostudies-literature
Project description:Dry fruits are convenient and nutritious snacks that can provide numerous health benefits. They are packed with vitamins, minerals, and fibres, which can help improve overall health, lower cholesterol levels, and reduce the risk of heart disease. Due to their health benefits, dry fruits are an essential part of a healthy diet. In addition to health advantage, dry fruits have high commercial worth. The value of the global dry fruit market is estimated to be USD 6.2 billion in 2021 and USD 7.7 billion by 2028. The appearance of dry fruits is utilized for assessing their quality to a great extent, requiring neat, appropriately tagged, and high-quality images. Hence, this dataset is a valuable resource for the classification and recognition of dry fruits. With over 11500+ high-quality processed images representing 12 distinct classes, this dataset is a comprehensive collection of different varieties of dry fruits. The four dry fruits included in this dataset are Almonds, Cashew Nuts, Raisins, and Dried Figs (Anjeer), along with three subtypes of each. This makes it a total of 12 distinct classes of dry fruits, each with its unique features, shape, and size. The dataset will be useful for building machine learning models that can classify and recognize different types of dry fruits under different conditions, and can also be beneficial for dry fruit research, education, and medicinal purposes. Due to their nutritional value and health advantages, dry fruits have been consumed for a very long time. One of the best strategies to improve general health is to include dry fruits in the diet.
| S-EPMC10333426 | biostudies-literature
Project description:AIM:To review how machine learning (ML) is applied to imaging biomarkers in neuro-oncology, in particular for diagnosis, prognosis, and treatment response monitoring. MATERIALS AND METHODS:The PubMed and MEDLINE databases were searched for articles published before September 2018 using relevant search terms. The search strategy focused on articles applying ML to high-grade glioma biomarkers for treatment response monitoring, prognosis, and prediction. RESULTS:Magnetic resonance imaging (MRI) is typically used throughout the patient pathway because routine structural imaging provides detailed anatomical and pathological information and advanced techniques provide additional physiological detail. Using carefully chosen image features, ML is frequently used to allow accurate classification in a variety of scenarios. Rather than being chosen by human selection, ML also enables image features to be identified by an algorithm. Much research is applied to determining molecular profiles, histological tumour grade, and prognosis using MRI images acquired at the time that patients first present with a brain tumour. Differentiating a treatment response from a post-treatment-related effect using imaging is clinically important and also an area of active study (described here in one of two Special Issue publications dedicated to the application of ML in glioma imaging). CONCLUSION:Although pioneering, most of the evidence is of a low level, having been obtained retrospectively and in single centres. Studies applying ML to build neuro-oncology monitoring biomarker models have yet to show an overall advantage over those using traditional statistical methods. Development and validation of ML models applied to neuro-oncology require large, well-annotated datasets, and therefore multidisciplinary and multi-centre collaborations are necessary.
| S-EPMC6927796 | biostudies-literature
Project description:Batteries are crucial for building a clean and sustainable society, and their performance is highly affected by aging status. Reliable battery health assessment, however, is currently restrained by limited access to sufficient aging data, resulting from not only complicated battery operations but also long aging experimental time (several months or even years). Refining industrial datasets (e.g., the field data from electric vehicle batteries) unlocks opportunities to acquire large-volume aging data with low experimental efforts. We introduce the potential of combining industrial data with accelerated aging tests to recover high-quality battery aging datasets, through a migration-based machine learning. A comprehensive dataset containing 8,947 aging cycles with 15 operational modes is collected for evaluation. While saving up to 90% experimental time, aging data can be recovered with ultra-low error within 1%. It provides an alternative solution to significantly improve data shortage issues for assessment of battery and energy storage aging.
| S-EPMC8369168 | biostudies-literature
Project description:Damage detection based on modal parameter changes has become popular in the last few decades. Nowadays, there are robust and reliable mathematical relations available to predict natural frequency changes if damage parameters are known. Using these relations, it is possible to create databases containing a large variety of damage scenarios. Damage can be thus assessed by applying an inverse method. The problem is the complexity of the database, especially for structures with more cracks. In this paper, we propose two machine learning methods, namely the random forest (RF), and the artificial neural network (ANN), as search tools. The databases we developed contain damage scenarios for a prismatic cantilever beam with one crack and ideal and non-ideal boundary conditions. The crack assessment was made in two steps. First, a coarse damage location was found from the networks trained for scenarios comprising the whole beam. Afterwards, the assessment was made involving a particular network trained for the segment of the beam on which the crack was previously found. Using the two machine learning methods, we succeeded in estimating the crack location and severity with high accuracy for both simulation and laboratory experiments. Regarding the location of the crack, which was the main goal of the practitioners, the errors were less than 0.6%. Based on these achievements, we concluded that the damage assessment we propose, in conjunction with the machine learning methods, is robust and reliable.
| S-EPMC8839218 | biostudies-literature
Project description:Machine Learning has been on the rise and healthcare is no exception to that. In healthcare, mental health is gaining more and more space. The diagnosis of mental disorders is based upon standardized patient interviews with defined set of questions and scales which is a time consuming and costly process. Our objective was to apply the machine learning model and to evaluate to see if there is predictive power of biomarkers data to enhance the diagnosis of depression cases. In this research paper, we aimed to explore the detection of depression cases among the sample of 11,081 Dutch citizen dataset. Most of the earlier studies have balanced datasets wherein the proportion of healthy cases and unhealthy cases are equal but in our study, the dataset contains only 570 cases of self-reported depression out of 11,081 cases hence it is a class imbalance classification problem. The machine learning model built on imbalance dataset gives predictions biased toward majority class hence the model will always predict the case as no depression case even if it is a case of depression. We used different resampling strategies to address the class imbalance problem. We created multiple samples by under sampling, over sampling, over-under sampling and ROSE sampling techniques to balance the dataset and then, we applied machine learning algorithm "Extreme Gradient Boosting" (XGBoost) on each sample to classify the mental illness cases from healthy cases. The balanced accuracy, precision, recall and F1 score obtained from over-sampling and over-under sampling were more than 0.90.
| S-EPMC7931945 | biostudies-literature
Project description:Radiomics extracts and mines large number of medical imaging features quantifying tumor phenotypic characteristics. Highly accurate and reliable machine-learning approaches can drive the success of radiomic applications in clinical care. In this radiomic study, fourteen feature selection methods and twelve classification methods were examined in terms of their performance and stability for predicting overall survival. A total of 440 radiomic features were extracted from pre-treatment computed tomography (CT) images of 464 lung cancer patients. To ensure the unbiased evaluation of different machine-learning methods, publicly available implementations along with reported parameter configurations were used. Furthermore, we used two independent radiomic cohorts for training (n = 310 patients) and validation (n = 154 patients). We identified that Wilcoxon test based feature selection method WLCX (stability = 0.84 ± 0.05, AUC = 0.65 ± 0.02) and a classification method random forest RF (RSD = 3.52%, AUC = 0.66 ± 0.03) had highest prognostic performance with high stability against data perturbation. Our variability analysis indicated that the choice of classification method is the most dominant source of performance variation (34.21% of total variance). Identification of optimal machine-learning methods for radiomic applications is a crucial step towards stable and clinically relevant radiomic biomarkers, providing a non-invasive way of quantifying and monitoring tumor-phenotypic characteristics in clinical practice.
| S-EPMC4538374 | biostudies-literature
Project description:Sugarcane, a vital crop for the global sugar industry, is susceptible to various diseases that significantly impact its yield and quality. Accurate and timely disease detection is crucial for effective management and prevention strategies. We persent the "Sugarcane Leaf Dataset" consisting of 6748 high-resolution leaf images classified into nine disease categories, a healthy leaves category, and a dried leaves category. The dataset covers diseases such as smut, yellow leaf disease, pokkah boeng, mosale, grassy shoot, brown spot, brown rust, banded cholorsis, and sett rot. The dataset's potential for reuse is significant. The provided dataset serves as a valuable resource for researchers and practitioners interested in developing machine learning algorithms for disease detection and classification in sugarcane leaves. By leveraging this dataset, various machine learning techniques can be applied, including deep learning, feature extraction, and pattern recognition, to enhance the accuracy and efficiency of automated sugarcane disease identification systems. The open availability of this dataset encourages collaboration within the scientific community, expediting research on disease control strategies and improving sugarcane production. By leveraging the "Sugarcane Leaf Dataset," we can advance disease detection, monitoring, and management in sugarcane cultivation, leading to enhanced agricultural practices and higher crop yields.
| S-EPMC10964057 | biostudies-literature