Preventing dataset shift from breaking machine-learning biomarkers.
Ontology highlight
ABSTRACT:
SUBMITTER: Dockes J
PROVIDER: S-EPMC8478611 | biostudies-literature | 2021 Sep
REPOSITORIES: biostudies-literature
Publications
Preventing dataset shift from breaking machine-learning biomarkers.
Dockès Jérôme J Varoquaux Gaël G Poline Jean-Baptiste JB
GigaScience 20210901 9
Machine learning brings the hope of finding new biomarkers extracted from cohorts with rich biomedical measurements. A good biomarker is one that gives reliable detection of the corresponding condition. However, biomarkers are often extracted from a cohort that differs from the target population. Such a mismatch, known as a dataset shift, can undermine the application of the biomarker to new individuals. Dataset shifts are frequent in biomedical research, e.g., because of recruitment biases. Wh ...[more]
Similar Datasets
Project description:ObjectiveThe change in performance of machine learning models over time as a result of temporal dataset shift is a barrier to machine learning-derived models facilitating decision-making in clinical practice. Our aim was to describe technical procedures used to preserve the performance of machine learning models in the presence of temporal dataset shifts.MethodsStudies were included if they were fully published articles that used machine learning and implemented a procedure to mitigate the effects of temporal dataset shift in a clinical setting. We described how dataset shift was measured, the procedures used to preserve model performance, and their effects.ResultsOf 4,457 potentially relevant publications identified, 15 were included. The impact of temporal dataset shift was primarily quantified using changes, usually deterioration, in calibration or discrimination. Calibration deterioration was more common (n = 11) than discrimination deterioration (n = 3). Mitigation strategies were categorized as model level or feature level. Model-level approaches (n = 15) were more common than feature-level approaches (n = 2), with the most common approaches being model refitting (n = 12), probability calibration (n = 7), model updating (n = 6), and model selection (n = 6). In general, all mitigation strategies were successful at preserving calibration but not uniformly successful in preserving discrimination.ConclusionThere was limited research in preserving the performance of machine learning models in the presence of temporal dataset shift in clinical medicine. Future research could focus on the impact of dataset shift on clinical decision making, benchmark the mitigation strategies on a wider range of datasets and tasks, and identify optimal strategies for specific settings.
| S-EPMC8410238 | biostudies-literature
Project description:Dry fruits are convenient and nutritious snacks that can provide numerous health benefits. They are packed with vitamins, minerals, and fibres, which can help improve overall health, lower cholesterol levels, and reduce the risk of heart disease. Due to their health benefits, dry fruits are an essential part of a healthy diet. In addition to health advantage, dry fruits have high commercial worth. The value of the global dry fruit market is estimated to be USD 6.2 billion in 2021 and USD 7.7 billion by 2028. The appearance of dry fruits is utilized for assessing their quality to a great extent, requiring neat, appropriately tagged, and high-quality images. Hence, this dataset is a valuable resource for the classification and recognition of dry fruits. With over 11500+ high-quality processed images representing 12 distinct classes, this dataset is a comprehensive collection of different varieties of dry fruits. The four dry fruits included in this dataset are Almonds, Cashew Nuts, Raisins, and Dried Figs (Anjeer), along with three subtypes of each. This makes it a total of 12 distinct classes of dry fruits, each with its unique features, shape, and size. The dataset will be useful for building machine learning models that can classify and recognize different types of dry fruits under different conditions, and can also be beneficial for dry fruit research, education, and medicinal purposes. Due to their nutritional value and health advantages, dry fruits have been consumed for a very long time. One of the best strategies to improve general health is to include dry fruits in the diet.
| S-EPMC10333426 | biostudies-literature
Project description:The use of machine learning for predicting ecotoxicological outcomes is promising, but underutilized. The curation of data with informative features requires both expertise in machine learning as well as a strong biological and ecotoxicological background, which we consider a barrier of entry for this kind of research. Additionally, model performances can only be compared across studies when the same dataset, cleaning, and splittings were used. Therefore, we provide ADORE, an extensive and well-described dataset on acute aquatic toxicity in three relevant taxonomic groups (fish, crustaceans, and algae). The core dataset describes ecotoxicological experiments and is expanded with phylogenetic and species-specific data on the species as well as chemical properties and molecular representations. Apart from challenging other researchers to try and achieve the best model performances across the whole dataset, we propose specific relevant challenges on subsets of the data and include datasets and splittings corresponding to each of these challenge as well as in-depth characterization and discussion of train-test splitting approaches.
| S-EPMC10584858 | biostudies-literature
Project description:The recent changes in policies in several countries regarding cannabis use has increased cannabis usage and research [1,2]. Cannabis is the second most used psychoactive substance word-wide [3]. Cannabis remains the subject of many research works. The cannabis can be classified into different classes according to their external features like colour, shape, and size using some computer vision and machine learning techniques. Precise classification or recognition is the unmet need of the agriculture business. This attracts many researchers to produce solutions with machine learning and deep learning techniques. Neat and clean dataset is the primary requirements to build accurate and robust machine learning model and minimize misclassification for the real-time environment. To achieve this objective, we have created an image dataset of cannabis seed. Accordingly, we have considered seventeen cannabis seeds to create dataset. The dataset contains 17 subfolders of cannabis seeds and folder is named with the category of seed. We strongly believe the cannabis seeds dataset will be very helpful for training, testing, and validation of cannabis classification or recognition with machine learning models.
| S-EPMC9925849 | biostudies-literature
Project description:AimTo review how machine learning (ML) is applied to imaging biomarkers in neuro-oncology, in particular for diagnosis, prognosis, and treatment response monitoring.Materials and methodsThe PubMed and MEDLINE databases were searched for articles published before September 2018 using relevant search terms. The search strategy focused on articles applying ML to high-grade glioma biomarkers for treatment response monitoring, prognosis, and prediction.ResultsMagnetic resonance imaging (MRI) is typically used throughout the patient pathway because routine structural imaging provides detailed anatomical and pathological information and advanced techniques provide additional physiological detail. Using carefully chosen image features, ML is frequently used to allow accurate classification in a variety of scenarios. Rather than being chosen by human selection, ML also enables image features to be identified by an algorithm. Much research is applied to determining molecular profiles, histological tumour grade, and prognosis using MRI images acquired at the time that patients first present with a brain tumour. Differentiating a treatment response from a post-treatment-related effect using imaging is clinically important and also an area of active study (described here in one of two Special Issue publications dedicated to the application of ML in glioma imaging).ConclusionAlthough pioneering, most of the evidence is of a low level, having been obtained retrospectively and in single centres. Studies applying ML to build neuro-oncology monitoring biomarker models have yet to show an overall advantage over those using traditional statistical methods. Development and validation of ML models applied to neuro-oncology require large, well-annotated datasets, and therefore multidisciplinary and multi-centre collaborations are necessary.
| S-EPMC6927796 | biostudies-literature
Project description:Changes in the timing of plant phenological phases are important proxies in contemporary climate research. However, most of the commonly used traditional phenological observations do not give any coherent spatial information. While consistent spatial data can be obtained from airborne sensors and preprocessed gridded meteorological data, not many studies robustly benefit from these data sources. Therefore, the main aim of this study is to create and evaluate different statistical models for reconstructing, predicting, and improving quality of phenological phases monitoring with the use of satellite and meteorological products. A quality-controlled dataset of the 13 BBCH plant phenophases in Poland was collected for the period 2007-2014. For each phenophase, statistical models were built using the most commonly applied regression-based machine learning techniques, such as multiple linear regression, lasso, principal component regression, generalized boosted models, and random forest. The quality of the models was estimated using a k-fold cross-validation. The obtained results showed varying potential for coupling meteorological derived indices with remote sensing products in terms of phenological modeling; however, application of both data sources improves models' accuracy from 0.6 to 4.6 day in terms of obtained RMSE. It is shown that a robust prediction of early phenological phases is mostly related to meteorological indices, whereas for autumn phenophases, there is a stronger information signal provided by satellite-derived vegetation metrics. Choosing a specific set of predictors and applying a robust preprocessing procedures is more important for final results than the selection of a particular statistical model. The average RMSE for the best models of all phenophases is 6.3, while the individual RMSE vary seasonally from 3.5 to 10 days. Models give reliable proxy for ground observations with RMSE below 5 days for early spring and late spring phenophases. For other phenophases, RMSE are higher and rise up to 9-10 days in the case of the earliest spring phenophases.
| S-EPMC6028898 | biostudies-other
Project description:BackgroundSeafood is increasingly traded worldwide, but its supply chain is particularly prone to frauds. To increase consumer confidence, prevent illegal trade, and provide independent validation for eco-labelling, accurate tools for seafood traceability are needed. Here we show that the use of microbiome profiling (MP) coupled with machine learning (ML) allows precise tracing the origin of Manila clams harvested in areas separated by small geographic distances. The study was designed to represent a real-world scenario. Clams were collected in different seasons across the most important production area in Europe (lagoons along the northern Adriatic coast) to cover the known seasonal variation in microbiome composition for the species. DNA extracted from samples underwent the same depuration process as commercial products (i.e. at least 12 h in open flow systems).ResultsMachine learning-based analysis of microbiome profiles was carried out using two completely independent sets of data (collected at the same locations but in different years), one for training the algorithm, and the other for testing its accuracy and assessing the temporal stability signal. Briefly, gills (GI) and digestive gland (DG) of clams were collected in summer and winter over two different years (i.e. from 2018 to 2020) in one banned area and four farming sites. 16S DNA metabarcoding was performed on clam tissues and the obtained amplicon sequence variants (ASVs) table was used as input for ML MP. The best-predicting performances were obtained using the combined information of GI and DG (consensus analysis), showing a Cohen K-score > 0.95 when the target was the classification of samples collected from the banned area and those harvested at farming sites. Classification of the four different farming areas showed slightly lower accuracy with a 0.76 score.ConclusionsWe show here that MP coupled with ML is an effective tool to trace the origin of shellfish products. The tool is extremely robust against seasonal and inter-annual variability, as well as product depuration, and is ready for implementation in routine assessment to prevent the trade of illegally harvested or mislabeled shellfish.
| S-EPMC11389296 | biostudies-literature
Project description:Machine Learning has been on the rise and healthcare is no exception to that. In healthcare, mental health is gaining more and more space. The diagnosis of mental disorders is based upon standardized patient interviews with defined set of questions and scales which is a time consuming and costly process. Our objective was to apply the machine learning model and to evaluate to see if there is predictive power of biomarkers data to enhance the diagnosis of depression cases. In this research paper, we aimed to explore the detection of depression cases among the sample of 11,081 Dutch citizen dataset. Most of the earlier studies have balanced datasets wherein the proportion of healthy cases and unhealthy cases are equal but in our study, the dataset contains only 570 cases of self-reported depression out of 11,081 cases hence it is a class imbalance classification problem. The machine learning model built on imbalance dataset gives predictions biased toward majority class hence the model will always predict the case as no depression case even if it is a case of depression. We used different resampling strategies to address the class imbalance problem. We created multiple samples by under sampling, over sampling, over-under sampling and ROSE sampling techniques to balance the dataset and then, we applied machine learning algorithm "Extreme Gradient Boosting" (XGBoost) on each sample to classify the mental illness cases from healthy cases. The balanced accuracy, precision, recall and F1 score obtained from over-sampling and over-under sampling were more than 0.90.
| S-EPMC7931945 | biostudies-literature
Project description:The utilisation of smart devices, such as smartwatches and smartphones, in the field of movement disorders research has gained significant attention. However, the absence of a comprehensive dataset with movement data and clinical annotations, encompassing a wide range of movement disorders including Parkinson's disease (PD) and its differential diagnoses (DD), presents a significant gap. The availability of such a dataset is crucial for the development of reliable machine learning (ML) models on smart devices, enabling the detection of diseases and monitoring of treatment efficacy in a home-based setting. We conducted a three-year cross-sectional study at a large tertiary care hospital. A multi-modal smartphone app integrated electronic questionnaires and smartwatch measures during an interactive assessment designed by neurologists to provoke subtle changes in movement pathologies. We captured over 5000 clinical assessment steps from 504 participants, including PD, DD, and healthy controls (HC). After age-matching, an integrative ML approach combining classical signal processing and advanced deep learning techniques was implemented and cross-validated. The models achieved an average balanced accuracy of 91.16% in the classification PD vs. HC, while PD vs. DD scored 72.42%. The numbers suggest promising performance while distinguishing similar disorders remains challenging. The extensive annotations, including details on demographics, medical history, symptoms, and movement steps, provide a comprehensive database to ML techniques and encourage further investigations into phenotypical biomarkers related to movement disorders.
| S-EPMC10770131 | biostudies-literature
Project description:Batteries are crucial for building a clean and sustainable society, and their performance is highly affected by aging status. Reliable battery health assessment, however, is currently restrained by limited access to sufficient aging data, resulting from not only complicated battery operations but also long aging experimental time (several months or even years). Refining industrial datasets (e.g., the field data from electric vehicle batteries) unlocks opportunities to acquire large-volume aging data with low experimental efforts. We introduce the potential of combining industrial data with accelerated aging tests to recover high-quality battery aging datasets, through a migration-based machine learning. A comprehensive dataset containing 8,947 aging cycles with 15 operational modes is collected for evaluation. While saving up to 90% experimental time, aging data can be recovered with ultra-low error within 1%. It provides an alternative solution to significantly improve data shortage issues for assessment of battery and energy storage aging.
| S-EPMC8369168 | biostudies-literature