Hands-on training about overfitting.
Ontology highlight
ABSTRACT:
SUBMITTER: Demsar J
PROVIDER: S-EPMC7932115 | biostudies-literature | 2021 Mar
REPOSITORIES: biostudies-literature
Publications
Hands-on training about overfitting.
PLoS computational biology 20210304 3
Overfitting is one of the critical problems in developing models by machine learning. With machine learning becoming an essential technology in computational biology, we must include training about overfitting in all courses that introduce this technology to students and practitioners. We here propose a hands-on training for overfitting that is suitable for introductory level courses and can be carried out on its own or embedded within any data science course. We use workflow-based design of mac ...[more]
Similar Datasets
Project description:Patients after stroke with paretic or plegic hands require frequent exercises to promote neuroplasticity and to improve hand joint mobilization. Available devices for hand exercising are intended for persons with some level of hand control or provide continuous passive motion with limited patient involvement. Patients can benefit from self-exercising where they use the other hand to exercise the plegic or paretic one. However, post-stroke neuropsychological complications, apathy, and cognitive impairments such as forgetfulness make regular self-exercising difficult. This paper describes Przypominajka v2-a system intended to support self-exercising, remind about it, and motivate patients. We propose a glove-based device with an on-device machine-learning-based exercise scoring, a tablet-based interface, and a web-based application for therapists. The feasibility of on-device inference and the accuracy of correct exercise classification was evaluated on four healthy participants. Whole system use was described in a case study with a patient with a paretic hand. The anomaly classification has an accuracy of 91.3% and f1 value of 91.6% but achieves poorer results for new users (78% and 81%). The case study showed that patients had a positive reaction to exercising with Przypominajka, but there were issues relating to sensor glove: ease of putting on and clarity of instructions. The paper presents a new way in which sensor systems can support the rehabilitation of after-stroke patients with an on-device machine-learning-based classification that can accurately score and contribute to patient motivation.
| S-EPMC8955878 | biostudies-literature
Project description:The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy training data. Motivated by this phenomenon, we consider when a perfect fit to training data in linear regression is compatible with accurate prediction. We give a characterization of linear regression problems for which the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization is in terms of two notions of the effective rank of the data covariance. It shows that overparameterization is essential for benign overfitting in this setting: the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size. By studying examples of data covariance properties that this characterization shows are required for benign overfitting, we find an important role for finite-dimensional data: the accuracy of the minimum norm interpolating prediction rule approaches the best possible accuracy for a much narrower range of properties of the data distribution when the data lie in an infinite-dimensional space vs. when the data lie in a finite-dimensional space with dimension that grows faster than the sample size.
| S-EPMC7720150 | biostudies-literature
Project description:We present a dataset of 13446 shell commands from 175 participants who attended cybersecurity training and solved assignments in the Linux terminal. Each acquired data record contains a command with its arguments and metadata, such as a timestamp, working directory, and host identification in the emulated training infrastructure. The commands were captured in Bash, ZSH, and Metasploit shells. The data are stored as JSON records, enabling vast possibilities for their further use in research and development. These include educational data mining, learning analytics, student modeling, and evaluating machine learning models for intrusion detection. The data were collected from 27 cybersecurity training sessions using an open-source logging toolset and two open-source interactive learning environments. Researchers and developers may use the dataset or deploy the learning environments with the logging toolset to generate their own data in the same format. Moreover, we provide a set of common analytical queries to facilitate the exploratory analysis of the dataset.
| S-EPMC8479389 | biostudies-literature
Project description:Bioprocess engineering is a highly interdisciplinary field of study which is strongly benefited by practical courses where students can actively experience the interconnection between biology, engineering, and physical sciences. This work describes a lab course developed for 2nd year undergraduate students of bioprocess engineering and related disciplines, where students are challenged with a real-life bioprocess-engineering application, the production of recombinant protein in a fed-batch process. The lab course was designed to introduce students to the subject of operating and supervising an experiment in a bioreactor, along with the analysis of collected data and a final critical evaluation of the experiment. To provide visual feedback of the experimental outcome, the organism used during class was Escherichia coli which carried a plasmid to recombinantly produce enhanced green fluorescent protein (eGFP) upon induction. This can easily be visualized in both the bioreactor and samples by using ultraviolet light. The lab course is performed with bioreactors of the simplest design, and is therefore highly flexible, robust and easy to reproduce. As part of this work the implementation and framework, the results, the evaluation and assessment of student learning combined with opinion surveys are presented, which provides a basis for instructors intending to implement a similar lab course at their respective institution.
| S-EPMC6680437 | biostudies-literature
Project description:Traditional university teaching formats are of limited use when it comes to conveying the inner workings and specific remit of general medicine in a practical way. Small supervised groups present themselves as a plausible and effective alternative for learning to interact with patients in low prevalence areas.A frontal lecture was transformed into an interactive seminar-like lecture for 280 students. Short kick-off presentations served as an introduction to rotating circuit stations. Knowledge, skills and professional attitude specific to general and family medicine were intensively trained by 28 small groups in and around the auditorium by means of activating didactic methods. The small groups were supervised by experienced GP's trained as tutors. During six days, consisting of 3.5 hour sessions per day, working methods, a large variety of common medical conditions, preventive primary care and care for the elderly were amongst the topics addressed.This new format was successfully implemented and developed with regards to content as well as organisation. Well over 90% of the 274 participating students evaluated tutors' commitment, coherent delivery of content, learner-oriented atmosphere and practical focus positively. Given the advantages of a small-group learning situation, the somewhat cramped conditions were accepted by 92%.The innovative concept was able to work successfully within the specific framework of patient care in a general and family medicine setting. A creative approach to the lack of space managed to mobilise resources for practical small-group work. Being able to work on specific general practice problems in a small-group setting and immediately reflect upon them was rated positively overall.Responsibilities and specific working methods in general practice / family medicine can be trained successfully even with large groups of students when limited space is used creatively. In a supervised circuit-training setting, students are able to apply their existing knowledge and skills in a practical way. Further research is needed to assess individual learning success and gains in competence under this novel learning situation.
| S-EPMC4152991 | biostudies-other
Project description:BackgroundSerious games enrich simulation-based health care trainings and improve knowledge, skills, and self-confidence of learners while entertaining them.ObjectiveA platform which can combine performance data from a basic life support (BLS) serious game app and hands-on data based on the same scoring system is not available in the market. The aim of this study was to create such a platform and investigate whether performance evaluation of BLS trainings would be more objective compared to conventional Objective Structured Clinical Examination (OSCE) examinations if these evaluations were carried out with the platform which combines OSCE scoring criteria with sensor data retrieved from the simulator's sensors.MethodsParticipants were 25 volunteers (11 men [44.0%] and 14 [56.0] women) among Ac?badem Mehmet Ali Ayd?nlar University students without prior knowledge of the BLS protocol. A serious game module has been created for teaching learners the European Resuscitation Council Basic Life Support 2015 protocol. A second module called the hands-on module was designed for educators. This module includes a checklist used for BLS OSCE examinations and can retrieve sensor data such as compression depth, compression frequency, and ventilation volume from the manikin (CPR Lilly; 3B Scientific GmbH) via Bluetooth. Data retrieved from the sensors of the manikin enable educators to evaluate learners in a more objective way. Performance data retrieved from the serious gaming module have been combined with the results of the hands-on module. Data acquired from the hands-on module have also been compared with the results of conventional OSCE scores of the participants, which were obtained by watching the videos of the same trainings.ResultsParticipants were considered successful in the game if they scored 80/100 or above. Overall, participants scored 80 or above in an average of 1.4 (SD 0.65) trials. The average BLS serious game score was 88.3/100 (SD 5.17) and hands-on average score was 70.7/100 (SD 17.3), whereas the OSCE average score was 84.4/100 (SD 12.9). There was no statistically significant correlation between success on trials (score ?80/100), serious game, hands-on training app, and OSCE scores (Spearman rho test, P>.05). The mean BLS serious game score of the participants was 88.3/100 (SD 5.17), whereas their mean hands-on training app score was 70.7/100 (SD 17.3) and OSCE score was 84.4/100 (SD 12.9).ConclusionsAlthough scoring criteria for OSCE and hands-on training app were identical, OSCE scores were 17% higher than hands-on training app scores. After analyzing the difference of scores between hands-on training app and OSCE, it has been revealed that these differences originate from scoring parameters such as compression depth, compression frequency, and ventilation volume. These data suggest that evaluation of BLS trainings would be more objective if these evaluations were carried out with the modality, which combines visual OSCE scoring criteria with sensor data retrieved from the simulator's sensors.Trial registrationClinicalTrials.gov NCT04533893; https://clinicaltrials.gov/ct2/show/NCT04533893.
| S-EPMC7725648 | biostudies-literature
Project description:This paper proposes solutions to three issues pertaining to the estimation of finite mixture models with an unknown number of components: the non-identifiability induced by overfitting the number of components, the mixing limitations of standard Markov Chain Monte Carlo (MCMC) sampling techniques, and the related label switching problem. An overfitting approach is used to estimate the number of components in a finite mixture model via a Zmix algorithm. Zmix provides a bridge between multidimensional samplers and test based estimation methods, whereby priors are chosen to encourage extra groups to have weights approaching zero. MCMC sampling is made possible by the implementation of prior parallel tempering, an extension of parallel tempering. Zmix can accurately estimate the number of components, posterior parameter estimates and allocation probabilities given a sufficiently large sample size. The results will reflect uncertainty in the final model and will report the range of possible candidate models and their respective estimated probabilities from a single run. Label switching is resolved with a computationally light-weight method, Zswitch, developed for overfitted mixtures by exploiting the intuitiveness of allocation-based relabelling algorithms and the precision of label-invariant loss functions. Four simulation studies are included to illustrate Zmix and Zswitch, as well as three case studies from the literature. All methods are available as part of the R package Zmix, which can currently be applied to univariate Gaussian mixture models.
| S-EPMC4503697 | biostudies-literature
Project description:Knowledge on the relationship between different biological modalities (RNA, chromatin, etc.) can help further our understanding of the processes through which biological components interact. The ready availability of multi-omics datasets has led to the development of numerous methods for identifying sources of common variation across biological modalities. However, evaluation of the performance of these methods, in terms of consistency, has been difficult because most methods are unsupervised. We present a comparison of sparse multiple canonical correlation analysis (Sparse mCCA), angle-based joint and individual variation explained (AJIVE) and multi-omics factor analysis (MOFA) using a cross-validation approach to assess overfitting and consistency. Both large and small-sample datasets were used to evaluate performance, and a permuted null dataset was used to identify overfitting through the application of our framework and approach. In the large-sample setting, we found that all methods demonstrated consistency and lack of overfitting; however, in the small-sample size setting, AJIVE provided the most stable results. We provide an R package so that our framework and approach can be applied to evaluate other methods and datasets.
| S-EPMC7373174 | biostudies-literature
Project description:Structural biology methods have delivered over 150 000 high-resolution structures of macromolecules, which have fundamentally altered our understanding of biology and our approach to developing new medicines. However, the description of molecular flexibility is instrinsically flawed and in almost all cases, regardless of the experimental method used for structure determination, there remains a strong overfitting bias during molecular model building and refinement. In the worst case this can lead to wholly incorrect structures and thus incorrect biological interpretations. Here, by reparametrizing the description of these complex structures in terms of bonds rather than atomic positions, and by modelling flexibility using a deterministic ensemble of structures, it is demonstrated that structures can be described using fewer parameters than in conventional refinement. The current implementation, applied to X-ray diffraction data, significantly reduces the extent of overfitting, allowing the experimental data to reveal more biological information in electron-density maps.
| S-EPMC8025884 | biostudies-literature
Project description:BACKGROUND:Integration of transesophageal echocardiography (TEE) with Focused Cardiac Ultrasound (FoCUS) can impact decision-making, assist in the diagnosis of reversible etiologies and help guiding resuscitation of patients with cardiac arrest. OBJECTIVE:To evaluate the ability of emergency physicians (EPs) to obtain and maintain skills in performing resusTEE after a course with clinical training in the cardiac surgery theatre. METHODS:Ten EPs without previous TEE experience underwent a resusTEE course, based on a 2-h workshop and an 8-h hands-on training. The training was performed in a cardiac surgery theatre tutored by cardiovascular anesthesiologists. The six taught views were mid-esophageal four-chamber (ME4CH), mid-esophageal long axis (MELAX), mid-esophageal two-chamber (ME2CH), mid-esophageal bicaval view (MEbicaval), transgastric short axis (TGSAX) and aorta view (AOview). The EPs were evaluated by a cardiovascular anesthesiologist at the end of the course as well as after 12 weeks according to a standardized evaluation method. Once the course was completed, resusTEE exams, performed by EPs in Emergency Department (ED), were monitored for a 12-week period. RESULTS:The average assessment of the ten EPs by the tutors was higher than 4 points out of 5, both at the end of the course and after 12 weeks. Probe insertion, acquisition and interpretation of the different views scored on average more than 4 points out of 5 except for TGSAX that showed worsening after 12 weeks. Trainees performed twelve resusTEE exams in ED in patients with out-of-hospital cardiac arrest (OHCA) over 12 weeks after the course. EPs used only four out of six taught views in clinical practice, in the following order of frequency: ME4CH, AOview, MEbicaval and MELAX. CONCLUSIONS:EPs, after a course with clinical training in the cardiac surgery theatre, can successfully acquire and maintain the skills needed to perform resusTEE. However, among the six views learned in the course, EPs used only four of them (ME4CH, MEbicaval, MELAX and AOview).
| S-EPMC7431479 | biostudies-literature