Browse
Submit Data
Databases
API
Help

Dataset Information

8 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Understanding Deep Learning Performance through an Examination of Test Set Difficulty: A Psychometric Case Study.

ABSTRACT: Interpreting the performance of deep learning models beyond test set accuracy is challenging. Characteristics of individual data points are often not considered during evaluation, and each data point is treated equally. We examine the impact of a test set question's difficulty to determine if there is a relationship between difficulty and performance. We model difficulty using well-studied psychometric methods on human response patterns. Experiments on Natural Language Inference (NLI) and Sentiment Analysis (SA) show that the likelihood of answering a question correctly is impacted by the question's difficulty. As DNNs are trained with more data, easy examples are learned more quickly than hard examples.

SUBMITTER: Lalor JP

PROVIDER: S-EPMC7685075 | biostudies-literature | 2018 Oct-Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

Understanding Deep Learning Performance through an Examination of Test Set Difficulty: A Psychometric Case Study.

Lalor John P JP Wu Hao H Munkhdalai Tsendsuren T Yu Hong H

Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing 20181001

Interpreting the performance of deep learning models beyond test set accuracy is challenging. Characteristics of individual data points are often not considered during evaluation, and each data point is treated equally. We examine the impact of a test set question's difficulty to determine if there is a relationship between difficulty and performance. We model difficulty using well-studied psychometric methods on human response patterns. Experiments on Natural Language Inference (NLI) and Sentim ...[more]

PMID: 33241233

Similar Datasets

DeNovo Peptide Identification Deep Learning Test Set

Project description:A set of bottom-up proteomics data for testing the deep learning network trained with data in PXD010000

2022-09-25 | PXD010613 | Pride

DeNovo Peptide Identification Deep Learning Training Set

Project description:A benchmark set of bottom-up proteomics data for training deep learning networks. It has data from 51 organisms and includes nearly 1 million peptides.

2018-06-13 | PXD010000 | Pride

Paradigm Test Set

Project description:This SuperSeries is composed of the SubSeries listed below.

2012-06-21 | GSE35191 | GEO

Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice.

Project description:RATIONALE AND OBJECTIVES:Evidence is inconsistent about whether radiologists' interpretive performance on a screening mammography test set reflects their performance in clinical practice. This study aimed to estimate the correlation between test set and clinical performance and determine if the correlation is influenced by cancer prevalence or lesion difficulty in the test set. MATERIALS AND METHODS:This institutional review board-approved study randomized 83 radiologists from six Breast Cancer Surveillance Consortium registries to assess one of four test sets of 109 screening mammograms each; 48 radiologists completed a fifth test set of 110 mammograms 2 years later. Test sets differed in number of cancer cases and difficulty of lesion detection. Test set sensitivity and specificity were estimated using woman-level and breast-level recall with cancer status and expert opinion as gold standards. Clinical performance was estimated using women-level recall with cancer status as the gold standard. Spearman rank correlations between test set and clinical performance with 95% confidence intervals (CI) were estimated. RESULTS:For test sets with fewer cancers (N?=?15) that were more difficult to detect, correlations were weak to moderate for sensitivity (woman level?=?0.46, 95% CI?=?0.16, 0.69; breast level?=?0.35, 95% CI?=?0.03, 0.61) and weak for specificity (0.24, 95% CI?=?0.01, 0.45) relative to expert recall. Correlations for test sets with more cancers (N?=?30) were close to 0 and not statistically significant. CONCLUSIONS:Correlations between screening performance on a test set and performance in clinical practice are not strong. Test set performance more accurately reflects performance in clinical practice if cancer prevalence is low and lesions are challenging to detect.

| S-EPMC5591765 | biostudies-literature

OSD TEST SET

Project description:test case for eosc life marine gos project

| PRJEB50795 | ENA

Protein function in precision medicine: deep understanding with machine learning.

Project description:Precision medicine and personalized health efforts propose leveraging complex molecular, medical and family history, along with other types of personal data toward better life. We argue that this ambitious objective will require advanced and specialized machine learning solutions. Simply skimming some low-hanging results off the data wealth might have limited potential. Instead, we need to better understand all parts of the system to define medically relevant causes and effects: how do particular sequence variants affect particular proteins and pathways? How do these effects, in turn, cause the health or disease-related phenotype? Toward this end, deeper understanding will not simply diffuse from deeper machine learning, but from more explicit focus on understanding protein function, context-specific protein interaction networks, and impact of variation on both.

| S-EPMC5937700 | biostudies-literature

DeNovo Peptide Identification Deep Learning Training Set

Project description:A benchmark set of bottom-up proteomics data for training deep learning networks. It has data from 51 organisms and includes nearly 1 million peptides.

2018-11-19 | MSV000083150 | MassIVE

Pyrene, a Test Case for Deep-Ultraviolet Molecular Photophysics.

Project description:We determined the complete relaxation dynamics of pyrene in ethanol from the second bright state, employing experimental and theoretical broadband heterodyne detected transient grating and two-dimensional photon echo (2DPE) spectroscopy, using pulses with duration of 6 fs and covering a spectral range spanning from 250 to 300 nm. Multiple lifetimes are assigned to conical intersections through a cascade of electronic states, eventually leading to a rapid population of the lowest long-living excited state and subsequent slow vibrational cooling. The lineshapes in the 2DPE spectra indicate that the efficiency of the population transfer depends on the kinetic energy deposited into modes required to reach a sloped conical intersection, which mediates the decay to the lowest electronic state. The presented experimental-theoretical protocol paves the way for studies on deep-ultraviolet-absorbing biochromophores ubiquitous in genomic and proteic systems.

| S-EPMC6774270 | biostudies-literature

Brain size affects performance in a reversal-learning test.

Project description:It has become increasingly clear that a larger brain can confer cognitive benefits. Yet not all of the numerous aspects of cognition seem to be affected by brain size. Recent evidence suggests that some more basic forms of cognition, for instance colour vision, are not influenced by brain size. We therefore hypothesize that a larger brain is especially beneficial for distinct and gradually more complex aspects of cognition. To test this hypothesis, we assessed the performance of brain size selected female guppies (Poecilia reticulata) in two distinct aspects of cognition that differ in cognitive complexity. In a standard reversal-learning test we first investigated basic learning ability with a colour discrimination test, then reversed the reward contingency to specifically test for cognitive flexibility. We found that large-brained females outperformed small-brained females in the reversed-learning part of the test but not in the colour discrimination part of the test. Large-brained individuals are hence cognitively more flexible, which probably yields fitness benefits, as they may adapt more quickly to social and/or ecological cognitive challenges. Our results also suggest that a larger brain becomes especially advantageous with increasing cognitive complexity. These findings corroborate the significance of brain size for cognitive evolution.

| S-EPMC5805926 | biostudies-literature

A Gastrointestinal Endoscopy Quality Control System Incorporated With Deep Learning Improved Endoscopist Performance in a Pretest and Post-Test Trial.

Project description:IntroductionGastrointestinal endoscopic quality is operator-dependent. To ensure the endoscopy quality, we constructed an endoscopic audit and feedback system named Endo.Adm and evaluated its effect in a form of pretest and posttest trial.MethodsEndo.Adm system was developed using Python and Deep Convolutional Neural Ne2rk models. Sixteen endoscopists were recruited from Renmin Hospital of Wuhan University and were randomly assigned to undergo feedback of Endo.Adm or not (8 for the feedback group and 8 for the control group). The feedback group received weekly quality report cards which were automatically generated by Endo.Adm. We then compared the adenoma detection rate (ADR) and gastric precancerous conditions detection rate between baseline and postintervention phase for endoscopists in each group to evaluate the impact of Endo.Adm feedback. In total, 1,191 colonoscopies and 3,515 gastroscopies were included for analysis.ResultsADR was increased after Endo.Adm feedback (10.8%-20.3%, P < 0.01, <odds ratio (OR) 2.13, 95% confidence interval (CI) 1.317-3.447), and endoscopists' ADR without feedback remained nearly unchanged (10.8%-10.9%, P = 0.57, OR 1.086, 95% CI 0.814-1.447). Gastric precancerous conditions detection rate increased in the feedback group (3%-7%, P < 0.01, OR 1.866, 95% CI 1.399-2.489) while no improvement was observed in the control group (3.9%-3.5%, P = 0.489, OR 0.856, 95% CI 0.550-1.332).DiscussionEndo.Adm feedback contributed to multifaceted gastrointestinal endoscopic quality improvement. This system is practical to implement and may serve as a standard model for quality improvement in routine work (http://www.chictr.org.cn/, ChiCTR1900024153).

| S-EPMC8208417 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data