Dataset Information

Crowdsourcing as a novel technique for retinal fundus photography classification: analysis of images in the EPIC Norfolk cohort on behalf of the UK Biobank Eye and Vision Consortium.

ABSTRACT:

Aim

Crowdsourcing is the process of outsourcing numerous tasks to many untrained individuals. Our aim was to assess the performance and repeatability of crowdsourcing for the classification of retinal fundus photography.

Methods

One hundred retinal fundus photograph images with pre-determined disease criteria were selected by experts from a large cohort study. After reading brief instructions and an example classification, we requested that knowledge workers (KWs) from a crowdsourcing platform classified each image as normal or abnormal with grades of severity. Each image was classified 20 times by different KWs. Four study designs were examined to assess the effect of varying incentive and KW experience in classification accuracy. All study designs were conducted twice to examine repeatability. Performance was assessed by comparing the sensitivity, specificity and area under the receiver operating characteristic curve (AUC).

Results

Without restriction on eligible participants, two thousand classifications of 100 images were received in under 24 hours at minimal cost. In trial 1 all study designs had an AUC (95%CI) of 0.701(0.680-0.721) or greater for classification of normal/abnormal. In trial 1, the highest AUC (95%CI) for normal/abnormal classification was 0.757 (0.738-0.776) for KWs with moderate experience. Comparable results were observed in trial 2. In trial 1, between 64-86% of any abnormal image was correctly classified by over half of all KWs. In trial 2, this ranged between 74-97%. Sensitivity was ? 96% for normal versus severely abnormal detections across all trials. Sensitivity for normal versus mildly abnormal varied between 61-79% across trials.

Conclusions

With minimal training, crowdsourcing represents an accurate, rapid and cost-effective method of retinal image analysis which demonstrates good repeatability. Larger studies with more comprehensive participant training are needed to explore the utility of this compelling technique in large scale medical image analysis.

SUBMITTER: Mitry D

PROVIDER: S-EPMC3749186 | biostudies-literature | 2013

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Crowdsourcing as a novel technique for retinal fundus photography classification: analysis of images in the EPIC Norfolk cohort on behalf of the UK Biobank Eye and Vision Consortium.

Mitry Danny D Peto Tunde T Hayat Shabina S Morgan James E JE Khaw Kay-Tee KT Foster Paul J PJ

PloS one 20130821 8

<h4>Aim</h4>Crowdsourcing is the process of outsourcing numerous tasks to many untrained individuals. Our aim was to assess the performance and repeatability of crowdsourcing for the classification of retinal fundus photography.<h4>Methods</h4>One hundred retinal fundus photograph images with pre-determined disease criteria were selected by experts from a large cohort study. After reading brief instructions and an example classification, we requested that knowledge workers (KWs) from a crowdsour ...[more]

PMID: 23990935

Similar Datasets

Project description:BackgroundHeavy drinkers of alcohol are reported to use hospitals more than non-drinkers, but it is unclear whether light-to-moderate drinkers use hospitals more than non-drinkers.ObjectiveWe examined the relationship between alcohol consumption in 10,883 men and 12,857 women aged 40-79 years in the general population and subsequent admissions to hospital and time spent in hospital.MethodsParticipants from the EPIC-Norfolk prospective population-based study were followed for ten years (1999-2009) using record linkage.ResultsCompared to current non-drinkers, men who reported any alcohol drinking had a lower risk of spending more than twenty days in hospital multivariable adjusted OR 0.80 (95%CI 0.68-0.94) after adjusting for age, smoking status, education, social class, body mass index and prevalent diseases. Women who were current drinkers were less likely to have any hospital admissions multivariable adjusted OR 0.84 (95%CI 0.74-0.95), seven or more admissions OR 0.77 (95% CI 0.66-0.88) or more than twenty hospital days OR 0.70 (95%CI 0.62-0.80). However, compared to lifelong abstainers, men who were former drinkers had higher risk of any hospital admissions multivariable adjusted OR 2.22 (95%CI 1.51-3.28) and women former drinkers had higher risk of seven or more admissions OR 1.30 (95%CI 1.01-1.67).ConclusionCurrent alcohol consumption was associated with lower risk of future hospital usage compared with non-drinkers in this middle aged and older population. In men, this association may in part be due to whether former drinkers are included in the non-drinker reference group but in women, the association was consistent irrespective of the choice of reference group. In addition, there were few participants in this cohort with very high current alcohol intake. The measurement of past drinking, the separation of non-drinkers into former drinkers and lifelong abstainers and the choice of reference group are all influential in interpreting the risk of alcohol consumption on future hospitalisation.

Project description:Studies have reported a higher mortality risk associated with weight loss, particularly in middle-aged and older adults, although some of these studies did find that gaining weight was also associated with an increased mortality risk. We examined changes in weight in relation to mortality in a prospective population-based cohort study of men and women, resident in Norfolk, UK. Participants were assessed at baseline (1993-1997) and at a second examination (1998-2000), as part of the European Prospective Investigation into Cancer and Nutrition (EPIC-Norfolk) study, and followed up to 2015 for mortality. Participants with a self-reported history of cancer or cardiovascular disease, body mass index < 18.5 kg/m2 or missing data on adjustment variables, at either time-point were excluded, leaving 12,580 participants, aged 39-78 in 1993-1997, eligible for analyses. Cox proportional hazards models were used to determine Hazard Ratios (HRs) for all-cause (2603 deaths), cardiovascular (749 deaths), cancer (981 deaths), respiratory (226 deaths) and other causes of mortality (647 deaths) by categories of weight change. After multivariate adjustment, the HRs (95% CIs) for all-cause mortality for men and women who lost more than 5 kg were 1.85 (1.48-2.31) and 1.64 (1.31-2.05) respectively. Higher hazards were also found for specific causes of mortality and weight loss > 5 kg. Similar associations were observed after excluding deaths in the first 5 years of follow-up. Results for weight gain were inconclusive. We conclude that objectively measured weight loss, but not weight gain, was associated with subsequent higher mortality risk in this population-based study of middle-aged and elderly men and women. However, undiagnosed, pre-existing disease and the inability to account for weight cycling need to be remembered when interpreting these results. Unravelling the causal pathways underlying this association will require more detailed studies, including that of changes in body composition.

Dataset Information

Crowdsourcing as a novel technique for retinal fundus photography classification: analysis of images in the EPIC Norfolk cohort on behalf of the UK Biobank Eye and Vision Consortium.

Aim

Methods

Results

Conclusions

Publications

Crowdsourcing as a novel technique for retinal fundus photography classification: analysis of images in the EPIC Norfolk cohort on behalf of the UK Biobank Eye and Vision Consortium.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets