Dataset Information

Improving Consensus Scoring of Crowdsourced Data Using the Rasch Model: Development and Refinement of a Diagnostic Instrument.

ABSTRACT:

Background

Diabetic retinopathy (DR) is a leading cause of vision loss in working age individuals worldwide. While screening is effective and cost effective, it remains underutilized, and novel methods are needed to increase detection of DR. This clinical validation study compared diagnostic gradings of retinal fundus photographs provided by volunteers on the Amazon Mechanical Turk (AMT) crowdsourcing marketplace with expert-provided gold-standard grading and explored whether determination of the consensus of crowdsourced classifications could be improved beyond a simple majority vote (MV) using regression methods.

Objective

The aim of our study was to determine whether regression methods could be used to improve the consensus grading of data collected by crowdsourcing.

Methods

A total of 1200 retinal images of individuals with diabetes mellitus from the Messidor public dataset were posted to AMT. Eligible crowdsourcing workers had at least 500 previously approved tasks with an approval rating of 99% across their prior submitted work. A total of 10 workers were recruited to classify each image as normal or abnormal. If half or more workers judged the image to be abnormal, the MV consensus grade was recorded as abnormal. Rasch analysis was then used to calculate worker ability scores in a random 50% training set, which were then used as weights in a regression model in the remaining 50% test set to determine if a more accurate consensus could be devised. Outcomes of interest were the percent correctly classified images, sensitivity, specificity, and area under the receiver operating characteristic (AUROC) for the consensus grade as compared with the expert grading provided with the dataset.

Results

Using MV grading, the consensus was correct in 75.5% of images (906/1200), with 75.5% sensitivity, 75.5% specificity, and an AUROC of 0.75 (95% CI 0.73-0.78). A logistic regression model using Rasch-weighted individual scores generated an AUROC of 0.91 (95% CI 0.88-0.93) compared with 0.89 (95% CI 0.86-92) for a model using unweighted scores (chi-square P value<.001). Setting a diagnostic cut-point to optimize sensitivity at 90%, 77.5% (465/600) were graded correctly, with 90.3% sensitivity, 68.5% specificity, and an AUROC of 0.79 (95% CI 0.76-0.83).

Conclusions

Crowdsourced interpretations of retinal images provide rapid and accurate results as compared with a gold-standard grading. Creating a logistic regression model using Rasch analysis to weight crowdsourced classifications by worker ability improves accuracy of aggregated grades as compared with simple majority vote.

SUBMITTER: Brady CJ

PROVIDER: S-EPMC5497070 | biostudies-literature | 2017 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Improving Consensus Scoring of Crowdsourced Data Using the Rasch Model: Development and Refinement of a Diagnostic Instrument.

Brady Christopher John CJ Mudie Lucy Iluka LI Wang Xueyang X Guallar Eliseo E Friedman David Steven DS

Journal of medical Internet research 20170620 6

<h4>Background</h4>Diabetic retinopathy (DR) is a leading cause of vision loss in working age individuals worldwide. While screening is effective and cost effective, it remains underutilized, and novel methods are needed to increase detection of DR. This clinical validation study compared diagnostic gradings of retinal fundus photographs provided by volunteers on the Amazon Mechanical Turk (AMT) crowdsourcing marketplace with expert-provided gold-standard grading and explored whether determinati ...[more]

PMID: 28634154

Similar Datasets

Project description:The NEI VFQ-25 has undergone psychometric evaluation in patients with varying ocular conditions and the general population. However, important limitations which may affect the interpretation of clinical trial results have been previously identified, such as concerns with reliability and validity. The purpose of this study was to evaluate the National Eye Institute Visual Functioning Questionnaire (NEI VFQ-25) and make recommendations for a revised scoring structure, with a view to improving its psychometric performance and interpretability.Rasch Measurement Theory analyses were conducted in two stages using pooled baseline NEI VFQ-25 data for 2487 participants with retinal diseases enrolled in six clinical trials. In stage 1, we examined: scale-to-sample targeting; thresholds for item response options; item fit statistics; stability; local dependence; and reliability. In stage 2, a post-hoc revision of the scoring structure (VFQ-28R) was created and psychometrically re-evaluated.In stage 1, we found that the NEI VFQ-25 was mis-targeted to the sample, and had disordered response thresholds (15/25 items) and mis-fitting items (8/25 items). However, items appeared to be stable (differential item functioning for three items), have minimal item dependency (one pair of items) and good reliability (person-separation index, 0.93). In stage 2, the modified Rasch-scored NEI VFQ-28-R was assessed. It comprised two broad domains: Activity Limitation (19 items) and Socio-Emotional Functioning (nine items). The NEI VFQ-28-R demonstrated improved performance with fewer disordered response thresholds (no items), less item misfit (three items) and improved population targeting (reduced ceiling effect) compared with the NEI VFQ-25.Compared with the original version, the proposed NEI VFQ-28-R, with Rasch-based scoring and a two-domain structure, appears to offer improved psychometric performance and interpretability of the vision-related quality of life scale for the population analysed.

Project description:BackgroundType 2 diabetes is an increasingly prevalent chronic illness and an important cause of avoidable mortality. Patients are managed by the integrated activities of clinical and non-clinical members of primary care teams. This study aimed to: investigate theoretically-based organisational, team, and individual factors determining the multiple behaviours needed to manage diabetes; and identify multilevel determinants of different diabetes management behaviours and potential interventions to improve them. This paper describes the instrument development, study recruitment, characteristics of the study participating practices and their constituent healthcare professionals and administrative staff and reports descriptive analyses of the data collected.MethodsThe study was a predictive study over a 12-month period. Practices (N = 99) were recruited from within the UK Medical Research Council General Practice Research Framework. We identified six behaviours chosen to cover a range of clinical activities (prescribing, non-prescribing), reflect decisions that were not necessarily straightforward (controlling blood pressure that was above target despite other drug treatment), and reflect recommended best practice as described by national guidelines. Practice attributes and a wide range of individually reported measures were assessed at baseline; measures of clinical outcome were collected over the ensuing 12 months, and a number of proxy measures of behaviour were collected at baseline and at 12 months. Data were collected by telephone interview, postal questionnaire (organisational and clinical) to practice staff, postal questionnaire to patients, and by computer data extraction query.ResultsAll 99 practices completed a telephone interview and responded to baseline questionnaires. The organisational questionnaire was completed by 931/1236 (75.3%) administrative staff, 423/529 (80.0%) primary care doctors, and 255/314 (81.2%) nurses. Clinical questionnaires were completed by 326/361 (90.3%) primary care doctors and 163/186 (87.6%) nurses. At a practice level, we achieved response rates of 100% from clinicians in 40 practices and > 80% from clinicians in 67 practices. All measures had satisfactory internal consistency (alpha coefficient range from 0.61 to 0.97; Pearson correlation coefficient (two item measures) 0.32 to 0.81); scores were generally consistent with good practice. Measures of behaviour showed relatively high rates of performance of the six behaviours, but with considerable variability within and across the behaviours and measures.DiscussionWe have assembled an unparalleled data set from clinicians reporting on their cognitions in relation to the performance of six clinical behaviours involved in the management of people with one chronic disease (diabetes mellitus), using a range of organisational and individual level measures as well as information on the structure of the practice teams and across a large number of UK primary care practices. We would welcome approaches from other researchers to collaborate on the analysis of this data.

Project description:BackgroundThe concomitant occurrence of the symptoms intravascular hypovolemia, peripheral edema and hemodynamic instability is typically named Capillary Leak Syndrome (CLS) and often occurs in surgical critical ill patients. However, neither a unitary definition nor standardized diagnostic criteria exist so far. We aimed to investigate common characteristics of this phenomenon with a subsequent scoring system, determining whether CLS contributes to mortality.MethodsWe conducted this single-center, observational, multidisciplinary, prospective trial in two separately run surgical ICUs of a tertiary academic medical center. 200 surgical patients admitted to the ICU and 30 healthy volunteers were included. Patients were clinically diagnosed as CLS or No-CLS group (each N = 100) according to the grade of edema, intravascular hypovolemia, hemodynamic instability, and positive fluid balance by two independent attending physicians with > 10 years of experience in ICU. We performed daily measurements with non-invasive body impedance electrical analysis, ultrasound and analysis of serum biomarkers to generate objective diagnostic criteria. Receiver operating characteristics were used, while we developed machine learning models to increase diagnostic specifications for our scoring model.ResultsThe 30-day mortility was increased among CLS patients (12 vs. 1%, P = 0.002), while showing higher SOFA-scores. Extracellular water was increased in patients with CLS with higher echogenicity of subcutaneous tissue [29(24-31) vs. 19(16-21), P < 0.001]. Biomarkers showed characteristic alterations, especially with an increased angiopoietin-2 concentration in CLS [9.9(6.2-17.3) vs. 3.7(2.6-5.6)ng/mL, P < 0.001]. We developed a score using seven parameters (echogenicity, SOFA-score, angiopoietin-2, syndecan-1, ICAM-1, lactate and interleukin-6). A Random Forest prediction model boosted its diagnostic characteristics (AUC 0.963, P < 0.001), while a two-parameter decision tree model showed good specifications (AUC 0.865).ConclusionsDiagnosis of CLS in critically ill patients is feasible by objective, non-invasive parameters using the CLS-Score. A simplified two-parameter diagnostic approach can enhance clinical utility. CLS contributes to mortality and should, therefore, classified as an independent entity.Trial registrationGerman Clinical Trials Registry (DRKS No. 00012713), Date of registration 10/05/2017, www.drks.de.

Dataset Information

Improving Consensus Scoring of Crowdsourced Data Using the Rasch Model: Development and Refinement of a Diagnostic Instrument.

Background

Objective

Methods

Results

Conclusions

Publications

Improving Consensus Scoring of Crowdsourced Data Using the Rasch Model: Development and Refinement of a Diagnostic Instrument.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets