Dataset Information

Crowdsourcing citation-screening in a mixed-studies systematic review: a feasibility study.

ABSTRACT:

Background

Crowdsourcing engages the help of large numbers of people in tasks, activities or projects, usually via the internet. One application of crowdsourcing is the screening of citations for inclusion in a systematic review. There is evidence that a 'Crowd' of non-specialists can reliably identify quantitative studies, such as randomized controlled trials, through the assessment of study titles and abstracts. In this feasibility study, we investigated crowd performance of an online, topic-based citation-screening task, assessing titles and abstracts for inclusion in a single mixed-studies systematic review.

Methods

This study was embedded within a mixed studies systematic review of maternity care, exploring the effects of training healthcare professionals in intrapartum cardiotocography. Citation-screening was undertaken via Cochrane Crowd, an online citizen science platform enabling volunteers to contribute to a range of tasks identifying evidence in health and healthcare. Contributors were recruited from users registered with Cochrane Crowd. Following completion of task-specific online training, the crowd and the review team independently screened 9546 titles and abstracts. The screening task was subsequently repeated with a new crowd following minor changes to the crowd agreement algorithm based on findings from the first screening task. We assessed the crowd decisions against the review team categorizations (the 'gold standard'), measuring sensitivity, specificity, time and task engagement.

Results

Seventy-eight crowd contributors completed the first screening task. Sensitivity (the crowd's ability to correctly identify studies included within the review) was 84% (N = 42/50), and specificity (the crowd's ability to correctly identify excluded studies) was 99% (N = 9373/9493). Task completion was 33 h for the crowd and 410 h for the review team; mean time to classify each record was 6.06 s for each crowd participant and 3.96 s for review team members. Replicating this task with 85 new contributors and an altered agreement algorithm found 94% sensitivity (N = 48/50) and 98% specificity (N = 9348/9493). Contributors reported positive experiences of the task.

Conclusion

It might be feasible to recruit and train a crowd to accurately perform topic-based citation-screening for mixed studies systematic reviews, though resource expended on the necessary customised training required should be factored in. In the face of long review production times, crowd screening may enable a more time-efficient conduct of reviews, with minimal reduction of citation-screening accuracy, but further research is needed.

SUBMITTER: Noel-Storr AH

PROVIDER: S-EPMC8077753 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundSystematic reviews (SRs) are often cited as the highest level of evidence available as they involve the identification and synthesis of published studies on a topic. Unfortunately, it is increasingly challenging for small teams to complete SR procedures in a reasonable time period, given the exponential rise in the volume of primary literature. Crowdsourcing has been postulated as a potential solution.ObjectiveThe feasibility objective of this study was to determine whether a crowd would be willing to perform and complete abstract and full text screening. The validation objective was to assess the quality of the crowd's work, including retention of eligible citations (sensitivity) and work performed for the investigative team, defined as the percentage of citations excluded by the crowd.MethodsWe performed a prospective study evaluating crowdsourcing essential components of an SR, including abstract screening, document retrieval, and full text assessment. Using CrowdScreenSR citation screening software, 2323 articles from 6 SRs were available to an online crowd. Citations excluded by less than or equal to 75% of the crowd were moved forward for full text assessment. For the validation component, performance of the crowd was compared with citation review through the accepted, gold standard, trained expert approach.ResultsOf 312 potential crowd members, 117 (37.5%) commenced abstract screening and 71 (22.8%) completed the minimum requirement of 50 citation assessments. The majority of participants were undergraduate or medical students (192/312, 61.5%). The crowd screened 16,988 abstracts (median: 8 per citation; interquartile range [IQR] 7-8), and all citations achieved the minimum of 4 assessments after a median of 42 days (IQR 26-67). Crowd members retrieved 83.5% (774/927) of the articles that progressed to the full text phase. A total of 7604 full text assessments were completed (median: 7 per citation; IQR 3-11). Citations from all but 1 review achieved the minimum of 4 assessments after a median of 36 days (IQR 24-70), with 1 review remaining incomplete after 3 months. When complete crowd member agreement at both levels was required for exclusion, sensitivity was 100% (95% CI 97.9-100) and work performed was calculated at 68.3% (95% CI 66.4-70.1). Using the predefined alternative 75% exclusion threshold, sensitivity remained 100% and work performed increased to 72.9% (95% CI 71.0-74.6; P<.001). Finally, when a simple majority threshold was considered, sensitivity decreased marginally to 98.9% (95% CI 96.0-99.7; P=.25) and work performed increased substantially to 80.4% (95% CI 78.7-82.0; P<.001).ConclusionsCrowdsourcing of citation screening for SRs is feasible and has reasonable sensitivity and specificity. By expediting the screening process, crowdsourcing could permit the investigative team to focus on more complex SR tasks. Future directions should focus on developing a user-friendly online platform that allows research teams to crowdsource their reviews.

Project description:BackgroundCrowdsourcing involves obtaining ideas, needed services, or content by soliciting Web-based contributions from a crowd. The 4 types of crowdsourced tasks (problem solving, data processing, surveillance or monitoring, and surveying) can be applied in the 3 categories of health (promotion, research, and care).ObjectiveThis study aimed to map the different applications of crowdsourcing in health to assess the fields of health that are using crowdsourcing and the crowdsourced tasks used. We also describe the logistics of crowdsourcing and the characteristics of crowd workers.MethodsMEDLINE, EMBASE, and ClinicalTrials.gov were searched for available reports from inception to March 30, 2016, with no restriction on language or publication status.ResultsWe identified 202 relevant studies that used crowdsourcing, including 9 randomized controlled trials, of which only one had posted results at ClinicalTrials.gov. Crowdsourcing was used in health promotion (91/202, 45.0%), research (73/202, 36.1%), and care (38/202, 18.8%). The 4 most frequent areas of application were public health (67/202, 33.2%), psychiatry (32/202, 15.8%), surgery (22/202, 10.9%), and oncology (14/202, 6.9%). Half of the reports (99/202, 49.0%) referred to data processing, 34.6% (70/202) referred to surveying, 10.4% (21/202) referred to surveillance or monitoring, and 5.9% (12/202) referred to problem-solving. Labor market platforms (eg, Amazon Mechanical Turk) were used in most studies (190/202, 94%). The crowd workers' characteristics were poorly reported, and crowdsourcing logistics were missing from two-thirds of the reports. When reported, the median size of the crowd was 424 (first and third quartiles: 167-802); crowd workers' median age was 34 years (32-36). Crowd workers were mainly recruited nationally, particularly in the United States. For many studies (58.9%, 119/202), previous experience in crowdsourcing was required, and passing a qualification test or training was seldom needed (11.9% of studies; 24/202). For half of the studies, monetary incentives were mentioned, with mainly less than US $1 to perform the task. The time needed to perform the task was mostly less than 10 min (58.9% of studies; 119/202). Data quality validation was used in 54/202 studies (26.7%), mainly by attention check questions or by replicating the task with several crowd workers.ConclusionsThe use of crowdsourcing, which allows access to a large pool of participants as well as saving time in data collection, lowering costs, and speeding up innovations, is increasing in health promotion, research, and care. However, the description of crowdsourcing logistics and crowd workers' characteristics is frequently missing in study reports and needs to be precisely reported to better interpret the study findings and replicate them.

Project description:BackgroundWith the accumulation of evidence regarding potential harms of cancer screening in recent years, researchers, policy-makers, and the public are becoming more critical of population-based cancer screening. Consequently, a high-quality cancer screening program should consider individuals’ values and preferences when determining recommendations. In cervical cancer screening, offering women autonomy is considered a “person-centered” approach to health care services; however, it may impact the effectiveness of the program should women choose to not participate. As part of a larger project to investigate women’s cervical screening preferences and correlates of these preferences, this systematic review will capture quantitative and qualitative investigations of women’s cervical screening preferences and the methods used to elicit them.Design and methodsThis mixed methods synthesis will use a thematic analysis approach to synthesize qualitative, quantitative, and mixed methods evidence. This protocol describes the methods that will be used in this investigation. A search strategy has been developed with a health librarian and peer reviewed using PRESS. Based on this strategy, five databases and the gray literature will be searched for studies that meet the inclusion criteria. The quality of the included individual studies will be examined using the Mixed Methods Appraisal Tool. Three reviewers will extract data from the primary studies on the tools or instruments used to elicit women’s preferences regarding cervical cancer screening, theoretical frameworks used, outcomes measured, the outstanding themes from quantitative and qualitative evidence, and the identified preferences for cervical cancer screening. We will describe the relationships between study results and the study population, “intervention” (e.g., tool or instrument), and context. We will follow the PRISMA reporting guideline. We will compare findings across studies and between study methods (e.g., qualitative versus quantitative study designs). The strength of the synthesized findings will be assessed using the validated GRADE and CERQual tool.DiscussionThis review will inform the development of a tool to elicit women’s cervical screening preferences. Understanding the methods used to elicit women’s preferences and what is known about women’s cervical screening preferences will be useful for guideline developers who wish to incorporate a woman-centered approach specifically for cervical screening guidelines.Systematic review registrationPROSPERO CRD42016035737Electronic supplementary materialThe online version of this article (doi:10.1186/s13643-016-0310-9) contains supplementary material, which is available to authorized users.

Project description:BackgroundThe value of citation searches as part of the systematic review process is currently unknown. While the major guides to conducting systematic reviews state that citation searching should be carried out in addition to searching bibliographic databases there are still few studies in the literature that support this view. Rather than using a predefined search strategy to retrieve studies, citation searching uses known relevant papers to identify further papers.MethodsWe describe a case study about the effectiveness of using the citation sources Google Scholar, Scopus, Web of Science and OVIDSP MEDLINE to identify records for inclusion in a systematic review.We used the 40 included studies identified by traditional database searches from one systematic review of interventions for multiple risk behaviours. We searched for each of the included studies in the four citation sources to retrieve the details of all papers that have cited these studies.We carried out two analyses; the first was to examine the overlap between the four citation sources to identify which citation tool was the most useful; the second was to investigate whether the citation searches identified any relevant records in addition to those retrieved by the original database searches.ResultsThe highest number of citations was retrieved from Google Scholar (1680), followed by Scopus (1173), then Web of Science (1095) and lastly OVIDSP (213). To retrieve all the records identified by the citation tracking searching all four resources was required. Google Scholar identified the highest number of unique citations.The citation tracking identified 9 studies that met the review's inclusion criteria. Eight of these had already been identified by the traditional databases searches and identified in the screening process while the ninth was not available in any of the databases when the original searches were carried out. It would, however, have been identified by two of the database search strategies if searches had been carried out later.ConclusionsBased on the results from this investigation, citation searching as a supplementary search method for systematic reviews may not be the best use of valuable time and resources. It would be useful to verify these findings in other reviews.

Project description:ObjectiveTo assess the feasibility of using a blood pressure (BP) self-measurement kiosk-a solid-cuff sphygmomanometer combined with technology to integrate the BP readings into patient electronic medical records- to improve hypertension detection.DesignA concurrent mixed-methods feasibility study incorporating observational and qualitative interview components.SettingTwo English general practitioner (GP) surgeries.ParticipantsAdult patients registered at participating surgeries. Staff working at these sites.InterventionsBP self-measurement kiosks were placed in the waiting rooms for a 12-month period between 2015 and 2016 and compared with a 12-month control period prior to installation.Outcome measures(1) The number of patients using the kiosk and agreeing to transfer of their data into their electronic medical records; (2) the cost of using a kiosk compared with GP/practice nurse BP screening; (3) qualitative themes regarding use of the equipment.ResultsOut of 15?624 eligible patients, only 186 (1.2%, 95% CI 1.0% to 1.4%) successfully used the kiosk to directly transfer a BP reading into their medical record. For a considerable portion of the intervention period, no readings were transferred, possibly indicating technical problems with the transfer link. A comparison of costs suggests that at least 52.6% of eligible patients would need to self-screen in order to bring costs below that of screening by GPs and practice nurses. Qualitative interviews confirmed that both patients and staff experienced technical difficulties, and used alternative methods to enter BP results into the medical record.ConclusionsWhile interviewees were generally positive about checking BP in the waiting room, the electronic transfer system as tested was neither robust, effective nor likely to be a cost-effective approach, thus may not be appropriate for a primary care environment. Since most of the cost of a kiosk system lies in the transfer mechanism, a solid-cuff sphygmomanometer and manual entry of results may be a suitable alternative.

Project description:BACKGROUND:Crowdsourcing is used increasingly in health and medical research. Crowdsourcing is the process of aggregating crowd wisdom to solve a problem. The purpose of this systematic review is to summarize quantitative evidence on crowdsourcing to improve health. METHODS:We followed Cochrane systematic review guidance and systematically searched seven databases up to September 4th 2019. Studies were included if they reported on crowdsourcing and related to health or medicine. Studies were excluded if recruitment was the only use of crowdsourcing. We determined the level of evidence associated with review findings using the GRADE approach. RESULTS:We screened 3508 citations, accessed 362 articles, and included 188 studies. Ninety-six studies examined effectiveness, 127 examined feasibility, and 37 examined cost. The most common purposes were to evaluate surgical skills (17 studies), to create sexual health messages (seven studies), and to provide layperson cardio-pulmonary resuscitation (CPR) out-of-hospital (six studies). Seventeen observational studies used crowdsourcing to evaluate surgical skills, finding that crowdsourcing evaluation was as effective as expert evaluation (low quality). Four studies used a challenge contest to solicit human immunodeficiency virus (HIV) testing promotion materials and increase HIV testing rates (moderate quality), and two of the four studies found this approach saved money. Three studies suggested that an interactive technology system increased rates of layperson initiated CPR out-of-hospital (moderate quality). However, studies analyzing crowdsourcing to evaluate surgical skills and layperson-initiated CPR were only from high-income countries. Five studies examined crowdsourcing to inform artificial intelligence projects, most often related to annotation of medical data. Crowdsourcing was evaluated using different outcomes, limiting the extent to which studies could be pooled. CONCLUSIONS:Crowdsourcing has been used to improve health in many settings. Although crowdsourcing is effective at improving behavioral outcomes, more research is needed to understand effects on clinical outcomes and costs. More research is needed on crowdsourcing as a tool to develop artificial intelligence systems in medicine. TRIAL REGISTRATION:PROSPERO: CRD42017052835. December 27, 2016.