Dataset Information

Setting health research priorities using the CHNRI method: V. Quantitative properties of human collective knowledge.

ABSTRACT: The CHNRI method for setting health research priorities has crowdsourcing as the major component. It uses the collective opinion of a group of experts to generate, assess and prioritize between many competing health research ideas. It is difficult to compare the accuracy of human individual and collective opinions in predicting uncertain future outcomes before the outcomes are known. However, this limitation does not apply to existing knowledge, which is an important component underlying opinion. In this paper, we report several experiments to explore the quantitative properties of human collective knowledge and discuss their relevance to the CHNRI method.We conducted a series of experiments in groups of about 160 (range: 122-175) undergraduate Year 2 medical students to compare their collective knowledge to their individual knowledge. We asked them to answer 10 questions on each of the following: (i) an area in which they have a degree of expertise (undergraduate Year 1 medical curriculum); (ii) an area in which they likely have some knowledge (general knowledge); and (iii) an area in which they are not expected to have any knowledge (astronomy). We also presented them with 20 pairs of well-known celebrities and asked them to identify the older person of the pair. In all these experiments our goal was to examine how the collective answer compares to the distribution of students' individual answers.When answering the questions in their own area of expertise, the collective answer (the median) was in the top 20.83% of the most accurate individual responses; in general knowledge, it was in the top 11.93%; and in an area with no expertise, the group answer was in the top 7.02%. However, the collective answer based on mean values fared much worse, ranging from top 75.60% to top 95.91%. Also, when confronted with guessing the older of the two celebrities, the collective response was correct in 18/20 cases (90%), while the 8 most successful individuals among the students had 19/20 correct answers (95%). However, when the system in which the students who were not sure of the correct answer were allowed to either choose an award of half of the point in all such instances, or withdraw from responding, in order to improve the score of the collective, the collective was correct in 19/20 cases (95%), while the 3 most successful individuals were correct in 17/20 cases (85%).Our experiments showed that the collective knowledge of a group with expertise in the subject should always be very close to the true value. In most cases and under most assumption, the collective knowledge will be more accurate than the knowledge of an "average" individual, but there always seems to be a small group of individuals who manage to out-perform the collective. The accuracy of collective prediction may be enhanced by allowing the individuals with low confidence in their answer to withdraw from answering.

SUBMITTER: Rudan I

PROVIDER: S-EPMC4920010 | biostudies-literature | 2016 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Setting health research priorities using the CHNRI method: V. Quantitative properties of human collective knowledge.

Rudan Igor I Yoshida Sachiyo S Wazny Kerri K Chan Kit Yee KY Cousens Simon S

Journal of global health 20160601 1

<h4>Introduction</h4>The CHNRI method for setting health research priorities has crowdsourcing as the major component. It uses the collective opinion of a group of experts to generate, assess and prioritize between many competing health research ideas. It is difficult to compare the accuracy of human individual and collective opinions in predicting uncertain future outcomes before the outcomes are known. However, this limitation does not apply to existing knowledge, which is an important compone ...[more]

PMID: 27350873

Similar Datasets

Project description:BackgroundSeveral recent reviews of the methods used to set research priorities have identified the CHNRI method (acronym derived from the "Child Health and Nutrition Research Initiative") as an approach that clearly became popular and widely used over the past decade. In this paper we review the first 50 examples of application of the CHNRI method, published between 2007 and 2016, and summarize the most important messages that emerged from those experiences.MethodsWe conducted a literature review to identify the first 50 examples of application of the CHNRI method in chronological order. We searched Google Scholar, PubMed and so-called grey literature.ResultsInitially, between 2007 and 2011, the CHNRI method was mainly used for setting research priorities to address global child health issues, although the first cases of application outside this field (eg, mental health, disabilities and zoonoses) were also recorded. Since 2012 the CHNRI method was used more widely, expanding into the topics such as adolescent health, dementia, national health policy and education. The majority of the exercises were focused on issues that were only relevant to low- and middle-income countries, and national-level applications are on the rise. The first CHNRI-based articles adhered to the five recommended priority-setting criteria, but by 2016 more than two-thirds of all conducted exercises departed from recommendations, modifying the CHNRI method to suit each particular exercise. This was done not only by changing the number of criteria used, but also by introducing some entirely new criteria (eg, "low cost", "sustainability", "acceptability", "feasibility", "relevance" and others).ConclusionsThe popularity of the CHNRI method in setting health research priorities can be attributed to several key conceptual advances that have addressed common concerns. The method is systematic in nature, offering an acceptable framework for handling many research questions. It is also transparent and replicable, because it clearly defines the context and priority-setting criteria. It is democratic, as it relies on "crowd-sourcing". It is inclusive, fostering "ownership" of the results by ensuring that various groups invest in the process. It is very flexible and adjustable to many different contexts and needs. Finally, it is simple and relatively inexpensive to conduct, which we believe is one of the main reasons for its uptake by many groups globally, particularly those in low- and middle-income countries.

Project description:BackgroundThe private health sector is an important source of sick child care, yet evidence gaps persist in best practices for integrated management of private sector child health services. Further, there is no prioritized research agenda to address these gaps. We used a Child Health and Nutrition Research Initiative (CHNRI) process to identify priority research questions in response to these evidence gaps. CHNRI is a consultative approach that entails prioritizing research questions by evaluating them against standardized criteria.MethodsWe engaged geographically and occupationally diverse experts in the private health sector and child health. Eighty-nine experts agreed to participate and provided 150 priority research questions. We consolidated submitted questions to reduce duplication into a final list of 50. We asked participants to complete an online survey to rank each question against 11 pre-determined criteria in four categories: (i) answerability, (ii) research feasibility, (iii) sustainability/equity, and (iv) importance/potential impact. Statistical data analysis was conducted in SAS 9.4 (SAS Institute Inc, Cary NC, USA). We weighted all 11 evaluation criteria equally to calculate the research priority score and average expert agreement for each question. We disaggregated results by location in high-income vs low- and middle-income countries.ResultsForty-nine participants (55.1%) completed the online survey, including 33 high-income and 16 low- and middle-income country respondents. The top, prioritized research question asks whether accreditation or regulation of private clinical and non-clinical sources of care would improve integrated management of childhood illness services. Four of the top ten research priorities were related to adherence to case management protocols. Other top research priorities were related to training and supportive supervision, digital health, and infant and newborn care. Research priorities among high-income and low- and middle-income country respondents were highly correlated.ConclusionTo our knowledge, this is the first systematic exercise conducted to define research priorities for the management of childhood illness in the private sector. The research priorities put forth in this CHNRI exercise aim to stimulate interest from policy makers, program managers, researchers, and donors to respond to and help close evidence gaps hindering the acceleration of reductions in child mortality through private sector approaches.

Project description:AimsTo systematically identify global research gaps and resource priorities for integrated community case management (iCCM).MethodsAn iCCM Child Health and Nutrition Research Initiative (CHNRI) Advisory Group, in collaboration with the Community Case Management Operational Research Group (CCM ORG) identified experts to participate in a CHNRI research priority setting exercise. These experts generated and systematically ranked research questions for iCCM. Research questions were ranked using a "Research Priority Score" (RPS) and the "Average Expert Agreement" (AEA) was calculated for every question. Our groups of experts were comprised of both individuals working in Ministries of Health or Non Governmental Organizations (NGOs) in low- and middle-income countries (LMICs) and individuals working in high-income countries (HICs) in academia or NGO headquarters. A Spearman's Rho was calculated to determine the correlation between the two groups' research questions' ranks.ResultsThe overall RPS ranged from 64.58 to 89.31, with a median score of 81.43. AEA scores ranged from 0.54 to 0.86. Research questions involving increasing the uptake of iCCM services, research questions concerning the motivation, retention, training and supervision of Community Health Workers (CHWs) and concerning adding additional responsibilities including counselling for infant and young child feeding (IYCF) and treatment of severe acute malnutrition (SAM) ranked highly. There was weak to moderate, statistically significant, correlation between scores by representatives of high-income countries and those working in-country or regionally (Spearman's ρ = 0.35034, P < 0.01).ConclusionsOperational research to determine optimal training, supervision and modes of motivation and retention for the CHW is vital for improving iCCM, globally, as is research to motivate caregivers to take advantage of iCCM services. Experts working in-country or regionally in LMICs prioritized different research questions than those working in organization headquarters in HICs. Further exploration is needed to determine the nature of this divergence.

Project description:BACKGROUND:Armed conflict, natural disaster, and forced displacement affect millions of children each year. Such humanitarian crises increase the risk of family separation, erode existing support networks, and often result in economic loss, increasing children's vulnerability to violence, exploitation, neglect, and abuse. Research is needed to understand these risks and vulnerabilities and guide donor investment towards the most effective interventions for improving the well-being of children in humanitarian contexts. METHODS:The Assessment, Measurement & Evidence (AME) Working Group of the Alliance for Child Protection in Humanitarian Action (ACPHA) identified experts to participate in a research priority setting exercise adapted from the Child Health and Nutrition Research Initiative (CHNRI). Experts individually identified key areas for research investment which were subsequently ranked by participants using a Likert scale. Research Priority Scores (RPS) and Average Expert Agreement (AEA) were calculated for each identified research topic, the top fifteen of which are presented within this paper. RESULTS:Intervention research, which aims to rigorously evaluate the effectiveness of standard child protection activities in humanitarian settings, ranked highly. Child labor was a key area of sector research with two of the top ten priorities examining the practice. Respondents also prioritized research efforts to understand how best to bridge humanitarian and development efforts for child protection as well as identifying most effective way to build the capacity of local systems in order to sustain child protection gains after a crisis. CONCLUSIONS:Rigorous, scientific research that assesses the scope of child protection risks, examines the effectiveness of interventions to improve child well-being, and translates evidence to practice is critical. Findings from this research priority setting exercise offer guidance for a global research agenda on child protection in humanitarian settings, encouraging cooperation among donors, implementers, and academics to pursue a coordinated approach to evidence generation.

Project description:BackgroundMillennium Development Goal 4 (MDGs) mobilised countries to reduce child mortality by two thirds the 1990 rate in 2015. While India did not reach MDG 4, it considerably reduced child mortality in the MDG-era. Efficient and targeted interventions and adequate monitoring are necessary to further progress in improvements to child health. Looking forward to the Sustainable Development Goal (SDG)-era, the Indian Council of Medical Research and The INCLEN Trust International conducted a national research priority setting exercise for maternal, child, newborn health, and maternal and child nutrition. Here, results are reported for child health.MethodsThe Child Health and Nutrition Research Initiative (CHNRI) method for research priority setting was employed. Research ideas were crowd-sourced from a network of child health experts from across India; these were refined and consolidated into research options (ROs) which were scored against five weighted criteria to arrive weighted Research Priority Scores (wRPS). National and regional priority lists were prepared.Results90 experts contributed 596 ideas that were consolidated into 101 research options (ROs). These were scored by 233 experts nationwide. National wRPS for ROs ranged between 0.92 and 0.51. The majority of the top research priorities related to development of cost-effective interventions and their implementation, and impact evaluations, improving data quality; and monitoring of existing programs, or improving the management of morbidities. The research priorities varied between regions, the Economic Action Group and North-Eastern states prioritised questions relating to delivering interventions at community- or household-level, whereas the North-Eastern states and Union Territories prioritised research questions involving managing and measuring malaria, and the Southern and Western states prioritised research questions involving pharmacovigilance of vaccines, impact of newly introduced vaccines, and delivery of vaccines to hard-to-reach populations.ConclusionsResearch priorities varied geographically, according the stage of development of the area and mostly pertained to implementation sciences, which was expected given diversity in epidemiological profiles. Priority setting should help guide investment decisions by national and international agencies, therefore encouraging researchers to focus on priority areas. The ICMR has launched a grants programme for implementation research on maternal and child health to pursue research priorities identified by this exercise.

Project description:BackgroundIn 2015, it was estimated that the burden of disease in Iran comprised of 19 million disability-adjusted life years (DALYs), 74% of which were due to non-communicable diseases (NCDs). The observed leading causes of death were cardiovascular diseases (41.9%), neoplasms (14.9%), and road traffic injuries (7.4%). Even so, the health research investment in Iran continues to remain limited. This study aims to identify national health research priorities in Iran for the next five years to assist the efficient use of resources towards achieving the long-term health targets.MethodsAdapting the Child Health and Nutrition Research Initiative (CHNRI) method, this study engaged 48 prominent Iranian academic leaders in the areas related to Iran's long-term health targets, a group of research funders and policy makers, and 68 stakeholders from the wider society. 128 proposed research questions were scored independently using a set of five criteria: feasibility, impact on health, impact on economy, capacity building, and equity.FindingsThe top-10 priorities were focused on the research questions relating to: health insurance system reforms to improve equity; integration of NCDs prevention strategy into primary health care; cost-effective population-level interventions for NCDs and road traffic injury prevention; tailoring medical qualifications; epidemiological assessment of NCDs by geographic areas; equality in the distribution of health resources and services; current and future common health problems in Iran's elderly and strategies to reduce their economic burden; the status of antibiotic resistance in Iran and strategies to promote rational use of antibiotics; the health impacts of water crisis; and research to replace the physician-centered health system with a team-based one.ConclusionsThese findings highlight consensus amongst various prominent Iranian researchers and stakeholders over the research priorities that require investment to generate information and knowledge relevant to the national health targets and policies. The exercise should assist in addressing the knowledge gaps to support both the National General Health Policies by 2025 and the health targets of the United Nations' Sustainable Development Goals by 2030.

Project description:BackgroundIn 2013, an estimated 2.8 million newborns died and 2.7 million were stillborn. A much greater number suffer from long term impairment associated with preterm birth, intrauterine growth restriction, congenital anomalies, and perinatal or infectious causes. With the approaching deadline for the achievement of the Millennium Development Goals (MDGs) in 2015, there was a need to set the new research priorities on newborns and stillbirth with a focus not only on survival but also on health, growth and development. We therefore carried out a systematic exercise to set newborn health research priorities for 2013-2025.MethodsWe used adapted Child Health and Nutrition Research Initiative (CHNRI) methods for this prioritization exercise. We identified and approached the 200 most productive researchers and 400 program experts, and 132 of them submitted research questions online. These were collated into a set of 205 research questions, sent for scoring to the 600 identified experts, and were assessed and scored by 91 experts.ResultsNine out of top ten identified priorities were in the domain of research on improving delivery of known interventions, with simplified neonatal resuscitation program and clinical algorithms and improved skills of community health workers leading the list. The top 10 priorities in the domain of development were led by ideas on improved Kangaroo Mother Care at community level, how to improve the accuracy of diagnosis by community health workers, and perinatal audits. The 10 leading priorities for discovery research focused on stable surfactant with novel modes of administration for preterm babies, ability to diagnose fetal distress and novel tocolytic agents to delay or stop preterm labour.ConclusionThese findings will assist both donors and researchers in supporting and conducting research to close the knowledge gaps for reducing neonatal mortality, morbidity and long term impairment. WHO, SNL and other partners will work to generate interest among key national stakeholders, governments, NGOs, and research institutes in these priorities, while encouraging research funders to support them. We will track research funding, relevant requests for proposals and trial registers to monitor if the priorities identified by this exercise are being addressed.

Project description:A focus on equity in health can be seen in many global development goals and reports, research and international declarations. With the development of a relevant framework and methods, the Campbell and Cochrane Equity Methods Group has encouraged the application of an 'equity lens' to systematic reviews, and many organizations publish reviews intended to address health equity. The purpose of the Evidence for Equity (E4E) project was to conduct a priority-setting exercise and apply an equity lens by developing a knowledge translation product comprising summaries of systematic reviews from the Cochrane Library. E4E translates evidence from systematic reviews into 'friendly front end' summaries for policy makers.The following topic areas with high burdens of disease globally, were selected for the pilot: diabetes/obesity, HIV/AIDS, malaria, nutrition, and mental health/depression. For each topic area, a "stakeholder panel" was assembled that included policymakers and researchers. A systematic search of Cochrane reviews was conducted for each area to identify equity-relevant interventions with a meaningful impact. Panel chairs developed a rating sheet which was used by all panels to rank the importance of these interventions by: 1) Ease of Implementation; 2) Health System Requirements; 3)Universality/Generalizability/Share of Burden; and 4) Impact on Inequities/Effect on equity. The ratings of panel members were averaged for each intervention and criterion, and interventions were ordered according to the average overall ratings.Stakeholder panels identified the top 10 interventions from their respective topic areas. The evidence on these interventions is being summarized with an equity focus and the results posted online, at http://methods.cochrane.org/equity/e4e-series .This method provides an explicit approach to setting priorities by systematic review groups and funders for providing decision makers with evidence for the most important equity-relevant interventions.

Project description:BackgroundAn estimated 70.8 million people are forcibly displaced worldwide, 75% of whom are women and children. Prioritizing a global research agenda to inform guidance, service delivery, access to and quality of services is essential to improve the survival and health of women, children and adolescents in humanitarian settings.MethodA mixed-methods design was adapted from the Child Health and Nutrition Research Initiative (CHNRI) methodology to solicit priority research questions across the sexual, reproductive, maternal, newborn, child and adolescent health (SRMNCAH) domains in humanitarian settings. The first step (CHNRI) involved data collection and scoring of perceived priority questions, using a web-based survey over two rounds (first, to generate the questions and secondly, to score them). Over 1000 stakeholders from across the globe were approached; 177 took part in the first survey and 69 took part in the second. These research questions were prioritized by generating a research prioritization score (RPP) across four dimensions: answerability, program feasibility, public health relevance and equity. A Delphi process of 29 experts followed, where the 50 scored and prioritized CHRNI research questions were shortlisted. The top five questions from the CHNRI scored list for each SRMNCAH domain were voted on, rendering a final list per domain.ResultsA total of 280 questions were generated. Generated questions covered sexual and reproductive health (SRH) (n = 90, 32.1%), maternal health (n = 75, 26.8%), newborn health (n = 42, 15.0%), child health (n = 43, 15.4%), and non-SRH aspects of adolescent health (n = 31, 11.1%). A shortlist of the top ten prioritized questions for each domain were generated on the basis of the computed RPPs. During the Delphi process, the prioritized questions, based on the CHNRI process, were further refined. Five questions from the shortlist of each of the SRMNCAH domain were formulated, resulting in 25 priority questions across SRMNCAH. For example, one of the prioritized SRH shortlisted and prioritized research question included: "What are effective strategies to implement good quality comprehensive contraceptive services (long-acting, short-acting and EC) for women and girls in humanitarian settings?"ConclusionData needs, effective intervention strategies and approaches, as well as greater efficiency and quality during delivery of care in humanitarian settings were prioritized. The findings from this research provide guidance for researchers, program implementers, as well as donor agencies on SRMNCAH research priorities in humanitarian settings. A global research agenda could save the lives of those who are at greatest risk and vulnerability as well as increase opportunities for translation and innovation for SRMNCAH in humanitarian settings.

Dataset Information

Setting health research priorities using the CHNRI method: V. Quantitative properties of human collective knowledge.

Publications

Setting health research priorities using the CHNRI method: V. Quantitative properties of human collective knowledge.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets