Dataset Information

Setting weights for fifteen CHNRI criteria at the global and regional level using public stakeholders: an Amazon Mechanical Turk study.

ABSTRACT:

Introduction

Stakeholder involvement has been described as an indispensable part of health research priority setting. Yet, more than 75% of the exercises using the Child Health and Nutrition Research Initiative (CHNRI) methodology have omitted the step involving stakeholders in priority setting. Those that have used stakeholders have rarely used the public, possibly due to the difficulty of assembling and/or accessing a public stakeholder group. In order to strengthen future exercises using the CHNRI methodology, we have used a public stakeholder group to weight 15 CHNRI criteria, and have explored regional differences or being a health stakeholder is influential, and whether the criteria are collapsible.

Methods

Using Amazon Mechanical Turk (AMT), an online crowdsourcing platform, we collected demographic information and conducted a Likert-scale format survey about the importance of the CHNRI criteria from 1051 stakeholders. The Kruskal-Wallis test, with Dunn's test for posthoc comparisons, was used to examine regional differences and Wilcoxon rank-sum test was used to analyse differences between stakeholders with health training/background and stakeholders without a health background and by region. A Factor Analysis (FA) was conducted on the criteria to identify the main domains connecting them. Criteria means were converted to weights.

Results

There were regional differences in thirteen of fifteen criteria according to the Kruskal-Wallis test and differences in responses from health stakeholders vs those who were not in eleven of fifteen criteria using the Wilcoxon rank-sum test. Three components were identified: improve and impact results; implementation and affordability; and, study design and dissemination. A formula is provided to convert means to weights for future studies.

Conclusion

In future CHNRI studies, researchers will need to ensure adequate representation from stakeholders to undue bias of CHNRI results. These results should be used in combination with other stakeholder groups, including government, donors, policy makers, and bilateral agencies. Global and regional stakeholder groups scored CHNRI criteria differently; due to this, researchers should consider which group to use in their CHNRI exercises.

SUBMITTER: Wazny K

PROVIDER: S-EPMC6445564 | biostudies-literature | 2019 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Setting weights for fifteen CHNRI criteria at the global and regional level using public stakeholders: an Amazon Mechanical Turk study.

Wazny Kerri K Ravenscroft John J Chan Kit Yee KY Bassani Diego G DG Anderson Niall N Rudan Igor I

Journal of global health 20190601 1

<h4>Introduction</h4>Stakeholder involvement has been described as an indispensable part of health research priority setting. Yet, more than 75% of the exercises using the Child Health and Nutrition Research Initiative (CHNRI) methodology have omitted the step involving stakeholders in priority setting. Those that have used stakeholders have rarely used the public, possibly due to the difficulty of assembling and/or accessing a public stakeholder group. In order to strengthen future exercises us ...[more]

PMID: 30992986

Similar Datasets

Project description:Internet interventions face significant challenges in recruitment and attrition rates are typically high and problematic. Finding innovative yet scientifically valid avenues for attaining and retaining participants is therefore of considerable importance. The main goal of this study was to compare recruitment process and participants characteristics between two similar randomized control trials of mood management interventions. One of the trials (Bunge et al., 2016) was conducted with participants recruited from Amazon's Mechanical Turk (AMT), and the other trial recruited via Unpaid Internet Resources (UIR). Methods:The AMT sample (Bunge et al., 2016) consisted of 765 adults, and the UIR sample (recruited specifically for this study) consisted of 329 adult US residents. Participants' levels of depression, anxiety, confidence, motivation, and perceived usefulness of the intervention were assessed. The AMT sample was financially compensated whereas the UIR was not. Results:AMT yielded higher recruitment rates per month (p?<?.05). At baseline, the AMT sample reported significantly lower depression and anxiety scores (p?<?.001 and p?<?.005, respectively) and significantly higher mood, motivation, and confidence (all p?<?.001) compared to the UIR sample. AMT participants spent significantly less time on the site (p?<?.05) and were more likely to complete follow-ups than the UIR sample (p?<?.05). Both samples reported a significant increase in their level of confidence and motivation from pre- to post-intervention. AMT participants showed a significant increase in perceived usefulness of the intervention (p?<?.0001), whereas the UIR sample did not (p?=?.1642). Conclusions:By using AMT, researchers can recruit very rapidly and obtain higher retention rates; however, these participants may not be representative of the general online population interested in clinical interventions. Considering that AMT and UIR participants differed in most baseline variables, data from clinical studies resulting from AMT samples should be interpreted with caution.

Project description:Several recent reviews of the methods used to set research priorities have identified the CHNRI method (acronym derived from the "Child Health and Nutrition Research Initiative") as an approach that clearly became popular and widely used over the past decade. In this paper we review the first 50 examples of application of the CHNRI method, published between 2007 and 2016, and summarize the most important messages that emerged from those experiences.We conducted a literature review to identify the first 50 examples of application of the CHNRI method in chronological order. We searched Google Scholar, PubMed and so-called grey literature.Initially, between 2007 and 2011, the CHNRI method was mainly used for setting research priorities to address global child health issues, although the first cases of application outside this field (eg, mental health, disabilities and zoonoses) were also recorded. Since 2012 the CHNRI method was used more widely, expanding into the topics such as adolescent health, dementia, national health policy and education. The majority of the exercises were focused on issues that were only relevant to low- and middle-income countries, and national-level applications are on the rise. The first CHNRI-based articles adhered to the five recommended priority-setting criteria, but by 2016 more than two-thirds of all conducted exercises departed from recommendations, modifying the CHNRI method to suit each particular exercise. This was done not only by changing the number of criteria used, but also by introducing some entirely new criteria (eg, "low cost", "sustainability", "acceptability", "feasibility", "relevance" and others).The popularity of the CHNRI method in setting health research priorities can be attributed to several key conceptual advances that have addressed common concerns. The method is systematic in nature, offering an acceptable framework for handling many research questions. It is also transparent and replicable, because it clearly defines the context and priority-setting criteria. It is democratic, as it relies on "crowd-sourcing". It is inclusive, fostering "ownership" of the results by ensuring that various groups invest in the process. It is very flexible and adjustable to many different contexts and needs. Finally, it is simple and relatively inexpensive to conduct, which we believe is one of the main reasons for its uptake by many groups globally, particularly those in low- and middle-income countries.

Project description:Introduction:The Child Health and Nutrition Research Initiative (CHNRI) method for health research prioritisation relies on stakeholders weighting criteria used to assess research options. These weights in turn impact on the final scores and ranks assigned to research options. Three quarters of CHNRI studies published to date have not involved stakeholders in criteria weighting. Of those that have, few incorporated members of the public into stakeholder groups. Those that have compared different stakeholder groups, such as donors, researchers, or policy makers, showed that different groups place different values upon CHNRI criteria. When choosing the composition of a stakeholder group, it may be important to understand factors that may influence weighting. Drawing upon a group of international public stakeholders, this study explores some of the effects of individual and demographic characteristics has on the weights assigned to the most commonly used CHNRI criteria, with the aim of informing future researchers on avoiding future biases. Methods:Individual and demographic information and 5-point Likert scale responses to questions about the importance of 15 CHNRI criteria were collected from 1031 "Turkers" (Amazon Mechanical Turk workers) via Amazon Mechanical Turk (AMT), which is an online crowdsourcing platform. Thirteen of the fifteen criteria were analysed using random-intercept models and the remaining two were analysed through logistic regression. Results:Self-reported health status explained most of the variability in participants' responses across criteria (11/15 criteria), followed by being female (10/15), ethnicity (9/15), employment (8/15), and religion (7/15). Differences across criteria indicate that when choosing stakeholder groups, researchers need to consider these factors to minimise bias. Conclusion:Researchers should collect and report more detailed information from stakeholders, including individual and demographic characteristics, and ensure participation from both genders, multiple ethnicities, religious beliefs, and people with differing health statuses to be transparent regarding possible biases in health research prioritisation. Our analyses indicate that these factors do influence the relative importance of these values, even when the data appears fairly homogeneous.

Project description:The CHNRI method for setting health research priorities has crowdsourcing as the major component. It uses the collective opinion of a group of experts to generate, assess and prioritize between many competing health research ideas. It is difficult to compare the accuracy of human individual and collective opinions in predicting uncertain future outcomes before the outcomes are known. However, this limitation does not apply to existing knowledge, which is an important component underlying opinion. In this paper, we report several experiments to explore the quantitative properties of human collective knowledge and discuss their relevance to the CHNRI method.We conducted a series of experiments in groups of about 160 (range: 122-175) undergraduate Year 2 medical students to compare their collective knowledge to their individual knowledge. We asked them to answer 10 questions on each of the following: (i) an area in which they have a degree of expertise (undergraduate Year 1 medical curriculum); (ii) an area in which they likely have some knowledge (general knowledge); and (iii) an area in which they are not expected to have any knowledge (astronomy). We also presented them with 20 pairs of well-known celebrities and asked them to identify the older person of the pair. In all these experiments our goal was to examine how the collective answer compares to the distribution of students' individual answers.When answering the questions in their own area of expertise, the collective answer (the median) was in the top 20.83% of the most accurate individual responses; in general knowledge, it was in the top 11.93%; and in an area with no expertise, the group answer was in the top 7.02%. However, the collective answer based on mean values fared much worse, ranging from top 75.60% to top 95.91%. Also, when confronted with guessing the older of the two celebrities, the collective response was correct in 18/20 cases (90%), while the 8 most successful individuals among the students had 19/20 correct answers (95%). However, when the system in which the students who were not sure of the correct answer were allowed to either choose an award of half of the point in all such instances, or withdraw from responding, in order to improve the score of the collective, the collective was correct in 19/20 cases (95%), while the 3 most successful individuals were correct in 17/20 cases (85%).Our experiments showed that the collective knowledge of a group with expertise in the subject should always be very close to the true value. In most cases and under most assumption, the collective knowledge will be more accurate than the knowledge of an "average" individual, but there always seems to be a small group of individuals who manage to out-perform the collective. The accuracy of collective prediction may be enhanced by allowing the individuals with low confidence in their answer to withdraw from answering.

Dataset Information

Setting weights for fifteen CHNRI criteria at the global and regional level using public stakeholders: an Amazon Mechanical Turk study.

Introduction

Methods

Results

Conclusion

Publications

Setting weights for fifteen CHNRI criteria at the global and regional level using public stakeholders: an Amazon Mechanical Turk study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure