Consensus-based guidance for conducting and reporting multi-analyst studies.
Ontology highlight
ABSTRACT:
SUBMITTER: Aczel B
PROVIDER: S-EPMC8626083 | biostudies-literature | 2021 Nov
REPOSITORIES: biostudies-literature
Publications
Consensus-based guidance for conducting and reporting multi-analyst studies.
Aczel Balazs B Szaszi Barnabas B Nilsonne Gustav G van den Akker Olmo R OR Albers Casper J CJ van Assen Marcel Alm MA Bastiaansen Jojanneke A JA Benjamin Daniel D Boehm Udo U Botvinik-Nezer Rotem R Bringmann Laura F LF Busch Niko A NA Caruyer Emmanuel E Cataldo Andrea M AM Cowan Nelson N Delios Andrew A van Dongen Noah Nn NN Donkin Chris C van Doorn Johnny B JB Dreber Anna A Dutilh Gilles G Egan Gary F GF Gernsbacher Morton Ann MA Hoekstra Rink R Hoffmann Sabine S Holzmeister Felix F Huber Juergen J Johannesson Magnus M Jonas Kai J KJ Kindel Alexander T AT Kirchler Michael M Kunkels Yoram K YK Lindsay D Stephen DS Mangin Jean-Francois JF Matzke Dora D Munafò Marcus R MR Newell Ben R BR Nosek Brian A BA Poldrack Russell A RA van Ravenzwaaij Don D Rieskamp Jörg J Salganik Matthew J MJ Sarafoglou Alexandra A Schonberg Tom T Schweinsberg Martin M Shanks David D Silberzahn Raphael R Simons Daniel J DJ Spellman Barbara A BA St-Jean Samuel S Starns Jeffrey J JJ Uhlmann Eric Luis EL Wicherts Jelte J Wagenmakers Eric-Jan EJ
eLife 20211109
Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst a ...[more]
Similar Datasets
Project description:ObjectiveTo improve the quality and consistency of intervention development reporting in health research.DesignThis was a consensus exercise consisting of two simultaneous and identical three-round e-Delphi studies (one with experts in intervention development and one with wider stakeholders including funders, journal editors and public involvement members), followed by a consensus workshop. Delphi items were systematically derived from two preceding systematic reviews and a qualitative interview study.ParticipantsIntervention developers (n=26) and wider stakeholders (n=18) from the UK, North America and Europe participated in separate e-Delphi studies. Intervention developers (n=13) and wider stakeholders (n=13) participated in a 1-day consensus workshop.Resultse-Delphi participants achieved consensus on 15 reporting items. Following feedback from the consensus meeting, the final inclusion and wording of 14 items with description and explanations for each item were agreed. Items focus on context, purpose, target population, approaches, evidence, theory, guiding principles, stakeholder contribution, changes in content or format during the development process, required changes for subgroups, continuing uncertainties, and open access publication. They form the GUIDED (GUIDance for the rEporting of intervention Development) checklist, which contains a description and explanation of each item, alongside examples of good reporting.ConclusionsConsensus-based reporting guidance for intervention development in health research is now available for publishers and researchers to use. GUIDED has the potential to lead to greater transparency, and enhance quality and improve learning about intervention development research and practice.
| S-EPMC7245409 | biostudies-literature
Project description:BackgroundThe concept of osteoarthritis (OA) heterogeneity is evolving and gaining renewed interest. According to this concept, distinct subtypes of OA need to be defined that will likely require recognition in research design and different approaches to clinical management. Although seemingly plausible, a wide range of views exist on how best to operationalize this concept. The current project aimed to provide consensus-based definitions and recommendations that together create a framework for conducting and reporting OA phenotype research.MethodsA panel of 25 members with expertise in OA phenotype research was composed. First, panel members participated in an online Delphi exercise to provide a number of basic definitions and statements relating to OA phenotypes and OA phenotype research. Second, panel members provided input on a set of recommendations for reporting on OA phenotype studies.ResultsFour Delphi rounds were required to achieve sufficient agreement on 11 definitions and statements. OA phenotypes were defined as subtypes of OA that share distinct underlying pathobiological and pain mechanisms and their structural and functional consequences. Reporting recommendations pertaining to the study characteristics, study population, data collection, statistical analysis, and appraisal of OA phenotype studies were provided.ConclusionsThis study provides a number of consensus-based definitions and recommendations relating to OA phenotypes. The resulting framework is intended to facilitate research on OA phenotypes and increase combined efforts to develop effective OA phenotype classification. Success in this endeavor will hopefully translate into more effective, differentiated OA management that will benefit a multitude of OA patients.
| S-EPMC7083005 | biostudies-literature
Project description:BackgroundWhen research evidence is limited, inconsistent, or absent, healthcare decisions and policies need to be based on consensus amongst interested stakeholders. In these processes, the knowledge, experience, and expertise of health professionals, researchers, policymakers, and the public are systematically collected and synthesised to reach agreed clinical recommendations and/or priorities. However, despite the influence of consensus exercises, the methods used to achieve agreement are often poorly reported. The ACCORD (ACcurate COnsensus Reporting Document) guideline was developed to help report any consensus methods used in biomedical research, regardless of the health field, techniques used, or application. This explanatory document facilitates the use of the ACCORD checklist.Methods and findingsThis paper was built collaboratively based on classic and contemporary literature on consensus methods and publications reporting their use. For each ACCORD checklist item, this explanation and elaboration document unpacks the pieces of information that should be reported and provides a rationale on why it is essential to describe them in detail. Furthermore, this document offers a glossary of terms used in consensus exercises to clarify the meaning of common terms used across consensus methods, to promote uniformity, and to support understanding for consumers who read consensus statements, position statements, or clinical practice guidelines. The items are followed by examples of reporting items from the ACCORD guideline, in text, tables and figures.ConclusionsThe ACCORD materials - including the reporting guideline and this explanation and elaboration document - can be used by anyone reporting a consensus exercise used in the context of health research. As a reporting guideline, ACCORD helps researchers to be transparent about the materials, resources (both human and financial), and procedures used in their investigations so readers can judge the trustworthiness and applicability of their results/recommendations.
| S-EPMC11198995 | biostudies-literature
Project description: Not available
| S-EPMC8481359 | biostudies-literature
Project description:BackgroundImplementation trials aim to test the effects of implementation strategies on the adoption, integration or uptake of an evidence-based intervention within organisations or settings. Feasibility and pilot studies can assist with building and testing effective implementation strategies by helping to address uncertainties around design and methods, assessing potential implementation strategy effects and identifying potential causal mechanisms. This paper aims to provide broad guidance for the conduct of feasibility and pilot studies for implementation trials.MethodsWe convened a group with a mutual interest in the use of feasibility and pilot trials in implementation science including implementation and behavioural science experts and public health researchers. We conducted a literature review to identify existing recommendations for feasibility and pilot studies, as well as publications describing formative processes for implementation trials. In the absence of previous explicit guidance for the conduct of feasibility or pilot implementation trials specifically, we used the effectiveness-implementation hybrid trial design typology proposed by Curran and colleagues as a framework for conceptualising the application of feasibility and pilot testing of implementation interventions. We discuss and offer guidance regarding the aims, methods, design, measures, progression criteria and reporting for implementation feasibility and pilot studies.ConclusionsThis paper provides a resource for those undertaking preliminary work to enrich and inform larger scale implementation trials.
| S-EPMC7603668 | biostudies-literature
Project description:Background and objectiveThe living systematic review (LSR) approach is based on ongoing surveillance of the literature and continual updating. Most currently available guidance documents address the conduct, reporting, publishing, and appraisal of systematic reviews (SRs), but are not suitable for LSRs per se and miss additional LSR-specific considerations. In this scoping review, we aim to systematically collate methodological guidance literature on how to conduct, report, publish, and appraise the quality of LSRs and identify current gaps in guidance.MethodsA standard scoping review methodology was used. We searched MEDLINE (Ovid), EMBASE (Ovid), and The Cochrane Library on August 28, 2021. As for searching gray literature, we looked for existing guidelines and handbooks on LSRs from organizations that conduct evidence syntheses. The screening was conducted by two authors independently in Rayyan, and data extraction was done in duplicate using a pilot-tested data extraction form in Excel. Data was extracted according to four pre-defined categories for (i) conducting, (ii) reporting, (iii) publishing, and (iv) appraising LSRs. We mapped the findings by visualizing overview tables created in Microsoft Word.ResultsOf the 21 included papers, methodological guidance was found in 17 papers for conducting, in six papers for reporting, in 15 papers for publishing, and in two papers for appraising LSRs. Some of the identified key items for (i) conducting LSRs were identifying the rationale, screening tools, or re-revaluating inclusion criteria. Identified items of (ii) the original PRISMA checklist included reporting the registration and protocol, title, or synthesis methods. For (iii) publishing, there was guidance available on publication type and frequency or update trigger, and for (iv) appraising, guidance on the appropriate use of bias assessment or reporting funding of included studies was found. Our search revealed major evidence gaps, particularly for guidance on certain PRISMA items such as reporting results, discussion, support and funding, and availability of data and material of a LSR.ConclusionImportant evidence gaps were identified for guidance on how to report in LSRs and appraise their quality. Our findings were applied to inform and prepare a PRISMA 2020 extension for LSR.
| S-EPMC10722674 | biostudies-literature
Project description:Background: The living systematic review (LSR) approach is based on an ongoing surveillance of the literature and continual updating. A few guidance documents address the conduct, reporting, publishing and appraisal of systematic reviews (SRs), but the methodology described is either not up-to date or not suitable for LSRs and misses additional LSR-specific considerations. The objective of this scoping review is to systematically collate methodological literature and guidance on how to conduct, report, publish and appraise the quality of LSRs. The scoping review will allow the mapping of the existing evidence on the topic to support LSRs authors seeking guidance and identify related gaps. Methods: To achieve our objectives, we will conduct a scoping review to survey and evaluate existing evidence, using the standard scoping review methodology. We will search MEDLINE, EMBASE, and Cochrane using the OVID interface. The search strategy was developed by a researcher experienced in developing literature search strategies with the help of an information specialist. As for searching grey literature, we will seek existing guidelines and handbooks on LSRs from organizations that conduct evidence syntheses using the Lens.org website. Two review authors will extract and catalogue the study data on LSR methodological aspects into a standardized and pilot-tested data extraction form. The main categories will reflect proposed methods for (i) conducting LSRs, (ii) reporting of LSRs, (iii) publishing and (iv) appraising the quality of LSRs. Data synthesis and conclusion: By collecting these data from methodological surveys and papers, as well as existing guidance documents and handbooks on LSRs, we might identify specific issues and components lacking within current LSR methodology. Thus, the systematically obtained findings of the scoping review could be used as basis for the revision of existing methods tools on LSR, for instance a PRISMA statement extension for LSRs.
| S-EPMC8822136 | biostudies-literature
Project description:ObjectiveTo identify evidence on the reporting quality of consensus methodology and to select potential checklist items for the ACcurate COnsensus Reporting Document (ACCORD) project to develop a consensus reporting guideline.DesignSystematic review.Data sourcesEmbase, MEDLINE, Web of Science, PubMed, Cochrane Library, Emcare, Academic Search Premier and PsycINFO from inception until 7 January 2022.Eligibility criteriaStudies, reviews and published guidance addressing the reporting quality of consensus methodology for improvement of health outcomes in biomedicine or clinical practice. Reports of studies using or describing consensus methods but not commenting on their reporting quality were excluded. No language restrictions were applied.Data extraction and synthesisScreening and data extraction of eligible studies were carried out independently by two authors. Reporting quality items addressed by the studies were synthesised narratively.ResultsEighteen studies were included: five systematic reviews, four narrative reviews, three research papers, three conference abstracts, two research guidance papers and one protocol. The majority of studies indicated that the quality of reporting of consensus methodology could be improved. Commonly addressed items were: consensus panel composition; definition of consensus and the threshold for achieving consensus. Items least addressed were: public patient involvement (PPI); the role of the steering committee, chair, cochair; conflict of interest of panellists and funding. Data extracted from included studies revealed additional items that were not captured in the data extraction form such as justification of deviation from the protocol or incentives to encourage panellist response.ConclusionThe results of this systematic review confirmed the need for a reporting checklist for consensus methodology and provided a range of potential checklist items to report. The next step in the ACCORD project builds on this systematic review and focuses on reaching consensus on these items to develop the reporting guideline.Protocol registrationhttps://osf.io/2rzm9.
| S-EPMC9462098 | biostudies-literature
Project description:Multi-attribute utility-based instruments (MAUIs) assess health status and provide an index score on the full health-dead scale, and are widely used to support reimbursement decisions for new healthcare interventions worldwide. A valuation study is a key part of the development of MAUIs, with the primary goal of developing a scoring algorithm through eliciting societal preferences. We developed the 21-item Checklist for REporting VAluaTion StudiEs (CREATE) by following a modified two-round Delphi panel approach plus an email survey. CREATE is intended to promote good reporting practice as well as guiding developers to thoroughly and carefully think through key methodological elements in designing valuation studies.
| S-EPMC4519579 | biostudies-literature
Project description:BackgroundTo our knowledge, no publication providing overarching guidance on the conduct of systematic reviews of observational studies of etiology exists.Methods and findingsConducting Systematic Reviews and Meta-Analyses of Observational Studies of Etiology (COSMOS-E) provides guidance on all steps in systematic reviews of observational studies of etiology, from shaping the research question, defining exposure and outcomes, to assessing the risk of bias and statistical analysis. The writing group included researchers experienced in meta-analyses and observational studies of etiology. Standard peer-review was performed. While the structure of systematic reviews of observational studies on etiology may be similar to that for systematic reviews of randomised controlled trials, there are specific tasks within each component that differ. Examples include assessment for confounding, selection bias, and information bias. In systematic reviews of observational studies of etiology, combining studies in meta-analysis may lead to more precise estimates, but such greater precision does not automatically remedy potential bias. Thorough exploration of sources of heterogeneity is key when assessing the validity of estimates and causality.ConclusionAs many reviews of observational studies on etiology are being performed, this document may provide researchers with guidance on how to conduct and analyse such reviews.
| S-EPMC6383865 | biostudies-literature