Dataset Information

Assessing risk of bias in human environmental epidemiology studies using three tools: different conclusions from different tools.

ABSTRACT: BACKGROUND:Systematic reviews are increasingly prevalent in environmental health due to their ability to synthesize evidence while reducing bias. Different systematic review methods have been developed by the US National Toxicology Program's Office of Health Assessment and Translation (OHAT), the US Environmental Protection Agency's (EPA) Integrated Risk Information System (IRIS), and by the US EPA under the Toxic Substances Control Act (TSCA), including the approach to assess risk of bias (ROB), one of the most vital steps which is used to evaluate internal validity of the studies. Our objective was to compare the performance of three tools (OHAT, IRIS, TSCA) in assessing ROB. METHODS:We selected a systematic review on polybrominated diphenyl ethers and intelligence quotient and/or attention deficit hyperactivity disorder because it had been endorsed by the National Academy of Sciences. Two reviewers followed verbatim instructions from the tools and independently applied each tool to assess ROB in 15 studies previously identified. We documented the time to apply each tool and the impact the ROB ratings for each tool had on the final rating of the quality of the overall body of evidence. RESULTS:The time to complete the ROB assessments varied widely (mean = 20, 32, and 40?min per study for the OHAT, IRIS, and TSCA tools, respectively). All studies were rated overall "low" or "uninformative" using IRIS, due to "deficient" or "critically deficient" ratings in one or two domains. Similarly, all studies were rated "unacceptable" using the TSCA tool because of one "unacceptable" rating in a metric related to statistical power. Approximately half of the studies had "low" or "probably low ROB" ratings across all domains with the OHAT and Navigation Guide tools. CONCLUSIONS:Tools that use overall ROB or study quality ratings, such as IRIS and TSCA, may reduce the available evidence to assess the harms of environmental exposures by erroneously excluding studies, which leads to inaccurate conclusions about the quality of the body of evidence. We recommend using ROB tools that circumvents these issues, such as OHAT and Navigation Guide. SYSTEMATIC REVIEW REGISTRATION:This review has not been registered as it is not a systematic review.

SUBMITTER: Eick SM

PROVIDER: S-EPMC7596989 | biostudies-literature | 2020 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Assessing risk of bias in human environmental epidemiology studies using three tools: different conclusions from different tools.

Eick Stephanie M SM Goin Dana E DE Chartres Nicholas N Lam Juleen J Woodruff Tracey J TJ

Systematic reviews 20201029 1

<h4>Background</h4>Systematic reviews are increasingly prevalent in environmental health due to their ability to synthesize evidence while reducing bias. Different systematic review methods have been developed by the US National Toxicology Program's Office of Health Assessment and Translation (OHAT), the US Environmental Protection Agency's (EPA) Integrated Risk Information System (IRIS), and by the US EPA under the Toxic Substances Control Act (TSCA), including the approach to assess risk of bi ...[more]

PMID: 33121530

Similar Datasets

Project description:BackgroundDietary exposure assessments are a critical issue in evaluating human nutrition studies; however, nutrition-specific criteria are not consistently included in existing bias assessment tools.ObjectivesOur objective was to develop a set of risk of bias (RoB) tools that integrated nutrition-specific criteria into validated generic assessment tools to address RoB issues, including those specific to dietary exposure assessment.MethodsThe Nutrition QUality Evaluation Strengthening Tools (NUQUEST) development and validation process included 8 steps. The first steps identified 1) a development strategy; 2) generic assessment tools with demonstrated validity; and 3) nutrition-specific appraisal issues. This was followed by 4) generation of nutrition-specific items and 5) development of guidance to aid users of NUQUEST. The final steps used established ratings of selected studies and feedback from independent raters to 6) assess reliability and validity; 7) assess formatting and usability; and 8) finalize NUQUEST.ResultsNUQUEST is based on the Scottish Intercollegiate Guidelines Network checklists for randomized controlled trials, cohort studies, and case-control studies. Using a purposive sample of 45 studies representing the 3 study designs, interrater reliability was high (Cohen's κ: 0.73; 95% CI: 0.52, 0.93) across all tools and at least moderate for individual tools (range: 0.57-1.00). The use of a worksheet improved usability and consistency of overall interrater agreement across all study designs (40% without worksheet, 80%-100% with worksheet). When compared to published ratings, NUQUEST ratings for evaluated studies demonstrated high concurrent validity (93% perfect or near-perfect agreement). Where there was disagreement, the nutrition-specific component was a contributing factor in discerning exposure methodological issues.ConclusionsNUQUEST integrates nutrition-specific criteria with generic criteria from assessment tools with demonstrated reliability and validity. NUQUEST represents a consistent and transparent approach for evaluating RoB issues related to dietary exposure assessment commonly encountered in human nutrition studies.

Project description:BackgroundEvidence based medicine aims to integrate scientific evidence, clinical experience, and patient values and preferences. Individual health care professionals need to appraise the evidence from randomized trials and observational studies when guidelines are not yet available. To date, tools for assessment of bias and terminologies for bias are specific for each study design. Moreover, most tools appeal only to methodological knowledge to detect bias, not to subject matter knowledge, i.e. in-depth medical knowledge about a topic. We propose a unified framework that enables the coherent assessment of bias across designs.MethodsEpidemiologists traditionally distinguish between three types of bias in observational studies: confounding, information bias, and selection bias. These biases result from a common cause, systematic error in the measurement or common effect of the intervention and outcome respectively. We applied this conceptual framework to randomized trials and show how it can be used to identify bias. The three sources of bias were illustrated with graphs that visually represent researchers' assumptions about the relationships between the investigated variables (causal diagrams).ResultsCritical appraisal of evidence started with the definition of the research question in terms of the population of interest, the compared interventions and the main outcome. Next, we used causal diagrams to illustrate how each source of bias can lead to over- or underestimated treatment effects. Then, we discussed how randomization, blinded outcome measurement and intention-to-treat analysis minimize bias in trials. Finally, we identified study aspects that can only be appraised with subject matter knowledge, irrespective of study design.ConclusionsThe unified framework encompassed the three main sources of bias for the effect of an assigned intervention on an outcome. It facilitated the integration of methodological and subject matter knowledge in the assessment of bias. We hope that graphical diagrams will help clarify debate among professionals by reducing misunderstandings based on different terminology for bias.

Project description:Aggregated species occurrence and abundance data from disparate sources are increasingly accessible to ecologists for the analysis of temporal trends in biodiversity. However, sampling biases relevant to any given research question are often poorly explored and infrequently reported; this can undermine statistical inference. In other disciplines, it is common for researchers to complete 'risk-of-bias' assessments to expose and document the potential for biases to undermine conclusions. The huge growth in available data, and recent controversies surrounding their use to infer temporal trends, indicate that similar assessments are urgently needed in ecology.We introduce ROBITT, a structured tool for assessing the 'Risk-Of-Bias In studies of Temporal Trends in ecology'. ROBITT has a similar format to its counterparts in other disciplines: it comprises signalling questions designed to elicit information on the potential for bias in key study domains. In answering these, users will define study inferential goal(s) and relevant statistical target populations. This information is used to assess potential sampling biases across domains relevant to the research question (e.g. geography, taxonomy, environment), and how these vary through time. If assessments indicate biases, then users must clearly describe them and/or explain what mitigating action will be taken.Everything that users need to complete a ROBITT assessment is provided: the tool, a guidance document and a worked example. Following other disciplines, the tool and guidance document were developed through a consensus-forming process across experts working in relevant areas of ecology and evidence synthesis.We propose that researchers should be strongly encouraged to include a ROBITT assessment when publishing studies of biodiversity trends, especially when using aggregated data. This will help researchers to structure their thinking, clearly acknowledge potential sampling issues, highlight where expert consultation is required and provide an opportunity to describe data checks that might go unreported. ROBITT will also enable reviewers, editors and readers to establish how well research conclusions are supported given a dataset combined with some analytical approach. In turn, it should strengthen evidence-based policy and practice, reduce differing interpretations of data and provide a clearer picture of the uncertainties associated with our understanding of reality.

Project description:BackgroundEstimations of causal effects from observational data are subject to various sources of bias. One method for adjusting for the residual biases in the estimation of treatment effects is through the use of negative control outcomes, which are outcomes not believed to be affected by the treatment of interest. The empirical calibration procedure is a technique that uses negative control outcomes to calibrate p-values. An extension of this technique calibrates the coverage of the 95% confidence interval of a treatment effect estimate by using negative control outcomes as well as positive control outcomes, which are outcomes for which the treatment of interest has known effects. Although empirical calibration has been used in several large observational studies, there is no systematic examination of its effect under different bias scenarios.MethodsThe effect of empirical calibration of confidence intervals was analyzed using simulated datasets with known treatment effects. The simulations consisted of binary treatment and binary outcome, with biases resulting from unmeasured confounder, model misspecification, measurement error, and lack of positivity. The performance of the empirical calibration was evaluated by determining the change in the coverage of the confidence interval and the bias in the treatment effect estimate.ResultsEmpirical calibration increased coverage of the 95% confidence interval of the treatment effect estimate under most bias scenarios but was inconsistent in adjusting the bias in the treatment effect estimate. Empirical calibration of confidence intervals was most effective when adjusting for the unmeasured confounding bias. Suitable negative controls had a large impact on the adjustment made by empirical calibration, but small improvements in the coverage of the outcome of interest were also observable when using unsuitable negative controls.ConclusionsThis work adds evidence to the efficacy of empirical calibration of the confidence intervals in observational studies. Calibration of confidence intervals is most effective where there are biases due to unmeasured confounding. Further research is needed on the selection of suitable negative controls.

Project description:ObjectivesTo compare the performance of Sequential Organ Failure Assessment, systemic inflammatory response syndrome, Red Flag Sepsis, and National Institute of Clinical Excellence sepsis risk stratification tools in the identification of patients at greatest risk of mortality from sepsis in nonintensive care environments.DesignSecondary analysis of three annual 24-hour point-prevalence study periods.SettingThe general wards and emergency departments of 14 acute hospitals across Wales. Studies were conducted on the third Wednesday of October in 2017, 2018, and 2019.PatientsWe screened all patients presenting to the emergency department and on the general wards.Measurements and main resultsWe recruited 1,271 patients, of which 724 (56.9%) had systemic inflammatory response syndrome greater than or equal to 2, 679 (53.4%) had Sequential Organ Failure Assessment greater than or equal to 2, and 977 (76.9%) had Red Flag Sepsis. When stratified according to National Institute of Clinical Excellence guidelines, 450 patients (35.4%) were in the "High risk" category in comparison with 665 (52.3%) in "Moderate to High risk" and 156 (12.3%) in "Low risk" category. In a planned sensitivity analysis, we found that none of the tools accurately predicted mortality at 90 days, and Sequential Organ Failure Assessment and National Institute of Clinical Excellence tools showed only moderate discriminatory power for mortality at 7 and 14 days. Furthermore, we could not find any significant correlation with any of the tools at any of the mortality time points.ConclusionsOur data suggest that the sepsis risk stratification tools currently utilized in emergency departments and on the general wards do not predict mortality adequately. This is illustrated by the disparity in mortality risk of the populations captured by each instrument, as well as the weak concordance between them. We propose that future studies on the development of sepsis identification tools should focus on identifying predicator values of both the short- and long-term outcomes of sepsis.

Dataset Information

Assessing risk of bias in human environmental epidemiology studies using three tools: different conclusions from different tools.

Publications

Assessing risk of bias in human environmental epidemiology studies using three tools: different conclusions from different tools.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets