Development of a Methodological Quality Criteria List for Observational Studies: The Observational Study Quality Evaluation.
Ontology highlight
ABSTRACT:
SUBMITTER: Drukker M
PROVIDER: S-EPMC8317224 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Background and objectiveObservational studies may provide valuable evidence on real-world causal effects of drug effectiveness in patients with coronavirus disease 2019 (COVID-19). As patients are usually observed from hospital admission to discharge and drug initiation starts during hospitalization, advanced statistical methods are needed to account for time-dependent drug exposure, confounding and competing events. Our objective is to evaluate the observational studies on the three common methodological pitfalls in time-to-event analyses: immortal time bias, confounding bias and competing risk bias.MethodsWe performed a systematic literature search on 23 October 2020, in the PubMed database to identify observational cohort studies that evaluated drug effectiveness in hospitalized patients with COVID-19. We included articles published in four journals: British Medical Journal, New England Journal of Medicine, Journal of the American Medical Association and The Lancet as well as their sub-journals.ResultsOverall, out of 255 articles screened, 11 observational cohort studies on treatment effectiveness with drug exposure-outcome associations were evaluated. All studies were susceptible to one or more types of bias in the primary study analysis. Eight studies had a time-dependent treatment. However, the hazard ratios were not adjusted for immortal time in the primary analysis. Even though confounders presented at baseline have been addressed in nine studies, time-varying confounding caused by time-varying treatment exposure and clinical variables was less recognized. Only one out of 11 studies addressed competing event bias by extending follow-up beyond patient discharge.ConclusionsIn the observational cohort studies on drug effectiveness for treatment of COVID-19 published in four high-impact journals, the methodological biases were concerningly common. Appropriate statistical tools are essential to avoid misleading conclusions and to obtain a better understanding of potential treatment effects.
| S-EPMC8015394 | biostudies-literature
Project description:Introduction and objectiveSystematic review tools that provide guidance on evaluating epidemiology studies are receiving increasing attention and support because their application facilitates improved quality of the review, consistency across reviewers, and transparency for readers. The U.S. Environmental Protection Agency's Integrated Risk Information System (IRIS) Program has developed an approach for systematic review of evidence of health effects from chemical exposures that includes structured approaches for literature search and screening, study evaluation, data extraction, and evidence synthesis and integration. This approach recognizes the need for developing outcome-specific criteria for study evaluation. Because studies are assessed at the outcome level, a study could be considered high quality for one investigated outcome, and low quality for another, due to differences in the outcome measures, analytic strategies, how relevant a certain bias is to the outcome, and how the exposure measure relates to the outcome. The objective of this paper is to illustrate the need for outcome-specific criteria in study evaluation or risk of bias evaluation, describe the process we used to develop the criteria, and summarize the resulting criteria.MethodsWe used a process of expert consultation to develop several sets of outcome-specific criteria to guide study reviewers, improve consistency, and ensure consideration of critical issues specific to the outcomes. The criteria were developed using the following domains: outcome assessment, exposure measurement (specifically timing of exposure in relation to outcome; other exposure measurement issues would be addressed in exposure-specific criteria), participant selection, confounding, analysis, and sensitivity (the study's ability to detect a true effect or hazard).ResultsWe discuss the application of this process to pregnancy-related outcomes (preterm birth, spontaneous abortion), other reproductive-related outcomes (male reproductive hormones, sperm parameters, time to pregnancy, pubertal development), chronic disease (diabetes, insulin resistance), and acute or episodic conditions (asthma, allergies), and provide examples of the criteria developed. For each outcome the most influential methodological considerations are highlighted including biological sample collection and quality control, sensitivity and specificity of ascertainment tools, optimal timing for recruitment into the study (e.g., preconception, specific trimesters), the etiologically relevant window for exposure assessments, and important potential confounders.ConclusionsOutcome-specific criteria are an important part of a systematic review and will facilitate study evaluations by epidemiologists with experience in evaluating studies using systematic review methods who may not have extensive discipline-specific experience in the outcomes being reviewed.
| S-EPMC8522891 | biostudies-literature
Project description:BACKGROUND:Despite a wide range of literature on emergency department (ED) overcrowding, scientific knowledge on emergency physicians' cognitive processes coping with overcrowding is limited. OBJECTIVE:This study aimed to develop and evaluate a virtual research environment that will allow us to study the effect of physicians' strategies and behaviors on quality of care in the context of ED overcrowding. METHODS:A simulation-based observational study was conducted over two stages: the development of a simulation model and its evaluation. A research environment in emergency medicine combining virtual reality and simulated patients was designed and developed. Afterwards, 12 emergency physicians took part in simulation scenarios and had to manage 13 patients during a 2-hour period. The study outcome was the authenticity of the environment through realism, consistency, and mastering. The realism was the resemblance perceived by the participants between virtual and real ED. The consistency of the scenario and the participants' mastering of the environment was expected for 90% (12/13) of the participants. RESULTS:The virtual ED was considered realistic with no significant difference from the real world with respect to facilities and resources, except for the length of time of procedures that was perceived to be shorter. A total of 100% (13/13) of participants deemed that patient information, decision making, and managing patient flow were similar to real clinical practice. The virtual environment was well-mastered by all participants over the course of the scenarios. CONCLUSIONS:The new simulation tool, Virtual Research Environment in Emergency Medicine, has been successfully designed and developed. It has been assessed as perfectly authentic by emergency physicians compared with real EDs and thus offers another way to study human factors, quality of care, and patient safety in the context of ED overcrowding.
| S-EPMC6705008 | biostudies-other
Project description:IntroductionMethodological studies (ie, studies that evaluate the design, conduct, analysis or reporting of other studies in health research) address various facets of health research including, for instance, data collection techniques, differences in approaches to analyses, reporting quality, adherence to guidelines or publication bias. As a result, methodological studies can help to identify knowledge gaps in the methodology of health research and strategies for improvement in research practices. Differences in methodological study names and a lack of reporting guidance contribute to lack of comparability across studies and difficulties in identifying relevant previous methodological studies. This paper outlines the methods we will use to develop an evidence-based tool-the MethodologIcal STudy reportIng Checklist-to harmonise naming conventions and improve the reporting of methodological studies.Methods and analysisWe will search for methodological studies in the Cumulative Index to Nursing and Allied Health Literature, Cochrane Library, Embase, MEDLINE, Web of Science, check reference lists and contact experts in the field. We will extract and summarise data on the study names, design and reporting features of the included methodological studies. Consensus on study terms and recommended reporting items will be achieved via video conference meetings with a panel of experts including researchers who have published methodological studies.Ethics and disseminationThe consensus study has been exempt from ethics review by the Hamilton Integrated Research Ethics Board. The results of the review and the reporting guideline will be disseminated in stakeholder meetings, conferences, peer-reviewed publications, in requests to journal editors (to endorse or make the guideline a requirement for authors), and on the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network and reporting guideline websites.RegistrationWe have registered the development of the reporting guideline with the EQUATOR Network and publicly posted this project on the Open Science Framework (www.osf.io/9hgbq).
| S-EPMC7747548 | biostudies-literature
Project description:ObjectivesTo assess the methodological quality of prognostic model development studies pertaining to post resection prognosis of pancreatic ductal adenocarcinoma (PDAC).Design/settingA narrative systematic review of international peer reviewed journals DATA SOURCE: Searches were conducted of: MEDLINE, Embase, PubMed, Cochrane database and Google Scholar for predictive modelling studies applied to the outcome of prognosis for patients with PDAC post resection. Predictive modelling studies in this context included prediction model development studies with and without external validation and external validation studies with model updating. Data was extracted following the Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) checklist.Primary and secondary outcome measuresPrimary outcomes were all components of the CHARMS checklist. Secondary outcomes included frequency of variables included across predictive models.Results263 studies underwent full text review. 15 studies met the inclusion criteria. 3 studies underwent external validation. Multivariable Cox proportional hazard regression was the most commonly employed modelling method (n=13). 10 studies were based on single centre databases. Five used prospective databases, seven used retrospective databases and three used cancer data registry. The mean number of candidate predictors was 19.47 (range 7 to 50). The most commonly included variables were tumour grade (n=9), age (n=8), tumour stage (n=7) and tumour size (n=5). Mean sample size was 1367 (range 50 to 6400). 5 studies reached statistical power. None of the studies reported blinding of outcome measurement for predictor values. The most common form of presentation was nomograms (n=5) and prognostic scores (n=5) followed by prognostic calculators (n=3) and prognostic index (n=2).ConclusionsAreas for improvement in future predictive model development have been highlighted relating to: general aspects of model development and reporting, applicability of models and sources of bias.Trial registration numberCRD42018105942.
| S-EPMC6707674 | biostudies-literature
Project description:This systematic review assessed the methodological quality of behavioural weight loss intervention studies conducted among adults and associations between quality and statistically significant weight loss outcome, strength of intervention effectiveness and sample size. Searches for trials published between January, 2009 and December, 2014 were conducted using PUBMED, MEDLINE and PSYCINFO and identified ninety studies. Methodological quality indicators included study design, anthropometric measurement approach, sample size calculations, intent-to-treat (ITT) analysis, loss to follow-up rate, missing data strategy, sampling strategy, report of treatment receipt and report of intervention fidelity (mean = 6.3). Indicators most commonly utilized included randomized design (100%), objectively measured anthropometrics (96.7%), ITT analysis (86.7%) and reporting treatment adherence (76.7%). Most studies (62.2%) had a follow-up rate > 75% and reported a loss to follow-up analytic strategy or minimal missing data (69.9%). Describing intervention fidelity (34.4%) and sampling from a known population (41.1%) were least common. Methodological quality was not associated with reporting a statistically significant result, effect size or sample size. This review found the published literature of behavioural weight loss trials to be of high quality for specific indicators, including study design and measurement. Identified for improvement include utilization of more rigorous statistical approaches to loss to follow up and better fidelity reporting.
| S-EPMC5996374 | biostudies-literature
Project description:BackgroundQUADOMICS is an adaptation of QUADAS (a quality assessment tool for use in systematic reviews of diagnostic accuracy studies), which takes into account the particular challenges presented by '-omics' based technologies. Our primary objective was to evaluate the applicability and consistency of QUADOMICS. Subsequently we evaluated and describe the methodological quality of a sample of recently published studies using the tool.Methodology/principal findings45'-omics'-based diagnostic studies were identified by systematic search of Pubmed using suitable MeSH terms ("Genomics", "Sensitivity and specificity", "Diagnosis"). Three investigators independently assessed the quality of the articles using QUADOMICS and met to compare observations and generate a consensus. Consistency and applicability was assessed by comparing each reviewer's original rating with the consensus. Methodological quality was described using the consensus rating. Agreement was above 80% for all three reviewers. Four items presented difficulties with application, mostly due to the lack of a clearly defined gold standard. Methodological quality of our sample was poor; studies met roughly half of the applied criteria (mean +/- sd, 54.7+/-18.4%). Few studies were carried out in a population that mirrored the clinical situation in which the test would be used in practice, (6, 13.3%); none described patient recruitment sufficiently; and less than half described clinical and physiological factors that might influence the biomarker profile (20, 44.4%).ConclusionsThe QUADOMICS tool can consistently be applied to diagnostic '-omics' studies presently published in biomedical journals. A substantial proportion of reports in this research field fail to address design issues that are fundamental to make inferences relevant for patient care.
| S-EPMC2896422 | biostudies-literature
Project description:Objectives: Rupture of a brain arteriovenous malformation (bAVM) can cause intracranial hemorrhage and severe clinical outcomes. At present, the mechanisms of bAVM-related hemorrhage are poorly understood. This study aimed to summarize the potential genetic risk factors for bAVM-related hemorrhage and appraise the methodological quality of existing genetic studies on bAVM-related hemorrhage using a cross-sectional design. Methods: A systematic literature search was conducted on genetic studies associated with bAVM-related hemorrhage published in PubMed, Embase, Web of Science, China National Knowledge Internet, and Wangfang databases, up to November 2022. Subsequently, a cross-sectional study was performed to describe the potential candidate genetic variants of bAVM associated with risk of hemorrhage and to evaluate the methodological quality of the identified studies using the Newcastle-Ottawa quality assessment scale and Q-genie tool. Results: Of the 1811 records identified in the initial search, nine studies met the filtering criteria and were included. Twelve single nucleotide polymorphisms (SNPs), including IL6 rs1800795, IL17A rs2275913, MMP9 rs9509, VEGFA rs1547651, and EPHB4 rs314353, rs314308, and rs314313, were associated with bAVM-related hemorrhage. However, only 12.5% of the evaluated SNPs showed statistical power> 0.80 (α = 0.05). Methodological quality assessment revealed significant flaws in the designs of the included studies, such as less reliable representativeness of recruited individuals, short follow-up periods in cohort studies, and less comparability between groups of hemorrhagic and non-hemorrhagic patients. Conclusion: IL1B, IL6, IL17A, APOE, MMP9, VEGFA and EPHB4 were potentially associated with bAVM-related hemorrhage. The methodological designs of the analyzed studies required improvement in order to obtain more reliable results. Regional alliances and rare disease banks need to be established to recruit large numbers of bAVM patients (especially familial and extreme-trait cases) in a multicenter, prospective cohort study with an adequate follow-up period. Furthermore, it is important to use advanced sequencing techniques and efficient measures to filter candidate genetic variants.
| S-EPMC10099571 | biostudies-literature
Project description:The methodological quality and reporting practices of laboratory studies of human eating behavior determine the validity and replicability of nutrition science. The aim of this research was to examine basic methodology and reporting practices in recent representative laboratory studies of human eating behavior. We examined laboratory studies of human eating behavior (N = 140 studies) published during 2016. Basic methodology (e.g., sample size, use of participant blinding) and reporting practices (e.g., information on participant characteristics) were assessed for each study. Some information relating to participant characteristics (e.g., age, gender) and study methodology (e.g., length of washout periods in within-subjects studies) were reported in the majority of studies. However, other aspects of study reporting, including participant eligibility criteria and how sample size was determined were frequently not reported. Studies often did not appear to standardize pre-test meal appetite or attempt to blind participants to study aims. The average sample size of studies was small (between-subjects design studies in particular) and the primary statistical analyses in a number of studies (24%) were reliant on very small sample sizes that would be likely to produce unreliable results. There are basic methodology and reporting practices in the laboratory study of human eating behavior that are sub-optimal and this is likely to be affecting the validity and replicability of research. Recommendations to address these issues are discussed.
| S-EPMC5890731 | biostudies-literature
Project description:The potential for conservation of individual species has been greatly advanced by the International Union for Conservation of Nature's (IUCN) development of objective, repeatable, and transparent criteria for assessing extinction risk that explicitly separate risk assessment from priority setting. At the IV World Conservation Congress in 2008, the process began to develop and implement comparable global standards for ecosystems. A working group established by the IUCN has begun formulating a system of quantitative categories and criteria, analogous to those used for species, for assigning levels of threat to ecosystems at local, regional, and global levels. A final system will require definitions of ecosystems; quantification of ecosystem status; identification of the stages of degradation and loss of ecosystems; proxy measures of risk (criteria); classification thresholds for these criteria; and standardized methods for performing assessments. The system will need to reflect the degree and rate of change in an ecosystem's extent, composition, structure, and function, and have its conceptual roots in ecological theory and empirical research. On the basis of these requirements and the hypothesis that ecosystem risk is a function of the risk of its component species, we propose a set of four criteria: recent declines in distribution or ecological function, historical total loss in distribution or ecological function, small distribution combined with decline, or very small distribution. Most work has focused on terrestrial ecosystems, but comparable thresholds and criteria for freshwater and marine ecosystems are also needed. These are the first steps in an international consultation process that will lead to a unified proposal to be presented at the next World Conservation Congress in 2012.
| S-EPMC3051828 | biostudies-literature