Dataset Information

Ignoring non-English-language studies may bias ecological meta-analyses.

ABSTRACT: Meta-analysis plays a crucial role in syntheses of quantitative evidence in ecology and biodiversity conservation. The reliability of estimates in meta-analyses strongly depends on unbiased sampling of primary studies. Although earlier studies have explored potential biases in ecological meta-analyses, biases in reported statistical results and associated study characteristics published in different languages have never been tested in environmental sciences. We address this knowledge gap by systematically searching published meta-analyses and comparing effect-size estimates between English- and Japanese-language studies included in existing meta-analyses. Of the 40 published ecological meta-analysis articles authored by those affiliated to Japanese institutions, we find that three meta-analysis articles searched for studies in the two languages and involved sufficient numbers of English- and Japanese-language studies, resulting in four eligible meta-analyses (i.e., four meta-analyses conducted in the three meta-analysis articles). In two of the four, effect sizes differ significantly between the English- and Japanese-language studies included in the meta-analyses, causing considerable changes in overall mean effect sizes and even their direction when Japanese-language studies are excluded. The observed differences in effect sizes are likely attributable to systematic differences in reported statistical results and associated study characteristics, particularly taxa and ecosystems, between English- and Japanese-language studies. Despite being based on a small sample size, our findings suggest that ignoring non-English-language studies may bias outcomes of ecological meta-analyses, due to systematic differences in study characteristics and effect-size estimates between English- and non-English languages. We provide a list of actions that meta-analysts could take in the future to reduce the risk of language bias.

SUBMITTER: Konno K

PROVIDER: S-EPMC7381574 | biostudies-literature | 2020 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Ignoring non-English-language studies may bias ecological meta-analyses.

Konno Ko K Akasaka Munemitsu M Koshida Chieko C Katayama Naoki N Osada Noriyuki N Spake Rebecca R Amano Tatsuya T

Ecology and evolution 20200529 13

Meta-analysis plays a crucial role in syntheses of quantitative evidence in ecology and biodiversity conservation. The reliability of estimates in meta-analyses strongly depends on unbiased sampling of primary studies. Although earlier studies have explored potential biases in ecological meta-analyses, biases in reported statistical results and associated study characteristics published in different languages have never been tested in environmental sciences. We address this knowledge gap by syst ...[more]

PMID: 32724519

Similar Datasets

Project description:BACKGROUND:Studies published in languages other than English are often neglected when research teams conduct systematic reviews. Literature on how to deal with non-English studies when conducting reviews have focused on the importance of including such studies, while less attention has been paid to the practical challenges of locating and assessing relevant non-English studies. We investigated the factors which might predict the inclusion of non-English studies in systematic reviews in the social sciences, to better understand how, when and why these are included/excluded. METHODS:We appraised all Campbell Collaboration systematic reviews (n = 123) published to July 2016, categorising each by its language inclusiveness. We sought additional information from review authors via a questionnaire and received responses concerning 47 reviews. Data were obtained for 17 factors and we explored correlations with the number of non-English studies in the reviews via statistical regression models. Additionally, we asked authors to identify factors that support or hinder the inclusion of non-English studies. RESULTS:Of 123 reviews, 108 did not explicitly exclude, and of these, 17 included non-English language studies. One factor correlated with the number of included non-English studies across all models: the number of countries in which the members of the review team work (B-value = 0.56; SE B = 0.24; 95% CI = 0.07-1.03; p = 0.02). This indicates that reviews which included non-English studies were more likely to be produced by international review teams. Our survey showed a dominance of researchers from English-speaking countries (52.9%) and review teams consisting only of team members from these countries (65.9%). The most frequently mentioned challenge to including non-English studies was a lack of resources (funding and time) followed by a lack of language resources (e.g. professional translators). CONCLUSION:Our findings may indicate a connection between the limited inclusion of non-English studies and a lack of resources, which forces review teams to rely on their limited language skills rather than the support of professional translators. If unaddressed, review teams risk ignoring key data and introduce bias in otherwise high-quality reviews. However, the validity and interpretation of our findings should be further assessed if we are to tackle the challenges of dealing with non-English studies.

Project description:Selective publication and reporting in individual papers compromise the scientific record, but are meta-analyses as compromised as their constituent studies? We systematically sampled 63 meta-analyses (each comprising at least 40 studies) in PLoS One, top medical journals, top psychology journals, and Metalab, an online, open-data database of developmental psychology meta-analyses. We empirically estimated publication bias in each, including only the peer-reviewed studies. Across all meta-analyses, we estimated that "statistically significant" results in the expected direction were only 1.17 times more likely to be published than "nonsignificant" results or those in the unexpected direction (95% CI: [0.93, 1.47]), with a confidence interval substantially overlapping the null. Comparable estimates were 0.83 for meta-analyses in PLoS One, 1.02 for top medical journals, 1.54 for top psychology journals, and 4.70 for Metalab. The severity of publication bias did differ across individual meta-analyses; in a small minority (10%; 95% CI: [2%, 21%]), publication bias appeared to favor "significant" results in the expected direction by more than threefold. We estimated that for 89% of meta-analyses, the amount of publication bias that would be required to attenuate the point estimate to the null exceeded the amount of publication bias estimated to be actually present in the vast majority of meta-analyses from the relevant scientific discipline (exceeding the 95th percentile of publication bias). Study-level measures ("statistical significance" with a point estimate in the expected direction and point estimate size) did not indicate more publication bias in higher-tier versus lower-tier journals, nor in the earliest studies published on a topic versus later studies. Overall, we conclude that the mere act of performing a meta-analysis with a large number of studies (at least 40) and that includes non-headline results may largely mitigate publication bias in meta-analyses, suggesting optimism about the validity of meta-analytic results.

Project description:BackgroundLanguage concordance can increase access to care for patients with language barriers and improve patient health outcomes. However, systematically assessing and tracking physician non-English language skills remains uncommon in most health systems. This is a missed opportunity for health systems to maximize language-concordant care.ObjectiveTo determine barriers and facilitators to participation in non-English language proficiency assessment among primary care physicians.DesignQualitative, semi-structured interviews.ParticipantsEleven fully and partially bilingual primary care physicians from a large academic health system with a language certification program (using a clinician oral proficiency interview).ApproachInterviews aimed to identify barriers and facilitators to participation in non-English language assessment. Two researchers independently and iteratively coded transcripts using a thematic analysis approach with constant comparison to identify themes.Key resultsMost participants were women (N= 9; 82%). Participants reported proficiency in Cantonese, Mandarin, Russian, and Spanish. All fully bilingual participants (n=5) had passed the language assessment; of the partially bilingual participants (n=6), four did not test, one passed with marginal proficiency, and one did not pass. Three themes emerged as barriers to assessment participation: (1) beliefs about the negative consequences (emotional and material) of not passing the test, (2) time constraints and competing demands, and (3) challenging test format and structure. Four themes emerged as facilitators to increase assessment adoption: (1) messaging consistent with professional ethos, (2) organizational culture that incentivizes certification, (3) personal empowerment about language proficiency, and (4) individuals championing certification.ConclusionsTo increase language assessment participation and thus ensure quality language-concordant care, health systems must address the identified barriers physicians experience and leverage potential facilitators. Findings can inform health system interventions to standardize the requirements and process, increase transparency, provide resources for preparation and remediation, utilize messaging focused on patient care quality and safety, and incentivize participation.

Dataset Information

Ignoring non-English-language studies may bias ecological meta-analyses.

Publications

Ignoring non-English-language studies may bias ecological meta-analyses.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets