When numbers fail: do researchers agree on operationalization of published research?
Ontology highlight
ABSTRACT:
SUBMITTER: Haucke M
PROVIDER: S-EPMC8424321 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:ObjectiveThe San Francisco Declaration on Research Assessment (DORA) advocates for assessing biomedical research quality and impact, yet academic organizations continue to employ traditional measures such as Journal Impact Factor. We aimed to identify and prioritize measures for assessing research quality and impact.MethodsWe conducted a review of published and grey literature to identify measures of research quality and impact, which we included in an online survey. We assembled a panel of researchers and research leaders, and conducted a two-round Delphi survey to prioritize measures rated as high (rated 6 or 7 by ≥ 80% of respondents) or moderate (rated 6 or 7 by ≥ 50% of respondents) importance.ResultsWe identified 50 measures organized in 8 domains: relevance of the research program, challenges to research program, or productivity, team/open science, funding, innovations, publications, other dissemination, and impact. Rating of measures by 44 panelists (60%) in Round One and 24 (55%) in Round Two of a Delphi survey resulted in consensus on the high importance of 5 measures: research advances existing knowledge, research plan is innovative, an independent body of research (or fundamental role) supported by peer-reviewed research funding, research outputs relevant to discipline, and quality of the content of publications. Five measures achieved consensus on moderate importance: challenges to research productivity, potential to improve health or healthcare, team science, collaboration, and recognition by professional societies or academic bodies. There was high congruence between researchers and research leaders across disciplines.ConclusionsOur work contributes to the field by identifying 10 DORA-compliant measures of research quality and impact, a more comprehensive and explicit set of measures than prior efforts. Research is needed to identify strategies to overcome barriers of use of DORA-compliant measures, and to "de-implement" traditional measures that do not uphold DORA principles yet are still in use.
| S-EPMC10180594 | biostudies-literature
Project description:BackgroundKnowledge translation (KT) is a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically sound application of knowledge to yield beneficial outcomes for society. Effective KT requires researchers to play an active role in promoting evidence uptake. This paper presents a systematised review of evidence on low- and middle-income country (LMIC) researchers' KT capacity, practice and interventions for enhancing their KT practice (support) with the aim of identifying gaps and informing future research and interventions.MethodsAn electronic search for peer-reviewed publications focusing on LMIC researchers' KT capacity, practice and support across all academic fields, authored in English and from the earliest records available to February 2019, was conducted using PubMed and Scopus. Selected studies were appraised using the Mixed Methods Appraisal Tool, data pertaining to publication characteristics and study design extracted, and an a priori thematic analysis of reported research findings completed.ResultsThe search resulted in 334 screened articles, of which 66 met the inclusion criteria. Most (n = 43) of the articles presented original research findings, 22 were commentaries and 1 was a structured review; 47 articles reported on researchers' KT practice, 12 assessed the KT capacity of researchers or academic/research institutions and 9 reported on KT support for researchers. More than half (59%) of the articles focused on sub-Saharan Africa and the majority (91%) on health research. Most of the primary studies used the case study design (41%). The findings suggest that LMIC researchers rarely conduct KT and face a range of barriers at individual and institutional levels that limit their KT practice, including inadequate KT knowledge and skills, particularly for communicating research and interacting with research end-users, insufficient funding, and inadequate institutional guidelines, structures and incentives promoting KT practice. Furthermore, the evidence-base on effective interventions for enhancing LMIC researchers' KT practice is insufficient and largely of weak quality.ConclusionsMore high-quality research on researchers' KT capacity, practice and effective KT capacity strengthening interventions is needed. Study designs that extend beyond case studies and descriptive studies are recommended, including better designed evaluation studies, e.g. use of realist approaches, pragmatic trials, impact evaluations, implementation research and participatory action research.
| S-EPMC7011245 | biostudies-literature
Project description:BackgroundReports of questionable or detrimental research practices (QRPs) call into question the reliability of scientific evidence and the trustworthiness of research. A critical component of the research ecosystem is the organization within which research takes place. We conducted a survey to explore the attitudes and beliefs of European and American researchers about the organisations in which they work, their own research practices and their attitudes towards research integrity and research integrity policies.MethodsWe administered an online survey (International Research Integrity Survey (IRIS)) to 2,300 active researchers based in the US and 45,000 in Europe (including UK, Norway, Iceland and Switzerland). We employed a stratified probability sample of the authors of research articles published between 2016 and 2020 included in Clarivate's Web of Science citation database. Coverage includes researchers in the humanities, social sciences, natural sciences and medical sciences, who hold at least a master's level degree.ResultsIn comparison to researchers in the US, European researchers admit to more QRPs and are less confident in maintaining high research integrity (RI) standards. In the US and Europe, many researchers judge their organization to fall short of best RI practice. All researchers recognize the benefits of RI, reliable knowledge and the trust of colleagues and the public, and there is support for RI training particularly among Europeans.ConclusionTo create and maintain a culture of integrity in scientific research, a collective commitment from researchers, their institutions and funders is needed. Researchers rely on many channels of communication about research integrity and thus the involvement of many different participants in the research system is required to make improvements. Policies must be developed to reinforce best practice rather than being seen as an irrelevance to the real business of research.
| S-EPMC10349267 | biostudies-literature
Project description:Participation in and opportunities for physical activity (PA) and sports (PA inclusively hereafter) are known to vary across individuals with different social positions. Intersectionality theory may help us to better understand the complex processes of multiple interlocking systems of oppression and privilege shaped by intersections of individuals' social categories. The objectives of this systematic scoping review were (1) to summarize the findings of articles examining PA claimed operationalization of intersectionality and (2) to identify the scope and gaps pertaining to the operationalization of intersectionality in PA research. A search was conducted in September 2019 in seven electronic databases (e.g., SPORTDiscus, Scopus, Web of Science) for relevant research articles written in English. Key search terms included "intersectionality" AND "physical activity" OR "sport". Database searches, data screening and extraction, and narrative synthesis were conducted between September 2019 and May 2020. Of 16564 articles identified, 45 articles were included in this review. The majority of included articles used qualitative methods (n = 41), with two quantitative and two mixed-methods articles. The most frequently observed intersectional social position was sex/gender + race/ethnicity (n = 11), followed by sex/gender + race/ethnicity + sexuality (n = 6) and sex/gender + race/ethnicity + religion (n = 6). Most qualitative studies (n = 38) explicitly claimed operationalization of intersectionality as a key theoretical framework, and over half of these studies (n = 27) implicitly used intra-categorical intersectionality. Two quantitative studies were identified which examined a number of intersections simultaneously using inter-categorical intersectionality. Complex processes of individual and social-structural level factors that drive inequalities in PA opportunities and participation could be better elucidated with the operationalization of intersectionality theory. Intersectionality theory may serve as a useful framework in both qualitative and quantitative investigations. Advancement in quantitative intersectionality is critical in order to produce knowledge that could inform more inclusive PA promotion efforts.
| S-EPMC8182115 | biostudies-literature
Project description:BackgroundThe objective of this study was to evaluate the nature and extent of reproducible and transparent research practices in neurology publications.MethodsThe NLM catalog was used to identify MEDLINE-indexed neurology journals. A PubMed search of these journals was conducted to retrieve publications over a 5-year period from 2014 to 2018. A random sample of publications was extracted. Two authors conducted data extraction in a blinded, duplicate fashion using a pilot-tested Google form. This form prompted data extractors to determine whether publications provided access to items such as study materials, raw data, analysis scripts, and protocols. In addition, we determined if the publication was included in a replication study or systematic review, was preregistered, had a conflict of interest declaration, specified funding sources, and was open access.ResultsOur search identified 223,932 publications meeting the inclusion criteria, from which 400 were randomly sampled. Only 389 articles were accessible, yielding 271 publications with empirical data for analysis. Our results indicate that 9.4% provided access to materials, 9.2% provided access to raw data, 0.7% provided access to the analysis scripts, 0.7% linked the protocol, and 3.7% were preregistered. A third of sampled publications lacked funding or conflict of interest statements. No publications from our sample were included in replication studies, but a fifth were cited in a systematic review or meta-analysis.ConclusionsCurrently, published neurology research does not consistently provide information needed for reproducibility. The implications of poor research reporting can both affect patient care and increase research waste. Collaborative intervention by authors, peer reviewers, journals, and funding sources is needed to mitigate this problem.
| S-EPMC7049215 | biostudies-literature
Project description:Many psychology studies are statistically underpowered. In part, this may be because many researchers rely on intuition, rules of thumb, and prior practice (along with practical considerations) to determine the number of subjects to test. In Study 1, we surveyed 291 published research psychologists and found large discrepancies between their reports of their preferred amount of power and the actual power of their studies (calculated from their reported typical cell size, typical effect size, and acceptable alpha). Furthermore, in Study 2, 89% of the 214 respondents overestimated the power of specific research designs with a small expected effect size, and 95% underestimated the sample size needed to obtain .80 power for detecting a small effect. Neither researchers' experience nor their knowledge predicted the bias in their self-reported power intuitions. Because many respondents reported that they based their sample sizes on rules of thumb or common practice in the field, we recommend that researchers conduct and report formal power analyses for their studies.
| S-EPMC4976648 | biostudies-literature
Project description:Background:There is increasing evidence that research misbehaviour is common, especially the minor forms. Previous studies on research misbehaviour primarily focused on biomedical and social sciences, and evidence from natural sciences and humanities is scarce. We investigated what academic researchers in Amsterdam perceived to be detrimental research misbehaviours in their respective disciplinary fields. Methods:We used an explanatory sequential mixed methods design. First, survey participants from four disciplinary fields rated perceived frequency and impact of research misbehaviours from a list of 60. We then combined these into a top five ranking of most detrimental research misbehaviours at the aggregate level, stratified by disciplinary field. Second, in focus group interviews, participants from each academic rank and disciplinary field were asked to reflect on the most relevant research misbehaviours for their disciplinary field. We used participative ranking methodology inducing participants to obtain consensus on which research misbehaviours are most detrimental. Results:In total, 1080 researchers completed the survey (response rate: 15%) and 61 participated in the focus groups (3 three to 8 eight researchers per group). Insufficient supervision consistently ranked highest in the survey regardless of disciplinary field and the focus groups confirmed this. Important themes in the focus groups were insufficient supervision, sloppy science, and sloppy peer review. Biomedical researchers and social science researchers were primarily concerned with sloppy science and insufficient supervision. Natural sciences and humanities researchers discussed sloppy reviewing and theft of ideas by reviewers, a form of plagiarism. Focus group participants further provided examples of particular research misbehaviours they were confronted with and how these impacted their work as a researcher. Conclusion:We found insufficient supervision and various forms of sloppy science to score highly on aggregate detrimental impact throughout all disciplinary fields. Researchers from the natural sciences and humanities also perceived nepotism to be of major impact on the aggregate level. The natural sciences regarded fabrication of data of major impact as well. The focus group interviews helped to understand how researchers interpreted 'insufficient supervision'. Besides, the focus group participants added insight into sloppy science in practice. Researchers from the natural sciences and humanities added new research misbehaviours concerning their disciplinary fields to the list, such as the stealing of ideas before publication. This improves our understanding of research misbehaviour beyond the social and biomedical fields.
| S-EPMC6886174 | biostudies-literature
Project description:With the increase in patient and consumer activism through the late twentieth century and into this century, patient roles in research evolved into a new model of research engagement, with patients serving as active advisors and co-leading or leading clinical research. By requiring active engagement of patients and other stakeholders, several government research funders have advanced this model, particularly in Canada, the United States (US), United Kingdom (UK), and Australia. A consortium of individuals from these countries formed a Multi-Stakeholder Engagement (MuSE) consortium to examine critical issues in engaged research, establish consensus on definitions, and provide guidance for the field, beginning with an overview of how to involve stakeholders in health research (Concannon et al. J Gen Intern Med. 2019;34(3):458-463) and continuing here with an examination of definitions of research engagement. The political and advocacy roots of engaged research are reflected in definitions. Engagement is conceptualized with reference to research project goals, from informing specific clinical decisions to informing health-system level decisions. Political and cultural differences across countries are evident. Some of these government funders focus on empirical rather than ethical rationales. In countries with centralized health technology assessment, the link between societal values and engaged research is explicit. Ethical rationales for engagement are explicit in most of the published literature on research engagement. Harmonization of definitions is recommended so that research engagement elements, methods, and outcomes and impacts can be clearly examined and understood, and so that the field of research engagement can proceed from a clear conceptual foundation. Specific recommendations for terminology definitions are provided. Placing engaged research on a continuum from specific clinical decisions to more global public and social justice concerns clarifies the type of engaged research, supports appropriate comparisons, and improves the rigor of engaged research methods. The results help identify knowledge gaps in this growing field.
| S-EPMC7048327 | biostudies-literature
Project description:Authorship is commonly used as the basis for the measurement of research productivity. It influences career progression and rewards, making it a valued commodity in a competitive scientific environment. To better understand authorship practices amongst collaborative teams, this study surveyed authors on collaborative journal articles published between 2011 and 2015. Of the 8364 respondents, 1408 responded to the final open-ended question, which solicited additional comments or remarks regarding the fair distribution of authorship in research teams. This paper presents the analysis of these comments, categorized into four main themes: (1) disagreements, (2) questionable behavior, (3) external influences regarding authorship, and (4) values promoted by researchers. Results suggest that some respondents find ways to effectively manage disagreements in a collegial fashion. Conversely, others explain how distribution of authorship can become a "blood sport" or a "horror story" which can negatively affect researchers' wellbeing, scientific productivity and integrity. Researchers fear authorship discussions and often try to avoid openly discussing the situation which can strain team interactions. Unethical conduct is more likely to result from deceit, favoritism, and questionable mentorship and may become more egregious when there is constant bullying and discrimination. Although values of collegiality, transparency and fairness were promoted by researchers, rank and need for success often overpowered ethical decision-making. This research provides new insight into contextual specificities related to fair authorship distribution that can be instrumental in developing applicable training tools to identify, prevent, and mitigate authorship disagreement.
| S-EPMC6891155 | biostudies-literature
Project description:BackgroundAlthough improving, development of drugs and devices for children is still less effective than for adults. Pediatric academicians play an important role in the bench-to-bedside research process, but much remains to be done to improve their contributions.ObjectiveTo provide a non-comprehensive review of selected literature based on my own personal experience as a U.S. based academic researcher who has spent over 4 decades doing pediatric drug and device development.MethodsThis commentary presents a summary of a talk given at a recent pediatric drug development conference. The observations and conclusions reached were based on the author's (largely US) experience and review of past history, the role of academicians in this process, some successful models of public-private collaboration, available funding, and barriers that remain to be overcome.ResultsPediatric-specific legislation and more available funding have increased participation from and successes of US academicians in the pediatric drug and device development process. Incentive based public-private collaborations have been particularly successful. However, academicians still face both attitude and practical barriers to success.ConclusionsChanges are needed if academicians are to maximize their involvement in pediatric drug and device development.
| S-EPMC6677781 | biostudies-literature