Dataset Information

Measuring the Impact of Research: Lessons from the UK's Research Excellence Framework 2014.

ABSTRACT: Impactful academic research plays a stellar role in society, pressing to ask the question of how one measures the impact created by different areas of academic research. Measuring the societal, cultural, economic and scientific impact of research is currently the priority of the National Science Foundation, European Commission and several research funding agencies. The recently concluded United Kingdom's national research quality exercise, the Research Excellence Framework (REF) 2014, which piloted impact assessment as part of the overall evaluation offers a lens to view how impact of research in different disciplines can be measured. Overall research quality was assessed through quality of outputs, 'impact' and research environment. We performed two studies using the REF 2014 as a case study. The first study on 363 Impact Case Studies (ICSs) submitted in 5 research areas (UoAs) reveals that, in general, the impact scores were constructed upon a combination of factors i.e. quantity of quartile-one (Q1) publications, quantity and value of grants/income, number of researchers stated in the ICSs, spin-offs created, discoveries/patents and presentation of esteem data, informing researchers/ academics of the factors to consider in order to achieve a better impact score in research impact assessments. However, there were differences among disciplines in terms of the role played by the factors in achieving their overall scores for the ICSs. The outcome of this study is thus a set of impact indicators, and their relationship with the overall score of impact of research in different disciplines as determined in REF2014, which would in the first instance provide some answers to impact measures that would be useful for researchers in different disciplines. The second study extracts the general themes of impact reported by universities by performing a word frequency analysis in all the ICSs submitted in the five chosen research areas, which were substantially varied owing to their fields.

SUBMITTER: Chowdhury G

PROVIDER: S-EPMC4898824 | biostudies-literature | 2016

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Measuring the Impact of Research: Lessons from the UK's Research Excellence Framework 2014.

Chowdhury Gobinda G Koya Kushwanth K Philipson Pete P

PloS one 20160608 6

Impactful academic research plays a stellar role in society, pressing to ask the question of how one measures the impact created by different areas of academic research. Measuring the societal, cultural, economic and scientific impact of research is currently the priority of the National Science Foundation, European Commission and several research funding agencies. The recently concluded United Kingdom's national research quality exercise, the Research Excellence Framework (REF) 2014, which pilo ...[more]

PMID: 27276219

Similar Datasets

Project description:INTRODUCTION:An impactful clinical trial will have real-life benefits for patients and society beyond the academic environment. This study analyses case studies of cancer trials to understand how impact is evidenced for cancer trials and how impact evaluation can be more routinely adopted and improved. METHODS:The United Kingdom (UK) Government allocates research funding to higher-education institutions based on an assessment of the institutions' previous research efforts, in an exercise known as the Research Excellence Framework (REF). In addition to each institution's journal publications and research environment, for the first time in 2014, allocation of funding was also dependent on an evaluation of the wider, societal impact of research conducted. In the REF2014, impact assessment was performed by evaluation of impact case studies. In this study, case studies (n?=?6637) submitted by institutions for the REF2014 were accessed and those focussing on cancer trials were identified. Manual content analysis was then used to assess the characteristics of the cancer trials discussed in the case studies, the impact described and the methods used by institutions to demonstrate impact. RESULTS:Forty-six case studies describing 106 individual cancer trials were identified. The majority were phase III randomised controlled trials and those recruiting patients with breast cancer. A list of indicators of cancer trial impact was generated using the previous literature and developed inductively using these case studies. The most common impact from a cancer trial identified in the case studies was on policy, in particular citation of trial findings in clinical guidelines. Impact on health outcomes and the economy were less frequent and health outcomes were often predicted rather than evidenced. There were few descriptions identified of trialists making efforts to maximise trial impact. DISCUSSION:Cancer trial impact narratives for the next REF assessment exercise in 2021 can be improved by evidencing actual rather than predicted Impact, with a clearer identification of the beneficiaries of cancer trials and the processes through which trial results are used. Clarification of the individuals responsible for performing impact evaluations of cancer trials and the provision of resources to do so needs to be addressed if impact evaluation is to be sustainable.

Project description:BackgroundProviding evidence of impact highlights the benefits of medical research to society. Such evidence is increasingly requested by research funders and commonly relies on citation analysis. However, other indicators may be more informative. Although frameworks to demonstrate the impact of clinical research have been reported, no complementary framework exists for methodological research. Therefore, we assessed the impact of methodological research projects conducted or completed between 2009 and 2012 at the UK Medical Research Council Clinical Trials Unit Hub for Trials Methodology Research Hub, with a view to developing an appropriate framework.MethodsVarious approaches to the collection of data on research impact were employed. Citation rates were obtained using Web of Science (http://www.webofknowledge.com/) and analyzed descriptively. Semistructured interviews were conducted to obtain information on the rates of different types of research output that indicated impact for each project. Results were then pooled across all projects. Finally, email queries pertaining to methodology projects were collected retrospectively and their content analyzed.ResultsSimple citation analysis established the citation rates per year since publication for 74 methodological publications; however, further detailed analysis revealed more about the potential influence of these citations. Interviews that spanned 20 individual research projects demonstrated a variety of types of impact not otherwise collated, for example, applications and further developments of the research; release of software and provision of guidance materials to facilitate uptake; formation of new collaborations and broad dissemination. Finally, 194 email queries relating to 6 methodological projects were received from 170 individuals across 23 countries. They provided further evidence that the methodologies were impacting on research and research practice, both nationally and internationally. We have used the information gathered in this study to adapt an existing framework for impact of clinical research for use in methodological research.ConclusionsGathering evidence on research impact of methodological research from a variety of sources has enabled us to obtain multiple indicators and thus to demonstrate broad impacts of methodological research. The adapted framework developed can be applied to future methodological research and thus provides a tool for methodologists to better assess and report research impacts.

Project description:BackgroundResearch impact is an emerging measure of research achievement alongside traditional academic outputs such as publications. We present the results of applying the Framework to Assess the Impact from Translational health research (FAIT) to the Centre for Research Excellence (CRE) in Stroke Rehabilitation and Brain Recovery (CRE-Stroke, 2014-2019) and report on the feasibility and lessons from the application of FAIT to a CRE rather than a discrete research project.MethodsData were gathered via online surveys, in-depth interviews, document analysis and review of relevant websites/databases to report on the three major FAIT methods: the modified Payback Framework, an assessment of costs against monetized consequences, and a narrative account of the impact generated from CRE-Stroke activities. FAIT was applied during the last 4 years of CRE-Stroke operation.ResultsWith an economic investment of AU$ 3.9 million over 5 years, CRE-Stroke delivered a return on investment that included AU$ 18.8 million in leveraged grants, fellowships and consultancies. Collectively, CRE-Stroke members produced 354 publications that were accessed 470,000 times and cited over 7220 times. CRE-Stroke supported 26 PhDs, 39 postdocs and seven novice clinician researchers. There were 59 capacity-building events benefiting 744 individuals including policy-makers and consumers. CRE-Stroke created research infrastructure (including a research register of stroke survivors and a brain biobank), and its global leadership produced international consensus recommendations to influence the stroke research landscape worldwide. Members contributed to the Australian Living Stroke Guidelines: four researchers' outputs were directly referenced. Based only on the consequences that could be monetized, CRE-Stroke returned AU$ 4.82 for every dollar invested in the CRE.ConclusionThis case example in the developing field of impact assessment illustrates how researchers can use evidence to demonstrate and report the impact of and returns on research investment. The prospective application of FAIT by a dedicated research impact team demonstrated impact in broad categories of knowledge-gain, capacity-building, new infrastructure, input to policy and economic benefits. The methods can be used by other research teams to provide comprehensive evidence to governments and other research funders about what has been generated from their research investment but requires dedicated resources to complete.

Project description:BackgroundMeasuring research impact is of critical interest to philanthropic and government funding agencies interested in ensuring that the research they fund is both scientifically excellent and has meaningful impact into health and other outcomes. The Beat Cancer Project (BCP) is a AUD $34 m cancer research funding scheme that commenced in 2011. It was initiated by an Australian charity (Cancer Council SA), and supported by the South Australian Government and the state's major universities.MethodsThis study applied Buxton and Hanney's Payback Framework to assess research impact generated from the BCP after 3 years of funding. Data sources were an audit of peer-reviewed publications from January 2011 to September 2014 from Web of Knowledge and a self-report survey of investigators awarded BCP research funding during its first 3 years of implementation (2011-2013). Of the 104 surveys, 92 (88%) were completed.ResultsThe BCP performed well across all five categories of the Payback Framework. In terms of knowledge production, 1257 peer-reviewed publications were generated and the mean impact factor of publishing journals increased annually. There were many benefits to future research with 21 respondents (23%) reporting career advancement, and 110 higher degrees obtained or expected (including 84 PhDs). Overall, 52% of funded projects generated tools for future research. The funded research attracted substantial further income yielding a very high rate of leverage. For every AUD $1 that the cancer charity invested, the BCP gained an additional AUD $6.06. Five projects (5%) had informed policy and 5 (5%) informed product development, with an additional 31 (34%) and 35 (38%) projects, respectively, anticipating doing so. In terms of health and sector and broader economic benefits, 8 (9%) projects had influenced practice or behaviour of health staff and 32 (34%) would reportedly to do so in the future.ConclusionsResearch impact was a priority of charity and government funders and led to a deliberate funding strategy. Emphasising research impact while maintaining rigorous, competitive processes can achieve the joint objectives of excellence in research, yielding good research impact and a high rate of leverage for philanthropic and public investment, as indicated by these early results.

Dataset Information

Measuring the Impact of Research: Lessons from the UK's Research Excellence Framework 2014.

Publications

Measuring the Impact of Research: Lessons from the UK's Research Excellence Framework 2014.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets