How has healthcare research performance been assessed?: a systematic review.
Ontology highlight
ABSTRACT:
SUBMITTER: Patel VM
PROVIDER: S-EPMC3110970 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:ObjectivesA key barrier in supporting health research capacity development (HRCD) is the lack of empirical measurement of competencies to assess skills and identify gaps in research activities. An effective tool to measure HRCD in healthcare workers would help inform teams to undertake more locally led research. The objective of this systematic review is to identify tools measuring healthcare workers' individual capacities to conduct research.DesignSystematic review and narrative synthesis using Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist for reporting systematic reviews and narrative synthesis and the Critical Appraisals Skills Programme (CASP) checklist for qualitative studies.Data sources11 databases were searched from inception to 16 January 2020. The first 10 pages of Google Scholar results were also screened.Eligibility criteriaWe included papers describing the use of tools/to measure/assess HRCD at an individual level among healthcare workers involved in research. Qualitative, mixed and quantitative methods were all eligible. Search was limited to English language only.Data extraction and synthesisTwo authors independently screened and reviewed studies using Covidence software, and performed quality assessments using the extraction log validated against the CASP qualitative checklist. The content method was used to define a narrative synthesis.ResultsThe titles and abstracts for 7474 unique records were screened and the full texts of 178 references were reviewed. 16 papers were selected: 7 quantitative studies; 1 qualitative study; 5 mixed methods studies; and 3 studies describing the creation of a tool. Tools with different levels of accuracy in measuring HRCD in healthcare workers at the individual level were described. The Research Capacity and Culture tool and the 'Research Spider' tool were the most commonly defined. Other tools designed for ad hoc interventions with good generalisability potential were identified. Three papers described health research core competency frameworks. All tools measured HRCD in healthcare workers at an individual level with the majority adding a measurement at the team/organisational level, or data about perceived barriers and motivators for conducting health research.ConclusionsCapacity building is commonly identified with pre/postintervention evaluations without using a specific tool. This shows the need for a clear distinction between measuring the outcomes of training activities in a team/organisation, and effective actions promoting HRCD. This review highlights the lack of globally applicable comprehensive tools to provide comparable, standardised and consistent measurements of research competencies.Prospero registration numberCRD42019122310.
| S-EPMC8264867 | biostudies-literature
Project description:A high proportion of people contact healthcare services in the 12 months prior to death by suicide. Identifying people at high-risk for suicide is therefore a key concern for healthcare services. Whilst there is extensive research on the validity and reliability of suicide risk assessment tools, there remains a lack of understanding of how suicide risk assessments are conducted by healthcare staff in practice. This scoping review examined the literature on how suicide risk assessments are conducted and experienced by healthcare practitioners, patients, carers, relatives, and friends of people who have died by suicide in the UK. Literature searches were conducted on key databases using a pre-defined search strategy pre-registered with the Open Science Framework and following the PRISMA extension for scoping reviews guidelines. Eligible for inclusion were original research, written in English, exploring how suicide risk is assessed in the UK, related to administering or undergoing risk assessment for suicide, key concepts relating to those experiences, or directly exploring the experiences of administering or undergoing assessment. Eighteen studies were included in the final sample. Information was charted including study setting and design, sampling strategy, sample characteristics, and findings. A narrative account of the literature is provided. There was considerable variation regarding how suicide risk assessments are conducted in practice. There was evidence of a lack of risk assessment training, low awareness of suicide prevention guidance, and a lack of evidence relating to patient perspectives of suicide risk assessments. Increased inclusion of patient perspectives of suicide risk assessment is needed to gain understanding of how the process can be improved. Limited time and difficulty in starting an open discussion about suicide with patients were noted as barriers to successful assessment. Implications for practice are discussed.
| S-EPMC10045566 | biostudies-literature
Project description:BackgroundOver the last decade, various pay-for-performance (P4P) programs have been implemented to improve quality in health systems, including the VHA. P4P programs are complex, and their effects may vary by design, context, and other implementation processes. We conducted a systematic review and key informant (KI) interviews to better understand the implementation factors that modify the effectiveness of P4P.MethodsWe searched PubMed, PsycINFO, and CINAHL through April 2014, and reviewed reference lists. We included trials and observational studies of P4P implementation. Two investigators abstracted data and assessed study quality. We interviewed P4P researchers to gain further insight.ResultsAmong 1363 titles and abstracts, we selected 509 for full-text review, and included 41 primary studies. Of these 41 studies, 33 examined P4P programs in ambulatory settings, 7 targeted hospitals, and 1 study applied to nursing homes. Related to implementation, 13 studies examined program design, 8 examined implementation processes, 6 the outer setting, 18 the inner setting, and 5 provider characteristics. Results suggest the importance of considering underlying payment models and using statistically stringent methods of composite measure development, and ensuring that high-quality care will be maintained after incentive removal. We found no conclusive evidence that provider or practice characteristics relate to P4P effectiveness. Interviews with 14 KIs supported limited evidence that effective P4P program measures should be aligned with organizational goals, that incentive structures should be carefully considered, and that factors such as a strong infrastructure and public reporting may have a large influence.DiscussionThere is limited evidence from which to draw firm conclusions related to P4P implementation. Findings from studies and KI interviews suggest that P4P programs should undergo regular evaluation and should target areas of poor performance. Additionally, measures and incentives should align with organizational priorities, and programs should allow for changes over time in response to data and provider input.
| S-EPMC4803682 | biostudies-literature
Project description:BackgroundThere is growing consensus on the importance of identifying age-related inequities in the receipt of public health and healthcare interventions, but concerns regarding conceptual and methodological rigour in this area of research. Establishing age inequity in receipt requires evidence of a difference that is not an artefact of poor measurement of need or receipt; is not warranted on the grounds of patient preference or clinical safety; and is judged to be unfair.MethodA systematic, thematic literature review was undertaken with the objective of characterising recent research approaches. Studies were eligible if the population was in a country within the Organisation for Economic Co-operation and Development and analyses included an explicit focus on age-related patterns of healthcare receipt including those 60 years or older. A structured extraction template was applied. Extracted material was synthesised in thematic memos. A set of categorical codes were then defined and applied to produce summary counts across key dimensions. This process was iterative to allow reconciliation of discrepancies and ensure reliability.ResultsForty nine studies met the eligibility criteria. A wide variety of concepts, terms and methodologies were used across these studies. Thirty five studies employed multivariable techniques to produce adjusted receipt-need ratios, though few clearly articulated their rationale, indicating the need for great conceptual clarity. Eighteen studies made reference to patient preference as a relevant consideration, but just one incorporated any kind of adjustment for this factor. Twenty five studies discussed effectiveness among older adults, with fourteen raising the possibility of differential effectiveness, and one differential cost-effectiveness, by age. Just three studies made explicit reference to the ethical nature of healthcare resource allocation by age. While many authors presented suitably cautious conclusions, some appeared to over-stretch their findings concluding that observed differences were 'inequitable'. Limitations include possible biases in the retrieved material due to inconsistent database indexing and a focus on OECD country populations and studies with English titles.ConclusionsCaution is needed among clinicians and other evidence-users in accepting claims of healthcare 'ageism' in some published papers. Principles for improved research practice are proposed.
| S-EPMC5505033 | biostudies-literature
Project description:BackgroundIncreasingly, researchers need to demonstrate the impact of their research to their sponsors, funders, and fellow academics. However, the most appropriate way of measuring the impact of healthcare research is subject to debate. We aimed to identify the existing methodological frameworks used to measure healthcare research impact and to summarise the common themes and metrics in an impact matrix.Methods and findingsTwo independent investigators systematically searched the Medical Literature Analysis and Retrieval System Online (MEDLINE), the Excerpta Medica Database (EMBASE), the Cumulative Index to Nursing and Allied Health Literature (CINAHL+), the Health Management Information Consortium, and the Journal of Research Evaluation from inception until May 2017 for publications that presented a methodological framework for research impact. We then summarised the common concepts and themes across methodological frameworks and identified the metrics used to evaluate differing forms of impact. Twenty-four unique methodological frameworks were identified, addressing 5 broad categories of impact: (1) 'primary research-related impact', (2) 'influence on policy making', (3) 'health and health systems impact', (4) 'health-related and societal impact', and (5) 'broader economic impact'. These categories were subdivided into 16 common impact subgroups. Authors of the included publications proposed 80 different metrics aimed at measuring impact in these areas. The main limitation of the study was the potential exclusion of relevant articles, as a consequence of the poor indexing of the databases searched.ConclusionsThe measurement of research impact is an essential exercise to help direct the allocation of limited research resources, to maximise research benefit, and to help minimise research waste. This review provides a collective summary of existing methodological frameworks for research impact, which funders may use to inform the measurement of research impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their research.
| S-EPMC5549933 | biostudies-literature
Project description:BACKGROUND: Organisational culture is an anthropological metaphor used to inform research and consultancy and to explain organisational environments. In recent years, increasing emphasis has been placed on the need to change organisational culture in order to improve healthcare performance. However, the precise function of organisational culture in healthcare policy often remains underspecified and the desirability and feasibility of strategies to be adopted have been called into question. The objective of this review was to determine the effectiveness of strategies to change organisational culture in order to improve healthcare performance. METHODS: We searched the following electronic databases: The Cochrane Central Register of Controlled Trials, MEDLINE, EMBASE, CINAHL, Sociological Abstracts, Web of Knowledge, PsycINFO, Business and Management, EThOS, Index to Theses, Intute, HMIC, SIGLE, and Scopus until October 2009. The Database of Abstracts of Reviews of Effectiveness (DARE) was searched for related reviews. We also searched the reference lists of all papers and relevant reviews identified, and we contacted experts in the field for advice on further potential studies. We considered randomised controlled trials (RCTs) or well designed quasi-experimental studies (controlled clinical trials (CCTs), controlled before and after studies (CBAs), and interrupted time series (ITS) analyses). Studies could be set in any type of healthcare organisation in which strategies to change organisational culture in order to improve healthcare performance were applied. Our main outcomes were objective measures of professional performance and patient outcome. RESULTS: The search strategy yielded 4,239 records. After the full text assessment, two CBA studies were included in the review. They both assessed the impact of interventions aimed at changing organisational culture, but one evaluated the impact on work-related and personal outcomes while the other measured clinical outcomes. Both were at high risk of bias. Both reported positive results. CONCLUSIONS: Current available evidence does not identify any effective, generalisable strategies to change organisational culture. Healthcare organisations considering implementing interventions aimed at changing culture should seriously consider conducting an evaluation (using a robust design, e.g., ITS) to strengthen the evidence about this topic.
| S-EPMC3080823 | biostudies-other
Project description:ObjectivesTo assess the use and utility of the Freedom of Information Act (2000) in healthcare research since 2005 and to determine if any particular feature of studies found led to greater data acquisition.DesignPRISMA compliant systematic review.ParticipantsAn extensive literature search was performed of EMBASE, MEDLINE, CINAHL, psychINFO, BNI, AMED, HMIC and Health business elite databases from January 2005 to January 2013 using terms 'Freedom of information', 'Freedom of information act' and 'Freedom of information act 2000'. Papers were considered for publication if they described utilising the UK Freedom of information act to gather data for healthcare research. 16 articles met these criteria.Primary and secondary outcome measuresPrimary outcome was the number and characteristics of studies utilising the Freedom of Information Act to collect the data for healthcare research. Secondary outcome measures were any features that improved data acquisition rates (including to whom the request was made, the number of questions asked per request, etc.)Results16 articles described utilising the Freedom of Information Act for healthcare research, and these investigated a broad range of topics. The median number of requests made was 86 (range 1-172), the total number of requests was 1732. A total of 15 817 pieces of data were retrieved by all studies. The amount of data collected was defined as the number of questions asked multiplied by the number of full responses. A median of five questions were asked per study (range 5-6.5) and the overall response rate was 86%. The National Health Service litigation authority responded to 100% of requests, while Primary Care Trusts had the lowest response rate of 81% for healthcare bodies. A positive correlation between number of requests made and data obtained (0.508, p<0.05) and number of requests made and increased response rate (0.737, p<0.01), both reached the statistical significance.ConclusionsResearchers should make greater use of the Act to access the information they need that is not otherwise disclosed. We discuss the issues with the research utilising the Act and how future research of this type could be optimised.
| S-EPMC3845048 | biostudies-literature
Project description:The objective of this systematic review was to systematically review papers in the United States that examine current practices in privacy and security when telehealth technologies are used by healthcare providers. A literature search was conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA-P). PubMed, CINAHL and INSPEC from 2003 - 2016 were searched and returned 25,404 papers (after duplications were removed). Inclusion and exclusion criteria were strictly followed to examine title, abstract, and full text for 21 published papers which reported on privacy and security practices used by healthcare providers using telehealth. Data on confidentiality, integrity, privacy, informed consent, access control, availability, retention, encryption, and authentication were all searched and retrieved from the papers examined. Papers were selected by two independent reviewers, first per inclusion/exclusion criteria and, where there was disagreement, a third reviewer was consulted. The percentage of agreement and Cohen's kappa was 99.04% and 0.7331 respectively. The papers reviewed ranged from 2004 to 2016 and included several types of telehealth specialties. Sixty-seven percent were policy type studies, and 14 percent were survey/interview studies. There were no randomized controlled trials. Based upon the results, we conclude that it is necessary to have more studies with specific information about the use of privacy and security practices when using telehealth technologies as well as studies that examine patient and provider preferences on how data is kept private and secure during and after telehealth sessions.
| S-EPMC5716616 | biostudies-other
Project description:Objective: To identify processes to engage stakeholders in healthcare Simulation Modeling (SM), and the impacts of this engagement on model design, model implementation, and stakeholder participants. To investigate how engagement process may lead to specific impacts. Data Sources: English-language articles on health SM engaging stakeholders in the MEDLINE, EMBASE, Scopus, Web of Science and Business Source Complete databases published from inception to February 2020. Study Design: A systematic review of the literature based on a priori protocol and reported according to PRISMA guidelines. Extraction Methods: Eligible articles were SM studies with a health outcome which engaged stakeholders in model design. Data were extracted using a data extraction form adapted to be specific for stakeholder engagement in SM studies. Data were analyzed using summary statistics, deductive and inductive content analysis, and narrative synthesis. Principal Findings: Thirty-two articles met inclusion criteria. Processes used to engage stakeholders in healthcare SM are heterogenous and often based on intuition rather than clear methodological frameworks. These processes most commonly involve stakeholders across multiple SM stages via discussion/dialogue, interviews, workshops and meetings. Key reported impacts of stakeholder engagement included improved model quality/accuracy, implementation, and stakeholder decision-making. However, for all but four studies, these reports represented author perceptions rather than formal evaluations incorporating stakeholder perspectives. Possible process enablers of impact included the use of models as "boundary objects" and structured facilitation via storytelling to promote effective communication and mutual understanding between stakeholders and modelers. Conclusions: There is a large gap in the current literature of formal evaluation of SM stakeholder engagement, and a lack of consensus about the processes required for effective SM stakeholder engagement. The adoption and clear reporting of structured engagement and process evaluation methodologies/frameworks are required to advance the field and produce evidence of impact.
| S-EPMC10012644 | biostudies-literature
Project description:Co-produced practices and publications in the healthcare sector are gaining momentum, since they can be a useful tool in addressing the sustainability and resilience challenges of health systems. However, the investigation of positive and, mainly, negative outcomes is still confused and fragmented, and above all, a comprehensive knowledge of the metrics used to assess these outcomes is lacking. To fill this gap, this study aims to systematically review the extant literature to map the methods, tools and metrics used to empirically evaluate co-production in health services. The search took place in six databases: Scopus, Web of Science, Psych INFO, PubMed, Cochrane and CINAHL. A total of 2311 articles were screened and 203 articles were included in the analysis, according to PRISMA guidelines. Findings show that outcomes are mainly investigated through qualitative methods and from the lay actor or provider perspective. Moreover, the detailed categorisation of the quantitative measures found offers a multidimensional performance measurement system and highlights the impact areas where research is needed to develop and test new measures. Findings should also promote improvements in empirical data collection on the multiple faceted co-produced activities and spur the consciousness of the adoption of sustainable co-productive initiatives.
| S-EPMC8037812 | biostudies-literature