Dataset Information

Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators.

ABSTRACT: BACKGROUND:Public health evaluation methods have been criticized for being overly reductionist and failing to generate suitable evidence for public health decision-making. A "complex systems approach" has been advocated to account for real world complexity. Qualitative methods may be well suited to understanding change in complex social environments, but guidance on applying a complex systems approach to inform qualitative research remains limited and underdeveloped. This systematic review aims to analyze published examples of process evaluations that utilize qualitative methods that involve a complex systems perspective and proposes a framework for qualitative complex system process evaluations. METHODS AND FINDINGS:We conducted a systematic search to identify complex system process evaluations that involve qualitative methods by searching electronic databases from January 1, 2014-September 30, 2019 (Scopus, MEDLINE, Web of Science), citation searching, and expert consultations. Process evaluations were included if they self-identified as taking a systems- or complexity-oriented approach, integrated qualitative methods, reported empirical findings, and evaluated public health interventions. Two reviewers independently assessed each study to identify concepts associated with the systems thinking and complexity science traditions. Twenty-one unique studies were identified evaluating a wide range of public health interventions in, for example, urban planning, sexual health, violence prevention, substance use, and community transformation. Evaluations were conducted in settings such as schools, workplaces, and neighborhoods in 13 different countries (9 high-income and 4 middle-income). All reported some utilization of complex systems concepts in the analysis of qualitative data. In 14 evaluations, the consideration of complex systems influenced intervention design, evaluation planning, or fieldwork. The identified studies used systems concepts to depict and describe a system at one point in time. Only 4 evaluations explicitly utilized a range of complexity concepts to assess changes within the system resulting from, or co-occurring with, intervention implementation over time. Limitations to our approach are including only English-language papers, reliance on study authors reporting their utilization of complex systems concepts, and subjective judgment from the reviewers relating to which concepts featured in each study. CONCLUSION:This study found no consensus on what bringing a complex systems perspective to public health process evaluations with qualitative methods looks like in practice and that many studies of this nature describe static systems at a single time point. We suggest future studies use a 2-phase framework for qualitative process evaluations that seek to assess changes over time from a complex systems perspective. The first phase involves producing a description of the system and identifying hypotheses about how the system may change in response to the intervention. The second phase involves following the pathway of emergent findings in an adaptive evaluation approach.

SUBMITTER: McGill E

PROVIDER: S-EPMC7605618 | biostudies-literature | 2020 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators.

McGill Elizabeth E Marks Dalya D Er Vanessa V Penney Tarra T Petticrew Mark M Egan Matt M

PLoS medicine 20201102 11

<h4>Background</h4>Public health evaluation methods have been criticized for being overly reductionist and failing to generate suitable evidence for public health decision-making. A "complex systems approach" has been advocated to account for real world complexity. Qualitative methods may be well suited to understanding change in complex social environments, but guidance on applying a complex systems approach to inform qualitative research remains limited and underdeveloped. This systematic revi ...[more]

PMID: 33137099

Similar Datasets

Project description:ObjectivesEnglish local authorities (LAs) are interested in reducing alcohol-related harms and may use discretionary powers such as the Late Night Levy (LNL) to do so. This study aims to describe how system stakeholders hypothesise the levy may generate changes and to explore how the system, its actors and the intervention adapt and co-evolve over time.DesignA process evaluation from a complex systems perspective, using qualitative methods.SettingA London LA with high densities of residential and commercial properties, which implemented the LNL in 2014.ParticipantsData were generated through interviews with LNL implementers and alcohol consumers, observations in bars and during LNL patrols and documentary review.InterventionThe LNL allows LAs to charge late-night alcohol retailers an annual fee (£299-£4440) to manage and police the night-time economy (NTE).ResultsWhen the LNL was being considered, stakeholders from different interest groups advanced diverse opinions about its likely impacts while rarely referencing supporting research evidence. Proponents of the levy argued it could reduce crime and anti-social behaviour by providing additional funds to police and manage the NTE. Critics of the levy hypothesised adverse consequences linked to claims that the intervention would force venues to vary their hours or close, cluster closing times, reduce NTE diversity and undermine public-private partnerships. In the first 2 years, levy-funded patrols developed relationships with the licensed trade and the public. The LNL did not undermine public-private partnerships and while some premises varied their hours, these changes did not undermine the intervention's viability, nor significantly cluster venue closing times, nor obviously damage the area's reputation for having a diverse NTE.ConclusionsThis study applies a framework for process evaluation from a complex systems perspective. The evaluation could be extended to measure alcohol-related outcomes and to consider the interplay between the national and local systems.

Project description:BackgroundQualitative Comparative Analysis (QCA) is a method for identifying the configurations of conditions that lead to specific outcomes. Given its potential for providing evidence of causality in complex systems, QCA is increasingly used in evaluative research to examine the uptake or impacts of public health interventions. We map this emerging field, assessing the strengths and weaknesses of QCA approaches identified in published studies, and identify implications for future research and reporting.MethodsPubMed, Scopus and Web of Science were systematically searched for peer-reviewed studies published in English up to December 2019 that had used QCA methods to identify the conditions associated with the uptake and/or effectiveness of interventions for public health. Data relating to the interventions studied (settings/level of intervention/populations), methods (type of QCA, case level, source of data, other methods used) and reported strengths and weaknesses of QCA were extracted and synthesised narratively.ResultsThe search identified 1384 papers, of which 27 (describing 26 studies) met the inclusion criteria. Interventions evaluated ranged across: nutrition/obesity (n = 8); physical activity (n = 4); health inequalities (n = 3); mental health (n = 2); community engagement (n = 3); chronic condition management (n = 3); vaccine adoption or implementation (n = 2); programme implementation (n = 3); breastfeeding (n = 2), and general population health (n = 1). The majority of studies (n = 24) were of interventions solely or predominantly in high income countries. Key strengths reported were that QCA provides a method for addressing causal complexity; and that it provides a systematic approach for understanding the mechanisms at work in implementation across contexts. Weaknesses reported related to data availability limitations, especially on ineffective interventions. The majority of papers demonstrated good knowledge of cases, and justification of case selection, but other criteria of methodological quality were less comprehensively met.ConclusionQCA is a promising approach for addressing the role of context in complex interventions, and for identifying causal configurations of conditions that predict implementation and/or outcomes when there is sufficiently detailed understanding of a series of comparable cases. As the use of QCA in evaluative health research increases, there may be a need to develop advice for public health researchers and journals on minimum criteria for quality and reporting.

Project description:BackgroundGrowing interest in public involvement in health research has led to organisational and policy change. Additionally, an emerging body of policy-oriented scholarship has begun to identify the organisational and network arrangements that shape public involvement activity. Such developments suggest the need to clearly conceptualise and characterise public involvement in health research in terms of governance.MethodsWe drew on an established health research system framework to analyse governance functions related to public involvement, adapting scoping review methods to identify evidence from a corpus of journal papers and policy reports. We drew on the logics of aggregation and top down configuration, using a qualitative interpretive approach to combine and link findings from different studies into framework categories.ResultsWe identified a total of 32 scholarly papers and 13 policy reports (n = 45 included papers) with relevance to governance for public involvement. Included papers were broadly consonant in identifying the need for activity to specify and support public involvement across all four governance functions of stewardship, financing, creating and sustaining resources, and research production and use. However, different visions for public involvement, and the activity required to implement it and achieve impact, were particularly evident with respect to the stewardship function, which seeks to set overall directions for research while addressing the potentially competing demands of a system's many constituents.ConclusionsA governance perspective has considerable value for public involvement in health research systems, supporting efforts to coordinate and institutionalise the burgeoning public involvement enterprise. Furthermore, it highlights challenges for what is, ultimately, a highly political intervention, suggesting that diverse publics must be both involved within health research systems and enrolled as governors of them.

Project description:BackgroundIncreasing awareness of the complexity of public health problems, including obesity, has led to growing interest in whole systems approaches (WSAs), defined as those that consider the multifactorial drivers of overweight and obesity, involve transformative co-ordinated action across a broad range of disciplines and stakeholders, operate across all levels of governance and throughout the life course. This paper reports a systematic review of WSAs targeting obesity and other complex public health and societal issues, such as healthy lifestyles for prevention of non-communicable disease.MethodsSeven electronic databases were searched from 1995 to 2018. Studies were included if there had been an effort to implement a WSA. Study selection was conducted by one reviewer with a random 20% double checked. Data extraction and validity assessment were undertaken by one reviewer and checked by a second reviewer. Narrative synthesis was undertaken.ResultsSixty-five articles were included; 33 about obesity. Most examined multicomponent community approaches, and there was substantial clinical and methodological heterogeneity. Nevertheless, a range of positive health outcomes were reported, with some evidence of whole systems thinking. Positive effects were seen on health behaviours, body mass index (BMI), parental and community awareness, community capacity building, nutrition and physical activity environments, underage drinking behaviour and health, safety and wellbeing of community members, self-efficacy, smoking and tobacco-related disease outcomes. Features of successful approaches reported in process evaluations included: full engagement of relevant partners and community; time to build relationships, trust and capacity; good governance; embedding within a broader policy context; local evaluation; finance.ConclusionsSystems approaches to tackle obesity can have some benefit, but evidence of how to operationalise a WSA to address public health problems is still in its infancy. Future research should: (a) develop an agreed definition of a WSA in relation to obesity, (b) look across multiple sectors to ensure consistency of language and definition, (c) include detailed descriptions of the approaches, and (d) include process and economic evaluations.

Project description:Introduction: Veterans frequently seek chronic pain care from their primary care providers (PCPs) who may not be adequately trained to provide pain management. To address this issue the Veterans Health Administration (VHA) Office of Specialty Care adopted the Specialty Care Access Network Extension for Community Healthcare Outcomes (VA-ECHO née SCAN-ECHO). The VA-ECHO program offered training and mentoring by specialists to PCPs and their staff. VA-ECHO included virtual sessions where expertise was shared in two formats: (1) didactics on common pain conditions, relevant psychological disorders, and treatment options and (2) real-time consultation on patient cases. Materials and methods: VA-ECHO participants' perspectives were obtained using a semi-structured interview guide designed to elicit responses based on the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework. A convenience sampling was used to recruit PCPs and non-physician support staff participants. Non-physicians from rural VHA sites were purposively sampled to gain diverse perspectives. Findings: This qualitative study yielded data on each RE-AIM domain except reach. Program reach was not measured as it is outside the scope of this study. Respondents reported program effectiveness as gains in knowledge and skills to improve pain care delivery. Effective incorporation of learning into practice was reflected in respondents' perceptions of improvements in: patient engagement, evidenced-based approaches, appropriate referrals, and opioid use. Program adoption included how participating health care systems selected trainees from a range of sites and roles to achieve a wide reach of pain expertise. Participation was limited by time to attend and facilitated by institutional support. Differences and similarities were noted in implementation between hub sites. Maintenance was revealed when respondents noted the importance of the lasting relationships formed between fellow participants. Discussion: This study highlights VA-ECHO program attributes and unintended consequences. These findings are expected to inform future use of VA-ECHO as a means to establish a supportive consultation network between primary and specialty care providers to promote the delivery evidence-based pain management practices.

Project description:BackgroundThe Assessing outcomes of enhanced Chronic disease Care through patient Education and a value-baSed formulary Study (ACCESS) is a randomized controlled trial evaluating two interventions targeting barriers to care among those at high risk of cardiovascular disease: copayment elimination for cardioprotective medications, and a tailored self-management support programme. We designed a process evaluation to better understand participant perspectives on the interventions.DesignWe used a qualitative descriptive study design, collecting patient and pharmacist feedback via individual semi-structured telephone interviews and in-person focus groups. Data were analysed inductively using thematic analysis.ResultsFifty-three patients (39 interviews and 14 in two focus groups) and 20 pharmacists participated. Copayment elimination provided quality of life benefits: minimizing the need to 'cut-back', allowing 'peace of mind' and providing emotional support. Health-related benefits included: improving adherence to covered medications, and helping to afford non-covered goods. The only criticism was that not all medications and testing supplies were covered. Patients reported that the educational materials provided helpful information, acted as a reminder, improved confidence, improved adherence to medication, and helped initiate conversations with providers about indicated medication. Some participants felt that the educational materials were repetitive, overly medication-focused and not tailored enough. Pharmacists felt that their patients benefitted from both interventions, which improved patient adherence and communication with their patients.ConclusionThe success of interventions intended to change behaviour is largely dependent upon participant's feelings that the intervention is helpful. This process evaluation provided insights into participants' perceptions on these interventions. Reception of both was largely positive with a few criticisms noted.

Dataset Information

Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators.

Publications

Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets