Dataset Information

Identification of Complex Health Interventions Suitable for Evaluation: Development and Validation of the 8-Step Scoping Framework.

ABSTRACT: BACKGROUND:There is extensive literature on the methodology of evaluation research and the development and evaluation of complex interventions but little guidance on the formative stages before evaluation and how to work with partner organizations that wish to have their provision evaluated. It is important to be able to identify suitable projects for evaluation from a range of provision and describe the steps required, often with academic institutions working in partnership with external organizations, in order to set up an evaluation. However, research evaluating programs or interventions rarely discusses these stages. OBJECTIVE:This study aimed to extend work on evaluability assessment and pre-evaluation planning by proposing an 8-Step Scoping Framework to enable the appraisal of multiple programs in order to identify interventions suitable for evaluation. We aimed to add to the literature on evaluability assessment and more recent evaluation guidance by describing the processes involved in working with partner organizations. METHODS:This paper documents the steps required to identify multiple complex interventions suitable for process and outcome evaluation. The steps were developed using an iterative approach by working alongside staff in a local government organization, to build an evidence base to demonstrate which interventions improve children's outcomes. The process of identifying suitable programs for evaluation, thereby establishing the pre-evaluation steps, was tested using all Flying Start provision. RESULTS:The 8-Step Scoping Framework was described using the example of the local government organization Flying Start to illustrate how each step contributes to finding projects suitable for process and outcome evaluation: (1) formulating overarching key questions that encompass all programs offered by an organization, (2) gaining an in-depth understanding of the work and provision of an organization and engaging staff, (3) completing a data template per project/program offered, (4) assessing the robustness/validity of data across all programs, (5) deciding on projects suitable for evaluation and those requiring additional data, (6) negotiating with chosen project leads, both within and outside the organization, (7) developing individual project evaluation protocols, and (8) applying for ethical approval from the university and partner organization. CONCLUSIONS:This paper describes the processes involved in identifying suitable projects for evaluation. It adds to the existing literature on the assessment of specific programs suitable for evaluation and guidance for conducting evaluations by establishing the formative steps required to identify suitable programs from a range of provision. This scoping framework particularly relates to academic partners and organizations tasked with delivering evidence-based services designed to meet local needs. The steps identified have been described in the context of early years provision but can be applied to a range of community-based evaluations, or more generally, to cases where an academic partner is working with external stakeholders to identify projects suitable for academic evaluation.

SUBMITTER: Davidson R

PROVIDER: S-EPMC6423464 | biostudies-literature | 2019 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Identification of Complex Health Interventions Suitable for Evaluation: Development and Validation of the 8-Step Scoping Framework.

Davidson Rosemary R Randhawa Gurch G Cash Stephanie S

JMIR research protocols 20190305 3

<h4>Background</h4>There is extensive literature on the methodology of evaluation research and the development and evaluation of complex interventions but little guidance on the formative stages before evaluation and how to work with partner organizations that wish to have their provision evaluated. It is important to be able to identify suitable projects for evaluation from a range of provision and describe the steps required, often with academic institutions working in partnership with externa ...[more]

PMID: 30835240

Similar Datasets

Project description:INTRODUCTION: The implementation of complex medical interventions in daily practice is often fraught with difficulties. According to the iterative phase model proposed by the British Medical Research Council (MRC), the development, implementation and evaluation of complex interventions should be theory-driven. A conceptual model that seems to be a promising framework is the Theory of planned behaviour (TPB). In our study we aimed to develop and validate a generic and multifaceted questionnaire based on the TPB to detect physicians' willingness to implement complex medical interventions and the factors influencing this willingness. METHODS: The questionnaire was developed according to the literature and was informed by previous qualitative research of our department. It was validated on the example of an electronic library of decision aids, arriba-lib. The sample consisted of 181 General Practitioners (GPs) who received a training regarding arriba-lib and subsequently filled in the questionnaire, assessing the TPB variables attitude, subjective norm, perceived behaviour control and intention. Follow-up assessments were conducted after two (assessing retest reliability) and eight weeks (assessing target behaviour). We performed a confirmatory factor analysis investigating the factorial structure of our questionnaire according to the TPB. Beside the calculation of the questionnaire's psychometric properties we conducted a structural equation model and an ordinal regression to predict actual behaviour regarding the installation and application of arriba-lib. RESULTS: The postulated three factorial model (attitude, subjective norm, perceived behaviour control) of our questionnaire based on the TPB was rejected. A two factorial model with a combined factor subjective norm/perceived behaviour control was accepted. The explained variance in the ordinal regression was low (Nagelkerke's R(2)=.12). Neither attitude nor intention were able to predict the use or non-use of arriba-lib (attitude: p=.68, intention: p=.44). For the combined factor subjective norm/perceived behaviour control a significant, but small effect (p=.03) was shown. CONCLUSIONS: The TPB is not an adequate theoretical framework to guide the development of a generic questionnaire in the context of the implementation of complex interventions. To enable the successful implementation of complex medical interventions evaluators have to go through the whole development and evaluation process according to the MRC-model, without short cuts. Further, it has to be discussed if a generic instrument can be valid and useful. Regarding the TPB a publication bias regarding the theory's applicability might have to be considered.

Project description:BackgroundThe use of process evaluations is a growing area of interest in research groups working on complex interventions. This methodology tries to understand how the intervention was implemented to inform policy and practice. A recent systematic review by Liu et al. on process evaluations of complex interventions addressing non-communicable diseases found few studies in low- and middle- income countries (LMIC) because it was restricted to randomized controlled trials, primary healthcare level and non-communicable diseases. Yet, LMICs face different barriers to implement interventions in comparison to high-income countries such as limited human resources, access to health care and skills of health workers to treat chronic conditions especially at primary health care level. Therefore, understanding the challenges of interventions for non-communicable diseases and neglected tropical diseases (diseases that affect poor populations and have chronic sequelae) will be important to improve how process evaluation is designed, conducted and used in research projects in LMICs. For these reasons, in comparison to the study of Liu et al., the current study will expand the search strategy to include different study designs, languages and settings.ObjectiveMap research using process evaluation in the areas of non-communicable diseases and neglected tropical diseases to inform the gaps in the design and conduct of this type of research in LMICs.MethodsScoping review of process evaluation studies of randomized controlled trials (RCTs) and non-RCTs of complex interventions implemented in LMICs including participants with non-communicable diseases or neglected tropical diseases and their health care providers (physicians, nurses, technicians and others) related to achieve better health for all through reforms in universal coverage, public policy, service delivery and leadership. The aspects that will be evaluated are as follows: (i) available evidence of process evaluation in the areas of non-communicable diseases and neglected tropical diseases such as frameworks and theories, (ii) methods applied to conduct process evaluations and (iii) gaps between the design of the intervention and its implementation that were identified through the process evaluation. Studies published from January 2008. Exclusion criteria are as follows: not peer reviewed articles, not a report based on empirical research, not reported in English or Spanish or Portuguese or French, reviews and non-human research.DiscussionThis scoping review will map the evidence of process evaluations conducted in LMICs. It will also identify the methods they used to collect and interpret data, how different theories and frameworks were used and lessons from the implementation of complex interventions. This information will allow researchers to conduct better process evaluations considering special characteristics from countries with limited human resources, scarce data available and limited access to health care.

Dataset Information

Identification of Complex Health Interventions Suitable for Evaluation: Development and Validation of the 8-Step Scoping Framework.

Publications

Identification of Complex Health Interventions Suitable for Evaluation: Development and Validation of the 8-Step Scoping Framework.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets