Dataset Information

An Approach to Assess Generalizability in Comparative Effectiveness Research: A Case Study of the Whole Systems Demonstrator Cluster Randomized Trial Comparing Telehealth with Usual Care for Patients with Chronic Health Conditions.

ABSTRACT:

Background

Policy makers require estimates of comparative effectiveness that apply to the population of interest, but there has been little research on quantitative approaches to assess and extend the generalizability of randomized controlled trial (RCT)-based evaluations. We illustrate an approach using observational data.

Methods

Our example is the Whole Systems Demonstrator (WSD) trial, in which 3230 adults with chronic conditions were assigned to receive telehealth or usual care. First, we used novel placebo tests to assess whether outcomes were similar between the RCT control group and a matched subset of nonparticipants who received usual care. We matched on 65 baseline variables obtained from the electronic medical record. Second, we conducted sensitivity analysis to consider whether the estimates of treatment effectiveness were robust to alternative assumptions about whether "usual care" is defined by the RCT control group or nonparticipants. Thus, we provided alternative estimates of comparative effectiveness by contrasting the outcomes of the RCT telehealth group and matched nonparticipants.

Results

For some endpoints, such as the number of outpatient attendances, the placebo tests passed, and the effectiveness estimates were robust to the choice of comparison group. However, for other endpoints, such as emergency admissions, the placebo tests failed and the estimates of treatment effect differed markedly according to whether telehealth patients were compared with RCT controls or matched nonparticipants.

Conclusions

The proposed placebo tests indicate those cases when estimates from RCTs do not generalize to routine clinical practice and motivate complementary estimates of comparative effectiveness that use observational data. Future RCTs are recommended to incorporate these placebo tests and the accompanying sensitivity analyses to enhance their relevance to policy making.

SUBMITTER: Steventon A

PROVIDER: S-EPMC4592957 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundTelehealth is increasingly used in the care of people with long term conditions. Whilst many studies look at the impacts of the technology on hospital use, few look at how it changes contacts with primary care professionals. The aim of this paper was to assess the impacts of home-based telehealth interventions on general practice contacts.MethodSecondary analysis of data from a Department of Health funded cluster-randomised trial with 179 general practices in three areas of England randomly assigned to offer telehealth or usual care to eligible patients. Telehealth included remote exchange of vitals signs and symptoms data between patients and healthcare professionals as part of the continuing management of patients. Usual care reflected the range of services otherwise available in the sites, excluding telehealth. Anonymised data from GP systems were used to construct person level histories for control and intervention patients. We tested for differences in numbers of general practitioner and practice nurse contacts over twelve months and in the number of clinical readings recorded on general practice systems over twelve months.Results3,230 people with diabetes, chronic obstructive pulmonary disease or heart failure were recruited in 2008 and 2009. 1219 intervention and 1098 control cases were available for analysis. No statistically significant differences were detected in the numbers of general practitioner or practice nurse contacts between intervention and control groups during the trial, or in the numbers of clinical readings recorded on the general practice systems.ConclusionsTelehealth did not appear associated with different levels of contact with general practitioners and practice nurses. We note that the way that telehealth impacts on primary care roles may be influenced by a number of other features in the health system. The challenge is to ensure that these systems lead to better integration of care than fragmentation.Trial registration numberInternational Standard Randomised Controlled Trial Number Register ISRCTN43002091.

Project description:BACKGROUND: The Whole Systems Demonstrator was a large, pragmatic, cluster randomised trial that compared telehealth with usual care among 3,230 patients with long-term conditions in three areas of England. Telehealth involved the regular transmission of physiological information such as blood glucose to health professionals working remotely. We examined whether telehealth led to changes in glycosylated haemoglobin (HbA1c) among the subset of patients with type 2 diabetes. METHODS: The general practice electronic medical record was used as the source of information on HbA1c. Effects on HbA1c were assessed using a repeated measures model that included all HbA1c readings recorded during the 12-month trial period, and adjusted for differences in HbA1c readings recorded before recruitment. Secondary analysis averaged multiple HbA1c readings recorded for each individual during the trial period. RESULTS: 513 of the 3,230 participants were identified as having type 2 diabetes and thus were included in the study. Telehealth was associated with lower HbA1c than usual care during the trial period (difference 0.21% or 2.3 mmol/mol, 95% CI, 0.04% to 0.38%, p = 0.013). Among the 457 patients in the secondary analysis, mean HbA1c showed little change for controls following recruitment, but fell for intervention patients from 8.38% to 8.15% (68 to 66 mmol/mol). A higher proportion of intervention patients than controls had HbA1c below the 7.5% (58 mmol/mol) threshold that was targeted by general practices (30.4% vs. 38.0%). This difference, however, did not quite reach statistical significance (adjusted odds ratio 1.63, 95% CI, 0.99 to 2.68, p = 0.053). CONCLUSIONS: Telehealth modestly improved glycaemic control in patients with type 2 diabetes over 12 months. The scale of the improvements is consistent with previous meta-analyses, but was relatively modest and seems unlikely to produce significant patient benefit. TRIAL REGISTRATION NUMBER: International Standard Randomized Controlled Trial Number Register ISRCTN43002091.

Project description:ObjectiveTo assess the effect of second generation, home based telehealth on health related quality of life, anxiety, and depressive symptoms over 12 months in patients with long term conditions.DesignA study of patient reported outcomes (the Whole Systems Demonstrator telehealth questionnaire study; baseline n=1573) was nested in a pragmatic, cluster randomised trial of telehealth (the Whole Systems Demonstrator telehealth trial, n=3230). General practice was the unit of randomisation, and telehealth was compared with usual care. Data were collected at baseline, four months (short term), and 12 months (long term). Primary intention to treat analyses tested treatment effectiveness; multilevel models controlled for clustering by general practice and a range of covariates. Analyses were conducted for 759 participants who completed questionnaire measures at all three time points (complete case cohort) and 1201 who completed the baseline assessment plus at least one other assessment (available case cohort). Secondary per protocol analyses tested treatment efficacy and included 633 and 1108 participants in the complete case and available case cohorts, respectively.SettingProvision of primary and secondary care via general practices, specialist nurses, and hospital clinics in three diverse regions of England (Cornwall, Kent, and Newham), with established integrated health and social care systems.ParticipantsPatients with chronic obstructive pulmonary disease (COPD), diabetes, or heart failure recruited between May 2008 and December 2009.Main outcome measuresGeneric, health related quality of life (assessed by physical and mental health component scores of the SF-12, and the EQ-5D), anxiety (assessed by the six item Brief State-Trait Anxiety Inventory), and depressive symptoms (assessed by the 10 item Centre for Epidemiological Studies Depression Scale).ResultsIn the intention to treat analyses, differences between treatment groups were small and non-significant for all outcomes in the complete case (0.480 ≤ P ≤ 0.904) or available case (0.181 ≤ P ≤ 0.905) cohorts. The magnitude of differences between trial arms did not reach the trial defined, minimal clinically important difference (0.3 standardised mean difference) for any outcome in either cohort at four or 12 months. Per protocol analyses replicated the primary analyses; the main effect of trial arm (telehealth v usual care) was non-significant for any outcome (complete case cohort 0.273 ≤ P ≤ 0.761; available case cohort 0.145 ≤ P ≤ 0.696).ConclusionsSecond generation, home based telehealth as implemented in the Whole Systems Demonstrator Evaluation was not effective or efficacious compared with usual care only. Telehealth did not improve quality of life or psychological outcomes for patients with chronic obstructive pulmonary disease, diabetes, or heart failure over 12 months. The findings suggest that concerns about potentially deleterious effect of telehealth are unfounded for most patients.Trial registrationISRCTN43002091.

Project description:BackgroundMuch is written about the promise of telehealth and there is great enthusiasm about its potential. However, many studies of telehealth do not meet orthodox quality standards and there are few studies examining quality of life in diabetes as an outcome.Objective To assess the impact of home-based telehealth (remote monitoring of physiological, symptom and self-care behavior data for long-term conditions) on generic and disease-specific health-related quality of life, anxiety, and depressive symptoms over 12 months in patients with diabetes. Remote monitoring provides the potential to improve quality of life, through the reassurance it provides patients.MethodsThe study focused on participant-reported outcomes of patients with diabetes within the Whole Systems Demonstrator (WSD) Telehealth Questionnaire Study, nested within a pragmatic cluster-randomized trial of telehealth (the WSD Telehealth Trial), held across 3 regions of England. Telehealth was compared with usual-care, with general practice as the unit of randomization. Participant-reported outcome measures (Short-Form 12, EuroQual-5D, Diabetes Health Profile scales, Brief State-Trait Anxiety Inventory, and Centre for Epidemiological Studies Depression Scale) were collected at baseline, short-term (4 months) and long-term (12months) follow-ups. Intention-to-treat analyses testing treatment effectiveness, were conducted using multilevel models controlling for practice clustering and a range of covariates. Analyses assumed participants received their allocated treatment and were conducted for participants who completed the baseline plus at least one follow-up assessment (n=317).Results Primary analyses showed differences between telehealth and usual care were small and only reached significance for 1 scale (diabetes health profile-disinhibited eating, P=.006). The magnitude of differences between trial arms did not reach the trial-defined minimal clinically important difference of 0.3 standard deviations for most outcomes. Effect sizes (Hedge's g) ranged from 0.015 to 0.143 for Generic quality of life (QoL) measures and 0.018 to 0.394 for disease specific measures.ConclusionsSecond generation home-based telehealth as implemented in the WSD evaluation was not effective in the subsample of people with diabetes. Overall, telehealth did not improve or have a deleterious effect quality of life or psychological outcomes for patients with diabetes over a 12-month period.