Dataset Information

Development and Evaluation of a Fidelity Instrument for PEARLS.

ABSTRACT:

Purpose

This manuscript describes the development and the preliminary evaluation of a fidelity instrument for the Program for Encouraging Active and Rewarding Lives (PEARLS), an evidence-based depression care management (DCM) program. The objective of the study was to find an effective, practical, multidimensional approach to measure fidelity of PEARLS programs to the original, research-driven PEARLS protocol in order to inform program implementation at various settings nationwide.

Methods

We conducted key informant interviews with PEARLS stakeholders, and held focus groups with former PEARLS clients, to identify core program components. These components were then ranked using a Q-sort process, and incorporated into a brief instrument. We tested the instrument at two time points with PEARLS counselors, other DCM program counselors, and non-DCM program counselors (n?=?56) in six states. Known-groups method was used to compare findings from PEARLS programs, other DCM programs, and non-DCM programs. We asked supervisors of the counselors to complete the fidelity instrument on behalf of their counselors to affirm the validity of the results. We examined the association of PEARLS program fidelity with individual client outcomes.

Results

Program for Encouraging Active and Rewarding Lives providers reported the highest fidelity scores compared to DCM program providers and non-DCM program providers. The sample size was too small to yield significant results on the comparison between counselor experience and fidelity. Scores varied between PEARLS counselors and their supervisors. PEARLS program fidelity was not significantly correlated with client outcomes, suggesting that other implementation factors may have influenced the outcomes and/or that the instrument needs refinement.

Conclusion

Our findings suggest that providers may be able to use the instrument to assess PEARLS program fidelity in various settings across the country. However, more rigorous research is needed to evaluate instrument effectiveness.

SUBMITTER: Farren L

PROVIDER: S-EPMC4410416 | biostudies-literature | 2014

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundEmployees insured in pension insurance, who are incapable of working due to ill health, are entitled to a disability pension. To assess whether an individual meets the medical requirements to be considered as disabled, a work capacity evaluation is conducted. However, there are no official guidelines on how to perform an external quality assurance for this evaluation process. Furthermore, the quality of medical reports in the field of insurance medicine can vary substantially, and systematic evaluations are scarce. Reliability studies using peer review have repeatedly shown insufficient ability to distinguish between high, moderate and low quality. Considering literature recommendations, we developed an instrument to examine the quality of medical experts' reports.MethodsThe peer review manual developed contains six quality domains (formal structure, clarity, transparency, completeness, medical-scientific principles, and efficiency) comprising 22 items. In addition, a superordinate criterion (survey confirmability) rank the overall quality and usefulness of a report. This criterion evaluates problems of inner logic and reasoning. Development of the manual was assisted by experienced physicians in a pre-test. We examined the observable variance in peer judgements and reliability as the most important outcome criteria. To evaluate inter-rater reliability, 20 anonymous experts' reports detailing the work capacity evaluation were reviewed by 19 trained raters (peers). Percentage agreement and Kendall's W, a reliability measure of concordance between two or more peers, were calculated. A total of 325 reviews were conducted.ResultsAgreement of peer judgements with respect to the superordinate criterion ranged from 29.2 to 87.5%. Kendall's W for the quality domain items varied greatly, ranging from 0.09 to 0.88. With respect to the superordinate criterion, Kendall's W was 0.39, which indicates fair agreement. The results of the percentage agreement revealed systemic peer preferences for certain deficit scale categories.ConclusionThe superordinate criterion was not sufficiently reliable. However, in comparison to other reliability studies, this criterion showed an equivalent reliability value. This report aims to encourage further efforts to improve evaluation instruments. To reduce disagreement between peer judgments, we propose the revision of the peer review instrument and the development and implementation of a standardized rater training to improve reliability.

Project description:OBJECTIVES:To develop a valid, reliable measure that reflected the environment of respectfulness within the ICU setting. DESIGN:We developed a preliminary survey instrument based on conceptual domains of respect identified through prior qualitative analyses of ICU patient, family member, and clinician perspectives. The initial instrument consisted of 21 items. After five cognitive interviews and 16 pilot surveys, we revised the instrument to include 23 items. We used standard psychometric methods to analyze the instrument. SETTINGS:Eight ICUs serving adult patients affiliated with a large university health system. SUBJECTS:ICU clinicians. INTERVENTIONS:None. MEASUREMENTS AND MAIN RESULTS:Based on 249 responses, we identified three factors and created subscales: General Respect, Respectful Behaviors, and Disrespectful Behaviors. The General Respect subscale had seven items (? = 0.932) and reflected how often patients in the ICU are treated with respect, in a dignified manner, as an individual, equally to all other patients, on the "same level" as the ICU team, as a person, and as you yourself would want to be treated. The Respectful Behaviors subscale had 10 items (? = 0.926) and reflected how often the ICU team responds to patient and/or family anxiety, makes an effort to get to know the patient and family as people, listens carefully, explains things thoroughly, gives the opportunity to provide input into care, protects patient modesty, greets when entering room, and talks to sedated patients. The subscale measuring disrespect has four items (? = 0.702) and reflects how often the ICU team dismisses family concerns, talks down to patients and families, speaks disrespectfully behind their backs, and gets frustrated with patients and families. CONCLUSIONS:We created a reliable set of scales to measure the climate of respectfulness in intensive care settings. These measures can be used for ongoing quality improvement that aim to enhance the experience of ICU patients and their families.

Project description:Entrustable Professional Activities (EPAs) are units of activity that early-stage professionals perform in the workplace that necessitate simultaneous integration of multiple competencies. EPA #6 requires students to perform a common surgical procedure on a stable patient, including pre-operative and post-operative management. Castration is one of the most common surgeries performed by equine primary care practitioners and is considered an "entry-level competency" for veterinary graduates entering equine private practice, however, to our knowledge there are no equine castration models available for veterinary student education. Therefore, we developed an inexpensive, low-fidelity model of equine field castration and evaluated it using a mixed-methods approach. Two different groups of students, with or without model experience, completed surveys before and after live horse castration. Students who used the model also completed model specific surveys. Videos of the students completing the model were evaluated by at least two different equine veterinary faculty using a 15-point rubric, and inter-rater reliability of the rubric was determined. After completing the model, students reflected on strengths and weaknesses of their performance. From our student survey results, we determined that student attitudes toward the model were mostly positive. Interestingly, there were several student attitudes toward the model that became significantly more favorable after live horse castration. Prior to live horse castration, there was no significant difference in confidence in model vs. no-model groups. Following live horse castration, students who used the model had higher confidence in procedure preparation and hand-ties than students who did not use the model, but they had lower scores for confidence during patient recovery. When reflecting on model castration, students most commonly cited preparation and surgical description as strengths, and ligature placement and hand-ties as weaknesses. Experts provided several suggestions to improve the model, including incorporation of emasculators and the need for better model stabilization. Our findings suggest that both students and veterinary educators feel that this low-fidelity model has educational value. Rubric performance metrics were favorable, but additional steps are needed to improve grading consistency among educators. Future research will determine whether student performance on the model is predictive of competence score during live-horse castration.