Dataset Information

Development and Inter-Rater Reliability of the Mealtime Scan for Long-Term Care.

ABSTRACT: Background and Objectives:Poor food intake is common in long-term care (LTC). The mealtime experience has been identified as influential, yet, research has been limited by lack of quality, standardized measures. The purpose of this study was to develop and test for inter-rater reliability the Mealtime Scan (MTS), an observational measure. Research Design and Methods:MTS was derived from the literature on ambiance, mealtime experience, social interactions at mealtimes, and social models of care. Three scales on person-centered care and physical and social environments are used to summarize key aspects observed with MTS. Two raters assessed MTS for reliability at 30 different meals conducted in 10 dining rooms, within three LTC residences. Intraclass correlation coefficient (ICC) was used to assess reliability. Results:MTS demonstrated good to excellent reliability on the three summative scales (physical ICC = 0.73, social ICC = 0.81, person-centered care ICC = 0.83) and other scalable items had good to excellent reliability (e.g., background noise ICC = 0.65, Mealtime Relational Care checklist: negative interactions ICC = 0.85). Discussion and Implications:MTS is reliable and face valid for assessing mealtime experience. Future work will explore construct validity of this measure. MTS can be used to support improving the mealtime experience for residents living in LTC.

SUBMITTER: Keller HH

PROVIDER: S-EPMC5946921 | biostudies-literature | 2018 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Development and Inter-Rater Reliability of the Mealtime Scan for Long-Term Care.

Keller Heather H HH Chaudhury Habib H Pfisterer Kaylen J KJ Slaughter Susan E SE

The Gerontologist 20180501 3

<h4>Background and objectives</h4>Poor food intake is common in long-term care (LTC). The mealtime experience has been identified as influential, yet, research has been limited by lack of quality, standardized measures. The purpose of this study was to develop and test for inter-rater reliability the Mealtime Scan (MTS), an observational measure.<h4>Research design and methods</h4>MTS was derived from the literature on ambiance, mealtime experience, social interactions at mealtimes, and social m ...[more]

PMID: 28329849

Similar Datasets

Project description:AimsTo refine the Cue Utilization and Engagement in Dementia mealtime video-coding scheme and examine its ease of use, feasibility, and inter-rater reliability in assessing the food intake process and dyadic verbal and nonverbal interactions.DesignThis study was a secondary analysis of 110 videotaped observations of mealtime interactions collected under usual care conditions from a dementia communication trial during 2011-2014.MethodsThe videos involved 29 staff and 25 residents with dementia (42 unique staff-resident dyads) in nine nursing homes. Data coding and analysis were performed in 2018-2019. Logs of coding challenges with matched solutions and coding time were collected. Inter-rater reliability was examined through rating of randomly selected 22 videos across four trained coders.ResultsIt took a mean of 10.81 hr to code a one-hour video using the refined coding scheme. Coding challenges, including identification of key intake process characteristics and differentiation of similar verbal or nonverbal behaviours, were identified with appropriate solutions. The refined coding scheme had good inter-rater reliability (Cohen's Kappa range = 0.93 - 0.99, 95% CI = 0.92 - 0.99).ConclusionFindings supported preliminary evidence on feasibility, usability and inter-rater reliability of the refined coding scheme. Future psychometric testing is needed in diverse populations with dementia across different care settings.ImpactExisting tools assessing the food intake process and dyadic interactions are few and have limited feasibility and/or reliability and fail to capture the complexity and dynamics of mealtime care. The refined coding scheme showed preliminary feasibility, usability, and inter-rater reliability. In consideration of the balance between time intensity and the richness of data obtained, the tool may be appropriate and useful in addressing certain research inquires (e.g., characterizing and clustering dyadic behaviours, temporal relationship between behaviours and intake) pertaining older adults with or without dementia and their formal or informal caregivers.

Project description:BackgroundEmployees insured in pension insurance, who are incapable of working due to ill health, are entitled to a disability pension. To assess whether an individual meets the medical requirements to be considered as disabled, a work capacity evaluation is conducted. However, there are no official guidelines on how to perform an external quality assurance for this evaluation process. Furthermore, the quality of medical reports in the field of insurance medicine can vary substantially, and systematic evaluations are scarce. Reliability studies using peer review have repeatedly shown insufficient ability to distinguish between high, moderate and low quality. Considering literature recommendations, we developed an instrument to examine the quality of medical experts' reports.MethodsThe peer review manual developed contains six quality domains (formal structure, clarity, transparency, completeness, medical-scientific principles, and efficiency) comprising 22 items. In addition, a superordinate criterion (survey confirmability) rank the overall quality and usefulness of a report. This criterion evaluates problems of inner logic and reasoning. Development of the manual was assisted by experienced physicians in a pre-test. We examined the observable variance in peer judgements and reliability as the most important outcome criteria. To evaluate inter-rater reliability, 20 anonymous experts' reports detailing the work capacity evaluation were reviewed by 19 trained raters (peers). Percentage agreement and Kendall's W, a reliability measure of concordance between two or more peers, were calculated. A total of 325 reviews were conducted.ResultsAgreement of peer judgements with respect to the superordinate criterion ranged from 29.2 to 87.5%. Kendall's W for the quality domain items varied greatly, ranging from 0.09 to 0.88. With respect to the superordinate criterion, Kendall's W was 0.39, which indicates fair agreement. The results of the percentage agreement revealed systemic peer preferences for certain deficit scale categories.ConclusionThe superordinate criterion was not sufficiently reliable. However, in comparison to other reliability studies, this criterion showed an equivalent reliability value. This report aims to encourage further efforts to improve evaluation instruments. To reduce disagreement between peer judgments, we propose the revision of the peer review instrument and the development and implementation of a standardized rater training to improve reliability.

Project description:Background: There has been a groundswell of national support for transparent tracking and dissemination of PhD career outcomes. In 2017, individuals from multiple institutions and professional organizations met to create the Unified Career Outcomes Taxonomy (UCOT 2017), a three-tiered taxonomy to help institutions uniformly classify career outcomes of PhD graduates. Early adopters of UCOT 2017, noted ambiguity in some categories of the career taxonomy, raising questions about its consistent application within and across institutions. Methods: To test and evaluate the consistency of UCOT 2017, we calculated inter-rater reliability across two rounds of iterative refinement of the career taxonomy, classifying over 800 PhD alumni records via nine coders. Results: We identified areas of discordance in the taxonomy, and progressively refined UCOT 2017 and an accompanying Guidance Document to improve inter-rater reliability across all three tiers of the career taxonomy. However, differing interpretations of the classifications, especially for faculty classifications in the third tier, resulted in continued discordance among the coders. We addressed this discordance with clarifying language in the Guidance Document, and proposed the addition of a flag system for identification of the title, rank, and prefix of faculty members. This labeling system provides the additional benefit of highlighting the granularity and the intersectionality of faculty job functions, while maintaining the ability to sort by - and report data on - faculty and postdoctoral trainee roles, as is required by some national and federal reporting guidelines. We provide specific crosswalk guidance for how a user may choose to incorporate our suggestions while maintaining the ability to report in accordance with UCOT 2017. Conclusions: Our findings underscore the importance of detailed guidance documents, coder training, and periodic collaborative review of career outcomes taxonomies as PhD careers evolve in the global workforce. Implications for coder-training and use of novice coders are also discussed.

Project description:PurposeFirst, to evaluate inter-rater reliability when human raters estimate the reading performance of visually impaired individuals using the MNREAD acuity chart. Second, to evaluate the agreement between computer-based scoring algorithms and compare them with human rating.MethodsReading performance was measured for 101 individuals with low vision, using the Portuguese version of the MNREAD test. Seven raters estimated the maximum reading speed (MRS) and critical print size (CPS) of each individual MNREAD curve. MRS and CPS were also calculated automatically for each curve using two different algorithms: the original standard deviation method (SDev) and a non-linear mixed effects (NLME) modeling. Intra-class correlation coefficients (ICC) were used to estimate absolute agreement between raters and/or algorithms.ResultsAbsolute agreement between raters was 'excellent' for MRS (ICC = 0.97; 95%CI [0.96, 0.98]) and 'moderate' to 'good' for CPS (ICC = 0.77; 95%CI [0.69, 0.83]). For CPS, inter-rater reliability was poorer among less experienced raters (ICC = 0.70; 95%CI [0.57, 0.80]) when compared to experienced ones (ICC = 0.82; 95%CI [0.76, 0.88]). Absolute agreement between the two algorithms was 'excellent' for MRS (ICC = 0.96; 95%CI [0.91, 0.98]). For CPS, the best possible agreement was found for CPS defined as the print size sustaining 80% of MRS (ICC = 0.77; 95%CI [0.68, 0.84]). Absolute agreement between raters and automated methods was 'excellent' for MRS (ICC = 0.96; 95% CI [0.88, 0.98] for SDev; ICC = 0.97; 95% CI [0.95, 0.98] for NLME). For CPS, absolute agreement between raters and SDev ranged from 'poor' to 'good' (ICC = 0.66; 95% CI [0.3, 0.80]), while agreement between raters and NLME was 'good' (ICC = 0.83; 95% CI [0.76, 0.88]).ConclusionFor MRS, inter-rater reliability is excellent, even considering the possibility of noisy and/or incomplete data collected in low-vision individuals. For CPS, inter-rater reliability is lower. This may be problematic, for instance in the context of multisite investigations or follow-up examinations. The NLME method showed better agreement with the raters than the SDev method for both reading parameters. Setting up consensual guidelines to deal with ambiguous curves may help improve reliability. While the exact definition of CPS should be chosen on a case-by-case basis depending on the clinician or researcher's motivations, evidence suggests that estimating CPS as the smallest print size sustaining about 80% of MRS would increase inter-rater reliability.

Dataset Information

Development and Inter-Rater Reliability of the Mealtime Scan for Long-Term Care.

Publications

Development and Inter-Rater Reliability of the Mealtime Scan for Long-Term Care.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets