Inter-Rater Reliability of Ergonomic Work Demands for Childcare Workers Using the Observation Instrument TRACK.
Ontology highlight
ABSTRACT:
SUBMITTER: Svendsen MJ
PROVIDER: S-EPMC7084378 | biostudies-literature | 2020 Mar
REPOSITORIES: biostudies-literature
Publications
Inter-Rater Reliability of Ergonomic Work Demands for Childcare Workers Using the Observation Instrument TRACK.
Svendsen Malene Jagd MJ Hendriksen Peter Fjeldstad PF Schmidt Kathrine Greby KG Stochkendahl Mette Jensen MJ Rasmussen Charlotte Nørregaard CN Holtermann Andreas A
International journal of environmental research and public health 20200302 5
The aim of this study is to evaluate the inter-rater reliability of a newly developed instrument-TRACK (observaTion woRk demAnds Childcare worK) for observations of ergonomic work demands in childcare work. Two trained raters conducted thirty hours of concurrent observation of fifteen childcare workers in three different day nurseries. Inter-rater reliability of ergonomic work demands was evaluated using Gwet's Agreement Coefficient (AC<sub>1</sub>) and interpreted by the Landis and Koch benchma ...[more]
Similar Datasets
Project description:BACKGROUND:Employees insured in pension insurance, who are incapable of working due to ill health, are entitled to a disability pension. To assess whether an individual meets the medical requirements to be considered as disabled, a work capacity evaluation is conducted. However, there are no official guidelines on how to perform an external quality assurance for this evaluation process. Furthermore, the quality of medical reports in the field of insurance medicine can vary substantially, and systematic evaluations are scarce. Reliability studies using peer review have repeatedly shown insufficient ability to distinguish between high, moderate and low quality. Considering literature recommendations, we developed an instrument to examine the quality of medical experts' reports. METHODS:The peer review manual developed contains six quality domains (formal structure, clarity, transparency, completeness, medical-scientific principles, and efficiency) comprising 22 items. In addition, a superordinate criterion (survey confirmability) rank the overall quality and usefulness of a report. This criterion evaluates problems of inner logic and reasoning. Development of the manual was assisted by experienced physicians in a pre-test. We examined the observable variance in peer judgements and reliability as the most important outcome criteria. To evaluate inter-rater reliability, 20 anonymous experts' reports detailing the work capacity evaluation were reviewed by 19 trained raters (peers). Percentage agreement and Kendall's W, a reliability measure of concordance between two or more peers, were calculated. A total of 325 reviews were conducted. RESULTS:Agreement of peer judgements with respect to the superordinate criterion ranged from 29.2 to 87.5%. Kendall's W for the quality domain items varied greatly, ranging from 0.09 to 0.88. With respect to the superordinate criterion, Kendall's W was 0.39, which indicates fair agreement. The results of the percentage agreement revealed systemic peer preferences for certain deficit scale categories. CONCLUSION:The superordinate criterion was not sufficiently reliable. However, in comparison to other reliability studies, this criterion showed an equivalent reliability value. This report aims to encourage further efforts to improve evaluation instruments. To reduce disagreement between peer judgments, we propose the revision of the peer review instrument and the development and implementation of a standardized rater training to improve reliability.
| S-EPMC6688267 | biostudies-other
Project description:BackgroundThe CHEERS is a self-administered tool to measure gaps, weaknesses, and strengths of an early childhood education and care (ECEC) centre-based nutrition and physical activity environment. ECEC settings have the potential to profoundly influence early dietary and physical activity behaviours. Content validation of the CHEERS tool has been previously reported. The purpose of this study was to develop reliability and validity evidence for the CHEERS audit tool and the proposed subscales of food served, healthy eating environment, program planning, and physical activity environment in ECEC centre-based programs.MethodsThis cross-sectional study consisted of 2 phases: Phase 1 included inter-, intra-rater and Cronbach's ?. A subset of this sample was invited to participate in a second survey (Trial 2) for intra-rater assessment within 3 weeks of completing the first survey (Trial 1). Phase 2 included concurrent validity assessment between a nutrition expert and the ECEC director using within a one-week period.ResultsOne hundred two directors and 85 educators (total of 187) returned the survey. Of these, there were 75 matched pairs for inter-rater reliability analysis providing a CHEERS ICC score of 0.59 and ICC scores ranging from 0.40 to 0.58 for the subscales. The ICC for intra-rater reliability of the CHEERS score was 0.81 for 40 participants completing the survey a second time and a range of 0.72 to 0.79 for the subscales. The CHEERS tool demonstrated very good internal consistency (??=?0.91) and ? scores ranging from 0.73 to 0.79 for the subscales. In phase 2, concurrent validation was ICC?=?0.65 (n?=?30) CHEERS scores with a range of 0.42 to 0.69 for the subscales.ConclusionsThis study provides evidence of inter-, intra-rater reliability, internal consistency, and concurrent validity of an environmental assessment audit tool to assess the nutrition and physical activity environment of ECEC centre-based programs. The results demonstrate that the self-administered CHEERS instrument is stable overtime and between evaluators at the same ECEC centre. The scores obtained with CHEERS self-administered audit tool are reasonably accurate compared to an expert rater (dietitian) assessment. This study adds additional support to establishing the psychometric soundness of the CHEERS tool.
| S-EPMC6816176 | biostudies-literature
Project description:Objective: To develop an instrument for the observation of therapeutic communication interactions during rehabilitation sessions and test its inter-rater reliability. Methods: The new instrument THER-I-ACT (THERapy-related Inter-ACTion) has been designed to assess both the frequency and timing of therapeutic interactions in the thematic fields information provision, feedback, other motivational interaction, and bonding. For this inter-rater reliability study, a sample of stroke survivors received arm rehabilitation as either arm ability training, arm basis training, or mirror therapy, or neglect training as individually indicated. Therapy sessions were video-recorded (one for each participant) and therapeutic interactions rated by two independent raters using THER-I-ACT. Results: With regard to the instrument's comprehensiveness to document therapeutic interactions with pre-defined categories the data from 29 sessions suggested almost complete coverage. Inter-rater reliability was very high both for individual categories of therapeutic interaction (frequency and time used for interaction) (intraclass correlation coefficient, ICC 0.91-1.00) and summary scores for the thematic fields of interaction (again for frequency and time used for interaction) (ICC 0.98-1.00). The inter-rater reliability for rating engagement and being focussed for both the therapist and patient was substantial (ICC 0.71 and 0.86). Conclusions: The observational study documented that by use of the newly designed THER-I-ACT various types of therapy-related communication interactions performed by therapists can be assessed with a very high inter-rater reliability. In addition, the thematic fields and categories of therapeutic interaction as defined by the instrument comprehensively covered the type of interaction that occurred in the therapeutic sessions observed.
| S-EPMC8377230 | biostudies-literature
Project description:This dataset belongs to a large-scale randomized controlled trial (RCT) in educational research targeting English learning students and their teachers' instructional capacity. The dataset includes ratings conducted through classroom observations of 45-minute English as a Second language (ESL) blocks. Each coder rated 60 recorded video segments collected from each teacher. During the 20-second segment, ratings of six domains of teachers' instruction (i.e., ESL Strategies, Group, Activity Structure, Mode, Language Content, Language of Teacher, Language of Student) were collected. The dataset is organized by teacher, by coder, and by domain, for researchers to analyze inter-rater reliability among coders by domain and/or cross-domain. This data article is related to the research article Tong et al. [3] on "The determination of appropriate coefficient indices for inter-rater reliability: using classroom observation instruments as fidelity measures in large-scale randomized research".
| S-EPMC7044508 | biostudies-literature
Project description:BACKGROUND:A new tool, "risk of bias (ROB) instrument for non-randomized studies of exposures (ROB-NRSE)," was recently developed. It is important to establish consistency in its application and interpretation across review teams. In addition, it is important to understand if specialized training and guidance will improve the reliability in the results of the assessments. Therefore, the objective of this cross-sectional study is to establish the inter-rater reliability (IRR), inter-consensus reliability (ICR), and concurrent validity of the new ROB-NRSE tool. Furthermore, as this is a relatively new tool, it is important to understand the barriers to using this tool (e.g., time to conduct assessments and reach consensus-evaluator burden). METHODS:Reviewers from four participating centers will apprise the ROB of a sample of NRSE publications using ROB-NRSE tool in two stages. For IRR and ICR, two pairs of reviewers will assess the ROB for each NRSE publication. In the first stage, reviewers will assess the ROB without any formal guidance. In the second stage, reviewers will be provided customized training and guidance. At each stage, each pair of reviewers will resolve conflicts and arrive at a consensus. To calculate the IRR and ICR, we will use Gwet's AC1 statistic. For concurrent validity, reviewers will appraise a sample of NRSE publications using both the Newcastle-Ottawa Scale (NOS) and ROB-NRSE tool. We will analyze the concordance between the two tools for similar domains and for the overall judgments using Kendall's tau coefficient. To measure evaluator burden, we will assess the time taken to apply ROB-NRSE tool (without and with guidance), and the NOS. To assess the impact of customized training and guidance on the evaluator burden, we will use the generalized linear models. We will use Microsoft Excel and SAS 9.4, to manage and analyze study data, respectively. DISCUSSION:The quality of evidence from systematic reviews that include NRSE depends partly on the study-level ROB assessments. The findings of this study will contribute to an improved understanding of ROB-NRSE and how best to use it.
| S-EPMC7017505 | biostudies-literature
Project description:BACKGROUND:To analyse the reliability, variance and execution time of the Extended Timed Up and Go (Extended TUG) test in three age groups of elderly participants (G1: 55-64?years; G2: 65-74?years; G3: 75-85?years). METHODS:An analytical cross-sectional study of 114 recruited participants (63 women) of average age 70.17 (± 7.3) years was undertaken. Each participant performed the Extended TUG three consecutive times, with a rest break between tests of 120?s. Both the intragroup and intergroup reliability of the measurements in the Extended TUG were analysed. RESULTS:The reliability of the Extended TUG test is excellent for the first and second decades but drops down to good for the third decade. Specifically, intragroup reliability ranged from 0.784 for G3 to 0.977 for G1 (G2?=?0.858). Intergroup reliability, compared with intragroup reliability, was slightly lower, ranging between 0.779 for G3 and 0.972 for G1 (G2?=?0.853). CONCLUSION:The reliability of the Extended TUG test progressively decreases with increasing age, being excellent for the younger age groups and good for the oldest age group.
| S-EPMC7020600 | biostudies-literature
Project description:Near falls (NFs) are more frequent than falls, and may occur before falls, potentially predicting fall risk. As such, identification of a NF is important. We aimed to assess intra and inter-rater reliability of the traditional definition of a NF and to demonstrate the potential utility of a new definition. To this end, 10 older adults, 10 idiopathic elderly fallers, and 10 patients with Parkinson's disease (PD) walked in an obstacle course while wearing a safety harness. All walks were videotaped. Forty-nine video segments were extracted to create 2 clips each of 8.48 min. Four raters scored each event using the traditional definition and, two weeks later, using the new definition. A fifth rater used only the new definition. Intra-rater reliability was determined using Kappa (K) statistics and inter-rater reliability was determined using ICC. Using the traditional definition, three raters had poor intra-rater reliability (K<0.054, p>0.137) and one rater had moderate intra-rater reliability (K=0.624, p<0.001). With the traditional definition, inter-rater reliability between the four raters was moderate (ICC=0.667, p<0.001). In contrast, the new NF definition showed high intra-rater (K>0.601, p<0.001) and excellent inter-rater reliability (ICC=0.815, p<0.001). A priori, it is easy to distinguish falls from usual walking and NFs, but it is more challenging to distinguish NFs from obstacle negotiation and usual walking. Therefore, a more precise definition of NF is required. The results of the present study suggest that the proposed new definition increases intra and inter-rater reliability, a critical step for using NFs to quantify fall risk.
| S-EPMC3842362 | biostudies-literature
Project description:Cerebral Performance Category (CPC) scores are often an outcome measure for post-arrest neurologic function, collected worldwide to compare performance, evaluate therapies, and formulate recommendations. At most institutions, no formal training is offered in their determination, potentially leading to misclassification.We identified 171 patients at 2 hospitals between 5/10/2005 and 8/31/2012 with two CPC scores at hospital discharge recorded independently - in an in-house quality improvement database and as part of a national registry. Scores were abstracted retrospectively from the same electronic medical record by two separate non-clinical researchers. These scores were compared to assess inter-rater reliability and stratified based on whether the score was concordant or discordant among reviewers to determine factors related to discordance.Thirty-nine CPC scores (22.8%) were discordant (kappa: 0.66), indicating substantial agreement. When dichotomized into "favorable" neurologic outcome (CPC 1-2)/"unfavorable" neurologic outcome (CPC 3-5), 20 (11.7%) scores were discordant (kappa: 0.70), also indicating substantial agreement. Patients discharged home (as opposed to nursing/other care facility) and patients with suspected cardiac etiology of arrest were statistically more likely to have concordant scores. For the quality improvement database, patients with discordant scores had a statistically higher median CPC score than those with concordant scores. The registry had statistically lower median CPC score (CPC 1) than the quality improvement database (CPC 2); p<0.01 for statistical significance.CPC scores have substantial inter-rater reliability, which is reduced in patients who have worse outcomes, have a non-cardiac etiology of arrest, and are discharged to a location other than home.
| S-EPMC5990009 | biostudies-literature
Project description:Background: There has been a groundswell of national support for transparent tracking and dissemination of PhD career outcomes. In 2017, individuals from multiple institutions and professional organizations met to create the Unified Career Outcomes Taxonomy (UCOT 2017), a three-tiered taxonomy to help institutions uniformly classify career outcomes of PhD graduates. Early adopters of UCOT 2017, noted ambiguity in some categories of the career taxonomy, raising questions about its consistent application within and across institutions. Methods: To test and evaluate the consistency of UCOT 2017, we calculated inter-rater reliability across two rounds of iterative refinement of the career taxonomy, classifying over 800 PhD alumni records via nine coders. Results: We identified areas of discordance in the taxonomy, and progressively refined UCOT 2017 and an accompanying Guidance Document to improve inter-rater reliability across all three tiers of the career taxonomy. However, differing interpretations of the classifications, especially for faculty classifications in the third tier, resulted in continued discordance among the coders. We addressed this discordance with clarifying language in the Guidance Document, and proposed the addition of a flag system for identification of the title, rank, and prefix of faculty members. This labeling system provides the additional benefit of highlighting the granularity and the intersectionality of faculty job functions, while maintaining the ability to sort by - and report data on - faculty and postdoctoral trainee roles, as is required by some national and federal reporting guidelines. We provide specific crosswalk guidance for how a user may choose to incorporate our suggestions while maintaining the ability to report in accordance with UCOT 2017. Conclusions: Our findings underscore the importance of detailed guidance documents, coder training, and periodic collaborative review of career outcomes taxonomies as PhD careers evolve in the global workforce. Implications for coder-training and use of novice coders are also discussed.
| S-EPMC7014580 | biostudies-literature
Project description:BackgroundInter-rater reliability (IRR) is mainly assessed based on only two reviewers of unknown expertise. The aim of this paper is to examine differences in the IRR of the Assessment of Multiple Systematic Reviews (AMSTAR) and R(evised)-AMSTAR depending on the pair of reviewers.MethodsFive reviewers independently applied AMSTAR and R-AMSTAR to 16 systematic reviews (eight Cochrane reviews and eight non-Cochrane reviews) from the field of occupational health. Responses were dichotomized and reliability measures were calculated by applying Holsti's method (r) and Cohen's kappa (?) to all potential pairs of reviewers. Given that five reviewers participated in the study, there were ten possible pairs of reviewers.ResultsInter-rater reliability varied for AMSTAR between r = 0.82 and r = 0.98 (median r = 0.88) using Holsti's method and ? = 0.41 and ? = 0.69 (median ? = 0.52) using Cohen's kappa and for R-AMSTAR between r = 0.77 and r = 0.89 (median r = 0.82) and ? = 0.32 and ? = 0.67 (median ? = 0.45) depending on the pair of reviewers. The same pair of reviewers yielded the highest IRR for both instruments. Pairwise Cohen's kappa reliability measures showed a moderate correlation between AMSTAR and R-AMSTAR (Spearman's ? =0.50). The mean inter-rater reliability for AMSTAR was highest for item 1 (? = 1.00) and item 5 (? = 0.78), while lowest values were found for items 3, 8, 9 and 11, which showed only fair agreement.ConclusionsInter-rater reliability varies widely depending on the pair of reviewers. There may be some shortcomings associated with conducting reliability studies with only two reviewers. Further studies should include additional reviewers and should probably also take account of their level of expertise.
| S-EPMC5504630 | biostudies-literature