Inter-rater agreement of a modified and extended Swedish version of the Frenchay Activities Index (FAI).
Ontology highlight
ABSTRACT:
SUBMITTER: Wendel KA
PROVIDER: S-EPMC5549121 | biostudies-other | 2013 Sep
REPOSITORIES: biostudies-other
Similar Datasets
Project description:Researchers have long been interested in how social evaluations are made based upon first impressions of faces. It is also important to consider the level of agreement we see in such evaluations across raters and what this may tell us. Typically, high levels of inter-rater agreement for facial judgements are reported, but the measures used may be misleading. At present, studies commonly report Cronbach's α as a way to quantify agreement, although problematically, there are various issues with the use of this measure. Most importantly, because researchers treat raters as items, Cronbach's α is inflated by larger sample sizes even when agreement between raters is fixed. Here, we considered several alternative measures and investigated whether these better discriminate between traits that were predicted to show low (parental resemblance), intermediate (attractiveness, dominance, trustworthiness), and high (age, gender) levels of agreement. Importantly, the level of inter-rater agreement has not previously been studied for many of these traits. In addition, we investigated whether familiar faces resulted in differing levels of agreement in comparison with unfamiliar faces. Our results suggest that alternative measures may prove more informative than Cronbach's α when determining how well raters agree in their judgements. Further, we found no apparent influence of familiarity on levels of agreement. Finally, we show that, like attractiveness, both trustworthiness and dominance show significant levels of private taste (personal or idiosyncratic rater perceptions), although shared taste (perceptions shared with other raters) explains similar levels of variance in people's perceptions. In conclusion, we recommend that researchers investigating social judgements of faces consider alternatives to Cronbach's α but should also be prepared to examine both the potential value and origin of private taste as these might prove informative.
| S-EPMC6097668 | biostudies-literature
Project description:The aim of this study was to assess on videographs the intra- and inter-rater reproducibility of the Smile Esthetic Index (SEI) that has been previously validated on photographs. Smile videographs were obtained using a smartphone associated with the Smile Lite MDP mounted on a tripod. They were then randomized and evaluated twice consecutively at a 1-week interval by three periodontists according to the SEI based on 10 variables. Cohen's Kappa and Fleiss' Kappa tests were performed to measure intra- and inter-rater agreement. Sixty-five smile videographs of 24 men and 41 women (mean age 33 ± 11.3 years) were scored. A mean intra-rater agreement of 0.68 (0.64-0.73) was obtained, representing substantial agreement. The inter-rater agreement calculated for each variable ranged from 0.31 for the variable "absence of visible excessive gingiva" to 0.90 for the variable "absence of diastema and/or missing inter-dental papilla." Within the limits of this study, we have demonstrated that it was possible to use videographs to reproducibly evaluate an aesthetic score (SEI) previously validated on photographs.
| S-EPMC9139191 | biostudies-literature
Project description:BackgroundIn inter-rater agreement studies, the assessment behaviour of raters can be influenced by their experience, training levels, the degree of willingness to take risks, and the availability of clear guidelines for the assessment. When the assessment behaviour of raters differentiates for some levels of an ordinal classification, a grey zone occurs between the corresponding adjacent cells to these levels around the main diagonal of the table. A grey zone introduces a negative bias to the estimate of the agreement level between the raters. In that sense, it is crucial to detect the existence of a grey zone in an agreement table.MethodsIn this study, a framework composed of a metric and the corresponding threshold is developed to identify grey zones in an agreement table. The symmetry model and Cohen's kappa are used to define the metric, and the threshold is based on a nonlinear regression model. A numerical study is conducted to assess the accuracy of the developed framework. Real data examples are provided to illustrate the use of the metric and the impact of identifying a grey zone.ResultsThe sensitivity and specificity of the proposed framework are shown to be very high under moderate, substantial, and near-perfect agreement levels for [Formula: see text] and [Formula: see text] tables and sample sizes greater than or equal to 100 and 50, respectively. Real data examples demonstrate that when a grey zone is detected in the table, it is possible to report a notably higher level of agreement in the studies.ConclusionsThe accuracy of the proposed framework is sufficiently high; hence, it provides practitioners with a precise way to detect the grey zones in agreement tables.
| S-EPMC9814438 | biostudies-literature
Project description:BackgroundWorldwide, there is a large and growing group of older adults. Frailty is known as an important discriminatory factor for poor outcomes. The Clinical Frailty Scale (CFS) has become a frequently used frailty instrument in different clinical settings and health care sectors, and it has shown good predictive validity. The aims of this study were to describe and validate the translation and cultural adaptation of the CFS into Swedish (CFS-SWE), and to test the inter-rater reliability (IRR) for registered nurses using the CFS-SWE.MethodsAn observational study design was employed. The ISPOR principles were used for the translation, linguistic validation and cultural adaptation of the scale. To test the IRR, 12 participants were asked to rate 10 clinical case vignettes using the CFS-SWE. The IRR was assessed using intraclass correlation and Krippendorff's alpha agreement coefficient test.ResultsThe Clinical Frailty Scale was translated and culturally adapted into Swedish and is presented in its final form. The IRR for all raters, measured by an intraclass correlation test, resulted in an absolute agreement value among the raters of 0.969 (95% CI: 0.929-0.991) and a consistency value of 0.979 (95% CI: 0.953-0.994), which indicates excellent reliability. Krippendorff's alpha agreement coefficient for all raters was 0.969 (95% CI: 0.917-0.988), indicating near-perfect agreement. The sensitivity of the reliability was examined by separately testing the IRR of the group of specialised registered nurses and non-specialised registered nurses respectively, with consistent and similar results.ConclusionThe Clinical Frailty Scale was translated, linguistically validated and culturally adapted into Swedish following a well-established standard technique. The IRR was excellent, judged by two established, separately used, reliability tests. The reliability test results did not differ between non-specialised and specialised registered nurses. However, the use of case vignettes might reduce the generalisability of the reliability findings to real-life settings. The CFS has the potential to be a common reference tool, especially when older adults are treated and rehabilitated in different care sectors.
| S-EPMC10696827 | biostudies-literature
Project description:BACKGROUND:To analyse the reliability, variance and execution time of the Extended Timed Up and Go (Extended TUG) test in three age groups of elderly participants (G1: 55-64?years; G2: 65-74?years; G3: 75-85?years). METHODS:An analytical cross-sectional study of 114 recruited participants (63 women) of average age 70.17 (± 7.3) years was undertaken. Each participant performed the Extended TUG three consecutive times, with a rest break between tests of 120?s. Both the intragroup and intergroup reliability of the measurements in the Extended TUG were analysed. RESULTS:The reliability of the Extended TUG test is excellent for the first and second decades but drops down to good for the third decade. Specifically, intragroup reliability ranged from 0.784 for G3 to 0.977 for G1 (G2?=?0.858). Intergroup reliability, compared with intragroup reliability, was slightly lower, ranging between 0.779 for G3 and 0.972 for G1 (G2?=?0.853). CONCLUSION:The reliability of the Extended TUG test progressively decreases with increasing age, being excellent for the younger age groups and good for the oldest age group.
| S-EPMC7020600 | biostudies-literature
Project description:Background and aimsVarious studies have been previously conducted on the diagnosis of lymphadenopathy as benign or malignant, but the results vary. These studies did not describe the inter-rater agreement on the EUS features of lymphadenopathy. In this study, we evaluate the inter-rater agreement on EUS features and propose EUS diagnostic norms for lymphadenopathy based on inter-rater agreement.MethodA total of 68 lymph nodes subjected to EUS-fine needle aspiration (FNA) were reviewed by five endoscopic experts. The EUS features evaluated lymph node size, shape, border, margin, echogenicity, homogeneity, and the hilum of the lymph node. Inter-rater agreement (multi-rater kappa statics) was performed. We established new criteria using results with a high degree of inter-rater agreement from EUS features and compared them with the former criteria.ResultThere was a moderate agreement on shape, kappa (K) = 0.44 (95% confidence interval [CI]: 0.34-0.54), and fair agreement on echogenicity, homogeneity, border, and hilum of the lymph node, K (95% CI) = 0.33 (0.17-0.38), 0.34 (0.26-0.35), 0.22 (0.21-0.31), and 0.22 (0.11-0.26), respectively. This resulted in the establishment of new EUS diagnostic criteria using shape, long axis > 20 mm and short axis > 10 mm. New criteria were superior to old criteria (area under the curve 0.82 vs 0.52, P < 0.001).ConclusionEUS diagnostic criteria for lymphadenopathy based on inter-rater agreement were more accurate than old criteria. This result will be useful for the diagnosis of lymphadenopathy.
| S-EPMC6386303 | biostudies-literature
Project description:ObjectivesElectroencephalography (EEG) technologists commonly screen continuous EEG. Until now, the inter-rater agreement or sensitivity for important EEG findings has been unknown in this group.MethodsTwenty-nine EEG technologists and three clinical neurophysiologists interpreted 90 five-minute samples of pediatric critical care EEG. Inter-rater agreement was examined with Cohen's kappa and Fleiss' kappa for EEG findings. A gold-standard consensus agreement was developed for examining sensitivity and specificity for seizures or discontinuity. Kruskal-Wallis tests with Benjamani-Hochberg corrections for multiple comparisons were utilized to examine associations between correct scoring and certification status and years of experience.ResultsAggregate agreement was moderate for seizures and fair for EEG background continuity among EEG technologists. Individual agreement for seizures and continuity varied from slight to substantial. For individual EEG technologists, sensitivity for seizures ranged from 44 to 93% and sensitivity for continuity ranged from 81 to 100%. Raters with Certified Long Term Monitoring credentials were more likely to identify seizures correctly.SignificanceThis is the first study to evaluate inter-rater agreement and interpretation correctness among EEG technologists interpreting pediatric critical care EEG. EEG technologists demonstrated better aggregate agreement for seizure detection than other EEG findings, yet individual results and internal consistency varied widely. These data provide important insight into the common practice of utilizing EEG technologists for screening critical care EEG.
| S-EPMC7856064 | biostudies-literature
Project description:Objectives To explore agreement among healthcare professionals assessing eligibility for work disability benefits.Design Systematic review and narrative synthesis of reproducibility studies.Data sources Medline, Embase, and PsycINFO searched up to 16 March 2016, without language restrictions, and review of bibliographies of included studies.Eligibility criteria Observational studies investigating reproducibility among healthcare professionals performing disability evaluations using a global rating of working capacity and reporting inter-rater reliability by a statistical measure or descriptively. Studies could be conducted in insurance settings, where decisions on ability to work include normative judgments based on legal considerations, or in research settings, where decisions on ability to work disregard normative considerations. : Teams of paired reviewers identified eligible studies, appraised their methodological quality and generalisability, and abstracted results with pretested forms. As heterogeneity of research designs and findings impeded a quantitative analysis, a descriptive synthesis stratified by setting (insurance or research) was performed.Results From 4562 references, 101 full text articles were reviewed. Of these, 16 studies conducted in an insurance setting and seven in a research setting, performed in 12 countries, met the inclusion criteria. Studies in the insurance setting were conducted with medical experts assessing claimants who were actual disability claimants or played by actors, hypothetical cases, or short written scenarios. Conditions were mental (n=6, 38%), musculoskeletal (n=4, 25%), or mixed (n=6, 38%). Applicability of findings from studies conducted in an insurance setting to real life evaluations ranged from generalisable (n=7, 44%) and probably generalisable (n=3, 19%) to probably not generalisable (n=6, 37%). Median inter-rater reliability among experts was 0.45 (range intraclass correlation coefficient 0.86 to κ-0.10). Inter-rater reliability was poor in six studies (37%) and excellent in only two (13%). This contrasts with studies conducted in the research setting, where the median inter-rater reliability was 0.76 (range 0.91-0.53), and 71% (5/7) studies achieved excellent inter-rater reliability. Reliability between assessing professionals was higher when the evaluation was guided by a standardised instrument (23 studies, P=0.006). No such association was detected for subjective or chronic health conditions or the studies' generalisability to real world evaluation of disability (P=0.46, 0.45, and 0.65, respectively).Conclusions Despite their common use and far reaching consequences for workers claiming disabling injury or illness, research on the reliability of medical evaluations of disability for work is limited and indicates high variation in judgments among assessing professionals. Standardising the evaluation process could improve reliability. Development and testing of instruments and structured approaches to improve reliability in evaluation of disability are urgently needed.
| S-EPMC5283380 | biostudies-literature
Project description:The objective of this study was to examine the content validity of commonly used muscle performance tests in military personnel and to investigate the reliability of a proposed test battery. For the content validity investigation, thirty selected tests were those described in the literature and/or commonly used in the Nordic and North Atlantic Treaty Organization (NATO) countries. Nine selected experts rated, on a four-point Likert scale, the relevance of these tests in relation to five different work tasks: lifting, carrying equipment on the body or in the hands, climbing, and digging. Thereafter, a content validity index (CVI) was calculated for each work task. The result showed excellent CVI (?0.78) for sixteen tests, which comprised of one or more of the military work tasks. Three of the tests; the functional lower-limb loading test (the Ranger test), dead-lift with kettlebells, and back extension, showed excellent content validity for four of the work tasks. For the development of a new muscle strength/endurance test battery, these three tests were further supplemented with two other tests, namely, the chins and side-bridge test. The inter-rater reliability was high (intraclass correlation coefficient, ICC2,1 0.99) for all five tests. The intra-rater reliability was good to high (ICC3,1 0.82-0.96) with an acceptable standard error of mean (SEM), except for the side-bridge test (SEM%>15). Thus, the final suggested test battery for a valid and reliable evaluation of soldiers' muscle performance comprised the following four tests; the Ranger test, dead-lift with kettlebells, chins, and back extension test. The criterion-related validity of the test battery should be further evaluated for soldiers exposed to varying physical workload.
| S-EPMC4503674 | biostudies-literature
Project description:Objectives: It has been recommended that clinical trials of Traditional Chinese Medicine (TCM) would be more ecologically valid if its characteristic mode of diagnostic reasoning were integrated into their design. In that context, however, it is also widely held that demonstrating a high level of agreement on initial TCM diagnoses is necessary for the replicability that the biomedical paradigm requires for the conclusions from such trials. Our aim was to review, summarize, and critique quantitative experimental studies of inter-rater agreement in TCM, and some of their underlying assumptions. Design: Systematic electronic searches were conducted for articles that reported a quantitative measure of inter-rater agreement across a number of rating choices based on examinations of human subjects in person by TCM practitioners, and published in English language peer-reviewed journals. Publications in languages other than English were not included, nor those appearing in other than peer-reviewed journals. Predefined categories of information were extracted from full texts by two investigators working independently. Each article was scored for methodological quality. Outcome measures: Design features across all studies and levels of inter-rater agreement across studies that reported the same type of outcome statistic were compared. Results: Twenty-one articles met inclusion criteria. Fourteen assessed inter-rater agreement on TCM diagnoses, two on diagnostic signs found upon traditional TCM examination, and five on novel rating schemes derived from TCM theory and practice. Raters were students of TCM colleges or graduates of TCM training programs with 3 or more years experience and licensure. Type of outcome statistic varied. Mean rates of pairwise agreement averaged 57% (median 65, range 19-96) across the 9 studies reporting them. Mean Cohen's kappa averaged 0.34 (median 0.34, range 0.07-0.59) across the seven studies reporting them. Meta-analysis was not possible due to variations in study design and outcome statistics. High risks of bias and confounding, and deficits in statistical reporting were common. Conclusions: With a few exceptions, the levels of agreement were low to moderate. Most studies had significant deficits of both methodology and reporting. Results overall suggest a few design features that might contribute to higher levels of agreement. These should be studied further with better experimental controls and more thorough reporting of outcomes. In addition, methods of complex systems analysis should be explored to more adequately model the relationship between clinical outcomes, and the series of diagnoses and treatments that are the norm in actual TCM practice.
| S-EPMC6864748 | biostudies-literature