Inter-Rater Agreement in the Assessment of Video Recordings of Eye Drop Instillation by Glaucoma Patients.
Ontology highlight
ABSTRACT: Purpose
To create a standardized method for evaluating the video recordings of patients self-instilling eye drops and to determine the level of agreement of eye drop instillation efficacy, safety and efficiency ratings by three masked graders.Design
Prospective cross-sectional study.Participants
78 patients with open-angle glaucoma or ocular hypertension who had at least 6 months of experience with the use of eye drop medications.Methods
Participants were video recorded while self-instilling artificial tears sequentially to both eyes. Three masked observers graded these video recordings on three criteria: efficacy (the determination of whether an eye drop was instilled on the ocular surface), safety (assessment of whether the tip of the medication bottle made contact with the ocular surface or eyelids), and efficiency (the number of eye drops expressed from the bottle).Main outcome measures
After grading the video recordings based on efficacy, safety, and efficiency, kappa statistics were used to estimate inter-rater agreement.Results
The mean kappa level of agreement for efficacy, safety, and efficiency was 0.64 (95% confidence interval (CI), 0.42-0.87), 0.73 (95% CI, 0.58-0.88), and 0.62 (95% CI, 0.42-0.81), respectively.Conclusions
We demonstrated good inter-rater reproducibility of the masked analysis of video recordings of patients self-instilling eye drops based on three criteria: efficiency, safety, and efficacy.
SUBMITTER: Park MS
PROVIDER: S-EPMC4711577 | biostudies-literature | 2016
REPOSITORIES: biostudies-literature
Publications
Inter-Rater Agreement in the Assessment of Video Recordings of Eye Drop Instillation by Glaucoma Patients.
Park Meghan S MS Patel Marguerite M MM Sarezky Daniel D Rojas Carin C Choo Clara C Choi Michael M Liu Dachao D Rademaker Alfred W AW Tanna Angelo P AP
PloS one 20160105 1
<h4>Purpose</h4>To create a standardized method for evaluating the video recordings of patients self-instilling eye drops and to determine the level of agreement of eye drop instillation efficacy, safety and efficiency ratings by three masked graders.<h4>Design</h4>Prospective cross-sectional study.<h4>Participants</h4>78 patients with open-angle glaucoma or ocular hypertension who had at least 6 months of experience with the use of eye drop medications.<h4>Methods</h4>Participants were video re ...[more]
Similar Datasets
Project description:Researchers have long been interested in how social evaluations are made based upon first impressions of faces. It is also important to consider the level of agreement we see in such evaluations across raters and what this may tell us. Typically, high levels of inter-rater agreement for facial judgements are reported, but the measures used may be misleading. At present, studies commonly report Cronbach's ? as a way to quantify agreement, although problematically, there are various issues with the use of this measure. Most importantly, because researchers treat raters as items, Cronbach's ? is inflated by larger sample sizes even when agreement between raters is fixed. Here, we considered several alternative measures and investigated whether these better discriminate between traits that were predicted to show low (parental resemblance), intermediate (attractiveness, dominance, trustworthiness), and high (age, gender) levels of agreement. Importantly, the level of inter-rater agreement has not previously been studied for many of these traits. In addition, we investigated whether familiar faces resulted in differing levels of agreement in comparison with unfamiliar faces. Our results suggest that alternative measures may prove more informative than Cronbach's ? when determining how well raters agree in their judgements. Further, we found no apparent influence of familiarity on levels of agreement. Finally, we show that, like attractiveness, both trustworthiness and dominance show significant levels of private taste (personal or idiosyncratic rater perceptions), although shared taste (perceptions shared with other raters) explains similar levels of variance in people's perceptions. In conclusion, we recommend that researchers investigating social judgements of faces consider alternatives to Cronbach's ? but should also be prepared to examine both the potential value and origin of private taste as these might prove informative.
| S-EPMC6097668 | biostudies-literature
Project description:PurposeTo assess the Support, Educate, Empower (SEE) personalized glaucoma coaching program impact on (1) eye drop instillation technique and (2) eye drop instillation self-efficacy.DesignProspective pre-post pilot study.ParticipantsPatients with a diagnosis of glaucoma or ocular hypertension taking ≥1 glaucoma medication, ≥40 years old, spoke English, self-administered their eye drops, and ≤80% adherent to their glaucoma medication by electronic monitoring.MethodsEye drop administration was video recorded before the first SEE in-person coaching session, which included teaching eye drop instillation techniques using a motivational interviewing-based approach. At the third and final in-person counseling session 6 months later, eye drop administration was video recorded. Participants' self-efficacy was assessed using the validated Eye Drop Technique Self-Efficacy Scale (EDTSES) survey at baseline and 1 month after completion of the program. Before and after intervention videos were assessed by an observer masked to time point. Before versus after intervention comparisons were made using McNemar's and paired t tests.Main outcome measuresThe main outcome was change in participants' eye drop instillation technique as measured by (1) accuracy of an eye drop landing on the eye, (2) ability to instill an eye drop on the first attempt, and (3) contaminating the bottle by contact with ocular surface, eyelashes, and skin. The secondary outcome measure was before versus after change in the EDTSES score (6 items, each assessed on a 3-point Likert scale, with higher scores indicating better self-efficacy).ResultsThirty-nine participants completed the SEE intervention, 38 with before and after EDTSES scores and 31 with video recordings. Six of 31 participants instilling drops outside the eye before intervention improved their technique after intervention, whereas 2 participants worsened (P = 0.157). From before to after intervention, participants demonstrated significant improvement in not touching the ocular surface (P = 0.046), the eyelashes (P = 0.020), or the skin (P = 0.025) with the bottle tip. A significant increase was found in eye drop instillation self-efficacy from an average score of 2.6 (standard deviation [SD], 0.3) to 2.8 (SD, 0.2) on the EDTSES score (P = 0.007).ConclusionsThe SEE program significantly decreased eye drop bottle contamination, increased eye drop instillation self-efficacy, and demonstrated an insignificant increase in ability to instill drops successfully and accurately.
| S-EPMC7854833 | biostudies-literature
Project description:OBJECTIVE:Surgical site infection (SSI) is the most common nosocomial infection, in vascular surgery patients, who experience a high rate of readmission. Facilitating transition from hospital to outpatient care with digital image-based wound monitoring has the potential to detect and to enable treatment of SSI at an early stage. In this study, we evaluated whether smartphone digital images can supplant in-person evaluation of postoperative vascular surgery wounds. METHODS:We developed a wound assessment checklist using previously validated criteria. We recruited adults who underwent a vascular surgical procedure between 2014 and 2015, involving an incision of at least 3 cm, from a high-volume academic vascular surgery service. Vascular surgery care providers evaluated wounds in person using the assessment checklist; a different group of providers evaluated wounds by a smartphone digital image. Inter-rater agreement coefficients for wound characteristics and treatment plan were calculated within and between the in-person group and the digital image group; the sensitivity and specificity of digital images relative to in-person evaluation were determined. RESULTS:We assessed a total of 80 wounds. Regardless of modality, inter-rater agreement was poor when wounds were evaluated for the presence of ecchymosis and redness; moderate for cellulitis; and high for the presence of a drain, necrosis, or dehiscence. As expected, the presence of drainage was more readily observed in person. Inter-rater agreement was high for both in-person and image-based assessment with respect to course of treatment, with near-perfect agreement for treatments ranging from antibiotics to surgical débridement to hospital readmission. No difference in agreement emerged when raters evaluated poor-quality compared with high-quality images. For most parameters, specificity was higher than sensitivity for image-based compared with "gold standard" in-person assessment. CONCLUSIONS:Using smartphone digital images is a valid method for evaluating postoperative vascular surgery wounds and is comparable to in-person evaluation with regard to most wound characteristics. The inter-rater reliability for determining treatment recommendations was universally high. Remote wound monitoring and assessment may play an integral role in future transitional care models to decrease readmission for SSI in vascular or other surgical patients. These findings will inform smartphone implementation in the clinical care setting as wound images transition from informal clinical communication to becoming part of the care standard.
| S-EPMC4913032 | biostudies-literature
Project description:Low concordance between drug-drug interaction (DDI) knowledge bases is a well-documented concern. One potential cause of inconsistency is variability between drug experts in approach to assessing evidence about potential DDIs. In this study, we examined the face validity and inter-rater reliability of a novel DDI evidence evaluation instrument designed to be simple and easy to use.MethodsA convenience sample of participants with professional experience evaluating DDI evidence was recruited. Participants independently evaluated pre-selected evidence items for 5 drug pairs using the new instrument. For each drug pair, participants labeled each evidence item as sufficient or insufficient to establish the existence of a DDI based on the evidence categories provided by the instrument. Participants also decided if the overall body of evidence supported a DDI involving the drug pair. Agreement was computed both at the evidence item and drug pair levels. A cut-off of ≥ 70% was chosen as the agreement threshold for percent agreement, while a coefficient > 0.6 was used as the cut-off for chance-corrected agreement. Open ended comments were collected and coded to identify themes related to the participants' experience using the novel approach.ResultsThe face validity of the new instrument was established by two rounds of evaluation involving a total of 6 experts. Fifteen experts agreed to participate in the reliability assessment, and 14 completed the study. Participant agreement on the sufficiency of 22 of the 34 evidence items (65%) did not exceed the a priori agreement threshold. Similarly, agreement on the sufficiency of evidence for 3 of the 5 drug pairs (60%) was poor. Chance-corrected agreement at the drug pair level further confirmed the poor interrater reliability of the instrument (Gwet's AC1 = 0.24, Conger's Kappa = 0.24). Participant comments suggested several possible reasons for the disagreements including unaddressed subjectivity in assessing an evidence item's type and study design, an infeasible separation of evidence evaluation from the consideration of clinical relevance, and potential issues related to the evaluation of DDI case reports.ConclusionsEven though the key findings were negative, the study's results shed light on how experts approach DDI evidence assessment, including the importance situating evidence assessment within the context of consideration of clinical relevance. Analysis of participant comments within the context of the negative findings identified several promising future research directions including: novel computer-based support for evidence assessment; formal evaluation of a more comprehensive evidence assessment approach that requires consideration of specific, explicitly stated, clinical consequences; and more formal investigation of DDI case report assessment instruments.
| S-EPMC7537787 | biostudies-literature
Project description:Background and aimsVarious studies have been previously conducted on the diagnosis of lymphadenopathy as benign or malignant, but the results vary. These studies did not describe the inter-rater agreement on the EUS features of lymphadenopathy. In this study, we evaluate the inter-rater agreement on EUS features and propose EUS diagnostic norms for lymphadenopathy based on inter-rater agreement.MethodA total of 68 lymph nodes subjected to EUS-fine needle aspiration (FNA) were reviewed by five endoscopic experts. The EUS features evaluated lymph node size, shape, border, margin, echogenicity, homogeneity, and the hilum of the lymph node. Inter-rater agreement (multi-rater kappa statics) was performed. We established new criteria using results with a high degree of inter-rater agreement from EUS features and compared them with the former criteria.ResultThere was a moderate agreement on shape, kappa (K) = 0.44 (95% confidence interval [CI]: 0.34-0.54), and fair agreement on echogenicity, homogeneity, border, and hilum of the lymph node, K (95% CI) = 0.33 (0.17-0.38), 0.34 (0.26-0.35), 0.22 (0.21-0.31), and 0.22 (0.11-0.26), respectively. This resulted in the establishment of new EUS diagnostic criteria using shape, long axis > 20 mm and short axis > 10 mm. New criteria were superior to old criteria (area under the curve 0.82 vs 0.52, P < 0.001).ConclusionEUS diagnostic criteria for lymphadenopathy based on inter-rater agreement were more accurate than old criteria. This result will be useful for the diagnosis of lymphadenopathy.
| S-EPMC6386303 | biostudies-literature
Project description:ObjectivesElectroencephalography (EEG) technologists commonly screen continuous EEG. Until now, the inter-rater agreement or sensitivity for important EEG findings has been unknown in this group.MethodsTwenty-nine EEG technologists and three clinical neurophysiologists interpreted 90 five-minute samples of pediatric critical care EEG. Inter-rater agreement was examined with Cohen's kappa and Fleiss' kappa for EEG findings. A gold-standard consensus agreement was developed for examining sensitivity and specificity for seizures or discontinuity. Kruskal-Wallis tests with Benjamani-Hochberg corrections for multiple comparisons were utilized to examine associations between correct scoring and certification status and years of experience.ResultsAggregate agreement was moderate for seizures and fair for EEG background continuity among EEG technologists. Individual agreement for seizures and continuity varied from slight to substantial. For individual EEG technologists, sensitivity for seizures ranged from 44 to 93% and sensitivity for continuity ranged from 81 to 100%. Raters with Certified Long Term Monitoring credentials were more likely to identify seizures correctly.SignificanceThis is the first study to evaluate inter-rater agreement and interpretation correctness among EEG technologists interpreting pediatric critical care EEG. EEG technologists demonstrated better aggregate agreement for seizure detection than other EEG findings, yet individual results and internal consistency varied widely. These data provide important insight into the common practice of utilizing EEG technologists for screening critical care EEG.
| S-EPMC7856064 | biostudies-literature
Project description:Objectives To explore agreement among healthcare professionals assessing eligibility for work disability benefits.Design Systematic review and narrative synthesis of reproducibility studies.Data sources Medline, Embase, and PsycINFO searched up to 16 March 2016, without language restrictions, and review of bibliographies of included studies.Eligibility criteria Observational studies investigating reproducibility among healthcare professionals performing disability evaluations using a global rating of working capacity and reporting inter-rater reliability by a statistical measure or descriptively. Studies could be conducted in insurance settings, where decisions on ability to work include normative judgments based on legal considerations, or in research settings, where decisions on ability to work disregard normative considerations. : Teams of paired reviewers identified eligible studies, appraised their methodological quality and generalisability, and abstracted results with pretested forms. As heterogeneity of research designs and findings impeded a quantitative analysis, a descriptive synthesis stratified by setting (insurance or research) was performed.Results From 4562 references, 101 full text articles were reviewed. Of these, 16 studies conducted in an insurance setting and seven in a research setting, performed in 12 countries, met the inclusion criteria. Studies in the insurance setting were conducted with medical experts assessing claimants who were actual disability claimants or played by actors, hypothetical cases, or short written scenarios. Conditions were mental (n=6, 38%), musculoskeletal (n=4, 25%), or mixed (n=6, 38%). Applicability of findings from studies conducted in an insurance setting to real life evaluations ranged from generalisable (n=7, 44%) and probably generalisable (n=3, 19%) to probably not generalisable (n=6, 37%). Median inter-rater reliability among experts was 0.45 (range intraclass correlation coefficient 0.86 to κ-0.10). Inter-rater reliability was poor in six studies (37%) and excellent in only two (13%). This contrasts with studies conducted in the research setting, where the median inter-rater reliability was 0.76 (range 0.91-0.53), and 71% (5/7) studies achieved excellent inter-rater reliability. Reliability between assessing professionals was higher when the evaluation was guided by a standardised instrument (23 studies, P=0.006). No such association was detected for subjective or chronic health conditions or the studies' generalisability to real world evaluation of disability (P=0.46, 0.45, and 0.65, respectively).Conclusions Despite their common use and far reaching consequences for workers claiming disabling injury or illness, research on the reliability of medical evaluations of disability for work is limited and indicates high variation in judgments among assessing professionals. Standardising the evaluation process could improve reliability. Development and testing of instruments and structured approaches to improve reliability in evaluation of disability are urgently needed.
| S-EPMC5283380 | biostudies-literature
Project description:Objectives: It has been recommended that clinical trials of Traditional Chinese Medicine (TCM) would be more ecologically valid if its characteristic mode of diagnostic reasoning were integrated into their design. In that context, however, it is also widely held that demonstrating a high level of agreement on initial TCM diagnoses is necessary for the replicability that the biomedical paradigm requires for the conclusions from such trials. Our aim was to review, summarize, and critique quantitative experimental studies of inter-rater agreement in TCM, and some of their underlying assumptions. Design: Systematic electronic searches were conducted for articles that reported a quantitative measure of inter-rater agreement across a number of rating choices based on examinations of human subjects in person by TCM practitioners, and published in English language peer-reviewed journals. Publications in languages other than English were not included, nor those appearing in other than peer-reviewed journals. Predefined categories of information were extracted from full texts by two investigators working independently. Each article was scored for methodological quality. Outcome measures: Design features across all studies and levels of inter-rater agreement across studies that reported the same type of outcome statistic were compared. Results: Twenty-one articles met inclusion criteria. Fourteen assessed inter-rater agreement on TCM diagnoses, two on diagnostic signs found upon traditional TCM examination, and five on novel rating schemes derived from TCM theory and practice. Raters were students of TCM colleges or graduates of TCM training programs with 3 or more years experience and licensure. Type of outcome statistic varied. Mean rates of pairwise agreement averaged 57% (median 65, range 19-96) across the 9 studies reporting them. Mean Cohen's kappa averaged 0.34 (median 0.34, range 0.07-0.59) across the seven studies reporting them. Meta-analysis was not possible due to variations in study design and outcome statistics. High risks of bias and confounding, and deficits in statistical reporting were common. Conclusions: With a few exceptions, the levels of agreement were low to moderate. Most studies had significant deficits of both methodology and reporting. Results overall suggest a few design features that might contribute to higher levels of agreement. These should be studied further with better experimental controls and more thorough reporting of outcomes. In addition, methods of complex systems analysis should be explored to more adequately model the relationship between clinical outcomes, and the series of diagnoses and treatments that are the norm in actual TCM practice.
| S-EPMC6864748 | biostudies-literature
Project description:Reproducible and accurate recognition of presence and severity of ataxia in horses with neurologic disease is important when establishing a diagnosis, assessing response to treatment, and making recommendations that might influence rider safety or a decision for euthanasia.To determine the reproducibility and validity of the gait assessment component in the neurologic examination of horses.Twenty-five horses referred to the Royal Veterinary College Equine Referral Hospital for neurological assessment (n = 15), purchased (without a history of gait abnormalities) for an unrelated study (n = 5), or donated because of perceived ataxia (n = 5).Utilizing a prospective study design; a group of board-certified medicine (n = 2) and surgery (n = 2) clinicians and residents (n = 2) assessed components of the equine neurologic examination (live and video recorded) and assigned individual and overall neurologic gait deficit grades (0-4). Inter-rater agreement and assessment-reassessment reliability were quantified using intraclass correlation coefficients (ICC).The ICCs of the selected components of the neurologic examination ranged from 0 to 0.69. "Backing up" and "recognition of mistakes over obstacle" were the only components with an ICC > 0.6. Assessment-reassessment agreement was poor to fair. The agreement on gait grading was good overall (ICC = 0.74), but poor for grades ? 1 (ICC = 0.08) and fair for ataxia grades ? 2 (ICC = 0.43). Clinicians with prior knowledge of a possible gait abnormality were more likely to assign a grade higher than the median grade.Clinicians should be aware of poor agreement even between skilled observers of equine gait abnormalities, especially when the clinical signs are subtle.
| S-EPMC4857973 | biostudies-literature
Project description:Reasons for performing studyLungeing is an important part of lameness examinations as the circular path may accentuate low-grade lameness. Movement asymmetries related to the circular path, to compensatory movements and to pain make the lameness evaluation complex. Scientific studies have shown high inter-rater variation when assessing lameness during straight line movement.ObjectivesThe aim was to estimate inter- and intra-rater agreement of equine veterinarians evaluating lameness from videos of sound and lame horses during lungeing and to investigate the influence of veterinarians' experience and the objective degree of movement asymmetry on rater agreement.Study designCross-sectional observational study.MethodsVideo recordings and quantitative gait analysis with inertial sensors were performed in 23 riding horses of various breeds. The horses were examined at trot on a straight line and during lungeing on soft or hard surfaces in both directions. One video sequence was recorded per condition and the horses were classified as forelimb lame, hindlimb lame or sound from objective straight line symmetry measurements. Equine veterinarians (n = 86), including 43 with >5 years of orthopaedic experience, participated in a web-based survey and were asked to identify the lamest limb on 60 videos, including 10 repeats. The agreements between (inter-rater) and within (intra-rater) veterinarians were analysed with κ statistics (Fleiss, Cohen).ResultsInter-rater agreement κ was 0.31 (0.38/0.25 for experienced/less experienced) and higher for forelimb (0.33) than for hindlimb lameness (0.11) or soundness (0.08) evaluation. Median intra-rater agreement κ was 0.57.ConclusionsInter-rater agreement was poor for less experienced raters, and for all raters when evaluating hindlimb lameness. Since identification of the lame limb/limbs is a prerequisite for successful diagnosis, treatment and recovery, the high inter-rater variation when evaluating lameness on the lunge is likely to influence the accuracy and repeatability of lameness examinations and, indirectly, the efficacy of treatment.
| S-EPMC4964936 | biostudies-literature