Dataset Information

Inter-rater and intra-raters' variability in evaluating complete dentures insertion procedure in senior undergraduates' prosthodontics clinics.

ABSTRACT: Background:Direct clinical assessment is the mainstay of evaluation in dentistry education. An effective evaluation method in prosthodontics should be equally valid and consistent; however, this is not attained frequently. A limited number of studies have applied an analytic evaluation in prosthodontics. Objective:To compare the intra- and inter-raters' variability in two evaluation methods: glance and grade (global), and checklist and criteria (analytical). Moreover, to identify the components of the analytical evaluation system and its applicability. Methods:This cross-sectional study was carried out on outpatients attending removable prosthodontics clinics affiliated with King Abdulaziz University (Jeddah, Saudi Arabia) from December 2017 to April 2018. Two prosthodontist examiners evaluated a sample of 35 complete denture cases (20 male, 15 female) twice over a period of five months. Inter-rater and intra-rater agreement were computed using reliability test (interclass correlation coefficient ICC). Data were analyzed in IBM SPSS version 23, using paired-samples t-test, weighted kappa, and Wilcoxon signed-rank test. The level of significance was set at p≤0.05. Results:The intra-rater agreement for the first and second exposures under global and analytical evaluation methods with Examiner A was outstanding with 90.7% and 99.8% agreement respectively. While with Examiner B, global was lower but still in the acceptable range with about 78.1%, and 96.1% for the analytical evaluation. Inter-rater reliability analysis showed high agreement between the two raters in the first exposure of the analytical evaluation with 97.3%, while it was 87.5% in the global evaluation. This trend goes the same with analytical in the second exposure with 93.2%; however, the second exposure under global evaluation failed the cut off, which is only 56.6% agreement. In evaluation of inter-raters agreement, the second exposure of the global method demonstrated inconsistency between the two examiners (p=0.002), while the analytical second exposure demonstrated more homogeneity (p=0.050). Intra-rater variability between first and second exposure in analytical evaluation was (0.711 for the first rater and 0.677 for the second rater). Intra-rater variability between first and second exposure in global evaluation was (<0.001 for the first rater and 0.075 for the second rater). Conclusion:A simple objective and detailed method to evaluate complete denture insertion procedure was developed, and it showed that both intra-rater and inter-rater agreement were excellent for the analytical method that might overcome errors and subjectivity in evaluation that result from the limitations of global method. Results recommend suitability of using the analytical evaluation to improve reliability between raters.

SUBMITTER: Alammari M

PROVIDER: S-EPMC6140989 | biostudies-other | 2018 Sep

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Inter-rater and intra-raters' variability in evaluating complete dentures insertion procedure in senior undergraduates' prosthodontics clinics.

Alammari Manal M Nawar El-Sayed ES

Electronic physician 20180909 9

<h4>Background</h4>Direct clinical assessment is the mainstay of evaluation in dentistry education. An effective evaluation method in prosthodontics should be equally valid and consistent; however, this is not attained frequently. A limited number of studies have applied an analytic evaluation in prosthodontics.<h4>Objective</h4>To compare the intra- and inter-raters' variability in two evaluation methods: glance and grade (global), and checklist and criteria (analytical). Moreover, to identify ...[more]

PMID: 30258562

Similar Datasets

Project description:ObjectivesTo investigate the intra- and inter-rater reliability of the total radiomics quality score (RQS) and the reproducibility of individual RQS items' score in a large multireader study.MethodsNine raters with different backgrounds were randomly assigned to three groups based on their proficiency with RQS utilization: Groups 1 and 2 represented the inter-rater reliability groups with or without prior training in RQS, respectively; group 3 represented the intra-rater reliability group. Thirty-three original research papers on radiomics were evaluated by raters of groups 1 and 2. Of the 33 papers, 17 were evaluated twice with an interval of 1 month by raters of group 3. Intraclass coefficient (ICC) for continuous variables, and Fleiss' and Cohen's kappa (k) statistics for categorical variables were used.ResultsThe inter-rater reliability was poor to moderate for total RQS (ICC 0.30-055, p < 0.001) and very low to good for item's reproducibility (k - 0.12 to 0.75) within groups 1 and 2 for both inexperienced and experienced raters. The intra-rater reliability for total RQS was moderate for the less experienced rater (ICC 0.522, p = 0.009), whereas experienced raters showed excellent intra-rater reliability (ICC 0.91-0.99, p < 0.001) between the first and second read. Intra-rater reliability on RQS items' score reproducibility was higher and most of the items had moderate to good intra-rater reliability (k - 0.40 to 1).ConclusionsReproducibility of the total RQS and the score of individual RQS items is low. There is a need for a robust and reproducible assessment method to assess the quality of radiomics research.Clinical relevance statementThere is a need for reproducible scoring systems to improve quality of radiomics research and consecutively close the translational gap between research and clinical implementation.Key points• Radiomics quality score has been widely used for the evaluation of radiomics studies. • Although the intra-rater reliability was moderate to excellent, intra- and inter-rater reliability of total score and point-by-point scores were low with radiomics quality score. • A robust, easy-to-use scoring system is needed for the evaluation of radiomics research.

Project description:Foot pain drawings (manikins) are commonly used to describe foot pain location in self-report health surveys. Respondents shade the manikin where they experience pain. The manikin is then scored via a transparent overlay that divides the drawings into areas. In large population based studies they are often scored by multiple raters. A difference in how different raters score manikins (inter-rater repeatability), or in how an individual rater scores manikins over time (intra-rater repeatability) can therefore affect data quality. This study aimed to assess inter- and intra-rater repeatability of scoring of the foot manikin.A random sample was generated of 50 respondents to a large population based survey of adults aged 50 years and older who experienced foot pain and completed a foot manikin. Manikins were initially scored by any one of six administrative staff (Rating 1). These manikins were re-scored by a second rater (Rating 2). The second rater then re-scored the manikins one week later (Rating 3). The following scores were compared: Rating 1 versus Rating 2 (inter-rater repeatability), and Rating 2 versus Rating 3 (intra-rater repeatability). A novel set of clinically relevant foot pain regions made up of one or more individual areas on the foot manikin were developed, and assessed for inter- and intra-rater repeatability.Scoring agreement of 100% (all 50 manikins) was seen in 69% (40 out of 58) of individual areas for inter-rater scoring (range 94 to 100%), and 81% (47 out of 58) of areas for intra-rater scoring (range 96 to 100%). All areas had a kappa value of ?0.70 for inter- and intra-rater scoring. Scoring agreement of 100% was seen in 50% (10 out of 20) of pain regions for inter-rater scoring (range 96 to 100%), and 95% (19 out of 20) of regions for intra-rater scoring (range 98 to 100%). All regions had a kappa value of >0.70 for inter- and intra-rater scoring.Individual and multiple raters can reliably score the foot pain manikin. In addition, our proposed regions may be used to reliably classify different patterns of foot pain using the foot manikin.

Project description:BACKGROUND:There is a growing trend in the use of mobile health (mHealth) technologies in traditional Chinese medicine (TCM) and telemedicine, especially during the coronavirus disease (COVID-19) outbreak. Tongue diagnosis is an important component of TCM, but also plays a role in Western medicine, for example in dermatology. However, the procedure of obtaining tongue images has not been standardized and the reliability of tongue diagnosis by smartphone tongue images has yet to be evaluated. OBJECTIVE:The first objective of this study was to develop an operating classification scheme for tongue coating diagnosis. The second and main objective of this study was to determine the intra-rater and inter-rater reliability of tongue coating diagnosis using the operating classification scheme. METHODS:An operating classification scheme for tongue coating was developed using a stepwise approach and a quasi-Delphi method. First, tongue images (n=2023) were analyzed by 2 groups of assessors to develop the operating classification scheme for tongue coating diagnosis. Based on clinicians' (n=17) own interpretations as well as their use of the operating classification scheme, the results of tongue diagnosis on a representative tongue image set (n=24) were compared. After gathering consensus for the operating classification scheme, the clinicians were instructed to use the scheme to assess tongue features of their patients under direct visual inspection. At the same time, the clinicians took tongue images of the patients with smartphones and assessed tongue features observed in the smartphone image using the same classification scheme. The intra-rater agreements of these two assessments were calculated to determine which features of tongue coating were better retained by the image. Using the finalized operating classification scheme, clinicians in the study group assessed representative tongue images (n=24) that they had taken, and the intra-rater and inter-rater reliability of their assessments was evaluated. RESULTS:Intra-rater agreement between direct subject inspection and tongue image inspection was good to very good (Cohen ? range 0.69-1.0). Additionally, when comparing the assessment of tongue images on different days, intra-rater reliability was good to very good (? range 0.7-1.0), except for the color of the tongue body (?=0.22) and slippery tongue fur (?=0.1). Inter-rater reliability was moderate for tongue coating (Gwet AC2 range 0.49-0.55), and fair for color and other features of the tongue body (Gwet AC2=0.34). CONCLUSIONS:Taken together, our study has shown that tongue images collected via smartphone contain some reliable features, including tongue coating, that can be used in mHealth analysis. Our findings thus support the use of smartphones in telemedicine for detecting changes in tongue coating.

Project description:ObjectivesThe objectives of this study were to asses (1) inter-rater and intrarater reliability of ultrasound imaging in patients with hip osteoarthritis, and (2) agreement between ultrasound and X-ray findings of hip osteoarthritis using validated Outcome Measures in Rheumatology ultrasound definitions for pathology.DesignAn inter-rater and intrarater reliability study.SettingA single-centre study conducted at a regional hospital.Participants50 patients >39 years of age referred for radiography due to hip pain and suspected hip osteoarthritis were included. Exclusion criteria were previous hip surgery in the painful hip, suspected fracture or malignant changes in the hip.InterventionBilateral ultrasound examinations (n=92) were performed continuously by two experienced operators blinded to clinical information and other imaging findings. After 4-6 weeks, one operator reassessed the images. X-rays were assessed by a third imaging specialist.Primary and secondary outcome measuresInter-rater and intrarater reliability and agreement between ultrasound imaging and X-ray were assessed using Cohen's ordinal kappa statistics for binary categorical variables and weighted kappa for ordered categorical variables.ResultsKappa values (κ) for inter-rater reliability were 0.9 and 0.8 for hip effusion/synovitis and osteoarthritis grading, respectively. For acetabular and femoral osteophytes, femoral cartilage changes and labrum changes κ ranged from 0.4 to 0.7. Intrarater reliability had κ equal or higher compared with inter-rater reliability. Agreement between ultrasound and X-ray findings ranged from κ=0.2 to κ=0.5.ConclusionThis study demonstrated substantial to almost perfect reliability on the most common ultrasound findings related to hip osteoarthritis and osteoarthritis grading. Agreement on the grade of osteoarthritis between ultrasound and X-ray was moderate. Overall, these results support ultrasound imaging as a reliable tool in the assessment of hip osteoarthritis.

Dataset Information

Inter-rater and intra-raters' variability in evaluating complete dentures insertion procedure in senior undergraduates' prosthodontics clinics.

Publications

Inter-rater and intra-raters' variability in evaluating complete dentures insertion procedure in senior undergraduates' prosthodontics clinics.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets