Dataset Information

Inter-rater reliability and validity of good pharmacy practices measures in inspection of public sector health facility pharmacies in Uganda.

ABSTRACT: Background:The National Drug Authority (NDA) inspects and certifies private and public sector pharmacies in Uganda using an indicator-based inspection tool that measures adherence to good pharmacy practices (GPP). 67 measures identify the situation in the domains of premises, dispensing quality, stores management, and operating requirements. Although the GPP measures are well-recognized and used internationally, little is known about their validity and reliability. The study aimed to assess validity, which measures agreement of GPP measures between a gold standard inspector and NDA inspector and inter-rater reliability (IRR), which measures agreement among NDA inspectors, of GPP measures. Methods:We assessed validity and IRR by four teams of inspectors in eight government health facilities that represent three levels of care. Each team inspected two facilities, resulting in 24 total inspections. Each team comprised one central-level inspector, one district-level inspector, and one gold-standard inspector (i.e., a very experienced central-level inspector). We calculated median validity and IRR for each GPP measure, overall, indicator categorized as either critical, major, or minor, by domains, by the inspection decision (i.e., certified or not certified) and by adequatevalidity and IRR score (i.e., score???75%). Results:The median validity for all GPP measures was 69%, with 29 (43%) measures having an adequate validity of ?75%. The median IRR for all GPP measures was 71%, with 31 (46%) having an adequate IRR measure of ?75%. Validity did not differ significantly by indicator category, domain or level of care. Adequate IRR and validity score (?75%) was lowest for critical measures, which are key determinants of the certification decision, scoring 20 and 40% respectively. District inspectors had lower median validity for critical indicators and premises and higher validity for store management. Compared to central inspectors, the validity of district inspectors'certification decisions was lower; in the eight facilities, three district inspectors agreed with gold standard inspector vs. all eight central inspectors. Conclusions:Our findings question the validity and reliability of many GPP inspection measures, particularly critical measures that greatly impact certification decision. This study demonstrates the need for assessments of, and interventions to improve, validity and reproducibility of GPP measures and inspections.

SUBMITTER: Sekayombya B

PROVIDER: S-EPMC6341575 | biostudies-literature | 2019

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Inter-rater reliability and validity of good pharmacy practices measures in inspection of public sector health facility pharmacies in Uganda.

Sekayombya Brian B Nahamya David D Garabedian Laura L Seru Morries M Trap Birna B

Journal of pharmaceutical policy and practice 20190122

<h4>Background</h4>The National Drug Authority (NDA) inspects and certifies private and public sector pharmacies in Uganda using an indicator-based inspection tool that measures adherence to good pharmacy practices (GPP). 67 measures identify the situation in the domains of premises, dispensing quality, stores management, and operating requirements. Although the GPP measures are well-recognized and used internationally, little is known about their validity and reliability. The study aimed to ass ...[more]

PMID: 30680220

Similar Datasets

Project description:Objective:This study aims to assess good pharmacy practice (GPP) aspects and compare GPP scores among community pharmacies in Lebanon, using a tool developed jointly by the International Pharmaceutical Federation (FIP) and the World Health Organization (WHO) to improve and maintain standards of pharmacy practice. Methods:Data collection was carried out between July and October 2018 by a team of 10 licensed inspectors who work at the Lebanese Order of Pharmacists (OPL) and visited community pharmacies across Lebanon. The questionnaire was adapted to the Lebanese context and included 109 questions organized under five sections: socio-demographics, Indicator A (data management and data recording), Indicator B (services and health promotion), Indicator C (dispensing, preparation and administration of medicines), and Indicator D (storage and facilities). The value of 75% was considered as the cutoff point for adherence to indicators. Results:Out of 276 pharmacies visited, a total of 250 (90.58%) pharmacists participated in the study with one pharmacist being interviewed in every pharmacy. Results showed that 18.8% of pharmacists were generally adherents to GPP guidelines (scores above the 75% cutoff): 23.3% were adherent to indicator A, 21.6% to indicator B, 14.8% to indicator C and 13.2% to indicator D. Moreover, comparison of GPP scores across geographical regions revealed a higher adherence among community pharmacists working in the Beirut region compared to the North region, the South region, Mount Lebanon, and the Bekaa. Conclusions:Our study shows that community pharmacists in Lebanon do not fulfill GPP criteria set by FIP/WHO, and that this poor adherence is a trend across the country's geographical regions. Therefore, efforts should be made to raise awareness among pharmacists about the necessity to adhere to GPP guidelines and standards, and train them and support them appropriately to reach that goal. This is the first indicator-based comprehensive pilot assessment to evaluate GPP adherence in community pharmacies across Lebanon. Working on the optimization of this assessment tool is also warranted.

Project description:BackgroundTo build capacity in medicines management, the Uganda Ministry of Health introduced a nationwide supervision, performance assessment and recognition strategy (SPARS) in 2012. Medicines management supervisors (MMS) assess performance using 25 indicators to identify problems, focus supervision, and monitor improvement in medicines stock and storage management, ordering and reporting, and prescribing and dispensing. Although the indicators are well-recognized and used internationally, little was known about the reliability of these indicators. An initial assessment of inter-rater reliability (IRR), which measures agreement among raters (i.e., MMS), showed poor IRR; subsequently, we implemented efforts to improve IRR. The aim of this study was to assess IRR for SPARS indicators at two subsequent time points to determine whether IRR increased following efforts to improve reproducibility.MethodsIRR was assessed in 2011 and again after efforts to improve IRR in 2012 and 2013. Efforts included targeted training, providing detailed guidelines and job aids, and refining indicator definitions and response categories. In the assessments, teams of three MMS measured 24 SPARS indicators in 26 facilities. We calculated IRR as a team agreement score (i.e., percent of the MMS teams in which all three MMS had the same score). Two sample tests for proportions were used to compare IRR scores for each indicator, domain, and overall for the initial assessment and the following two assessments. We also compared the IRR scores for indicators classified as simple (binary) versus complex (multi-component). Logistic regression was used to identify supervisor group characteristics associated with domain-specific and overall IRR scores.ResultsInitially only five (21%) indicators had acceptable reproducibility, defined as an IRR score ≥ 75%. At the initial assessment, prescribing quality indicators had the lowest and stock management indicators had the highest IRR. By the third IRR assessment, 12 (50%) indicators had acceptable reproducibility, and the overall IRR score improved from 57% to 72%. The IRR of simple indicators was consistently higher than that of complex indicators in the three assessment periods. We found no correlation between IRR scores and MMS experience or professional background.ConclusionsAssessments of indicator reproducibility are needed to improve IRR. Using simple indicators is recommended.

Project description:Background: There has been a groundswell of national support for transparent tracking and dissemination of PhD career outcomes. In 2017, individuals from multiple institutions and professional organizations met to create the Unified Career Outcomes Taxonomy (UCOT 2017), a three-tiered taxonomy to help institutions uniformly classify career outcomes of PhD graduates. Early adopters of UCOT 2017, noted ambiguity in some categories of the career taxonomy, raising questions about its consistent application within and across institutions. Methods: To test and evaluate the consistency of UCOT 2017, we calculated inter-rater reliability across two rounds of iterative refinement of the career taxonomy, classifying over 800 PhD alumni records via nine coders. Results: We identified areas of discordance in the taxonomy, and progressively refined UCOT 2017 and an accompanying Guidance Document to improve inter-rater reliability across all three tiers of the career taxonomy. However, differing interpretations of the classifications, especially for faculty classifications in the third tier, resulted in continued discordance among the coders. We addressed this discordance with clarifying language in the Guidance Document, and proposed the addition of a flag system for identification of the title, rank, and prefix of faculty members. This labeling system provides the additional benefit of highlighting the granularity and the intersectionality of faculty job functions, while maintaining the ability to sort by - and report data on - faculty and postdoctoral trainee roles, as is required by some national and federal reporting guidelines. We provide specific crosswalk guidance for how a user may choose to incorporate our suggestions while maintaining the ability to report in accordance with UCOT 2017. Conclusions: Our findings underscore the importance of detailed guidance documents, coder training, and periodic collaborative review of career outcomes taxonomies as PhD careers evolve in the global workforce. Implications for coder-training and use of novice coders are also discussed.

Project description:PurposeFirst, to evaluate inter-rater reliability when human raters estimate the reading performance of visually impaired individuals using the MNREAD acuity chart. Second, to evaluate the agreement between computer-based scoring algorithms and compare them with human rating.MethodsReading performance was measured for 101 individuals with low vision, using the Portuguese version of the MNREAD test. Seven raters estimated the maximum reading speed (MRS) and critical print size (CPS) of each individual MNREAD curve. MRS and CPS were also calculated automatically for each curve using two different algorithms: the original standard deviation method (SDev) and a non-linear mixed effects (NLME) modeling. Intra-class correlation coefficients (ICC) were used to estimate absolute agreement between raters and/or algorithms.ResultsAbsolute agreement between raters was 'excellent' for MRS (ICC = 0.97; 95%CI [0.96, 0.98]) and 'moderate' to 'good' for CPS (ICC = 0.77; 95%CI [0.69, 0.83]). For CPS, inter-rater reliability was poorer among less experienced raters (ICC = 0.70; 95%CI [0.57, 0.80]) when compared to experienced ones (ICC = 0.82; 95%CI [0.76, 0.88]). Absolute agreement between the two algorithms was 'excellent' for MRS (ICC = 0.96; 95%CI [0.91, 0.98]). For CPS, the best possible agreement was found for CPS defined as the print size sustaining 80% of MRS (ICC = 0.77; 95%CI [0.68, 0.84]). Absolute agreement between raters and automated methods was 'excellent' for MRS (ICC = 0.96; 95% CI [0.88, 0.98] for SDev; ICC = 0.97; 95% CI [0.95, 0.98] for NLME). For CPS, absolute agreement between raters and SDev ranged from 'poor' to 'good' (ICC = 0.66; 95% CI [0.3, 0.80]), while agreement between raters and NLME was 'good' (ICC = 0.83; 95% CI [0.76, 0.88]).ConclusionFor MRS, inter-rater reliability is excellent, even considering the possibility of noisy and/or incomplete data collected in low-vision individuals. For CPS, inter-rater reliability is lower. This may be problematic, for instance in the context of multisite investigations or follow-up examinations. The NLME method showed better agreement with the raters than the SDev method for both reading parameters. Setting up consensual guidelines to deal with ambiguous curves may help improve reliability. While the exact definition of CPS should be chosen on a case-by-case basis depending on the clinician or researcher's motivations, evidence suggests that estimating CPS as the smallest print size sustaining about 80% of MRS would increase inter-rater reliability.

Project description:PurposeThere is limited knowledge on the reliability of risk of bias (ROB) tools for assessing internal validity in systematic reviews of exposure and frequency studies. We aimed to identify and then compare the inter-rater reliability (IRR) of six commonly used tools for frequency (Loney scale, Gyorkos checklist, American Academy of Neurology [AAN] tool) and exposure (Newcastle-Ottawa scale, SIGN50 checklist, AAN tool) studies.MethodsSix raters independently assessed the ROB of 30 frequency and 30 exposure studies using the three respective ROB tools. Articles were rated as low, intermediate, or high ROB. We calculated an intraclass correlation coefficient (ICC) for each tool and category of ROB tool. We compared the IRR between ROB tools and tool type by inspection of overlapping ICC 95% CIs and by comparing their coefficients after transformation to Fisher's Z values. We assessed the criterion validity of the AAN ROB tools by calculating an ICC for each rater in comparison with the original ratings from the AAN.ResultsAll individual ROB tools had an IRR in the substantial range or higher (ICC point estimates between 0.61 and 0.80). The IRR was almost perfect (ICC point estimate > 0.80) for the AAN frequency tool and the SIGN50 checklist. All tools were comparable in IRR, except for the AAN frequency tool which had a significantly higher ICC than the Gyorkos checklist (p = 0.021) and trended towards a higher ICC when compared to the Loney scale (p = 0.085). When examined by category of ROB tool, scales, and checklists had a substantial IRR, whereas the AAN tools had an almost perfect IRR. For the criterion validity of the AAN ROB tools, the average agreement between our raters and the original AAN ratings was moderate.ConclusionAll tools had substantial IRRs except for the AAN frequency tool and the SIGN50 checklist, which both had an almost perfect IRR. The AAN ROB tools were the only category of ROB tools to demonstrate an almost perfect IRR. This category of ROB tools had fewer and simpler criteria. Overall, parsimonious tools with clear instructions, such as those from the AAN, may provide more reliable ROB assessments.

Project description:ObjectivesTo investigate the intra- and inter-rater reliability of the total radiomics quality score (RQS) and the reproducibility of individual RQS items' score in a large multireader study.MethodsNine raters with different backgrounds were randomly assigned to three groups based on their proficiency with RQS utilization: Groups 1 and 2 represented the inter-rater reliability groups with or without prior training in RQS, respectively; group 3 represented the intra-rater reliability group. Thirty-three original research papers on radiomics were evaluated by raters of groups 1 and 2. Of the 33 papers, 17 were evaluated twice with an interval of 1 month by raters of group 3. Intraclass coefficient (ICC) for continuous variables, and Fleiss' and Cohen's kappa (k) statistics for categorical variables were used.ResultsThe inter-rater reliability was poor to moderate for total RQS (ICC 0.30-055, p < 0.001) and very low to good for item's reproducibility (k - 0.12 to 0.75) within groups 1 and 2 for both inexperienced and experienced raters. The intra-rater reliability for total RQS was moderate for the less experienced rater (ICC 0.522, p = 0.009), whereas experienced raters showed excellent intra-rater reliability (ICC 0.91-0.99, p < 0.001) between the first and second read. Intra-rater reliability on RQS items' score reproducibility was higher and most of the items had moderate to good intra-rater reliability (k - 0.40 to 1).ConclusionsReproducibility of the total RQS and the score of individual RQS items is low. There is a need for a robust and reproducible assessment method to assess the quality of radiomics research.Clinical relevance statementThere is a need for reproducible scoring systems to improve quality of radiomics research and consecutively close the translational gap between research and clinical implementation.Key points• Radiomics quality score has been widely used for the evaluation of radiomics studies. • Although the intra-rater reliability was moderate to excellent, intra- and inter-rater reliability of total score and point-by-point scores were low with radiomics quality score. • A robust, easy-to-use scoring system is needed for the evaluation of radiomics research.

Dataset Information

Inter-rater reliability and validity of good pharmacy practices measures in inspection of public sector health facility pharmacies in Uganda.

Publications

Inter-rater reliability and validity of good pharmacy practices measures in inspection of public sector health facility pharmacies in Uganda.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets