Reliability of Quality Assessments in Research Synthesis: Securing the Highest Quality Bioinformation for HIT.
Ontology highlight
ABSTRACT:
SUBMITTER: Chiappelli F
PROVIDER: S-EPMC3449364 | biostudies-literature | 2012
REPOSITORIES: biostudies-literature
Publications
Reliability of Quality Assessments in Research Synthesis: Securing the Highest Quality Bioinformation for HIT.
Chiappelli Francesco F Phil André Barkhordarian C AB Arora Rashi R Phi Linda L Giroux Amy A Uyeda Molly M Kung Jason J Ramchandani Manisha M
Bioinformation 20120721 14
Current trends in bio-medicine include research synthesis and dissemination of bioinformation by means of health (bio) information technology (H[b] IT). Research must secure the validity and reliability of assessment tools to quantify research quality in the pursuit of the best available evidence. Our concerted work in this domain led to the revision of three instruments for that purpose, including the stringent characterization of inter-rater reliability and coefficient of agreement. It is time ...[more]
Similar Datasets
Project description:BackgroundThe objective of this study was to test the data quality, test-retest reliability and hospital-level reliability of the Patient-Reported Incident in Hospital Instrument (PRIH-I).Methods13 incident questions were included in a national patient-experience survey in Norway during the spring of 2011. All questions and a composite incident index were assessed by calculating missing-item rates, test-retest reliability and hospital-level reliability. A multivariate linear regression on a global item regarding incorrect treatment was used to assess the main sources of variation in patient-perceived incorrect treatment at hospitals.ResultsFive of the 13 patient-incident questions had a missing-item rate of >20%. Only one item met the criterion of 0.7 for test-retest reliability (wrong or delayed diagnosis), seven items had a score of >0.5, while the remainder had a reliability score of <0.5. However, the reliability was >0.7 for six of 10 items tested at the hospital level, and >0.6 for the remaining four items. A patient-incident index based on 12 of the incident items had no missing data, the test-retest reliability was 0.6 and the hospital-level reliability was 0.85.ConclusionsThe PRIH-I comprises 13 questions about patient-perceived incidents in hospitals, and can be easily and cost-effectively included in national patient-experience surveys with an acceptable increase in respondent burden. Although the missing-item rate and test-retest reliability were poor for several items, the hospital-level reliability was satisfactory for most of the items. The incident items contribute to a patient-reported incident index, with excellent data quality and hospital-level reliability.
| S-EPMC3756465 | biostudies-literature
Project description:Improvements in technology often drive scientific discovery. Therefore, research requires sustained investment in the latest equipment and training for the researchers who are going to use it. Prioritising and administering infrastructure investment is challenging because future needs are difficult to predict. In the past, highly computationally demanding research was associated primarily with particle physics and astronomy experiments. However, as biology becomes more quantitative and bioscientists generate more and more data, their computational requirements may ultimately exceed those of physical scientists. Computation has always been central to bioinformatics, but now imaging experiments have rapidly growing data processing and storage requirements. There is also an urgent need for new modelling and simulation tools to provide insight and understanding of these biophysical experiments. Bioscience communities must work together to provide the software and skills training needed in their areas. Research-active institutions need to recognise that computation is now vital in many more areas of discovery and create an environment where it can be embraced. The public must also become aware of both the power and limitations of computing, particularly with respect to their health and personal data.
| S-EPMC6521984 | biostudies-literature
Project description: Not available
| S-EPMC3569606 | biostudies-literature
Project description:Quality control is a critical step in the processing and analysis of functional magnetic resonance imaging data. Its purpose is to remove problematic data that could otherwise lead to downstream errors in the analysis and reporting of results. The manual inspection of data can be a laborious and error-prone process that is susceptible to human error. The development of automated tools aims to mitigate these issues. One such tool is pyfMRIqc, which we previously developed as a user-friendly method for assessing data quality. Yet, these methods still generate output that requires subjective interpretations about whether the quality of a given dataset meets an acceptable standard for further analysis. Here we present a quality control protocol using pyfMRIqc and assess the inter-rater reliability of four independent raters using this protocol for data from the fMRI Open QC project (https://osf.io/qaesm/). Data were classified by raters as either "include," "uncertain," or "exclude." There was moderate to substantial agreement between raters for "include" and "exclude," but little to no agreement for "uncertain." In most cases only a single rater used the "uncertain" classification for a given participant's data, with the remaining raters showing agreement for "include"/"exclude" decisions in all but one case. We suggest several approaches to increase rater agreement and reduce disagreement for "uncertain" cases, aiding classification consistency.
| S-EPMC9936142 | biostudies-literature
Project description:The approximate number system (ANS) is the hypothesized cognitive mechanism that allows adults, infants, and animals to enumerate large sets of items approximately. Researchers usually assess the ANS by having subjects compare two sets and indicate which is larger. Accuracy or Weber fraction is taken as an index of the acuity of the system. However, as Clayton et al. (2015) have highlighted, the stimulus parameters used when assessing the ANS vary widely. In particular, the numerical ratio between the pairs, and the way in which non-numerical features are varied often differ radically between studies. Recently, Clayton et al. (2015) found that accuracy measures derived from two commonly used stimulus sets are not significantly correlated. They argue that a lack of inter-test reliability threatens the validity of the ANS construct. Here we apply a recently developed modeling technique to the same data set. The model, by explicitly accounting for the effect of numerical ratio and non-numerical features, produces dependent measures that are less perturbed by stimulus protocol. Contrary to their conclusion we find a significant correlation in Weber fraction across the two stimulus sets. Nevertheless, in agreement with Clayton et al. (2015) we find that different protocols do indeed induce differences in numerical acuity and the degree of influence of non-numerical stimulus features. These findings highlight the need for a systematic investigation of how protocol idiosyncrasies affect ANS assessments.
| S-EPMC4781867 | biostudies-literature
Project description:BACKGROUND:The grades of recommendation, assessment, development and evaluation (GRADE) approach is widely implemented in systematic reviews, health technology assessment and guideline development organisations throughout the world. We have previously reported on the development of the Semi-Automated Quality Assessment Tool (SAQAT), which enables a semi-automated validity assessment based on GRADE criteria. The main advantage to our approach is the potential to improve inter-rater agreement of GRADE assessments particularly when used by less experienced researchers, because such judgements can be complex and challenging to apply without training. This is the first study examining the inter-rater agreement of the SAQAT. METHODS:We conducted two studies to compare: a) the inter-rater agreement of two researchers using the SAQAT independently on 28 meta-analyses and b) the inter-rater agreement between a researcher using the SAQAT (who had no experience of using GRADE) and an experienced member of the GRADE working group conducting a standard GRADE assessment on 15 meta-analyses. RESULTS:There was substantial agreement between independent researchers using the Quality Assessment Tool for all domains (for example, overall GRADE rating: weighted kappa 0.79; 95% CI 0.65 to 0.93). Comparison between the SAQAT and a standard GRADE assessment suggested that inconsistency was parameterised too conservatively by the SAQAT. Therefore the tool was amended. Following amendment we found fair-to-moderate agreement between the standard GRADE assessment and the SAQAT (for example, overall GRADE rating: weighted kappa 0.35; 95% CI 0.09 to 0.87). CONCLUSIONS:Despite a need for further research, the SAQAT may aid consistent application of GRADE, particularly by less experienced researchers.
| S-EPMC4696848 | biostudies-literature
Project description:Hundreds of thousands of individuals have been genotyped in the past decades using genotyping arrays, representing both a valuable data resource for future biomedical research and a substantial investment in human genetic research. However, novel chip designs and their altered sets of single-nucleotide polymorphisms (SNPs) pose the question of how well established data resources, such as large samples of healthy controls genotyped on legacy arrays, can be combined with newer samples genotyped on those novel arrays using genotype imputation. We exemplarily investigated this question based on genotype data of 30 European and 30 African unrelated samples from the 1000 Genomes project and on markers present on two legacy SNP arrays, namely Affymetrix's Human SNP 6.0 and Illumina's 550k array, and three newer arrays, namely two Axiom arrays from Affymetrix and an OmniExpress array from Illumina. We cross-compared the imputation accuracy as well as efficacy and assessed genotype concordance among these arrays. Although the accuracy of genotype prediction was uniformly high across all arrays, the imputation efficacy, that is, the proportion of successfully imputed markers, differed considerably between array combinations in both sample sets, with legacy arrays showing a trend towards lower efficacy values compared with newer arrays when serving as imputation basis. We conclude that, given the substantial losses of markers covered by the legacy arrays, the re-genotyping of existing samples sets, in particular those of healthy population controls, would be a worthwhile endeavor to secure their continued use in the future.
| S-EPMC5437900 | biostudies-other
Project description:BACKGROUND:Connected health (CH) technologies have resulted in a paradigm shift, moving health care steadily toward a more patient-centered delivery approach. CH requires a broad range of disciplinary expertise from across the spectrum to work in a cohesive and productive way. Building this interdisciplinary relationship at an earlier stage of career development may nurture and accelerate the CH developments and innovations required for future health care. OBJECTIVE:This study aimed to explore the perceptions of interdisciplinary CH researchers regarding the design and delivery of an interdisciplinary education (IDE) module for disciplines currently engaged in CH research (engineers, computer scientists, health care practitioners, and policy makers). This study also investigated whether this module should be delivered as a taught component of an undergraduate, master's, or doctoral program to facilitate the development of interdisciplinary learning. METHODS:A qualitative, cross-institutional, multistage research approach was adopted, which involved a background study of fundamental concepts, individual interviews with CH researchers in Greece (n=9), and two structured group feedback sessions with CH researchers in Ireland (n=10/16). Thematic analysis was used to identify the themes emerging from the interviews and structured group feedback sessions. RESULTS:A total of two sets of findings emerged from the data. In the first instance, challenges to interdisciplinary work were identified, including communication challenges, divergent awareness of state-of-the-art CH technologies across disciplines, and cultural resistance to interdisciplinarity. The second set of findings were related to the design for interdisciplinarity. In this regard, the need to link research and education with real-world practice emerged as a key design concern. Positioning within the program context was also considered to be important with a need to balance early intervention to embed integration with later repeat interventions that maximize opportunities to share skills and experiences. CONCLUSIONS:The authors raise and address challenges to interdisciplinary program design for CH based on an abductive approach combining interdisciplinary and interprofessional education literature and the collection of qualitative data. This recipe approach for interdisciplinary design offers guidelines for policy makers, educators, and innovators in the CH space. Gaining insight from CH researchers regarding the development of an IDE module has offered the designers a novel insight regarding the curriculum, timing, delivery, and potential challenges that may be encountered.
| S-EPMC6881783 | biostudies-literature
Project description:Considering the vast physical and neural developments experienced throughout adolescence, the reliability of physical performance may vary in youth populations. This study aimed to examine the reliability of change of direction (COD) and agility tests in youth soccer players. Altogether, 86 youth soccer players, aged 13.6 ± 2.0 years, volunteered to participate. Data were collected from a modified 505 COD test (m505COD) and the Y-sprint drill in both pre-planned (Y-SprintPRE) and reactive (Y-SprintREACT) conditions during 2 sessions, 7 days apart. Anthropometric data including body mass, standing stature, and sitting height were also collected. COD and agility tests demonstrated good reliability (ICC = 0.81-0.91; CV = 1.2-2.0; d = 0.00-0.31; p < 0.01) for our entire sample. However, we observed a small negative relationship between age and intersession differences for the Y-SprintPRE (r = -0.28; p = 0.04), and moderate negative relationships between both age (r = -0.41; p < 0.01), and maturity offset (r = -0.39; p < 0.01) for the Y-SprintREACT. Although the COD and agility tests adopted within this study possess good intersession reliability, we observed greater intersession differences for younger and less mature individuals. We suggest that while COD and agility tests may provide meaningful objective data for monitoring the development of youth soccer players, these tests should be used with caution when evaluating younger, more immature athletes.
| S-EPMC7240391 | biostudies-literature
Project description:Clinical prediction models for risk stratification of older adults with syncope or near syncope may improve resource utilization and management. Predictors considered for inclusion into such models must be reliable. Our primary objective was to evaluate the inter-rater agreement of historical, physical examination, and electrocardiogram (ECG) findings in older adults undergoing emergency department (ED) evaluation for syncope or near syncope. Our secondary objective was to assess the level of agreement between clinicians on the patient's overall risk for death or serious cardiac outcomes.We conducted a cross-sectional study at 11 EDs in adults 60 years of age or older who presented with unexplained syncope or near syncope. We excluded patients with a presumptive cause of syncope (e.g., seizure) or if they were unable or unwilling to follow-up. Evaluations of the patient's past medical history and current medication use were completed by treating provider and trained research associate pairs. Evaluations of the patient's physical examination and ECG interpretation were completed by attending/resident, attending/advanced practice provider, or attending/attending pairs. All evaluations were blinded to the responses from the other rater. We calculated the percent agreement and kappa statistic for binary variables. Inter-rater agreement was considered acceptable if the kappa statistic was 0.6 or higher.We obtained paired observations from 255 patients; mean (±SD) age was 73 (±9) years, 137 (54%) were male, and 204 (80%) were admitted to the hospital. Acceptable agreement was achieved in 18 of the 21 (86%) past medical history and current medication findings, none of the 10 physical examination variables, and three of the 13 (23%) ECG interpretation variables. There was moderate agreement (Spearman correlation coefficient, r = 0.40) between clinicians on the patient's probability of 30-day death or serious cardiac outcome, although as the probability increased, there was less agreement.Acceptable agreement between raters was more commonly achieved with historical rather than physical examination or ECG interpretation variables. Clinicians had moderate agreement in assessing the patient's overall risk for a serious outcome at 30 days. Future development of clinical prediction models in older adults with syncope should account for variability of assessments between raters and consider the use of objective clinical variables.
| S-EPMC5102262 | biostudies-literature