Assessing the Reliability of the Framework for Equitable and Effective Teaching With the Many-Facet Rasch Model.
Ontology highlight
ABSTRACT:
SUBMITTER: Govindasamy P
PROVIDER: S-EPMC6587337 | biostudies-literature | 2019
REPOSITORIES: biostudies-literature
Publications
Assessing the Reliability of the Framework for Equitable and Effective Teaching With the Many-Facet Rasch Model.
Govindasamy Priyalatha P Del Carmen Salazar Maria M Lerner Jessica J Green Kathy E KE
Frontiers in psychology 20190614
This manuscript reports results of an empirical assessment of a newly developed measure designed to assess apprentice teaching proficiency. In this study, Many Facets Rasch model software was used to evaluate the psychometric quality of the Framework for Equitable and Effective Teaching (FEET), a rater-mediated assessment. The analysis focused on examining variability in (1) supervisor severity in ratings, (2) level of item difficulty, (3) time of assessment, and (4) teacher apprentice proficien ...[more]
Similar Datasets
Project description:Purpose: This study aimed to analyze interview data collected from a series of in-basket tests during managerial personnel recruitment in a local Chinese company, taking advantage of the use of combination of Generalizability theory (GT) and Many-facet Rasch Model (MFRM), rather than the Classical Test Theory (CTT). Design/Methodology/Approach: Participants included 78 candidates (M age = 37.76, SD = 6.36; 35.9% female) interviewed for eight managerial positions in a local Chinese company. Data were collected based on a series of 10 in-basket interview tests, and a 20-item rating scale (the in-basket test rating scale; IBTRS) was developed and piloted, and five expert raters rated the participants on their performance in five aspects (planning, communication and coordination, capital operations and management, analysis and problem solving, and empowerment and controlling). The data were analyzed using a crossed design of p × i × r, where p represents person, i represents item, and r represents rater. Effects of candidate (person), test item, rater, and the interaction of item and rater were examined. Findings: The use of the combination of GT and MFRM was able to provide accurate, comprehensive information over the process of in-basket interview tests. Specifically, GT analysis showed good generalization coefficient and reliability index (0.893 and 0.871, respectively), and the variation of candidates explained most of the total variance (53.22%). The candidates scored high in the dimension of empowerment and controlling. There were differences in the severity of raters. Three raters should be sufficient to ensure good scoring stability. Originality/Value: This study used the combination of GT and MFRM to assess the interview data instead of using a CTT approach.
| S-EPMC8359922 | biostudies-literature
Project description:IntroductionThe science of team science (SciTS) is an emerging research area that studies the processes and outcomes of team-based research. A well-established conceptual framework and appropriate methodology for examining the effectiveness of team science are critically important for promoting and advancing collaborative and interdisciplinary research. Although many instruments have been developed and used in the SciTS field, psychometric evidence has not been routinely assessed or reported for these scales. In addition, commonly used psychometric methods were mainly limited to internal consistency and factor analysis. To fill the gaps, this study introduces a framework based on Rasch measurement theory for creating and evaluating measures for team sciences.MethodsWe illustrate the application of Rasch measurement theory through the creation of valid measures to evaluate the processes of interdisciplinary scientific teams. Data were collected from 16 interdisciplinary teams through a university-wide initiative for promoting interdisciplinary team collaboration. Psychometric evidence based on a many-facet Rasch model was obtained for assessing the quality of the measures.ResultsThe interdisciplinary teams differed in their clarity measures. Significant differences were also found between gender groups, racial groups, and academic ranks. We reported the reliability of measures and identified items that do not fit the model and may present potential threat to validity and fairness of SciTS measures.ConclusionThis study shows the great potential of using Rasch measurement theory for developing and evaluating SciTS measures. Applying Rasch measurement theory produces objective measures that are comparable across individuals, interdisciplinary teams, institutions, time, and various demographic groups.
| S-EPMC9794954 | biostudies-literature
Project description:OBJECTIVES:Sources of bias, such as the examiners, domains and stations, can influence the student marks in objective structured clinical examination (OSCE). This study describes the extent to which the facets modelled in an OSCE can contribute to scoring variance and how they fit into a Many-Facet Rasch Model (MFRM) of OSCE performance. A further objective is to identify the functioning of the rating scale used. DESIGN:A non-experimental cross-sectional design. PARTICIPANTS AND SETTINGS:An MFRM was used to identify sources of error (eg, examiner, domain and station), which may influence the student outcome. A 16-station OSCE was conducted for 329 final year medical students. Domain-based marking was applied, each station using a sample from eight defined domains across the whole OSCE. The domains were defined as follows: communication skills, professionalism, information gathering, information giving, clinical interpretation, procedure, diagnosis and management. The domains in each station were weighted to ensure proper attention to the construct of the individual station. Four facets were assessed: students, examiners, domains and stations. RESULTS:The results suggest that the OSCE data fit the model, confirming that an MFRM approach was appropriate to use. The variable map allows a comparison with and between the facets of students, examiners, domains and stations and the 5-point score for each domain with each station as they are calibrated to the same scale. Fit statistics showed that the domains map well to the performance of the examiners. No statistically significant difference between examiner sensitivity (3.85 logits) was found. However, the results did suggest examiners were lenient and that some behaved inconsistently. The results also suggest that the functioning of response categories on the 5-point rating scale need further examination and optimisation. CONCLUSIONS:The results of the study have important implications for examiner monitoring and training activities, to aid assessment improvement.
| S-EPMC6731829 | biostudies-other
Project description:Applied bioinformatics skills are becoming ever more indispensable for biologists, yet incorporation of these skills into the undergraduate biology curriculum is lagging behind, in part due to a lack of instructors willing and able to teach basic bioinformatics in classes that don't specifically focus on quantitative skill development, such as statistics or computer sciences. To help undergraduate course instructors who themselves did not learn bioinformatics as part of their own education and are hesitant to plunge into teaching big data analysis, a module was developed that is written in plain-enough language, using publicly available computing tools and data, to allow novice instructors to teach next-generation sequence analysis to upper-level undergraduate students. To determine if the module allowed students to develop a better understanding of and appreciation for applied bioinformatics, various tools were developed and employed to assess the impact of the module. This article describes both the module and its assessment. Students found the activity valuable for their education and, in focus group discussions, emphasized that they saw a need for more and earlier instruction of big data analysis as part of the undergraduate biology curriculum.
| S-EPMC5764237 | biostudies-literature
Project description:Collaborative Cloud Classroom (3CR) is an educational application that provides flexibility in the learning process. This specialized tool is designed to improve analytical and teambuilding skills, as well as enable students to document ideas, which are subsequently developed into bubble diagrams and text outlines. This study aims to assess and analyze the menu qualities possessed by the 3CR application using the Many-Facet Rasch Model (MFRM) analysis using 5 raters and 6 menus. A rating assessment was applied to 5 criteria, namely usability, functionality, visual communication, learning design, and security. In this case, a total of 135 from 150 data points were coded into FACETS software. The results showed that the six menus in the 3CR application were valid and reliable, as well as met the quality standards of learning tools. Security criteria were also the most difficult standards for raters to measure in all the application menus. The study also found that Rater A was more severe, and Rater E was more lenient in rating the 3CR menu. Rater B has overfitting and detected bias between rater gender and 3CR's menu (Discussion Forum and Learning Reflection menu) based on MFRM analysis.
| S-EPMC10570598 | biostudies-literature
Project description:ObjectiveThis study aimed to assess a multipronged strategy using primarily digital methods to equitably recruit asthma patients into a clinical trial of a digital health intervention.MethodsWe approached eligible patients using at least one of eight recruitment strategies. We recorded approach dates and the strategy that led to completion of a web-based eligibility questionnaire that was reported during the verbal consent phone call. Study team members conducted monthly sessions using a structured guide to identify recruitment barriers and facilitators. The proportion of participants who reported being recruited by a portal or nonportal strategy was measured as our outcomes. We used Fisher's exact test to compare outcomes by equity variable, and multivariable logistic regression to control for each covariate and adjust effect size estimates. Using grounded theory, we coded and extracted themes regarding recruitment barriers and facilitators.ResultsThe majority (84.4%) of patients who met study inclusion criteria were patient portal enrollees. Of 6,366 eligible patients who were approached, 627 completed the eligibility questionnaire and were less frequently Hispanic, less frequently Spanish-speaking, and more frequently patient portal enrollees. Of 445 patients who consented to participate, 241 (54.2%) reported completing the eligibility questionnaire after being contacted by a patient portal message. In adjusted analysis, only race (odds ratio [OR]: 0.46, 95% confidence interval [CI]: 0.28-0.77, p = 0.003) and college education (OR: 0.60, 95% CI: 0.39-0.91, p = 0.016) remained significant. Key recruitment barriers included technology issues (e.g., lack of email access) and facilitators included bilingual study staff, Spanish-language recruitment materials, targeted phone calls, and clinician-initiated "1-click" referrals.ConclusionA primarily digital strategy to recruit patients into a digital health trial is unlikely to achieve equitable participation, even in a population overrepresented by patient portal enrollees. Nondigital recruitment methods that address racial and educational disparities and less active portal enrollees are necessary to ensure equity in clinical trial enrollment.
| S-EPMC10412068 | biostudies-literature
Project description:The purpose of this article is to provide raw data and measure-validation data pertaining to a co-submission published in European Journal of Radiology and entitled: Development and validation of a novel measure of adverse patient positioning in mammography. This Data in Brief article serves not only to provide greater detail than its companion article but also as an educational worked example of the Rasch measurement framework. Rasch measurement is a form of modern psychometric technique and our articles provide the first known example of its use in the evaluation of clinical radiological image quality. The data consist of observations of mammographic images, plus limited participant parameters relevant to the measure validation process. Also provided are validation indices produced by subjecting the primary data to Rasch analysis. An expert observer generated the primary data by reviewing mammographic images to judge the presence or absence of a set of features developed through theory and consultation with other experts. The validation data were generated through Rasch analysis, performed using Winsteps® software, which mathematically models the probability of having a correct response (or a present feature in this dataset) to an item in a given measurement instrument (e.g. questionnaire), as a function of the participant's ability/position on the underlying construct under study. The data can be reused by anyone wishing to learn and practice psychometric validation techniques. They can also form a basis for researchers wishing to build on our preliminary measure for the assessment of mammographic clinical image quality.
| S-EPMC8479236 | biostudies-literature
Project description:BackgroundClinical teaching evaluations are common in health profession education programs to ensure students are receiving a quality clinical education experience. Questionnaires students use to evaluate their clinical teachers have been developed in professions such as medicine and nursing. The development of a questionnaire that is specifically for the osteopathy on-campus, student-led clinic environment is warranted. Previous work developed the 30-item Osteopathy Clinical Teaching Questionnaire. The current study utilised Rasch analysis to investigate the construct validity of the Osteopathy Clinical Teaching Questionnaire and provide evidence for the validity argument through fit to the Rasch model.MethodsSenior osteopathy students at four institutions in Australia, New Zealand and the United Kingdom rated their clinical teachers using the Osteopathy Clinical Teaching Questionnaire. Three hundred and ninety-nine valid responses were received and the data were evaluated for fit to the Rasch model. Reliability estimations (Cronbach's alpha and McDonald's omega) were also evaluated for the final model.ResultsThe initial analysis demonstrated the data did not fit the Rasch model. Accordingly, modifications to the questionnaire were made including removing items, removing person responses, and rescoring one item. The final model contained 12 items and fit to the Rasch model was adequate. Support for unidimensionality was demonstrated through both the Principal Components Analysis/t-test, and the Cronbach's alpha and McDonald's omega reliability estimates. Analysis of the questionnaire using McDonald's omega hierarchical supported a general factor (quality of clinical teaching in osteopathy).ConclusionThe evidence for unidimensionality and the presence of a general factor support the calculation of a total score for the questionnaire as a sufficient statistic. Further work is now required to investigate the reliability of the 12-item Osteopathy Clinical Teaching Questionnaire to provide evidence for the validity argument.
| S-EPMC5932865 | biostudies-literature
Project description:The usual method for assessing the reliability of survey data has been to conduct reinterviews a short interval (such as one to two weeks) after an initial interview and to use these data to estimate relatively simple statistics, such as gross difference rates (GDRs). More sophisticated approaches have also been used to estimate reliability. These include estimates from multi-trait, multi-method experiments, models applied to longitudinal data, and latent class analyses. To our knowledge, no prior study has systematically compared these different methods for assessing reliability. The Population Assessment of Tobacco and Health Reliability and Validity (PATH-RV) Study, done on a national probability sample, assessed the reliability of answers to the Wave 4 questionnaire from the PATH Study. Respondents in the PATH-RV were interviewed twice about two weeks apart. We examined whether the classic survey approach yielded different conclusions from the more sophisticated methods. We also examined two ex ante methods for assessing problems with survey questions and item nonresponse rates and response times to see how strongly these related to the different reliability estimates. We found that kappa was highly correlated with both GDRs and over-time correlations, but the latter two statistics were less highly correlated, particularly for adult respondents; estimates from longitudinal analyses of the same items in the main PATH study were also highly correlated with the traditional reliability estimates. The latent class analysis results, based on fewer items, also showed a high level of agreement with the traditional measures. The other methods and indicators had at best weak relationships with the reliability estimates derived from the reinterview data. Although the Question Understanding Aid seems to tap a different factor from the other measures, for adult respondents, it did predict item nonresponse and response latencies and thus may be a useful adjunct to the traditional measures.
| S-EPMC8519302 | biostudies-literature
Project description:BACKGROUND: Protein-protein interactions play vital roles in nearly all cellular processes and are involved in the construction of biological pathways such as metabolic and signal transduction pathways. Although large-scale experiments have enabled the discovery of thousands of previously unknown linkages among proteins in many organisms, the high-throughput interaction data is often associated with high error rates. Since protein interaction networks have been utilized in numerous biological inferences, the inclusive experimental errors inevitably affect the quality of such prediction. Thus, it is essential to assess the quality of the protein interaction data. RESULTS: In this paper, a novel Bayesian network-based integrative framework is proposed to assess the reliability of protein-protein interactions. We develop a cross-species in silico model that assigns likelihood scores to individual protein pairs based on the information entirely extracted from model organisms. Our proposed approach integrates multiple microarray datasets and novel features derived from gene ontology. Furthermore, the confidence scores for cross-species protein mappings are explicitly incorporated into our model. Applying our model to predict protein interactions in the human genome, we are able to achieve 80% in sensitivity and 70% in specificity. Finally, we assess the overall quality of the experimentally determined yeast protein-protein interaction dataset. We observe that the more high-throughput experiments confirming an interaction, the higher the likelihood score, which confirms the effectiveness of our approach. CONCLUSION: This study demonstrates that model organisms certainly provide important information for protein-protein interaction inference and assessment. The proposed method is able to assess not only the overall quality of an interaction dataset, but also the quality of individual protein-protein interactions. We expect the method to continually improve as more high quality interaction data from more model organisms becomes available and is readily scalable to a genome-wide application.
| S-EPMC2681066 | biostudies-literature