Dataset Information

Validity and reproducibility of a tool for assessing clinical competencies in physical therapy students.

ABSTRACT: BACKGROUND:The evaluation of competencies in the clinical field is essential for health professionals, as it allows the acquisition of these competencies to be tracked. The objective of this study was to create and evaluate the validity and reliability of a tool for measuring clinical competencies in physical therapy (PT) students to assess the quality of their performance in a professional context. METHODS:A descriptive study was designed. The Measurement Tool for Clinical Competencies in PT (MTCCP) was developed based on the evaluation of 39 experts: 15 clinicians and 24 instructors. The content validity was evaluated using the Content Validity Index (CVI). Three professors were invited to apply the tool to 10 students. Cronbach's alpha, exploratory factor analysis, and the intraclass correlation coefficient were used to determine the reliability and validity of the scale. RESULTS:The CVI was positive-higher than 0.8. Principal component analysis confirmed the construct validity of the tool for two main factors: clinical reasoning (first factor) and professional behavior (second factor). With regard to reliability, the MTCCP achieved an internal congruence of 0.982. The inter-evaluator reproducibility for clinical reasoning, professional behavior, and the total MTCCP score was almost perfect; the ICCs were 0.984, 0.930, and 0.983, respectively. CONCLUSIONS:The MTCCP is a valid and reliable instrument for assessing the performance of PT students in hospital settings and can be used to determine what skills students feel less confident using and what additional training/learning opportunities could be provided. Further research is needed to determine whether the MTCCP has similar validity and reproducibility in other Spanish-speaking national and international PT programs.

SUBMITTER: Torres-Narvaez MR

PROVIDER: S-EPMC6260878 | biostudies-other | 2018 Nov

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Validity and reproducibility of a tool for assessing clinical competencies in physical therapy students.

Torres-Narváez Martha-Rocío MR Vargas-Pinilla Olga-Cecilia OC Rodríguez-Grande Eliana-Isabel EI

BMC medical education 20181123 1

<h4>Background</h4>The evaluation of competencies in the clinical field is essential for health professionals, as it allows the acquisition of these competencies to be tracked. The objective of this study was to create and evaluate the validity and reliability of a tool for measuring clinical competencies in physical therapy (PT) students to assess the quality of their performance in a professional context.<h4>Methods</h4>A descriptive study was designed. The Measurement Tool for Clinical Compet ...[more]

PMID: 30470221

Similar Datasets

Project description:Objective:The study examined the efficacy of an interprofessional information and historical literacy project implemented by an occupational therapy educator and a librarian. Methods:A graduate course was revised to include information and historical literacy objectives and instruction. A course-specific questionnaire administered on the first and last day of class, assignment grades, and course evaluations provided measures of project outcomes for six years. Differences between questionnaire pre- and post-test means were determined using t-tests. Course evaluation comments were analyzed to obtain qualitative perceptions. Results:A significant difference (p<0.0001) was found between pre-test (M=3.93, SD=0.48) and post-test (M=4.67, SD=0.30) scores of total information and historical literacy competence across all years (n=242). Responses to individual items also differed significantly (p<0.0001). Student ratings (n=189) from the course evaluation historical literacy objectives were high (M=4.6 on a 5-point scale). Assignment quality and grades improved, and course evaluation comments reflected student satisfaction. Conclusions:The findings supported the hypothesis that students' self-reported information and historical literacy competencies would increase after project participation. Acquired skills were evident in students' assignments. Research to determine if these capabilities were used post-graduation is needed. Because this was a course-specific project, findings are not generalizable; however, the instructional methods developed for this project can serve as a model for effective interprofessional collaboration. The broadening of information literacy instruction to include discipline-specific historical literacy provides a unique opportunity for health sciences librarians and educators. Developing students' historical literacy in their chosen fields can help them understand their profession's present status and be informed participants in shaping its future.

Project description:PurposeDeveloping medical students' clinical reasoning requires a structured longitudinal curriculum with frequent targeted assessment and feedback. Performance-based assessments, which have the strongest validity evidence, are currently not feasible for this purpose because they are time-intensive to score. This study explored the potential of using machine learning technologies to score one such assessment-the diagnostic justification essay.MethodFrom May to September 2018, machine scoring algorithms were trained to score a sample of 700 diagnostic justification essays written by 414 third-year medical students from the Southern Illinois University School of Medicine classes of 2012-2017. The algorithms applied semantically based natural language processing metrics (e.g., coherence, readability) to assess essay quality on 4 criteria (differential diagnosis, recognition and use of findings, workup, and thought process); the scores for these criteria were summed to create overall scores. Three sources of validity evidence (response process, internal structure, and association with other variables) were examined.ResultsMachine scores correlated more strongly with faculty ratings than faculty ratings did with each other (machine: .28-.53, faculty: .13-.33) and were less case-specific. Machine scores and faculty ratings were similarly correlated with medical knowledge, clinical cognition, and prior diagnostic justification. Machine scores were more strongly associated with clinical communication than were faculty ratings (.43 vs .31).ConclusionsMachine learning technologies may be useful for assessing medical students' long-form written clinical reasoning. Semantically based machine scoring may capture the communicative aspects of clinical reasoning better than faculty ratings, offering the potential for automated assessment that generalizes to the workplace. These results underscore the potential of machine scoring to capture an aspect of clinical reasoning performance that is difficult to assess with traditional analytic scoring methods. Additional research should investigate machine scoring generalizability and examine its acceptability to trainees and educators.

Dataset Information

Validity and reproducibility of a tool for assessing clinical competencies in physical therapy students.

Publications

Validity and reproducibility of a tool for assessing clinical competencies in physical therapy students.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets