Dataset Information

Assessing professional competence in optometry - a review of the development and validity of the written component of the competency in optometry examination (COE).

ABSTRACT:

Background

Credentialing assessment for overseas-educated optometrists seeking registration in Australia and New Zealand is administered by the Optometry Council of Australia and New Zealand. The aim was to review the validation and outcomes of the written components of this exam to demonstrate credentialing meets entry-level competency standards.

Methods

The Competency in Optometry Examination consists of two written and two clinical parts. Part 1 of the written exam comprises multiple choice questions (MCQ) covering basic and clinical science, while Part 2 has 18 short answer questions (SAQ) examining diagnosis and management. Candidates must pass both written components to progress to the clinical exam. Validity was evaluated using Kane's framework for scoring (marking criteria, item analysis), generalization (blueprint), extrapolation (standard setting), and implications (outcome, including pass rates). A competency-based blueprint, the Optometry Australia Entry-level Competency Standards for Optometry 2014, guided question selection with the number of items weighted towards key competencies. A standard setting exercise, last conducted in 2017, was used to determine the minimum standard for both written exams. Item response theory (Rasch) was used to analyse exams, produce reliability metrics, apply consistent standards to the results, calibrate difficulty across exams, and score candidates.

Results

Data is reported on 12 administrations of the written examination since 2014. Of the 193 candidates who sat the exam over the study period, 133 (68.9%) passed and moved on to the practical component. Ninety-one (47.2%) passed both the MCQ and SAQ exams on their first attempt. The MCQ exam has displayed consistently high reliability (reliability index range 0.71 to 0.93, average 0.88) across all 12 administrations. Prior to September 2017 the SAQ had a set cutscore of 50%, and the difficulty of the exam was variable. Since the introduction of Rasch analysis to calibrate difficulty across exams, the reliability and power of the SAQ exam has been consistently high (separation index range 0.82 to 0.93, average 0.86).

Conclusions

The findings from collective evidence support the validity of the written components (MCQ and SAQ) of the credentialing of the competency of overseas-educated optometrists in Australia and New Zealand.

SUBMITTER: Backhouse S

PROVIDER: S-EPMC7786977 | biostudies-literature | 2021 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Assessing professional competence in optometry - a review of the development and validity of the written component of the competency in optometry examination (COE).

Backhouse S S Chiavaroli N G NG Schmid K L KL McKenzie T T Cochrane A L AL Phillips G G Jalbert I I

BMC medical education 20210106 1

<h4>Background</h4>Credentialing assessment for overseas-educated optometrists seeking registration in Australia and New Zealand is administered by the Optometry Council of Australia and New Zealand. The aim was to review the validation and outcomes of the written components of this exam to demonstrate credentialing meets entry-level competency standards.<h4>Methods</h4>The Competency in Optometry Examination consists of two written and two clinical parts. Part 1 of the written exam comprises mu ...[more]

PMID: 33407393

Similar Datasets

Project description:ObjectiveDevelop new performance evaluation standards for the clinical breast examination (CBE).Summary background dataThere are several, technical aspects of a proper CBE. Our recent work discovered a significant, linear relationship between palpation force and CBE accuracy. This article investigates the relationship between other technical aspects of the CBE and accuracy.MethodsThis performance assessment study involved data collection from physicians (n = 553) attending 3 different clinical meetings between 2013 and 2014: American Society of Breast Surgeons, American Academy of Family Physicians, and American College of Obstetricians and Gynecologists. Four, previously validated, sensor-enabled breast models were used for clinical skills assessment. Models A and B had solitary, superficial, 2 cm and 1 cm soft masses, respectively. Models C and D had solitary, deep, 2 cm hard and moderately firm masses, respectively. Finger movements (search technique) from 1137 CBE video recordings were independently classified by 2 observers. Final classifications were compared with CBE accuracy.ResultsAccuracy rates were model A = 99.6%, model B = 89.7%, model C = 75%, and model D = 60%. Final classification categories for search technique included rubbing movement, vertical movement, piano fingers, and other. Interrater reliability was (k = 0.79). Rubbing movement was 4 times more likely to yield an accurate assessment (odds ratio 3.81, P < 0.001) compared with vertical movement and piano fingers. Piano fingers had the highest failure rate (36.5%). Regression analysis of search pattern, search technique, palpation force, examination time, and 6 demographic variables, revealed that search technique independently and significantly affected CBE accuracy (P < 0.001).ConclusionsOur results support measurement and classification of CBE techniques and provide the foundation for a new paradigm in teaching and assessing hands-on clinical skills. The newly described piano fingers palpation technique was noted to have unusually high failure rates. Medical educators should be aware of the potential differences in effectiveness for various CBE techniques.

Project description:Background To ascertain if undergraduate medical students attain adequate knowledge to practice in paediatrics, we designed the minimum accepted competency (MAC) examination. This was a set of MCQ’s designed to test the most basic, ‘must know’ knowledge as determined by non-faculty paediatric clinicians. Only two-thirds of undergraduate students passed this exam, despite 96% of the same cohort passing their official university paediatric examination. We aim to describe the psychometric properties of the MAC examination to explore why there was a difference in student performance between these two assessments which should, in theory, be testing the same subject area. We will also investigate if the MAC examination is a potentially reliable method of assessing undergraduate knowledge. Methods The MAC examination was sat by three groups of undergraduate medical students and paediatric trainee doctors. Test item analysis was performed using facility index, discrimination index and Cronbach’s alpha. Results Test item difficulty on the MAC between each group was positively correlated. Correlation of item difficulty with the standard set for each item showed a statistically significant positive relationship. However, for 10 of the items, the mean score achieved by the candidates did not even reach two standard deviations below the standard set by the faculty. Medical students outperformed the trainee doctors on three items. 18 of 30 items achieved a discrimination index > 0.2. Cronbach’s alpha ranged from 0.22–0.59. Conclusion Despite faculty correctly judging that this would be a difficult paper for the candidates, there were a significant number of items on which students performed particularly badly. It is possible that the clinical emphasis in these non-faculty derived questions was juxtaposed with the factual recall often required for university examinations. The MAC examination highlights the difference in the level of knowledge expected of a junior doctor starting work in paediatrics between faculty and non-faculty clinicians and can identify gaps between the current curriculum and the ‘hidden curriculum’ required for real world clinical practice. The faculty comprises physicians in employment by the University whose role it is to design the paediatric curriculum and deliver teaching to undergraduate students. Non-faculty clinicians are paediatric physicians who work soley as clinicians with no affiliation to an educational institution. The concept of a MAC examination to test basic medical knowledge is feasible and the study presented is an encouraging first step towards this method of assessment. Supplementary information The online version contains supplementary material available at 10.1186/s12909-022-03475-8.

Project description:PurposeDeveloping medical students' clinical reasoning requires a structured longitudinal curriculum with frequent targeted assessment and feedback. Performance-based assessments, which have the strongest validity evidence, are currently not feasible for this purpose because they are time-intensive to score. This study explored the potential of using machine learning technologies to score one such assessment-the diagnostic justification essay.MethodFrom May to September 2018, machine scoring algorithms were trained to score a sample of 700 diagnostic justification essays written by 414 third-year medical students from the Southern Illinois University School of Medicine classes of 2012-2017. The algorithms applied semantically based natural language processing metrics (e.g., coherence, readability) to assess essay quality on 4 criteria (differential diagnosis, recognition and use of findings, workup, and thought process); the scores for these criteria were summed to create overall scores. Three sources of validity evidence (response process, internal structure, and association with other variables) were examined.ResultsMachine scores correlated more strongly with faculty ratings than faculty ratings did with each other (machine: .28-.53, faculty: .13-.33) and were less case-specific. Machine scores and faculty ratings were similarly correlated with medical knowledge, clinical cognition, and prior diagnostic justification. Machine scores were more strongly associated with clinical communication than were faculty ratings (.43 vs .31).ConclusionsMachine learning technologies may be useful for assessing medical students' long-form written clinical reasoning. Semantically based machine scoring may capture the communicative aspects of clinical reasoning better than faculty ratings, offering the potential for automated assessment that generalizes to the workplace. These results underscore the potential of machine scoring to capture an aspect of clinical reasoning performance that is difficult to assess with traditional analytic scoring methods. Additional research should investigate machine scoring generalizability and examine its acceptability to trainees and educators.

Project description:BackgroundNursing competency frameworks describe the competencies; knowledge, skills and attitudes nurses should possess. Countries have their own framework. Knowledge of the content of professional competency frameworks in different countries can enhance the development of these frameworks and international collaborations.ObjectiveThis study examines how competencies and task divisions are described in the current professional competency frameworks for registered nurses (RNs with a Bachelor's degree) in the Netherlands, Belgium, the United Kingdom (UK), Canada and the United States (US).MethodsQualitative document analysis was conducted using the most recently published professional competency frameworks for registered nurses in the above-mentioned five countries.ResultsAll the competency frameworks distinguished categories of competencies. Three of the five frameworks explicitly mentioned the basis for the categorization: an adaptation of the CanMEDS model (Netherlands), European directives on the recognition of professional qualifications (Belgium) and an adapted inter-professional framework (US). Although there was variation in how competencies were grouped, we inductively identified ten generic competency domains: (1) Professional Attitude, (2) Clinical Care in Practice, (3) Communication and Collaboration, (4) Health Promotion and Prevention, (5) Organization and Planning of Care, (6) Leadership, (7) Quality and Safety of Care, (8) Training and (continuing) Education, (9) Technology and e-Health, (10) Support of Self-Management and Patient Empowerment. Country differences were found in some more specific competency descriptions. All frameworks described aspects related to the division of tasks between nurses on the one hand and physicians and other healthcare professionals on the other hand. However, these descriptions were rather limited and often imprecise.ConclusionsAlthough ten generic domains could be identified when analysing and comparing the competency frameworks, there are country differences in the categorizations and the details of the competencies described in the frameworks. These differences and the limited attention paid to the division of tasks might lead to cross-country differences in nursing practice and barriers to the international labour mobility of Bachelor-educated RNs.

Dataset Information

Assessing professional competence in optometry - a review of the development and validity of the written component of the competency in optometry examination (COE).

Background

Methods

Results

Conclusions

Publications

Assessing professional competence in optometry - a review of the development and validity of the written component of the competency in optometry examination (COE).

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets