Dataset Information

Development of an Instrument for Preceptor Evaluation of Medical Graduates' Performance: the Psychometric Properties.

ABSTRACT:

Background

A preceptor has a key role in evaluating medical graduate's performance in the clinical setting. This study is conducted to develop an instrument for preceptor evaluation of medical graduates' performance in the clinical setting.

Method

A mixed-method study design, sequential exploratory approach was chosen to develop the instrument. Initial semi-structured interviews were conducted with 4 preceptors at the teaching hospitals. Five main themes emerged from the interviews. The themes were developed into a 23-item questionnaire. Nineteen Head or Assistant Head of clinical departments were asked to review the relevance of the content. The questionnaire was later sent out to 34 preceptors and 35 paramedic staffs to participate in the construct validity study by conducting exploratory factor analysis (EFA). SPSS version 21 software was used to analyze the data and Varimax rotation method was performed to simplify and describe the data structure.

Result

Review of the factor structures suggested that the most appropriate fit was 5 factors. Most of the questionnaire items were relevant to assess performance (4.65 + 0.15), except in item 4 of the clinical skill factor. The 23 items of the evaluation instruments showed that five factors were extracted which explained 73.9% of the variance between them. Construct validity was achieved after the instrument was run for an iteration of eight times, with Cronbach's alpha of 0.951.

Discussion

The instrument has achieved the desired content and construct validity score. It can be used by other institutions to assess their medical graduates' performance in the clinical setting.

SUBMITTER: Kusmiati M

PROVIDER: S-EPMC8368111 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Nighttime symptoms can negatively impact the quality of life of patients with chronic obstructive pulmonary disease (COPD). The Nighttime Symptoms of COPD Instrument (NiSCI) was designed to measure the occurrence and severity of nighttime symptoms in patients with COPD, the impact of symptoms on nighttime awakenings, and rescue medication use. The objective of this study was to explore item reduction, inform scoring recommendations, and evaluate the psychometric properties of the NiSCI.COPD patients participating in a Phase III clinical trial completed the NiSCI daily. Item analyses were conducted using weekly mean and single day scores. Descriptive statistics (including percentage of respondents at floor/ceiling and inter-item correlations), factor analyses, and Rasch model analyses were conducted to examine item performance and scoring. Test-retest reliability was assessed for the final instrument using the intraclass correlation coefficient (ICC). Correlations with assessments conducted during study visits were used to evaluate convergent and known-groups validity.Data from 1,663 COPD patients aged 40-93 years were analyzed. Item analyses supported the generation of four scores. A one-factor structure was confirmed with factor analysis and Rasch analysis for the symptom severity score. Test-retest reliability was confirmed for the six-item symptom severity (ICC, 0.85), number of nighttime awakenings (ICC, 0.82), and rescue medication (ICC, 0.68) scores. Convergent validity was supported by significant correlations between the NiSCI, St George's Respiratory Questionnaire, and Exacerbations of Chronic Obstructive Pulmonary Disease Tool-Respiratory Symptoms scores.The results suggest that the NiSCI can be used to determine the severity of nighttime COPD symptoms, the number of nighttime awakenings due to COPD symptoms, and the nighttime use of rescue medication. The NiSCI is a reliable and valid instrument to evaluate these concepts in COPD patients in clinical trials and clinical practice. Scoring recommendations and steps for further research are discussed.

Project description:BackgroundEarly morning respiratory symptoms impact quality of life and are often the most troublesome for patients with COPD. Reduction in symptoms and their impact are important treatment outcomes for COPD. The Early Morning Symptoms of COPD Instrument (EMSCI) is a daily diary designed to collect patients' report of the occurrence, severity, and impact of morning COPD symptoms.MethodsTo assess the psychometric properties of the EMSCI, a split-half sample of data from a COPD clinical trial where participants completed the EMSCI daily was used for conducting descriptive statistics, factor analyses, and Rasch model analyses to examine item performance and inform scoring. Once the final scoring algorithm was determined, data from the second split-half sample were used to examine the properties of the EMSCI. Test-retest reliability was assessed using intraclass correlation coefficient (ICC). Correlations with other study assessments were used to evaluate convergent and known-groups validity.ResultsData from 1,663 patients with COPD aged 40-93 years were analyzed. Factor analysis and Rasch analysis confirmed a one-factor structure for the 6 individual symptom items. Item analyses supported the generation of 4 scores. All scores demonstrated good test-retest reliability: 6-item symptom severity (ICC, 0.84); overall morning symptom severity (ICC, 0.84); activity limitation (ICC, 0.85); and rescue medication (ICC, 0.62). Significant correlations between EMSCI scores, St George's Respiratory Questionnaire scores, and EXAcerbations of Chronic pulmonary disease Tool (EXACT)-Respiratory Symptoms scores supported the tool's convergent validity. Significant differences (p<0.0001) in all EMSCI domain scores were found between known-groups based on median split St George's Respiratory Questionnaire and EXACT-Respiratory Symptoms scores.ConclusionThe EMSCI consists of 4 scores: 6-item symptom severity, overall symptom severity, activity limitation, and rescue medication. The EMSCI is a reliable and valid instrument for evaluating patients' experience of early morning COPD symptoms.

Project description:Introduction: It has been shown that communication skills acquired during undergraduate medical education are of great importance. Hence, many countries require teaching communication as part of their medical curricula. To assess students' learning progress, "Catering to the Patient", as an aspect of showing empathy, should be evaluated. Since there was no description of a validated instrument fitting for this purpose, one had to be developed. To describe its process of development and its psychometric properties were the aims of this study. Methods: Based on the Calgary-Cambridge Observation Guide (CCOG), items describing catering to the patient were selected and modified. Cognitive pretest interviews were conducted to check understandability. Therefore, 7 raters assessed 1 video each (R=7, V=1). In a following pilot study (R=3, V=10) first psychometric properties were evaluated and necessary corrections in the preliminary evaluation form were carried out before the final evaluation form was used to assess students' ability to cater to the patient and psychometric properties were described in detail (R=2, V=35). Results: The final assessment instrument, "catering to the patient - Marburg evaluation form", contains 11 checklist items and two global ratings (items 12 and 13). In the final evaluation the inter-rater reliability (IRR) ranged from 0 to 0.562, the median was r=0.305. Concerning item 13 (a global rating), 88.6% of the videos were scored with the maximum difference of one point. The internal consistency was very high (Cronbach's α: α=0.937 and α=0.962), and the correlation between the checklist items and the global rating was high (Pearson's r: r=0.856 and r=0.898). Discussion: The assessment instrument "catering to the patient" is suitable for giving feedback and for using it in formative examinations. Its use for summative examinations can be considered. Further examinations should evaluate if a three-point Likert scale could reach higher values and if item 13 can be used as a stand-alone item.

Project description:BackgroundAssessing the early development of children at a population level in educational settings, may be useful for public health and policy decision making. In this study, we evaluated the psychometric properties and the contextual appropriateness of a German language version of the Early Development Instrument (EDI), a survey-based instrument originally developed in Canada, which assesses developmental vulnerability for children in preschool settings.MethodsSixty preschool teachers from six preschool organizations (22% of organizations contacted) in three cities in southwest Germany participated. They administered a German version of the EDI (GEDI) to 225 children (51% of eligible children). We assessed internal consistency, test-retest and interrater reliability. Preschool teachers assisted in determining face-validity by reviewing item coverage and comprehensibility. Exploratory factor analysis (EFA) was used to evaluate convergent validity. Concurrent validity was measured using correlations and agreements (Bland-Altman plots) between GEDI and other validated instrument scores. Additionally, we compared associations between GEDI domain scores and sociodemographic characteristics with similar associations in EDI studies worldwide.ResultsGEDI domains showed good to excellent internal consistency (0.73 < α > 0.99) and moderate to good test-retest and interrater reliability (0.50 to 0.81 and 0.48 to 0.71, respectively [p-value < 0.05]). Face validity was considered acceptable. EFA showed a factor structure similar to the original EDI. Correlations (range: 0.32 to 0.67) and agreements between GEDI scores and other German language instruments suggested good external reliability. Scoring within the lowest 10th percentile was strongly associated with age.ConclusionsOur psychometric assessment suggests good reliability and consistency of the GEDI. Differences in the age distribution of children, pedagogical objectives and educational system features of German preschools require future work to determine score thresholds indicative of vulnerability. Aside from dropping selected items from the original EDI that were inconsistent with features of the German educational system, the distribution of values in the language and cognitive development domain also suggested that context-specific cut-offs must be established for the German version. Such efforts are needed to account for relevant contextual differences between the educational systems.

Project description:BackgroundJob satisfaction refers to a person's attitude toward his/her job and its various aspects. Job satisfaction improves the quality of service and employees' physical and mental health. The present study aimed to design a valid and reliable instrument to assess Iranian midwives job satisfaction instrument (MJSI).MethodsThis is a sequential exploratory study for tool design. This study in two phases; (qualitative and tool's psychometric evaluation) was conducted in Ilam, Iran, 2019 years. In the first phase, a qualitative content analysis was carried out by in-depth and semi-structured individual interviews with 10 experts. Then, the pool of items extracted from the qualitative phase was completed by reviewing the existing texts and tools. The second phase of the study involved reducing the overlapping items and validating the tool. In order to investigate the construct validity, a cross sectional study was conducted with the participation of 121 midwives with census sampling. Data analysis was performed by SPSS-19 software using exploratory factor analysis and reliability tests (Cronbach's alpha).ResultsIn the qualitative phase and after reviewing the existing texts and tools by the research team, a 58-item questionnaire was developed and then entered into the psychometric phase. Then, the tool was finalized with five factors, including: 1) communication features, 2) professional features, 3) responsibility aspects, 4) physical-mental aspects and 5) social aspects, respectively. After the psychometric process, by removing the items in different stages, a specific questionnaire was developed to measure the midwives' job satisfaction with 25 items which explained a total of 49.95% of the total variance. Reliability of the tool was approved by Cronbach's alpha = 0.71 and test-retest with 2-weeks intervals, indicating an appropriate stability for the scale (ICC = 0.898).ConclusionThe 25-item self-reporting midwives job satisfaction tool had acceptable validity and reliability. We recommend the use of this tool for evaluating the job satisfaction of midwives, as well as management and research purposes.

Project description:BackgroundInvesting in children's early years can have a lasting positive effect, such as better academic outcomes throughout their school careers. In Jordan, investments have been made in early childhood development and early childhood care and education to improve children's school readiness. School readiness comprises a range of abilities needed to succeed in school, including physical, emotional, social, and cognitive skills. To measure the impact of these investments on children's school readiness, Jordan has been implementing the Early Development Instrument (EDI), a population-level, teacher-completed checklist of children's school readiness, assessing children's development in five main areas, referred to as domains.MethodsThe goal of the current study was to examine the psychometric properties of the Arabic version of the EDI, using data collected in 2018 on a sample of 5952 children in Jordan. The EDI was translated from the original English version to Arabic and adapted for use in Jordan. We conducted a categorical confirmatory factor analysis (CFA) for each of the five domains of the EDI and examined the reliability of the domains and subdomains using Cronbach's alpha reliability coefficient.ResultsWith few exceptions, the study results are in line with those of the analysis of the psychometric properties found with the original, Canadian English version of the EDI in a population of Canadian children. Results of CFAs demonstrated, for the most part, good model fits. Internal consistency indices of the domains ranged from 0.60 for physical health and well-being to 0.96 for social competence. For the subdomains, they ranged from 0.26 to 0.94.ConclusionsOur results provide empirical support for the adaptation of the EDI for population monitoring of school readiness in Jordan. Validation of the Arabic adaptation opens up the possibility of assessing school readiness of young children in Jordan in comparison to the many other countries that have successfully adapted and applied the EDI.