Dataset Information

Web-based and mixed-mode cognitive large-scale assessments in higher education: An evaluation of selection bias, measurement bias, and prediction bias.

ABSTRACT: Educational large-scale studies typically adopt highly standardized settings to collect cognitive data on large samples of respondents. Increasing costs alongside dwindling response rates in these studies necessitate exploring alternative assessment strategies such as unsupervised web-based testing. Before respective assessment modes can be implemented on a broad scale, their impact on cognitive measurements needs to be quantified. Therefore, an experimental study on N = 17,473 university students from the German National Educational Panel Study has been conducted. Respondents were randomly assigned to a supervised paper-based, a supervised computerized, and an unsupervised web-based mode to work on a test of scientific literacy. Mode-specific effects on selection bias, measurement bias, and predictive bias were examined. The results showed a higher response rate in web-based testing as compared to the supervised modes, without introducing a pronounced mode-specific selection bias. Analyses of differential test functioning showed systematically larger test scores in paper-based testing, particularly among low to medium ability respondents. Prediction bias for web-based testing was observed for one out of four criteria on study-related success factors. Overall, the results indicate that unsupervised web-based testing is not strictly equivalent to other assessment modes. However, the respective bias introduced by web-based testing was generally small. Thus, unsupervised web-based assessments seem to be a feasible option in cognitive large-scale studies in higher education.

SUBMITTER: Zinn S

PROVIDER: S-EPMC8219565 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundAmong the processes to be experienced by any organization during its establishment is the formation of an organizational identity. This process can be understood as the activity and event through which an organization becomes unique in the mind of its members. An organizational identity leads to an identification and both are directly associated with the success of an institution. This study is about a public higher education institution in health in its early years, with distinctive characteristics in the country where it is situated. In spite of having been successful in the graduation of its students it has fragile institutional bases, lack of autonomy and internal problems common to other institutions of this type. Thus, this study was conducted to understand how this institution defined itself among its own members, the elements of its identity and what justified its relative success despite its weaknesses.MethodsA mixed-method approach was used to evaluate how a representative portion of this organization identifies with it. For the qualitative study two focus groups were conducted with transcripts submitted to content analysis proposed by Bardin, culminating in results from which a Likert scale-based questionnaire was elaborated and applied to 297 subjects.ResultsThere were six central elements of the organizational identity made evident by the focus groups: political / ideological conflict; active teaching and learning methodologies; location / separation of campuses; time of existence; teaching career; political-administrative transformations. The quantitative analysis revealed in more detail the general impressions raised in the focus groups. Most results were able to demonstrate distinct identifications of the same identity with its exposed weaknesses.ConclusionsLack of autonomy, administrative and structural shortcomings and ideological or political conflicts presented themselves as problems capable of destabilizing the identity of a public higher education institution. On the other hand, one way to combat such problems is through the development of the institution itself, particularly by becoming more active and useful to the community and seeking in a common interest to the higher administration agencies.

Project description:ObjectiveRoutine measurement of patient safety from the patients' perspective receives increasing attention as an important component of safety measurement systems. The aim of this study was to examine patients' experience with patient safety in ambulatory care and the results' implications for routine patient safety measurement in ambulatory care.DesignCross-sectional mixed-mode survey.SettingGeneral practitioner and specialist practices.ParticipantsPatients aged >18 years seeking care in ambulatory care practices between February and June 2020.MethodsA 22-item-questionnaire was completed in the practice or at home either on paper or online. Multivariate logistic regression was used to analyse the influence of survey mode and patient characteristics on patient experience with patient safety.ResultsThe overall response rate was 71.1%. Most patients completed the questionnaire on site (76.6%) and on paper (96.1%). Between 30.1% to 68.5% of the respondents report the most positive option for patient experience with the main domains of patient safety. A total of 2.9% of patients reported having experienced a patient-safety event (PSE) during the last 12 months. Patients who filled in the questionnaire off site were more likely to report negative experiences for the scales communication & information (OR 1.2, 95% CI 1.0-1.5), rapport & participation (OR 1.4, 95% CI 1.1-1.7) and access (OR 1.3, 95% CI 0.9-1.4) than those who completed it on site. Those who chose a paper questionnaire were more likely to report negative experiences for all five scales compared to web responders.ConclusionRoutine measurement of patient experience with factors contributing to the occurrence of PSEs can achieve high response rates by offering flexible participation options. Results gained from mixed-mode surveys need to take mode-effects into account when interpreting and using the results. Further research is needed in how to adequately assess number and type of experienced events in routine measurements.

Project description:Introduction ?In the United Kingdom, work-based assessments (WBAs) including procedure-based assessments (PBAs), case-based discussions (CBDs), clinical evaluation exercises (CEXs), and direct observation of procedural skills (DOPS) have been used in Higher General Surgical Training Program (HGSTP) since the introduction of Modernising Medical Careers. Although the Intercollegiate Surgical Curriculum Project states that they should be used for the formative development of trainees using feedback and reflection, there is no study to look at the perception of their usefulness and barriers in using them, particularly in HGSTP. The aim of this study is to investigate trainer's and trainee's perception of their usefulness, barriers in using them, and way forward for their improvement in HGSTP. Methods ?This was a mixed method study. In phase I, after ethics committee approval, an online survey was sent to 83 trainers and 104 trainees, with a response rate of 33 and 37%, respectively, using Online Surveys (formerly Bristol Online Survey) from July 2018 to December 2018. After analysis of this result, in phase II, semistructured interviews were conducted with five trainees and five trainers who had volunteered to take part from phase I. Thematic analysis was performed to develop overarching themes. Results ?For professional formative development, 15% of the trainers and 53% of the trainees felt that WBAs had a low value. Among 4 WBAs-CEX, CBD, PBA, and DOPS-PBA was thought to be the most useful WBA by 52% trainers and 74% trainees. More trainers than trainees felt that it was being used as a formative tool (33 vs. 16%). The total number of WBAs thought to be required was between 20 and 40 per year, with 46% of the trainers and 53% of the trainees preferring these numbers. The thematic analysis generated four themes with subthemes in each: theme 1, "factors affecting usefulness," including the mode of validation, trainer/trainee engagement, and time spent in validating; theme 2, "doubt on utility" due to doubt on validity and being used as a tick-box exercise; theme 3, "pitfalls/difficulties" due to lack of time to validate, late validation, e-mail rather than face-to-face validation, trainer and trainee behavior, variability in feedback given, and emphasis on number than quality; and theme 4, "improvement strategies." Conclusions ?The WBAs are not being used in a way they are supposed to be used. The perception of educational impact (Kirkpatrick levels 1 and 2) by trainers was more optimistic than by trainees. Improvements can be made by giving/finding more time, trainer training, more face-to-face validation, and better trainer trainee interactions.

Dataset Information

Web-based and mixed-mode cognitive large-scale assessments in higher education: An evaluation of selection bias, measurement bias, and prediction bias.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets