Dataset Information

Web-based and mixed-mode cognitive large-scale assessments in higher education: An evaluation of selection bias, measurement bias, and prediction bias.

ABSTRACT: Educational large-scale studies typically adopt highly standardized settings to collect cognitive data on large samples of respondents. Increasing costs alongside dwindling response rates in these studies necessitate exploring alternative assessment strategies such as unsupervised web-based testing. Before respective assessment modes can be implemented on a broad scale, their impact on cognitive measurements needs to be quantified. Therefore, an experimental study on N = 17,473 university students from the German National Educational Panel Study has been conducted. Respondents were randomly assigned to a supervised paper-based, a supervised computerized, and an unsupervised web-based mode to work on a test of scientific literacy. Mode-specific effects on selection bias, measurement bias, and predictive bias were examined. The results showed a higher response rate in web-based testing as compared to the supervised modes, without introducing a pronounced mode-specific selection bias. Analyses of differential test functioning showed systematically larger test scores in paper-based testing, particularly among low to medium ability respondents. Prediction bias for web-based testing was observed for one out of four criteria on study-related success factors. Overall, the results indicate that unsupervised web-based testing is not strictly equivalent to other assessment modes. However, the respective bias introduced by web-based testing was generally small. Thus, unsupervised web-based assessments seem to be a feasible option in cognitive large-scale studies in higher education.

SUBMITTER: Zinn S

PROVIDER: S-EPMC8219565 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundHigher education institutions need to put change management as a pivotal part of their strategy. The challenge is to effectively contextualize existing change management models to the respective work environment. Failing to properly adapt existing models to match the intricacies of the environment could lead to plenty of setbacks. For such a contextualization to take place, gauging employees' engagement and satisfaction becomes of paramount importance. As such, the overall purpose of the current study is to explore the perception of employees of a medical and health sciences university in Middle East and North Africa (MENA) region, in relation to change management and agility, and to showcase how the captured perspectives can be systemically interpreted to inform decision-making in the context of the study.MethodThis research study relied on a sequential mixed methods design, which started with an exploration of the perception of Mohammed Bin Rashid University of Medicine and Health Sciences (MBRU) leaders. Qualitative data was collected through a focus group session and was inductively analysed (based on constructivist epistemology). The output of the qualitative analysis contributed to the development of the quantitative data collection tool. The quantitative data was analysed by SPSS-version-27.FindingsThe qualitative analysis generated three key themes: Trigger, Execution, and Results, along with a thorough outline of lessons learned and opportunities for improvement. The Cronbach's Alpha reliability score was 92.8%. The percentage of the total average of agreement was 72.3%, and it appeared that 83.2% of the variance can be explained by the instrument (p<0.001).ConclusionThe current study generated a novel conceptual framework that can be leveraged by educational leadership and administration to reinforce their decisions and optimize their agility in terms of managing change. The study also introduces a data collection tool which captures the perception of higher education stakeholders regarding the way their respective institutions handle change. This tool proved to be reliable and valid in the context of the study.

Project description:BackgroundAmong the processes to be experienced by any organization during its establishment is the formation of an organizational identity. This process can be understood as the activity and event through which an organization becomes unique in the mind of its members. An organizational identity leads to an identification and both are directly associated with the success of an institution. This study is about a public higher education institution in health in its early years, with distinctive characteristics in the country where it is situated. In spite of having been successful in the graduation of its students it has fragile institutional bases, lack of autonomy and internal problems common to other institutions of this type. Thus, this study was conducted to understand how this institution defined itself among its own members, the elements of its identity and what justified its relative success despite its weaknesses.MethodsA mixed-method approach was used to evaluate how a representative portion of this organization identifies with it. For the qualitative study two focus groups were conducted with transcripts submitted to content analysis proposed by Bardin, culminating in results from which a Likert scale-based questionnaire was elaborated and applied to 297 subjects.ResultsThere were six central elements of the organizational identity made evident by the focus groups: political / ideological conflict; active teaching and learning methodologies; location / separation of campuses; time of existence; teaching career; political-administrative transformations. The quantitative analysis revealed in more detail the general impressions raised in the focus groups. Most results were able to demonstrate distinct identifications of the same identity with its exposed weaknesses.ConclusionsLack of autonomy, administrative and structural shortcomings and ideological or political conflicts presented themselves as problems capable of destabilizing the identity of a public higher education institution. On the other hand, one way to combat such problems is through the development of the institution itself, particularly by becoming more active and useful to the community and seeking in a common interest to the higher administration agencies.

Dataset Information

Web-based and mixed-mode cognitive large-scale assessments in higher education: An evaluation of selection bias, measurement bias, and prediction bias.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets