Dataset Information

Detecting cheating in written medical examinations by statistical analysis of similarity of answers: pilot study.

ABSTRACT:

Objective

To assess whether a computer program using a variant of Angoff's method can detect anomalous behaviour indicative of cheating in multiple choice medical examinations.

Design

Statistical analysis of 11 examinations held by the Royal College of Paediatrics and Child Health.

Setting

UK postgraduate medical examination.

Participants

Examination candidates.

Main outcome measures

Detection of anomalous candidate pairs by regression of similarity of correct answers in all possible pairs of candidates on the overall proportion of correct answers. Anomalous pairs were subsequently assessed in terms of examination centres and the seating plan of candidates, to assess adjacency.

Results

The 11 examinations were taken by a total of 11,518 candidates, and Acinonyx examined 6,178,628 pairs of candidates. Two examinations showed no anomalies, and one examination found an anomaly resulting from a scanning error. The other eight examinations showed 13 anomalies compatible with cheating, and in each pair the two candidates had sat the examination at the same centre, and for six examinations with seating plans, the candidates in the anomalous pairs had been seated side by side. The raw probabilities of the anomalies varied from 3.9x10(-11) to 9.3x10(-30) (median = 1.1x10(-17)), with Bonferroni-corrected probabilities in the range 2.4x10(-5) to 4.1x10(-24) (median = 1.6x10(-11)). This suggests that one anomalous pair is found for every 1000 or so candidates taking this postgraduate examination.

Conclusions

This statistical technique identified a small proportion of candidates who had very similar patterns of correctly answered questions. The likelihood is that one candidate has copied from the other, or that there was collusion, or that a technical error occurred in the exams department (as happened in a single case). Analysis of similarities can be used to identify cheating and as part of the quality assurance process of postgraduate medical examinations.

SUBMITTER: McManus IC

PROVIDER: S-EPMC557229 | biostudies-literature | 2005 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Detecting cheating in written medical examinations by statistical analysis of similarity of answers: pilot study.

McManus I C IC Lissauer Tom T Williams S E SE

BMJ (Clinical research ed.) 20050501 7499

<h4>Objective</h4>To assess whether a computer program using a variant of Angoff's method can detect anomalous behaviour indicative of cheating in multiple choice medical examinations.<h4>Design</h4>Statistical analysis of 11 examinations held by the Royal College of Paediatrics and Child Health.<h4>Setting</h4>UK postgraduate medical examination.<h4>Participants</h4>Examination candidates.<h4>Main outcome measures</h4>Detection of anomalous candidate pairs by regression of similarity of correct ...[more]

PMID: 15879391

Similar Datasets

Project description:ObjectiveLarge language models (LLMs) such as ChatGPT are being developed for use in research, medical education and clinical decision systems. However, as their usage increases, LLMs face ongoing regulatory concerns. This study aims to analyse ChatGPT's performance on a postgraduate examination to identify areas of strength and weakness, which may provide further insight into their role in healthcare.DesignWe evaluated the performance of ChatGPT 4 (24 May 2023 version) on official MRCP (Membership of the Royal College of Physicians) parts 1 and 2 written examination practice questions. Statistical analysis was performed using Python. Spearman rank correlation assessed the relationship between the probability of correctly answering a question and two variables: question difficulty and question length. Incorrectly answered questions were analysed further using a clinical reasoning framework to assess the errors made.SettingOnline using ChatGPT web interface.Primary and secondary outcome measuresPrimary outcome was the score (percentage questions correct) in the MRCP postgraduate written examinations. Secondary outcomes were qualitative categorisation of errors using a clinical decision-making framework.ResultsChatGPT achieved accuracy rates of 86.3% (part 1) and 70.3% (part 2). Weak but significant correlations were found between ChatGPT's accuracy and both just-passing rates in part 2 (r=0.34, p=0.0001) and question length in part 1 (r=-0.19, p=0.008). Eight types of error were identified, with the most frequent being factual errors, context errors and omission errors.ConclusionChatGPT performance greatly exceeded the passing mark for both exams. Multiple choice examinations provide a benchmark for LLM performance which is comparable to human demonstrations of knowledge, while also highlighting the errors LLMs make. Understanding the reasons behind ChatGPT's errors allows us to develop strategies to prevent them in medical devices that incorporate LLM technology.

Project description:Many proteins function as homo-oligomers and are regulated via their oligomeric state. For some proteins, the stoichiometry of homo-oligomeric states under various conditions has been studied using gel filtration or analytical ultracentrifugation experiments. The interfaces involved in these assemblies may be identified using cross-linking and mass spectrometry, solution-state NMR, and other experiments. However, for most proteins, the actual interfaces that are involved in oligomerization are inferred from X-ray crystallographic structures using assumptions about interface surface areas and physical properties. Examination of interfaces across different Protein Data Bank (PDB) entries in a protein family reveals several important features. First, similarities in space group, asymmetric unit size, and cell dimensions and angles (within 1%) do not guarantee that two crystals are actually the same crystal form, containing similar relative orientations and interactions within the crystal. Conversely, two crystals in different space groups may be quite similar in terms of all the interfaces within each crystal. Second, NMR structures and an existing benchmark of PDB crystallographic entries consisting of 126 dimers as well as larger structures and 132 monomers were used to determine whether the existence or lack of common interfaces across multiple crystal forms can be used to predict whether a protein is an oligomer or not. Monomeric proteins tend to have common interfaces across only a minority of crystal forms, whereas higher-order structures exhibit common interfaces across a majority of available crystal forms. The data can be used to estimate the probability that an interface is biological if two or more crystal forms are available. Finally, the Protein Interfaces, Surfaces, and Assemblies (PISA) database available from the European Bioinformatics Institute is more consistent in identifying interfaces observed in many crystal forms compared with the PDB and the European Bioinformatics Institute's Protein Quaternary Server (PQS). The PDB, in particular, is missing highly likely biological interfaces in its biological unit files for about 10% of PDB entries.

Dataset Information

Detecting cheating in written medical examinations by statistical analysis of similarity of answers: pilot study.

Objective

Design

Setting

Participants

Main outcome measures

Results

Conclusions

Publications

Detecting cheating in written medical examinations by statistical analysis of similarity of answers: pilot study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets