Dataset Information

Automatic analysis of summary statements in virtual patients - a pilot study evaluating a machine learning approach.

ABSTRACT: BACKGROUND:The ability to compose a concise summary statement about a patient is a good indicator for the clinical reasoning abilities of healthcare students. To assess such summary statements manually a rubric based on five categories - use of semantic qualifiers, narrowing, transformation, accuracy, and global rating has been published. Our aim was to explore whether computer-based methods can be applied to automatically assess summary statements composed by learners in virtual patient scenarios based on the available rubric in real-time to serve as a basis for immediate feedback to learners. METHODS:We randomly selected 125 summary statements in German and English composed by learners in five different virtual patient scenarios. Then we manually rated these statements based on the rubric plus an additional category for the use of the virtual patients' name. We implemented a natural language processing approach in combination with our own algorithm to automatically assess 125 randomly selected summary statements and compared the results of the manual and automatic rating in each category. RESULTS:We found a moderate agreement of the manual and automatic rating in most of the categories. However, some further analysis and development is needed, especially for a more reliable assessment of the factual accuracy and the identification of patient names in the German statements. CONCLUSIONS:Despite some areas of improvement we believe that our results justify a careful display of the computer-calculated assessment scores as feedback to the learners. It will be important to emphasize that the rating is an approximation and give learners the possibility to complain about supposedly incorrect assessments, which will also help us to further improve the rating algorithms.

SUBMITTER: Hege I

PROVIDER: S-EPMC7565765 | biostudies-literature | 2020 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Automatic analysis of summary statements in virtual patients - a pilot study evaluating a machine learning approach.

Hege Inga I Kiesewetter Isabel I Adler Martin M

BMC medical education 20201016 1

<h4>Background</h4>The ability to compose a concise summary statement about a patient is a good indicator for the clinical reasoning abilities of healthcare students. To assess such summary statements manually a rubric based on five categories - use of semantic qualifiers, narrowing, transformation, accuracy, and global rating has been published. Our aim was to explore whether computer-based methods can be applied to automatically assess summary statements composed by learners in virtual patient ...[more]

PMID: 33066789

Similar Datasets

Project description:Bee-mediated pollination greatly increases the size and weight of tomato fruits. Therefore, distinguishing between the local set of bees-those that are efficient pollinators-is essential to improve the economic returns for farmers. To achieve this, it is important to know the identity of the visiting bees. Nevertheless, the traditional taxonomic identification of bees is not an easy task, requiring the participation of experts and the use of specialized equipment. Due to these limitations, the development and implementation of new technologies for the automatic recognition of bees become relevant. Hence, we aim to verify the capacity of Machine Learning (ML) algorithms in recognizing the taxonomic identity of visiting bees to tomato flowers based on the characteristics of their buzzing sounds. We compared the performance of the ML algorithms combined with the Mel Frequency Cepstral Coefficients (MFCC) and with classifications based solely on the fundamental frequency, leading to a direct comparison between the two approaches. In fact, some classifiers powered by the MFCC-especially the SVM-achieved better performance compared to the randomized and sound frequency-based trials. Moreover, the buzzing sounds produced during sonication were more relevant for the taxonomic recognition of bee species than analysis based on flight sounds alone. On the other hand, the ML classifiers performed better in recognizing bees genera based on flight sounds. Despite that, the maximum accuracy obtained here (73.39% by SVM) is still low compared to ML standards. Further studies analyzing larger recording samples, and applying unsupervised learning systems may yield better classification performance. Therefore, ML techniques could be used to automate the taxonomic recognition of flower-visiting bees of the cultivated tomato and other buzz-pollinated crops. This would be an interesting option for farmers and other professionals who have no experience in bee taxonomy but are interested in improving crop yields by increasing pollination.

Project description:Object To evaluate the difference between multiple primary lung adenocarcinoma (MPLA) and solitary primary lung adenocarcinoma (SPLA) by delta-radiomics based machine learning algorithms in CT images. Methods A total of 1094 patients containing 268 MPLAs and 826 SPLAs were recruited for this retrospective study between 2014 to 2020. After the segmentation of volume of interest, the radiomic features were automatically calculated. The patients were categorized into the training set and testing set by a random proportion of 7:3. After feature selection, the relevant classifiers were constructed by the machine learning algorithms of Bayes, forest, k-nearest neighbor, logistic regression, support vector machine, and decision tree. The relative standard deviation (RSD) was calculated and the classification model with minimal RSD was chosen for delta-radiomics analysis to explore the variation of tumor during follow-up surveillance in the cohort of 225 MPLAs and 320 SPLAs. According to the different follow-up duration, it was divided into group A (3–12 months), group B (13–24 months), and group C (25–48 months). Then the corresponding delta-radiomics classifiers were developed to predict MPLAs. The area under the receiver operator characteristic curve (AUC) with 95% confidence interval (CI) was quantified to evaluate the efficiency of the model. Results To radiomics analysis, the forest classifier (FC-radio) with the minimal RSD showed the better stability with AUCs of 0.840 (95%CI, 0.810–0.867) and 0.670 (95%CI, 0.611–0.724) in the training and testing set. The AUCs of the forest classifier based on delta-radiomics (FC-delta) were higher than those of FC-radio. In addition, with the extension of follow-up duration, the performance of FC-delta in Group C were the best with AUCs of 0.998 (95%CI, 0.993–1.000) in the training set and 0.853 (95%CI, 0.752–0.940) in the testing set. Conclusions The machine-learning approach based on radiomics and delta-radiomics helped to differentiate SPLAs from MPLAs. The FC-delta with a longer follow-up duration could better distinguish between SPLAs and MPLAs. Supplementary Information The online version contains supplementary material available at 10.1186/s12885-022-10036-1.

Dataset Information

Automatic analysis of summary statements in virtual patients - a pilot study evaluating a machine learning approach.

Publications

Automatic analysis of summary statements in virtual patients - a pilot study evaluating a machine learning approach.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets