Dataset Information

User-centered design of a web-based crowdsourcing-integrated semantic text annotation tool for building a mental health knowledge base.

ABSTRACT:

Background

One in five U.S. adults lives with some kind of mental health condition and 4.6% of all U.S. adults have a serious mental illness. The Internet has become the first place for these people to seek online mental health information for help. However, online mental health information is not well-organized and often of low quality. There have been efforts in building evidence-based mental health knowledgebases curated with information manually extracted from the high-quality scientific literature. Manual extraction is inefficient. Crowdsourcing can potentially be a low-cost mechanism to collect labeled data from non-expert laypeople. However, there is not an existing annotation tool integrated with popular crowdsourcing platforms to perform the information extraction tasks. In our previous work, we prototyped a Semantic Text Annotation Tool (STAT) to address this gap.

Objective

We aimed to refine the STAT prototype (1) to improve its usability and (2) to enhance the crowdsourcing workflow efficiency to facilitate the construction of evidence-based mental health knowledgebase, following a user-centered design (UCD) approach.

Methods

Following UCD principles, we conducted four design iterations to improve the initial STAT prototype. In the first two iterations, usability testing focus groups were conducted internally with 8 participants recruited from a convenient sample, and the usability was evaluated with a modified System Usability Scale (SUS). In the following two iterations, usability testing was conducted externally using the Amazon Mechanical Turk (MTurk) platform. In each iteration, we summarized the usability testing results through thematic analysis, identified usability issues, and conducted a heuristic evaluation to map identified usability issues to Jakob Nielsen's usability heuristics. We collected suggested improvements in the usability testing sessions and enhanced STAT accordingly in the next UCD iteration. After four UCD iterations, we conducted a case study of the system on MTurk using mental health related scientific literature. We compared the performance of crowdsourcing workers with two expert annotators from two aspects: efficiency and quality.

Results

The SUS score increased from 70.3 ± 12.5 to 81.1 ± 9.8 after the two internal UCD iterations as we improved STAT's functionality based on the suggested improvements. We then evaluated STAT externally through MTurk in the following two iterations. The SUS score decreased to 55.7 ± 20.1 in the third iteration, probably because of the complexity of the tasks. After further simplification of STAT and the annotation tasks with an improved annotation guideline, the SUS score increased to 73.8 ± 13.8 in the fourth iteration of UCD. In the evaluation case study, on average, the workers spent 125.5 ± 69.2 s on the onboarding tutorial and the crowdsourcing workers spent significantly less time on the annotation tasks compared to the two experts. In terms of annotation quality, the workers' annotation results achieved average F1-scores ranged from 0.62 to 0.84 for the different sentences.

Conclusions

We successfully developed a web-based semantic text annotation tool, STAT, to facilitate the curation of semantic web knowledgebases through four UCD iterations. The lessons learned from the UCD process could serve as a guide to further enhance STAT and the development and design of other crowdsourcing-based semantic text annotation tasks. Our study also showed that a well-organized, informative annotation guideline is as important as the annotation tool itself. Further, we learned that a crowdsourcing task should consist of multiple simple microtasks rather than a complicated task.

SUBMITTER: He X

PROVIDER: S-EPMC7577935 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundAlthough eHealth technology makes it possible to improve the management of complex health care systems and follow up on chronic patients, it is not without challenges, thus requiring the development of efficient programs and graphic user interface (GUI) features. Similar information technology tools are crucial, as health care populations are going to have to endure social distancing measures in the forthcoming months and years.ObjectiveThis study aims to provide adequate and personalized support to complex health care populations by developing a specific web-based mobile app. The app is designed around the patient and adapted to specific groups, for example, people with complex or rare diseases, autism, or disabilities (especially among children) as well as Alzheimer or senile dementia. The app's core features include the collection, labeling, analysis, and sorting of clinical data. Furthermore, it authorizes a network of people around the patient to securely access the data contained in his or her electronic health record.MethodsThe application was designed according to the paradigms of patient-centered care and user-centered design (UCD). It considers the patient as the main empowered and motivating factor in the management of his or her well-being. Implementation was informed through a family needs and technology perception assessment. We used 3 interdisciplinary focus groups and 2 assessment surveys to study the contexts of app use, subpopulation management, and preferred functions. Finally, we developed an observational study involving 116 enrolled patients and 253 system users, followed by 2 feedback surveys to evaluate the performance and impact of the app.ResultsIn the validated general GUI, we developed 10 user profiles with different privacy settings. We tested 81 functions and studied a modular structure based on disease or medical area. This allowed us to identify replicable methods to be applied to module design. The observational study not only showed good family and community engagement but also revealed some limitations that need to be addressed. In total, 42 of 51 (82%) patients described themselves as satisfied or very satisfied. Health care providers reported facilitated communication with colleagues and the need to support data quality.ConclusionsThe experimented solution addressed some of the health system challenges mentioned by the World Health Organization: usability appears to be significantly improved when the GUI is designed according to patients' UCD mental models and when new media and medical literacy are promoted. This makes it possible to maximize the impact of eHealth products, thereby overcoming some crucial gaps reported in the literature. Two main features seemed to have potential benefit compared with other eHealth products: the modeling, within the app, of both the formal and informal health care support networks and the modular structure allowing for comorbidity management, both of which require further implementation.