Dataset Information

"Yes, but will it work for my patients?" Driving clinically relevant research with benchmark datasets.

ABSTRACT: Benchmark datasets have a powerful normative influence: by determining how the real world is represented in data, they define which problems will first be solved by algorithms built using the datasets and, by extension, who these algorithms will work for. It is desirable for these datasets to serve four functions: (1) enabling the creation of clinically relevant algorithms; (2) facilitating like-for-like comparison of algorithmic performance; (3) ensuring reproducibility of algorithms; (4) asserting a normative influence on the clinical domains and diversity of patients that will potentially benefit from technological advances. Without benchmark datasets that satisfy these functions, it is impossible to address two perennial concerns of clinicians experienced in computational research: "the data scientists just go where the data is rather than where the needs are," and, "yes, but will this work for my patients?" If algorithms are to be developed and applied for the care of patients, then it is prudent for the research community to create benchmark datasets proactively, across specialties. As yet, best practice in this area has not been defined. Broadly speaking, efforts will include design of the dataset; compliance and contracting issues relating to the sharing of sensitive data; enabling access and reuse; and planning for translation of algorithms to the clinical environment. If a deliberate and systematic approach is not followed, not only will the considerable benefits of clinical algorithms fail to be realized, but the potential harms may be regressively incurred across existing gradients of social inequity.

SUBMITTER: Panch T

PROVIDER: S-EPMC7305156 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

"Yes, but will it work for <i>my</i> patients?" Driving clinically relevant research with benchmark datasets.

Panch Trishan T Pollard Tom J TJ Mattie Heather H Lindemer Emily E Keane Pearse A PA Celi Leo Anthony LA

NPJ digital medicine 20200619

Benchmark datasets have a powerful normative influence: by determining how the real world is represented in data, they define which problems will first be solved by algorithms built using the datasets and, by extension, who these algorithms will work <i>for</i>. It is desirable for these datasets to serve four functions: (1) enabling the creation of clinically relevant algorithms; (2) facilitating like-for-like comparison of algorithmic performance; (3) ensuring reproducibility of algorithms; (4 ...[more]

PMID: 32577534

Similar Datasets

Project description:BackgroundDigital clinical measures collected via various digital sensing technologies such as smartphones, smartwatches, wearables, ingestibles, and implantables are increasingly used by individuals and clinicians to capture health outcomes or behavioral and physiological characteristics of individuals. Although academia is taking an active role in evaluating digital sensing products, academic contributions to advancing the safe, effective, ethical, and equitable use of digital clinical measures are poorly characterized.ObjectiveWe performed a systematic review to characterize the nature of academic research on digital clinical measures and to compare and contrast the types of sensors used and the sources of funding support for specific subareas of this research.MethodsWe conducted a PubMed search using a range of search terms to retrieve peer-reviewed articles reporting US-led academic research on digital clinical measures between January 2019 and February 2021. We screened each publication against specific inclusion and exclusion criteria. We then identified and categorized research studies based on the types of academic research, sensors used, and funding sources. Finally, we compared and contrasted the funding support for these specific subareas of research and sensor types.ResultsThe search retrieved 4240 articles of interest. Following the screening, 295 articles remained for data extraction and categorization. The top five research subareas included operations research (research analysis; n=225, 76%), analytical validation (n=173, 59%), usability and utility (data visualization; n=123, 42%), verification (n=93, 32%), and clinical validation (n=83, 28%). The three most underrepresented areas of research into digital clinical measures were ethics (n=0, 0%), security (n=1, 0.5%), and data rights and governance (n=1, 0.5%). Movement and activity trackers were the most commonly studied sensor type, and physiological (mechanical) sensors were the least frequently studied. We found that government agencies are providing the most funding for research on digital clinical measures (n=192, 65%), followed by independent foundations (n=109, 37%) and industries (n=56, 19%), with the remaining 12% (n=36) of these studies completely unfunded.ConclusionsSpecific subareas of academic research related to digital clinical measures are not keeping pace with the rapid expansion and adoption of digital sensing products. An integrated and coordinated effort is required across academia, academic partners, and academic funders to establish the field of digital clinical measures as an evidence-based field worthy of our trust.

Project description:My father was diagnosed with stomach cancer recently. Luckily, it was still at an early stage, and endoscopic surgery successfully took care of it. My father was fortunate; since people with stomach cancer do not show clear symptoms in the early stages, the disease is often not diagnosed until it becomes advanced. In his case, the diagnosis started from a suggestion by his doctor to check whether he had a gastric infection with Helicobacter pylori, a bacterial species found in the digestive tract. In Japan, where he lives, a majority of gastric cancer patients (more than 99%) have been infected with H. pylori [1], and the causative role of this bacterial species in promoting gastric cancer is very well established. Now, scientific understanding connecting gastric cancer to H. pylori is saving the lives of many people, including my father. Thinking about this recent personal experience, I wonder if the connection between bacteria and cancer might have been considered a crazy idea decades ago. Research makes it possible to connect seemingly unrelated matters. My laboratory works on seemingly unrelated research topics, such as fungal infections and autoimmunity. However, my question is the same whatever the topic: How do leukocytes elicit and regulate inflammation when they detect infections or endogenous signals? In fact, host receptors detecting pathogens can induce autoimmunity, and autoimmunity alters host sensitivity to pathogens due to the imbalance in the immune system. We are beginning to gain some insight into this question, as revealed by some of our recent studies. For example, the NLR family, pyrin domain containing 3 (NLRP3) inflammasome, which is known to sense a wide variety of pathogens, can also change the course of experimental autoimmune encephalomyelitis (EAE), an animal model of multiple sclerosis (MS). In particular, our study suggested that disease treatment approaches need to be changed based on the activation status of the NLRP3 inflammasome [2]. Another recent study from our laboratory demonstrated that a protein, termed osteopontin (OPN), skews the balance of population sizes between myeloid cells (i.e., innate immunity) and lymphoid cells (i.e., adaptive immunity) during infections and other biological insults [3]. An intracellular isoform of OPN (iOPN) negatively regulates emergency myelopoiesis. Thus, OPN attenuates host resistance by limiting neutrophil supply at the early stage of systemic Candida infection. In contrast, a secreted OPN (sOPN) isoform positively regulates the expansion of T lymphocytes and ends up triggering autoimmune colitis. I am an immunologist but obtained my PhD in mycology. Nevertheless, it took some time for me to appreciate that research enables us to connect the dots placed far apart. This is a truly exciting time to connect seemingly unrelated biological phenomena, because scientists are exponentially increasing our understanding of nature. This is particularly true in innate immunity, which is not only the central alarming system in host-microbe interactions but also relates to almost any human disease we can imagine. However, we are facing a dark age for science and research, in which certain interests wrongfully discredit some research fields. There are things that can be achieved only by research. I am always ready to tell anyone, "Yes, research matters!".

Dataset Information

"Yes, but will it work for my patients?" Driving clinically relevant research with benchmark datasets.

Publications

"Yes, but will it work for <i>my</i> patients?" Driving clinically relevant research with benchmark datasets.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets