Dataset Information

A comparative case study of two models of a clinical informaticist service.

ABSTRACT:

Objectives

To describe and evaluate two different models of a clinical informaticist service.

Design

A case study approach, using various qualitative methods to illuminate the complexity of the project groups' experiences.

Setting

UK primary health care.

Interventions

Two informaticist projects to provide evidence based answers to questions arising in clinical practice and thereby support high quality clinical decision making by practitioners.

Results

The projects took contrasting and complementary approaches to establishing the service. One was based in an academic department of primary health care. The service was academically highly rigorous, remained true to its original proposal, included a prominent research component, and involved relatively little personal contact with practitioners. This group achieved the aim of providing general information and detailed guidance to others intending to set up a similar service. The other group was based in a service general practice and took a much more pragmatic, flexible, and facilitative approach. They achieved the aim of a credible, acceptable, and sustainable service that engaged local practitioners beyond the innovators and enthusiasts and secured continued funding.

Conclusion

An informaticist service should be judged on at least two aspects of quality---an academic dimension (the technical quality of the evidence based answers) and a service dimension (the facilitation of questioning behaviour and implementation). This study suggests that, while the former may be best achieved within an academic environment, the latter requires a developmental approach in which pragmatic service considerations are addressed.

SUBMITTER: Greenhalgh T

PROVIDER: S-EPMC67768 | biostudies-literature | 2002 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A comparative case study of two models of a clinical informaticist service.

Greenhalgh Trisha T Hughes Jane J Humphrey Charlotte C Rogers Stephen S Swinglehurst Deborah D Martin Peter P

BMJ (Clinical research ed.) 20020301 7336

<h4>Objectives</h4>To describe and evaluate two different models of a clinical informaticist service.<h4>Design</h4>A case study approach, using various qualitative methods to illuminate the complexity of the project groups' experiences.<h4>Setting</h4>UK primary health care.<h4>Interventions</h4>Two informaticist projects to provide evidence based answers to questions arising in clinical practice and thereby support high quality clinical decision making by practitioners.<h4>Results</h4>The proj ...[more]

PMID: 11872553

Similar Datasets

Project description:BackgroundWorking together efficiently and effectively presents a significant challenge in large-scale, complex, interdisciplinary research projects. Collaboratories are a nascent method to help meet this challenge. However, formal collaboratories in biomedical research centers are the exception rather than the rule.ObjectiveThe main purpose of this paper is to compare and describe two collaboratories that used off-the-shelf tools and relatively modest resources to support the scientific activity of two biomedical research centers. The two centers were the Great Lakes Regional Center for AIDS Research (HIV/AIDS Center) and the New York University Oral Cancer Research for Adolescent and Adult Health Promotion Center (Oral Cancer Center).MethodsIn each collaboratory, we used semistructured interviews, surveys, and contextual inquiry to assess user needs and define the technology requirements. We evaluated and selected commercial software applications by comparing their feature sets with requirements and then pilot-testing the applications. Local and remote support staff cooperated in the implementation and end user training for the collaborative tools. Collaboratory staff evaluated each implementation by analyzing utilization data, administering user surveys, and functioning as participant observers.ResultsThe HIV/AIDS Center primarily required real-time interaction for developing projects and attracting new participants to the center; the Oral Cancer Center, on the other hand, mainly needed tools to support distributed and asynchronous work in small research groups. The HIV/AIDS Center's collaboratory included a center-wide website that also served as the launch point for collaboratory applications, such as NetMeeting, Timbuktu Conference, PlaceWare Auditorium, and iVisit. The collaboratory of the Oral Cancer Center used Groove and Genesys Web conferencing. The HIV/AIDS Center was successful in attracting new scientists to HIV/AIDS research, and members used the collaboratory for developing and implementing new research studies. The Oral Cancer Center successfully supported highly distributed and asynchronous research, and the collaboratory facilitated real-time interaction for analyzing data and preparing publications.ConclusionsThe two collaboratory implementations demonstrated the feasibility of supporting biomedical research centers using off-the-shelf commercial tools, but they also identified several barriers to successful collaboration. These barriers included computing platform incompatibilities, network infrastructure complexity, variable availability of local versus remote IT support, low computer and collaborative software literacy, and insufficient maturity of available collaborative software. Factors enabling collaboratory use included collaboration incentives through funding mechanism, a collaborative versus competitive relationship of researchers, leadership by example, and tools well matched to tasks and technical progress. Integrating electronic collaborative tools into routine scientific practice can be successful but requires further research on the technical, social, and behavioral factors influencing the adoption and use of collaboratories.

Project description:BackgroundAlthough the importance of clinical ethics in contemporary clinical environments is established, development of formal clinical ethics services in the Australia health system has, to date, been ad hoc. This study was designed to systematically follow and reflect upon the first 18 months of activity by a newly established service, to examine key barriers and facilitators to establishing a new service in an Australian hospital setting.MethodsHOW THE STUDY WAS PERFORMED AND STATISTICAL TESTS USED: A qualitative case study approach was utilised. The study gathered and analysed data using observations of service committee meetings, document analysis of agendas and minutes, and semi-structured interviews with committee members to generate semantic themes. By interpreting the thematic findings in reference to national capacity building resources, this study also aimed to provide practice-based reflections for other health agencies.ResultsTHE MAIN FINDINGS: An overarching theme identified in the data was a strong commitment to supporting clinicians facing difficult patient care decisions and navigating difficult discussions with patients and families. Another key theme was the role of the new clinical ethics support service in providing clinicians with a pathway to raise system-wide issues with the organisation Executive. While there was strong clinical engagement, consumer and community participation remained a challenge, as did unresolved governance issues and a need for clearer policy relationship between the service and the organisation. Considering these themes in relation to the national capacity building resources, the study identifies three areas likely to require ongoing development and negotiation. These are: the role of the clinical ethics support service as a link between the workforce and the Executive; the incorporation of consumers and patients; and ethical reasoning. To improve the effectiveness of the service, it is necessary to increase clarity on the service's role at the governance and policy level, as well as develop strategies for engaging consumers, patients and families. Finally, the capacity of the service to reflect on complex cases may be enhanced through explicit discussions of various different ethical frameworks and ways of deliberating.

Project description:BACKGROUND:Harm reduction services, despite their proved effectiveness in the prevention of infectious diseases, are still underdeveloped in several European states. The situation in the Visegrad Group countries is especially interesting. Notwithstanding the shared history, culture and political situation in the last decades, there are significant differences in the state of harm reduction between the countries. METHODS:The research applies the ecological systems model to identify the structural barriers and facilitators affecting organisations providing needle exchange services. It uses a comparative multiple case study design with embedded units of analysis complemented by within-case analysis to establish the relationship between the number and scope of identified factors and the performance of needle exchange services. The qualitative data were collected through semi-structured interviews with professionals working in needle exchange services in the Czech Republic, Poland, Slovakia and Hungary. Additionally, relevant documents, reports and online sources were analysed. RESULTS:A total of 24 themes (structural factors) were identified across 11 categories on 3 levels (mesosystem, exosystem, macrosystem). The list includes themes related to the broader society, politics and policy on state and local level, frameworks and amounts of funding, the situation on the education labour market, and attitudes of local communities, among others. The data shows that in the Czech Republic, many facilitators can be identified. In the three remaining countries, on the contrary, one can observe mostly barriers in NSP services delivery. CONCLUSIONS:The study addresses a highly unexplored topic of the functioning of harm reduction organisations in East-Central Europe. It sheds light on the environment of analysed services, identifying a number of structural factors in effective service delivery in the Czech Republic, Poland, Slovakia and Hungary. The research confirms the significant role of the barriers and facilitators in the services' performance. It highlights the relationships between various elements of the needle exchange programmes' environment, suggesting holistic strategies for addressing them. It also provides a potential starting point for further research.

Project description:BackgroundAs advancements in artificial intelligence (AI) continue, large language models (LLMs) have emerged as promising tools for generating medical information. Their rapid adaptation and potential benefits in health care require rigorous assessment in terms of the quality, accuracy, and safety of the generated information across diverse medical specialties.ObjectiveThis study aimed to evaluate the performance of 4 prominent LLMs, namely, Claude-instant-v1.0, GPT-3.5-Turbo, Command-xlarge-nightly, and Bloomz, in generating medical content spanning the clinical specialties of ophthalmology, orthopedics, and dermatology.MethodsThree domain-specific physicians evaluated the AI-generated therapeutic recommendations for a diverse set of 60 diseases. The evaluation criteria involved the mDISCERN score, correctness, and potential harmfulness of the recommendations. ANOVA and pairwise t tests were used to explore discrepancies in content quality and safety across models and specialties. Additionally, using the capabilities of OpenAI's most advanced model, GPT-4, an automated evaluation of each model's responses to the diseases was performed using the same criteria and compared to the physicians' assessments through Pearson correlation analysis.ResultsClaude-instant-v1.0 emerged with the highest mean mDISCERN score (3.35, 95% CI 3.23-3.46). In contrast, Bloomz lagged with the lowest score (1.07, 95% CI 1.03-1.10). Our analysis revealed significant differences among the models in terms of quality (P<.001). Evaluating their reliability, the models displayed strong contrasts in their falseness ratings, with variations both across models (P<.001) and specialties (P<.001). Distinct error patterns emerged, such as confusing diagnoses; providing vague, ambiguous advice; or omitting critical treatments, such as antibiotics for infectious diseases. Regarding potential harm, GPT-3.5-Turbo was found to be the safest, with the lowest harmfulness rating. All models lagged in detailing the risks associated with treatment procedures, explaining the effects of therapies on quality of life, and offering additional sources of information. Pearson correlation analysis underscored a substantial alignment between physician assessments and GPT-4's evaluations across all established criteria (P<.01).ConclusionsThis study, while comprehensive, was limited by the involvement of a select number of specialties and physician evaluators. The straightforward prompting strategy ("How to treat…") and the assessment benchmarks, initially conceptualized for human-authored content, might have potential gaps in capturing the nuances of AI-driven information. The LLMs evaluated showed a notable capability in generating valuable medical content; however, evident lapses in content quality and potential harm signal the need for further refinements. Given the dynamic landscape of LLMs, this study's findings emphasize the need for regular and methodical assessments, oversight, and fine-tuning of these AI tools to ensure they produce consistently trustworthy and clinically safe medical advice. Notably, the introduction of an auto-evaluation mechanism using GPT-4, as detailed in this study, provides a scalable, transferable method for domain-agnostic evaluations, extending beyond therapy recommendation assessments.

Project description:ObjectiveSepsis recognition among infants in the Neonatal Intensive Care Unit (NICU) is challenging and delays in recognition can result in devastating consequences. Although predictive models may improve sepsis outcomes, clinical adoption has been limited. Our focus was to align model behavior with clinician information needs by developing a machine learning (ML) pipeline with two components: (1) a model to predict baseline sepsis risk and (2) a model to detect evolving (dynamic) sepsis risk due to physiologic changes. We then compared the performance of this two-component pipeline to a single model that combines all features reflecting both baseline risk and evolving risk.Materials and methodsWe developed prediction models (two-stage pipeline and a single model) using logistic regression and XGBoost trained on electronic healthcare record data of an NICU cohort (1706 observations from 1094 patients, with a 1:1 ratio of cases to controls). We used nested 10-fold cross-validation to evaluate model performance on predictions made 1 h (T -1) before actual clinical recognition.ResultsThe single model (XGBoost) achieved the best performance with a sensitivity of 0.77 (0.74, 0.80), specificity of 0.83 (0.80, 0.85), and positive predictive value (PPV) of 0.82 (0.79, 0.84), at 1 h prior to clinical sepsis recognition (T -1). The pipeline model (XGBoost) achieved a sensitivity of 0.72 (0.69, 0.75), specificity of 0.84 (0.82, 0.87), and PPV of 0.82 (0.80, 0.85) at T -1.DiscussionOur findings highlight the challenges of aligning machine learning with NICU clinical decision-making processes. The two-stage pipeline, designed to mirror clinicians' reasoning, underperformed compared to the single model. Future work should explore integrating continuous physiological data to enhance real-time risk assessment.ConclusionAlthough a pipeline model that separately estimates baseline and dynamic sepsis risk aligns with clinical information needs, at similar levels of specificity the observed sensitivity of the pipeline is inferior to that of a single model. Additional research is needed to better align model outputs with clinician information needs.

Dataset Information

A comparative case study of two models of a clinical informaticist service.

Objectives

Design

Setting

Interventions

Results

Conclusion

Publications

A comparative case study of two models of a clinical informaticist service.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets