Co-Production Performance Evaluation in Healthcare. A Systematic Review of Methods, Tools and Metrics.
Ontology highlight
ABSTRACT:
SUBMITTER: Marsilio M
PROVIDER: S-EPMC8037812 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundPatient, public, consumer, and community (P2C2) engagement in organization-, community-, and systemlevel healthcare decision-making is increasing globally, but its formal evaluation remains challenging. To define a taxonomy of possible P2C2 engagement metrics and compare existing evaluation tools against this taxonomy, we conducted a systematic review.MethodsA broad search strategy was developed for English language publications available from January 1962 through April 2015 in PubMed, Embase, Sociological Abstracts, PsycINFO, EconLit, and the gray literature. A publication was excluded if: (1) the setting was not healthcare delivery (ie, we excluded non-health sectors, such as urban planning; research settings; and public health settings not involving clinical care delivery); (2) the P2C2 engagement was episodic; or (3) the concept of evaluation or possible evaluation metrics were absent. To be included as an evaluation tool, publications had to contain an evaluative instrument that could be employed with minimal modification by a healthcare organization.ResultsA total of 199 out of 3953 publications met exclusion and inclusion criteria. These were qualitatively analyzed using inductive content analysis to create a comprehensive taxonomy of 116 possible metrics for evaluating P2C2 engagement. 44 outcome metrics were grouped into three domains (internal, external, and aggregate outcomes) that included six subdomains: impact on engagement participants, impact on services provided by the healthcare organization, impact on the organization itself, influence on the broader public, influence on population health, and engagement cost-effectiveness. The 72 process metrics formed four domains (direct process metrics; surrogate process metrics; aggregate process metrics; and preconditions for engagement) that comprised sixteen subdomains. We identified 23 potential tools for evaluating P2C2 engagement. The identified tools were published between 1973-2015 and varied in their coverage of the taxonomy, methodology used (qualitative, quantitative, or mixed), and intended evaluators (organizational leaders, P2C2 participants, external evaluators, or some combination). Parts of the metric taxonomy were absent from all tools.ConclusionBy comprehensively mapping potential outcome and process metrics as well as existing P2C2 engagement tools, this review supports high-quality P2C2 engagement globally by informing the selection of existing evaluation tools and identifying gaps where new tools are needed.
| S-EPMC6186472 | biostudies-literature
Project description:Focus on predictive algorithm and its performance evaluation is extensively covered in most research studies to determine best or appropriate predictive model with Optimum prediction solution indicated by prediction accuracy score, precision, recall, f1score etc. Prediction accuracy score from performance evaluation has been used extensively as the main determining metric for performance recommendation. It is one of the most widely used metric for identifying optimal prediction solution irrespective of dataset class distribution context or nature of dataset and output class distribution between the minority and majority variables. The key research question however is the impact of class inequality on prediction accuracy score in such datasets with output class distribution imbalance as compared to balanced accuracy score in the determination of model performance in healthcare and other real-world application systems. Answering this question requires an appraisal of current state of knowledge in both prediction accuracy score and balanced accuracy score use in real-world applications where there is unequal class distribution. Review of related works that highlight the use of imbalanced class distribution datasets with evaluation metrics will assist in contextualizing this systematic review.
| S-EPMC10688675 | biostudies-literature
Project description:This SuperSeries is composed of the SubSeries listed below.
2019-06-27 | GSE131398 | GEO
Project description:Sigma metrics were applied to evaluate the performance of 20 routine chemistry assays, and individual quality control criteria were established based on the sigma values of different assays. Precisions were expressed as the average coefficient variations (CVs) of long-term two-level chemistry controls. The biases of the 20 assays were obtained from the results of trueness programs organized by National Center for Clinical Laboratories (NCCL, China) in 2016. Four different allowable total error (TEa) targets were chosen from biological variation (minimum, desirable, optimal), Clinical Laboratory Improvements Amendments (CLIA, US), Analytical Quality Specification for Routine Analytes in Clinical Chemistry (WS/T 403-2012, China) and the National Cholesterol Education Program (NECP). The sigma values from different TEa targets varied. The TEa targets for ALT, AMY, Ca, CHOL, CK, Crea, GGT, K, LDH, Mg, Na, TG, TP, UA and Urea were chosen from WS/T 403-2012; the targets for ALP, AST and GLU were chosen from CLIA; the target for K was chosen from desirable biological variation; and the targets for HDL and LDL were chosen from the NECP. Individual quality criteria were established based on different sigma values. Sigma metrics are an optimal tool to evaluate the performance of different assays. An assay with a high value could use a simple internal quality control rule, while an assay with a low value should be monitored strictly.
| S-EPMC6816973 | biostudies-literature
Project description:BackgroundEarly warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. In the past few years, there has been a proliferation of studies that describe the development and validation of novel machine learning-based EWS. Systematic reviews of published studies which focus on evaluating performance of both well-established and novel EWS have shown conflicting conclusions. A possible reason is the heterogeneity in validation methods applied. In this review, we aim to examine the methodologies and metrics used in studies which perform EWS validation.MethodsA systematic review of all eligible studies from the MEDLINE database and other sources, was performed. Studies were eligible if they performed validation on at least one EWS and reported associations between EWS scores and inpatient mortality, intensive care unit (ICU) transfers, or cardiac arrest (CA) of adults. Two reviewers independently did a full-text review and performed data abstraction by using standardized data-worksheet based on the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist. Meta-analysis was not performed due to heterogeneity.ResultsThe key differences in validation methodologies identified were (1) validation dataset used, (2) outcomes of interest, (3) case definition, time of EWS use and aggregation methods, and (4) handling of missing values. In terms of case definition, among the 48 eligible studies, 34 used the patient episode case definition while 12 used the observation set case definition, and 2 did the validation using both case definitions. Of those that used the patient episode case definition, 18 studies validated the EWS at a single point of time, mostly using the first recorded observation. The review also found more than 10 different performance metrics reported among the studies.ConclusionsMethodologies and performance metrics used in studies performing validation on EWS were heterogeneous hence making it difficult to interpret and compare EWS performance. Standardizing EWS validation methodology and reporting can potentially address this issue.
| S-EPMC7301346 | biostudies-literature
Project description:In this study, we present a comprehensive evaluation of four RNA-Seq library preparation methods. We used three standard input protocols, the Illumina TruSeq Stranded Total RNA and TruSeq Stranded mRNA kits, and a modified NuGEN Ovation v2 kit; and an ultra-low-input RNA protocol, the TaKaRa SMARTer Ultra Low RNA Kit v3. Our evaluation of these kits included quality control measures such as overall reproducibility, 5’ and 3’ end-bias, and the identification of DEGs, lncRNAs, and alternatively spliced transcripts. Overall, we found that the two Illumina kits were most similar in terms of recovering DEGs, and the Illumina, modified NuGEN, and TaKaRa kits allowed identification of a similar set of DEGs. However, we also discovered that the Illumina, NuGEN and TaKaRa kits each enriched for different sets of genes.
2019-06-27 | GSE131397 | GEO
Project description:In this study, we present a comprehensive evaluation of four RNA-Seq library preparation methods. We used three standard input protocols, the Illumina TruSeq Stranded Total RNA and TruSeq Stranded mRNA kits, and a modified NuGEN Ovation v2 kit; and an ultra-low-input RNA protocol, the TaKaRa SMARTer Ultra Low RNA Kit v3. Our evaluation of these kits included quality control measures such as overall reproducibility, 5’ and 3’ end-bias, and the identification of DEGs, lncRNAs, and alternatively spliced transcripts. Overall, we found that the two Illumina kits were most similar in terms of recovering DEGs, and the Illumina, modified NuGEN, and TaKaRa kits allowed identification of a similar set of DEGs. However, we also discovered that the Illumina, NuGEN and TaKaRa kits each enriched for different sets of genes.
2019-06-27 | GSE131396 | GEO
Project description:Many injury severity scoring tools have been developed over the past few decades. These tools include the Injury Severity Score (ISS), New ISS (NISS), Trauma and Injury Severity Score (TRISS) and International Classification of Diseases (ICD)-based Injury Severity Score (ICISS). Although many studies have endeavored to determine the ability of these tools to predict the mortality of injured patients, their results have been inconsistent. We conducted a systematic review to summarize the predictive performances of these tools and explore the heterogeneity among studies. We defined a relevant article as any research article that reported the area under the Receiver Operating Characteristic curve as a measure of predictive performance. We conducted an online search using MEDLINE and Embase. We evaluated the quality of each relevant article using a quality assessment questionnaire consisting of 10 questions. The total number of positive answers was reported as the quality score of the study. Meta-analysis was not performed due to the heterogeneity among studies. We identified 64 relevant articles with 157 AUROCs of the tools. The median number of positive answers to the questionnaire was 5, ranging from 2 to 8. Less than half of the relevant studies reported the version of the Abbreviated Injury Scale (AIS) and/or ICD (37.5%). The heterogeneity among the studies could be observed in a broad distribution of crude mortality rates of study data, ranging from 1% to 38%. The NISS was mostly reported to perform better than the ISS when predicting the mortality of blunt trauma patients. The relative performance of the ICSS against the AIS-based tools was inconclusive because of the scarcity of studies. The performance of the ICISS appeared to be unstable because the performance could be altered by the type of formula and survival risk ratios used. In conclusion, high-quality studies were limited. The NISS might perform better in the mortality prediction of blunt injuries than the ISS. Additional studies are required to standardize the derivation of the ICISS and determine the relative performance of the ICISS against the AIS-based tools.
| S-EPMC3511252 | biostudies-literature
Project description:BackgroundSocial network analysis quantifies and visualizes relationships between and among individuals or organizations. Applications in the health sector remain underutilized. This systematic review seeks to analyze what social network methods have been used to study professional communication and performance among healthcare providers.MethodsTen databases were searched from 1990 through April 2016, yielding 5970 articles screened for inclusion by two independent reviewers who extracted data and critically appraised each study. Inclusion criteria were study of health care worker professional communication, network methods used, and patient outcomes measured. The search identified 10 systematic reviews. The final set of articles had their citations prospectively and retrospectively screened. We used narrative synthesis to summarize the findings.ResultsThe six articles meeting our inclusion criteria described unique health sectors: one at primary healthcare level and five at tertiary level; five conducted in the USA, one in Australia. Four studies looked at multidisciplinary healthcare workers, while two focused on nurses. Two studies used mixed methods, four quantitative methods only, and one involved an experimental design. Four administered network surveys, one coded observations, and one used an existing survey to extract network data. Density and centrality were the most common network metrics although one study did not calculate any network properties and only visualized the network. Four studies involved tests of significance, and two used modeling methods. Social network analysis software preferences were evenly split between ORA and UCINET. All articles meeting our criteria were published in the past 5 years, suggesting that this remains in clinical care a nascent but emergent research area. There was marked diversity across all six studies in terms of research questions, health sector area, patient outcomes, and network analysis methods.ConclusionNetwork methods are underutilized for the purposes of understanding professional communication and performance among healthcare providers. The paucity of articles meeting our search criteria, lack of studies in middle- and low-income contexts, limited number in non-tertiary settings, and few longitudinal, experimental designs, or network interventions present clear research gaps.Systematic review registrationPROSPERO CRD42015019328.
| S-EPMC5651641 | biostudies-literature
Project description:BACKGROUND:Patient prioritization is a strategy used to manage access to healthcare services. Patient prioritization tools (PPT) contribute to supporting the prioritization decision process, and to its transparency and fairness. Patient prioritization tools can take various forms and are highly dependent on the particular context of application. Consequently, the sets of criteria change from one context to another, especially when used in non-emergency settings. This paper systematically synthesizes and analyzes the published evidence concerning the development and challenges related to the validation and implementation of PPTs in non-emergency settings. METHODS:We conducted a systematic mixed studies review. We searched evidence in five databases to select articles based on eligibility criteria, and information of included articles was extracted using an extraction grid. The methodological quality of the studies was assessed by using the Mixed Methods Appraisal Tool. The article selection process, data extraction, and quality appraisal were performed by at least two reviewers independently. RESULTS:We included 48 studies listing 34 different patient prioritization tools. Most of them are designed for managing access to elective surgeries in hospital settings. Two-thirds of the tools were investigated based on reliability or validity. Inconclusive results were found regarding the impact of PPTs on patient waiting times. Advantages associated with PPT use were found mostly in relationship to acceptability of the tools by clinicians and increased transparency and equity for patients. CONCLUSIONS:This review describes the development and validation processes of PPTs used in non-urgent healthcare settings. Despite the large number of PPTs studied, implementation into clinical practice seems to be an open challenge. Based on the findings of this review, recommendations are proposed to develop, validate, and implement such tools in clinical settings. SYSTEMATIC REVIEW REGISTRATION:PROSPERO CRD42018107205.
| S-EPMC7541289 | biostudies-literature