Dataset Information

Assessing a New Prescreening Score for the Simplified Evaluation of the Clinical Quality and Relevance of eHealth Apps: Instrument Validation Study.

ABSTRACT:

Background

In 2020, more than 250 eHealth solutions were added to app stores each day, or 90,000 in the year; however, the vast majority of these solutions have not undergone clinical validation, their quality is unknown, and the user does not know if they are effective and safe. We sought to develop a simple prescreening scoring method that would assess the quality and clinical relevance of each app. We designed this tool with 3 health care stakeholder groups in mind: eHealth solution designers seeking to evaluate a potential competitor or their own tool, investors considering a fundraising candidate, and a hospital clinician or IT department wishing to evaluate a current or potential eHealth solution.

Objective

We built and tested a novel prescreening scoring tool (the Medical Digital Solution scoring tool). The tool, which consists of 26 questions that enable the quick assessment and comparison of the clinical relevance and quality of eHealth apps, was tested on 68 eHealth solutions.

Methods

The Medical Digital Solution scoring tool is based on the 2021 evaluation criteria of the French National Health Authority, the 2022 European Society of Medical Oncology recommendations, and other provided scores. We built the scoring tool with patient association and eHealth experts and submitted it to eHealth app creators, who evaluated their apps via the web-based form in January 2022. After completing the evaluation criteria, their apps obtained an overall score and 4 categories of subscores. These criteria evaluated the type of solution and domain, the solution's targeted population size, the level of clinical assessment, and information about the provider.

Results

In total, 68 eHealth solutions were evaluated with the scoring tool. Oncology apps (22%, 20/90) and general health solutions (23%, 21/90) were the most represented. Of the 68 apps, 32 (47%) were involved in remote monitoring by health professionals. Regarding clinical outcomes, 5% (9/169) of the apps assessed overall survival. Randomized studies had been conducted for 21% (23/110) of the apps to assess their benefit. Of the 68 providers, 38 (56%) declared the objective of obtaining reimbursement, and 7 (18%) out of the 38 solutions seeking reimbursement were assessed as having a high probability of reimbursement. The median global score was 11.2 (range 4.7-17.4) out of 20 and the distribution of the scores followed a normal distribution pattern (Shapiro-Wilk test: P=.33).

Conclusions

This multidomain prescreening scoring tool is simple, fast, and can be deployed on a large scale to initiate an assessment of the clinical relevance and quality of a clinical eHealth app. This simple tool can help a decision-maker determine which aspects of the app require further analysis and improvement.

SUBMITTER: Wagneur N

PROVIDER: S-EPMC9297133 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BACKGROUND:Regular physical activity can not only help with weight management, but also lower cardiovascular risks, cancer rates, and chronic disease burden. Yet, only approximately 20% of Americans currently meet the physical activity guidelines recommended by the US Department of Health and Human Services. With the rapid development of mobile technologies, mobile apps have the potential to improve participation rates in exercise programs, particularly if they are evidence-based and are of sufficient content quality. OBJECTIVE:The goal of this study was to develop and test an instrument, which was designed to score the content quality of exercise program apps with respect to the exercise guidelines set forth by the American College of Sports Medicine (ACSM). METHODS:We conducted two focus groups (N=14) to elicit input for developing a preliminary 27-item scoring instruments based on the ACSM exercise prescription guidelines. Three reviewers who were no sports medicine experts independently scored 28 exercise program apps using the instrument. Inter- and intra-rater reliability was assessed among the 3 reviewers. An expert reviewer, a Fellow of the ACSM, also scored the 28 apps to create criterion scores. Criterion validity was assessed by comparing nonexpert reviewers' scores to the criterion scores. RESULTS:Overall, inter- and intra-rater reliability was high with most coefficients being greater than .7. Inter-rater reliability coefficients ranged from .59 to .99, and intra-rater reliability coefficients ranged from .47 to 1.00. All reliability coefficients were statistically significant. Criterion validity was found to be excellent, with the weighted kappa statistics ranging from .67 to .99, indicating a substantial agreement between the scores of expert and nonexpert reviewers. Finally, all apps scored poorly against the ACSM exercise prescription guidelines. None of the apps received a score greater than 35, out of a possible maximal score of 70. CONCLUSIONS:We have developed and presented valid and reliable scoring instruments for exercise program apps. Our instrument may be useful for consumers and health care providers who are looking for apps that provide safe, progressive general exercise programs for health and fitness.

Project description:IntroductionThis study aimed at generating a new simplified prognostic score (SPS) using common clinical and biological variables to discriminate a limited number of subgroups of patients with SCLC differing by their overall survival (OS).MethodsThe SPS was developed exploring the Montpellier University Hospital retrospective database of 401 patients over a 16-year period. All patients had received etoposide - platinum-based chemotherapy as first-line treatment. The SPS development took into account significant determinants of OS in the Cox model, weighted by their regression β coefficients. Validation of the consequent SPS has been done separately in a combined population of 213 patients accrued from two different published trials (NCT03059667 and NCT00930891).ResultsThe significant independent determinants of OS included the following: (1) American Joint Committee on Cancer TNM stage IV (hazard ratio [HR]: 2.52; 95% confidence interval [CI]: 1.91-3.33); (2) Eastern Cooperative Oncology Group performance status greater than 1 (HR: 2.27; 95% CI: 1.79-2.87); (3) the presence of liver metastases (HR: 1.66; 95% CI: 1.29-2.15); and (4) neutrophil-to-lymphocyte ratio greater than 4 (HR: 1.39; 95% CI: 1.11-1.92). The SPS generated with these four variables, segregated three groups (good, intermediate, and poor prognosis) with respective median OS of 26.9 months (95% CI: 20.1-38.9), 11.5 months (95% CI: 9.8-13.0), and 6.8 months (95% CI: 5.8-8.3; log-rank p < 10-4). Harrell's C statistic estimate was 0.68 ± 0.012, suggesting goodness of calibration. In the validation cohort, the SPS segregated the aforementioned three subgroups in a nearly similar manner, with respective median OS: 27.2, 12.3, and 8.6 months (log-rank p < 10-3; Harrell's C statistic: 0.58 ± 0.02).ConclusionsThe SPS is easy to calculate in real-life practice and efficiently discriminates three populations with different prognoses. This study deserves further validation of this score in patients with SCLC receiving immunochemotherapy.

Project description:BackgroundThe rapid proliferation of web-based information on health and health care has profoundly changed individuals' health-seeking behaviors, with individuals choosing the internet as their first source of information on their health conditions before seeking professional advice. However, barriers to the evaluation of people's eHealth literacy present some difficulties for decision makers with respect to encouraging and empowering patients to use web-based resources.ObjectiveThis study aims to examine the psychometric properties of a simplified Chinese version of the eHealth Literacy Scale (SC-eHEALS).MethodsData used for analysis were obtained from a cross-sectional multicenter survey. Confirmatory factor analysis (CFA) was used to examine the structure of the SC-eHEALS. Correlations between the SC-eHEALS and ICEpop capability measure for adults (ICECAP-A) items and overall health status were estimated to assess the convergent validity. Internal consistency reliability was confirmed using Cronbach alpha (α), McDonald omega (ω), and split-half reliability (λ). A general partial credit model was used to perform the item response theory (IRT) analysis. Item difficulty, discrimination, and fit were reported. Item-category characteristic curves (ICCs) and item and test information curves were used to graphically assess the validity and reliability based on the IRT analysis. Differential item functioning (DIF) was used to check for possible item bias on gender and age.ResultsA total of 574 respondents from 5 cities in China completed the SC-eHEALS. CFA confirmed that the one-factor model was acceptable. The internal consistency reliability was good, with α=0.96, ω=0.92, and λ=0.96. The item-total correlation coefficients ranged between 0.86 and 0.91. Items 8 and 4 showed the lowest and highest mean scores, respectively. The correlation coefficients between the SC-eHEALS and ICECAP-A items and overall health status were significant, but the strength was mild. The discrimination of SC-eHEALS items ranged between 2.63 and 5.42. ICCs indicated that the order of categories' thresholds for all items was as expected. In total, 70% of the information provided by SC-eHEALS was below the average level of the latent trait. DIF was found for item 6 on age.ConclusionsThe SC-eHEALS has been demonstrated to have good psychometric properties and can therefore be used to evaluate people's eHealth literacy in China.

Project description:BackgroundThis paper describes the development of the Chinese Quality of Life Instrument (ChQOL) which is a self-report health status instrument. Chinese Medicine relies very much on asking subjective feelings of patients in the process of diagnosis and monitoring of treatment. For thousands of years, Chinese Medicine practitioners have accumulated a good wealth of experiences in asking questions about health of their patients based on the concept of health in Chinese Medicine. These experiences were then transformed into questions for the ChQOL. It is believed that ChQOL can contribute to the existing Patient Report Outcome measures. This paper outlines the concept of health and disease in Traditional Chinese Medicine, the building of the conceptual framework of the ChQOL, the steps of drafting, selecting and validating the items, and the psychometric properties of the ChQOL.MethodsThe development of the ChQOL was based on the concept of health in Traditional Chinese Medicine with a theory driven approach. Based on the results of literature review, the research team developed an initial model of health which encompassed the concept of health in TCM. An expert panel was then invited to comment and give suggestions for improvement of the initial model. According to their suggestions, the model was refined and a set of initial items for the ChQOL was drafted. The refined model, together with the key domains, facets and initial items of the ChQOL were then mailed to a sample of about 100 Chinese medicine practitioners throughout Mainland China for their comments and advice. A revised set of items were developed for linguistic testing by a convenience sample consisting of both healthy people and people who attended Chinese Medicine treatment. After that, an item pool was developed for field-testing. Field test was conducted on a convenience sample of healthy and patient subjects to determine the construct validity and psychometric properties of the ChQOL.ResultsConstruct validity was established by various methods, i.e. the internal consistency in all facets and domains were good; the correlation between facets to domain, and domains to overall ChQOL correlation were high; confirmatory factor analysis showed that the structure fitness of all facets, domain and overall structure were good with CFI > 0.9. Test-retest reliability was also good, especially in the domain scores with ICC value ranging from 0.83 to 0.90. No ceiling or floor effect was noted which indicated that ChQOL can be applied to subjects with a wide range of health status. Most facet scores, domain scores and the overall CHQOL scores were able to discriminate groups of subjects with known differences in health status. The ChQOL had mild positive convergence with the other generic health related QOL measures, i.e. the WHOQOL-100 and the SF-36, with moderate correlations.ConclusionIn conclusion, the study indicated that the ChQOL is conceptually valid with satisfactory psychometric properties. It can provide additional information on health and QOL on top of the existing generic health related QOL measures. Furthermore, it forms basis for further testing and applications in clinical trials.