Dataset Information

Questionnaire validation practice within a theoretical framework: a systematic descriptive literature review of health literacy assessments.

ABSTRACT: OBJECTIVE:Validity refers to the extent to which evidence and theory support the adequacy and appropriateness of inferences based on score interpretations. The health sector is lacking a theoretically-driven framework for the development, testing and use of health assessments. This study used the Standards for Educational and Psychological Testing framework of five sources of validity evidence to assess the types of evidence reported for health literacy assessments, and to identify studies that referred to a theoretical validity testing framework. METHODS:A systematic descriptive literature review investigated methods and results in health literacy assessment development, application and validity testing studies. Electronic searches were conducted in EBSCOhost, Embase, Open Access Theses and Dissertations and ProQuest Dissertations. Data were coded to the Standards' five sources of validity evidence, and for reference to a validity testing framework. RESULTS:Coding on 46 studies resulted in 195 instances of validity evidence across the five sources. Only nine studies directly or indirectly referenced a validity testing framework. Evidence based on relations to other variables is most frequently reported. CONCLUSIONS:The health and health equity of individuals and populations are increasingly dependent on decisions based on data collected through health assessments. An evidence-based theoretical framework provides structure and coherence to existing evidence and stipulates where further evidence is required to evaluate the extent to which data are valid for an intended purpose. This review demonstrates the use of the Standards' theoretical validity testing framework to evaluate sources of evidence reported for health literacy assessments. Findings indicate that theoretical validity testing frameworks are rarely used to collate and evaluate evidence in validation practice for health literacy assessments. Use of the Standards' theoretical validity testing framework would improve evaluation of the evidence for inferences derived from health assessment data on which public health and health equity decisions are based.

SUBMITTER: Hawkins M

PROVIDER: S-EPMC7265003 | biostudies-literature | 2020 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Questionnaire validation practice within a theoretical framework: a systematic descriptive literature review of health literacy assessments.

Hawkins Melanie M Elsworth Gerald R GR Hoban Elizabeth E Osborne Richard H RH

BMJ open 20200601 6

<h4>Objective</h4>Validity refers to the extent to which evidence and theory support the adequacy and appropriateness of inferences based on score interpretations. The health sector is lacking a theoretically-driven framework for the development, testing and use of health assessments. This study used the <i>Standards for Educational and Psychological Testing</i> framework of five sources of validity evidence to assess the types of evidence reported for health literacy assessments, and to identif ...[more]

PMID: 32487577

Similar Datasets

Project description:IntroductionContemporary validity testing theory holds that validity lies in the extent to which a proposed interpretation and use of test scores is justified, the evidence for which is dependent on both quantitative and qualitative research methods. Despite this, we hypothesise that development and validation studies for assessments in the field of health primarily report a limited range of statistical properties, and that a systematic theoretical framework for validity testing is rarely applied. Using health literacy assessments as an exemplar, this paper outlines a protocol for a systematic descriptive literature review about types of validity evidence being reported and if the evidence is reported within a theoretical framework.Methods and analysisA systematic descriptive literature review of qualitative and quantitative research will be used to investigate the scope of validation practice in the rapidly growing field of health literacy assessment. This review method employs a frequency analysis to reveal potentially interpretable patterns of phenomena in a research area; in this study, patterns in types of validity evidence reported, as assessed against the criteria of the 2014 Standards for Educational and Psychological Testing, and in the number of studies using a theoretical validity testing framework. The search process will be consistent with the Preferred Reporting Items for Systematic Reviews and Meta-analyses statement. Outcomes of the review will describe patterns in reported validity evidence, methods used to generate the evidence and theoretical frameworks underpinning validation practice and claims. This review will inform a theoretical basis for future development and validity testing of health assessments in general.Ethics and disseminationEthics approval is not required for this systematic review because only published research will be examined. Dissemination of the review findings will be through publication in a peer-reviewed journal, at conference presentations and in the lead author's doctoral thesis.

Project description:BackgroundThe need for digital literacy in aging populations is increasing in the digitalizing society. Digital literacy involves the identification, evaluation, and communication of information through various digital devices or relevant programs.ObjectiveThe aims of this study were to develop an Everyday Digital Literacy Questionnaire (EDLQ), a digital literacy assessment scale, and subsequently evaluate its psychometric properties using a population of community-dwelling older adults in South Korea.MethodsThe EDLQ was developed using an instrument development design. A nationwide survey was conducted, and the study included 1016 community-dwelling older adults (age ≥60 years). To evaluate the psychometric properties, the participants were randomly divided into 2 groups (n=508 each), and the internal consistency (Cronbach α and McDonald ω), structural validity (exploratory factor analysis and confirmatory factor analysis), hypothesis-testing construct validity using the eHealth Literacy Scale (eHEALS), and measurement invariance were analyzed.ResultsAmong the initial 30 items of the EDLQ, 22 items with a 3-factor solution had a total explained variance of 77%. The domains included "information and communication" (9 items), "content creation and management" (4 items), and "safety and security" (9 items). Confirmatory factor analysis was conducted with this 3-factor solution (χ2206=345.1; normed χ2206=1.7; comparative fit index=0.997; Tucker-Lewis index=0.997; root-mean-square error of approximation=0.036; standardized root-mean-square residual=0.050; composite reliability=0.903-0.959; average variance extracted=0.699-0.724; R2=0.616-0.773). Hypothesis-testing construct validity with the eHEALS revealed a strong correlation (r=0.75). Cronbach α and McDonald ω coefficients were .98 and 0.98, respectively. The fit indices for measurement invariance, including the configural, metric, and scalar invariance models, demonstrated a satisfactory fit to the data. Our findings suggest that the psychometric properties of the 22-item EDLQ are valid and reliable for assessing digital literacy among older Korean adults.ConclusionsIn this study, we developed a digital literacy measure with strong psychometric properties that made it suitable for assessing the digital literacy of community-dwelling older adults in Korea. To broaden its applicability, however, further assessment of its feasibility for use with different languages and cultures is necessary. Moreover, more empirical research on digital literacy and related factors in older adults can facilitate the development of personalized digital health care services and educational interventions in the digital society.

Project description:BackgroundHealth literacy (HL) is implicated in improved health decision-making and health promotion, and reduced racial, ethnic, and socioeconomic health disparities. Three major areas of HL include functional, interactive, and critical HL. HL skills develop throughout the lifespan as individuals' psychosocial and cognitive capacities develop and as they accumulate experiences with navigating health systems. Though adolescence is marked by increased involvement in health decision-making, most HL studies and measures of HL have focused on adults. Both the adult and adolescent HL literature are also limited by the paucity of validated test-based measures for assessing HL. The existing test-based validated HL measures for adolescents were originally designed for adults. However, adolescents are at an earlier phase of developing their HL skills (e.g., fewer experiences navigating the health system) compared to adults and measures originally designed for adults may assume prior knowledge that adolescents may lack therein underestimating adolescents' HL. This study developed and validated test-based assessments of adolescents' functional, interactive, and critical HL.MethodsItems were generated in an iterative process: focus groups with adolescents informed item content, cognitive interviews with adolescents and expert consultation established content and face validity of the initial items, and items were revised or removed where indicated. High school students (n = 355) completed a measurement battery including the revised HL items. The items were evaluated and validated using Rasch measurement models.ResultsThe final 6-item functional, 10-item interactive, and 7-item critical HL assessments and their composite (23 items) fit their respective Rasch models. Item-level invariance was established for gender (male vs. female), age (12-15-year-olds vs. 16-18-year-olds), and ethnicity in all assessments. The assessments had good convergent validity with an established measure of functional HL and scores on the assessments were positively related to reading instructions before taking medicine and questioning the truthfulness of health information found online.ConclusionsThese assessments are the first test-based measures of adolescents' interactive and critical HL, the first test-based measure of functional HL designed for adolescents, and the first composite test-based assessment of all three major areas of HL. These assessments should be used to inform strategies for improving adolescents' HL, decision-making, and behaviors.

Project description:BACKGROUND:For people to be able to access, understand, and benefit from the increasing digitalization of health services, it is critical that services are provided in a way that meets the user's needs, resources, and competence. OBJECTIVE:The objective of the study was to develop a questionnaire that captures the 7-dimensional eHealth Literacy Framework (eHLF). METHODS:Draft items were created in parallel in English and Danish. The items were generated from 450 statements collected during the conceptual development of eHLF. In all, 57 items (7 to 9 items per scale) were generated and adjusted after cognitive testing. Items were tested in 475 people recruited from settings in which the scale was intended to be used (community and health care settings) and including people with a range of chronic conditions. Measurement properties were assessed using approaches from item response theory (IRT) and classical test theory (CTT) such as confirmatory factor analysis (CFA) and reliability using composite scale reliability (CSR); potential bias due to age and sex was evaluated using differential item functioning (DIF). RESULTS:CFA confirmed the presence of the 7 a priori dimensions of eHLF. Following item analysis, a 35-item 7-scale questionnaire was constructed, covering (1) using technology to process health information (5 items, CSR=.84), (2) understanding of health concepts and language (5 items, CSR=.75), (3) ability to actively engage with digital services (5 items, CSR=.86), (4) feel safe and in control (5 items, CSR=.87), (5) motivated to engage with digital services (5 items, CSR=.84), (6) access to digital services that work (6 items, CSR=.77), and (7) digital services that suit individual needs (4 items, CSR=.85). A 7-factor CFA model, using small-variance priors for cross-loadings and residual correlations, had a satisfactory fit (posterior productive P value: .27, 95% CI for the difference between the observed and replicated chi-square values: -63.7 to 133.8). The CFA showed that all items loaded strongly on their respective factors. The IRT analysis showed that no items were found to have disordered thresholds. For most scales, discriminant validity was acceptable; however, 2 pairs of dimensions were highly correlated; dimensions 1 and 5 (r=.95), and dimensions 6 and 7 (r=.96). All dimensions were retained because of strong content differentiation and potential causal relationships between these dimensions. There is no evidence of DIF. CONCLUSIONS:The eHealth Literacy Questionnaire (eHLQ) is a multidimensional tool based on a well-defined a priori eHLF framework with robust properties. It has satisfactory evidence of construct validity and reliable measurement across a broad range of concepts (using both CTT and IRT traditions) in various groups. It is designed to be used to understand and evaluate people's interaction with digital health services.

Dataset Information

Questionnaire validation practice within a theoretical framework: a systematic descriptive literature review of health literacy assessments.

Publications

Questionnaire validation practice within a theoretical framework: a systematic descriptive literature review of health literacy assessments.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets