Dataset Information

Validity and reliability of exposure assessors' ratings of exposure intensity by type of occupational questionnaire and type of rater.

ABSTRACT:

Background

In epidemiologic studies that rely on professional judgment to assess occupational exposures, the raters' accurate assessment is vital to detect associations. We examined the influence of the type of questionnaire, type of industry, and type of rater on the raters' ability to reliably and validly assess within-industry differences in exposure. Our aim was to identify areas where improvements in exposure assessment may be possible.

Methods

Subjects from three foundries (n = 72) and three textile plants (n = 74) in Shanghai, China, completed an occupational history (OH) and an industry-specific questionnaire (IQ). Six total dust measurements were collected per subject and were used to calculate a subject-specific measurement mean, which was used as the gold standard. Six raters independently ranked the intensity of each subject's current job on an ordinal scale (1-4) based on the OH alone and on the OH and IQ together. Aggregate ratings were calculated for the group, for industrial hygienists, and for occupational physicians. We calculated intra-class correlation coefficients (ICCs) to evaluate the reliability of the raters. We calculated the correlation between the subject-specific measurement means and the ratings to evaluate the raters' validity. Analyses were stratified by industry, type of questionnaire, and type of rater. We also examined the agreement between the ratings by exposure category, where the subject-specific measurement means were categorized into two and four categories.

Results

The reliability and validity measures were higher for the aggregate ratings than for the ratings from the individual raters. The group's performance was maximized with three raters. Both the reliability and validity measures were higher for the foundry industry than for the textile industry. The ICCs were consistently lower in the OH/IQ round than in the OH round in both industries. In contrast, the correlations with the measurement means were higher in the OH/IQ round than in the OH round for the foundry industry (group rating, OH/IQ: Spearman rho = 0.77; OH: rho = 0.64). No pattern by questionnaire type was observed for the textile industry (group rating, Spearman rho = 0.50, both assessment rounds). For both industries, the agreement by exposure category was higher when the task was reduced to discriminating between two versus four exposure categories.

Conclusions

Assessments based on professional judgment may reduce misclassification by using two or three raters, by using questionnaires that systematically collect task information, and by defining intensity categories that are distinguishable by the raters. However, few studies have the resources to use multiple raters and these additional efforts may not be adequate for obtaining valid subjective ratings. Thus, improving exposure assessment approaches for studies that rely on professional judgment remain an important research need.

SUBMITTER: Friesen MC

PROVIDER: S-EPMC3131504 | biostudies-literature | 2011 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Validity and reliability of exposure assessors' ratings of exposure intensity by type of occupational questionnaire and type of rater.

Friesen Melissa C MC Coble Joseph B JB Katki Hormuzd A HA Ji Bu-Tian BT Xue Shouzheng S Lu Wei W Stewart Patricia A PA

The Annals of occupational hygiene 20110421 6

<h4>Background</h4>In epidemiologic studies that rely on professional judgment to assess occupational exposures, the raters' accurate assessment is vital to detect associations. We examined the influence of the type of questionnaire, type of industry, and type of rater on the raters' ability to reliably and validly assess within-industry differences in exposure. Our aim was to identify areas where improvements in exposure assessment may be possible.<h4>Methods</h4>Subjects from three foundries ( ...[more]

PMID: 21511891

Similar Datasets

Project description:BACKGROUND: Available evidence suggest that perceptions or ratings of the neighborhood, e.g. as being green, walkable or noisy, are important for effects on health and wellbeing, also after controlling for objective measures of identical or similar features. When evaluating effects of the perceived environment, it is important that measurement properties and the reliability of the environmental ratings are evaluated before decisions about how these ratings should be handled in the statistical analyses are made. In this paper we broaden the usage of two association measures, the well-known kappa statistic and the novel colocation quotient (CLQ), to studies of inter-rater reliability and of associations between different categorical ratings in spatial contexts. METHODS: We conducted reliability analysis of a survey instrument for assessing perceived greenness at geographical point locations, here the close outdoor environment within 5-10 minutes walking distance from home. Data were obtained from a public health survey conducted in 2008 in Scania, southern Sweden (n =27 967 participants). RESULTS: The results demonstrate the usefulness of kappa and CLQ as tools for assessing reliability and measurement properties of environmental rating scales when used at geographical point locations. We further show that the two measures are interchangeable, i.e. kappa can be accurately approximated from CLQ and vice versa, but can be used for somewhat different purposes in reliability analyses. Inter-rater reliability between the nearest neighbors was demonstrated for all five items of the evaluated instrument for assessing perceived greenness, albeit with clear differences across the items. CONCLUSION: Reliability analysis employing kappa and CLQ can be used as a basis for informed decisions about, for instance, how dichotomizations of the ratings should be defined and how missing or indefinite ratings should be handled. Such reliability analyses can thus serve as guidance for subsequent epidemiological studies of associations between environmental ratings, health and wellbeing.

Project description:BackgroundThe CHEERS is a self-administered tool to measure gaps, weaknesses, and strengths of an early childhood education and care (ECEC) centre-based nutrition and physical activity environment. ECEC settings have the potential to profoundly influence early dietary and physical activity behaviours. Content validation of the CHEERS tool has been previously reported. The purpose of this study was to develop reliability and validity evidence for the CHEERS audit tool and the proposed subscales of food served, healthy eating environment, program planning, and physical activity environment in ECEC centre-based programs.MethodsThis cross-sectional study consisted of 2 phases: Phase 1 included inter-, intra-rater and Cronbach's ?. A subset of this sample was invited to participate in a second survey (Trial 2) for intra-rater assessment within 3 weeks of completing the first survey (Trial 1). Phase 2 included concurrent validity assessment between a nutrition expert and the ECEC director using within a one-week period.ResultsOne hundred two directors and 85 educators (total of 187) returned the survey. Of these, there were 75 matched pairs for inter-rater reliability analysis providing a CHEERS ICC score of 0.59 and ICC scores ranging from 0.40 to 0.58 for the subscales. The ICC for intra-rater reliability of the CHEERS score was 0.81 for 40 participants completing the survey a second time and a range of 0.72 to 0.79 for the subscales. The CHEERS tool demonstrated very good internal consistency (??=?0.91) and ? scores ranging from 0.73 to 0.79 for the subscales. In phase 2, concurrent validation was ICC?=?0.65 (n?=?30) CHEERS scores with a range of 0.42 to 0.69 for the subscales.ConclusionsThis study provides evidence of inter-, intra-rater reliability, internal consistency, and concurrent validity of an environmental assessment audit tool to assess the nutrition and physical activity environment of ECEC centre-based programs. The results demonstrate that the self-administered CHEERS instrument is stable overtime and between evaluators at the same ECEC centre. The scores obtained with CHEERS self-administered audit tool are reasonably accurate compared to an expert rater (dietitian) assessment. This study adds additional support to establishing the psychometric soundness of the CHEERS tool.

Dataset Information

Validity and reliability of exposure assessors' ratings of exposure intensity by type of occupational questionnaire and type of rater.

Background

Methods

Results

Conclusions

Publications

Validity and reliability of exposure assessors' ratings of exposure intensity by type of occupational questionnaire and type of rater.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets