Dataset Information

Comparison of ordinal and nominal classification trees to predict ordinal expert-based occupational exposure estimates in a case-control study.

ABSTRACT:

Objectives

To evaluate occupational exposures in case-control studies, exposure assessors typically review each job individually to assign exposure estimates. This process lacks transparency and does not provide a mechanism for recreating the decision rules in other studies. In our previous work, nominal (unordered categorical) classification trees (CTs) generally successfully predicted expert-assessed ordinal exposure estimates (i.e. none, low, medium, high) derived from occupational questionnaire responses, but room for improvement remained. Our objective was to determine if using recently developed ordinal CTs would improve the performance of nominal trees in predicting ordinal occupational diesel exhaust exposure estimates in a case-control study.

Methods

We used one nominal and four ordinal CT methods to predict expert-assessed probability, intensity, and frequency estimates of occupational diesel exhaust exposure (each categorized as none, low, medium, or high) derived from questionnaire responses for the 14983 jobs in the New England Bladder Cancer Study. To replicate the common use of a single tree, we applied each method to a single sample of 70% of the jobs, using 15% to test and 15% to validate each method. To characterize variability in performance, we conducted a resampling analysis that repeated the sample draws 100 times. We evaluated agreement between the tree predictions and expert estimates using Somers' d, which measures differences in terms of ordinal association between predicted and observed scores and can be interpreted similarly to a correlation coefficient.

Results

From the resampling analysis, compared with the nominal tree, an ordinal CT method that used a quadratic misclassification function and controlled tree size based on total misclassification cost had a slightly better predictive performance that was statistically significant for the frequency metric (Somers' d: nominal tree = 0.61; ordinal tree = 0.63) and similar performance for the probability (nominal = 0.65; ordinal = 0.66) and intensity (nominal = 0.65; ordinal = 0.65) metrics. The best ordinal CT predicted fewer cases of large disagreement with the expert assessments (i.e. no exposure predicted for a job with high exposure and vice versa) compared with the nominal tree across all of the exposure metrics. For example, the percent of jobs with expert-assigned high intensity of exposure that the model predicted as no exposure was 29% for the nominal tree and 22% for the best ordinal tree.

Conclusions

The overall agreements were similar across CT models; however, the use of ordinal models reduced the magnitude of the discrepancy when disagreements occurred. As the best performing model can vary by situation, researchers should consider evaluating multiple CT methods to maximize the predictive performance within their data.

SUBMITTER: Wheeler DC

PROVIDER: S-EPMC4365762 | biostudies-literature | 2015 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Comparison of ordinal and nominal classification trees to predict ordinal expert-based occupational exposure estimates in a case-control study.

Wheeler David C DC Archer Kellie J KJ Burstyn Igor I Yu Kai K Stewart Patricia A PA Colt Joanne S JS Baris Dalsu D Karagas Margaret R MR Schwenn Molly M Johnson Alison A Armenti Karla K Silverman Debra T DT Friesen Melissa C MC

The Annals of occupational hygiene 20141127 3

<h4>Objectives</h4>To evaluate occupational exposures in case-control studies, exposure assessors typically review each job individually to assign exposure estimates. This process lacks transparency and does not provide a mechanism for recreating the decision rules in other studies. In our previous work, nominal (unordered categorical) classification trees (CTs) generally successfully predicted expert-assessed ordinal exposure estimates (i.e. none, low, medium, high) derived from occupational qu ...[more]

PMID: 25433003

Similar Datasets

Project description:Robust maximum likelihood (RML) and asymptotically generalized least squares (AGLS) methods have been recommended for fitting ordinal structural equation models. Studies show that some of these methods underestimate standard errors. However, these studies have not investigated the coverage and bias of interval estimates. An estimate with a reasonable standard error could still be severely biased. This can only be known by systematically investigating the interval estimates. The present study compares Bayesian, RML, and AGLS interval estimates of factor correlations in ordinal confirmatory factor analysis models (CFA) for small sample data. Six sample sizes, 3 factor correlations, and 2 factor score distributions (multivariate normal and multivariate mildly skewed) were studied. Two Bayesian prior specifications, informative and relatively less informative were studied. Undercoverage of confidence intervals and underestimation of standard errors was common in non-Bayesian methods. Underestimated standard errors may lead to inflated Type-I error rates. Non-Bayesian intervals were more positive biased than negatively biased, that is, most intervals that did not contain the true value were greater than the true value. Some non-Bayesian methods had non-converging and inadmissible solutions for small samples and non-normal data. Bayesian empirical standard error estimates for informative and relatively less informative priors were closer to the average standard errors of the estimates. The coverage of Bayesian credibility intervals was closer to what was expected with overcoverage in a few cases. Although some Bayesian credibility intervals were wider, they reflected the nature of statistical uncertainty that comes with the data (e.g., small sample). Bayesian point estimates were also more accurate than non-Bayesian estimates. The results illustrate the importance of analyzing coverage and bias of interval estimates, and how ignoring interval estimates can be misleading. Therefore, editors and policymakers should continue to emphasize the inclusion of interval estimates in research.

Project description:ObjectivesAlgorithm-based exposure assessments based on patterns in questionnaire responses and professional judgment can readily apply transparent exposure decision rules to thousands of jobs quickly. However, we need to better understand how algorithms compare to a one-by-one job review by an exposure assessor. We compared algorithm-based estimates of diesel exhaust exposure to those of three independent raters within the New England Bladder Cancer Study, a population-based case-control study, and identified conditions under which disparities occurred in the assessments of the algorithm and the raters.MethodsOccupational diesel exhaust exposure was assessed previously using an algorithm and a single rater for all 14 983 jobs reported by 2631 study participants during personal interviews conducted from 2001 to 2004. Two additional raters independently assessed a random subset of 324 jobs that were selected based on strata defined by the cross-tabulations of the algorithm and the first rater's probability assessments for each job, oversampling their disagreements. The algorithm and each rater assessed the probability, intensity and frequency of occupational diesel exhaust exposure, as well as a confidence rating for each metric. Agreement among the raters, their aggregate rating (average of the three raters' ratings) and the algorithm were evaluated using proportion of agreement, kappa and weighted kappa (κw). Agreement analyses on the subset used inverse probability weighting to extrapolate the subset to estimate agreement for all jobs. Classification and Regression Tree (CART) models were used to identify patterns in questionnaire responses that predicted disparities in exposure status (i.e., unexposed versus exposed) between the first rater and the algorithm-based estimates.ResultsFor the probability, intensity and frequency exposure metrics, moderate to moderately high agreement was observed among raters (κw = 0.50-0.76) and between the algorithm and the individual raters (κw = 0.58-0.81). For these metrics, the algorithm estimates had consistently higher agreement with the aggregate rating (κw = 0.82) than with the individual raters. For all metrics, the agreement between the algorithm and the aggregate ratings was highest for the unexposed category (90-93%) and was poor to moderate for the exposed categories (9-64%). Lower agreement was observed for jobs with a start year <1965 versus ≥1965. For the confidence metrics, the agreement was poor to moderate among raters (κw = 0.17-0.45) and between the algorithm and the individual raters (κw = 0.24-0.61). CART models identified patterns in the questionnaire responses that predicted a fair-to-moderate (33-89%) proportion of the disagreements between the raters' and the algorithm estimates.DiscussionThe agreement between any two raters was similar to the agreement between an algorithm-based approach and individual raters, providing additional support for using the more efficient and transparent algorithm-based approach. CART models identified some patterns in disagreements between the first rater and the algorithm. Given the absence of a gold standard for estimating exposure, these patterns can be reviewed by a team of exposure assessors to determine whether the algorithm should be revised for future studies.

Project description:The National Institute for Environmental Health Sciences (NIEHS) is conducting an epidemiologic study (GuLF STUDY) to investigate the health of the workers and volunteers who participated from April to December of 2010 in the response and cleanup of the oil release after the Deepwater Horizon explosion in the Gulf of Mexico. The exposure assessment component of the study involves analyzing thousands of personal monitoring measurements that were collected during this effort. A substantial portion of these data has values reported by the analytic laboratories to be below the limits of detection (LOD). A simulation study was conducted to evaluate three established methods for analyzing data with censored observations to estimate the arithmetic mean (AM), geometric mean (GM), geometric standard deviation (GSD), and the 95th percentile (X0.95) of the exposure distribution: the maximum likelihood (ML) estimation, the β-substitution, and the Kaplan-Meier (K-M) methods. Each method was challenged with computer-generated exposure datasets drawn from lognormal and mixed lognormal distributions with sample sizes (N) varying from 5 to 100, GSDs ranging from 2 to 5, and censoring levels ranging from 10 to 90%, with single and multiple LODs. Using relative bias and relative root mean squared error (rMSE) as the evaluation metrics, the β-substitution method generally performed as well or better than the ML and K-M methods in most simulated lognormal and mixed lognormal distribution conditions. The ML method was suitable for large sample sizes (N ≥ 30) up to 80% censoring for lognormal distributions with small variability (GSD = 2-3). The K-M method generally provided accurate estimates of the AM when the censoring was <50% for lognormal and mixed distributions. The accuracy and precision of all methods decreased under high variability (GSD = 4 and 5) and small to moderate sample sizes (N < 20) but the β-substitution was still the best of the three methods. When using the ML method, practitioners are cautioned to be aware of different ways of estimating the AM as they could lead to biased interpretation. A limitation of the β-substitution method is the absence of a confidence interval for the estimate. More research is needed to develop methods that could improve the estimation accuracy for small sample sizes and high percent censored data and also provide uncertainty intervals.

Project description:ObjectivesInvestigating the agreement between an expert-rated mini job exposure matrix (JEM) of lower body exposures and technical measurements of worktime spent standing/walking and observation-based estimates of time spent kneeling/squatting and total load lifted per workday.MethodsWe chose 16 job titles from the 121 job groups in the lower body JEM and included them in the mini JEM. New expert ratings for the mini JEM were performed by the same five occupational physicians who performed the ratings for the lower body JEM. For each job title and type of exposure, the exposure estimates were a mean of the five independent ratings. Technical measurements of standing/walking for all 16 job titles, and for 8 job titles workplace observations were performed of kneeling/squatting and total load lifted per workday. Data were collected from September to December 2015 and supplemented by data from the NOMAD and DPhacto studies collected between 2011 and 2013. All data were collected in Denmark. Agreement between expert-based and measured/observed lower body exposures by job titles was evaluated using Spearman's rank correlation, Bland-Altman plots evaluated systematic deviations and limits of agreement (LoA).ResultsStanding/walking showed a rank correlation of 0.55, kneeling/squatting 0.83 and total load lifted per workday 0.71. The mini JEM estimates did not systematically deviate from the technical measurements/observations for time spent standing/walking (mean difference 0.20 hours/workday, LoA -1.63, 2.03 hours/workday) and kneeling/squatting (mean difference -0.35 hours/workday, LoA -1.21, 0.51 hours/workday). For total load lifted per workday, the mini JEM systematically overestimated the exposures compared with the observations (mean difference -909 kg/workday, LoA -3000, 1147 kg/workday).ConclusionsThere was moderate to very high agreement between an expert-rated mini JEM of standing/walking, kneeling/squatting, and lifting exposures and corresponding technical measurements/observations. This method comparison study supports the use of the expert-based lower body JEM in large-scale occupational epidemiological studies.

Dataset Information

Comparison of ordinal and nominal classification trees to predict ordinal expert-based occupational exposure estimates in a case-control study.

Objectives

Methods

Results

Conclusions

Publications

Comparison of ordinal and nominal classification trees to predict ordinal expert-based occupational exposure estimates in a case-control study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets