Dataset Information

Reporting methods in studies developing prognostic models in cancer: a review.

ABSTRACT:

Background

Development of prognostic models enables identification of variables that are influential in predicting patient outcome and the use of these multiple risk factors in a systematic, reproducible way according to evidence based methods. The reliability of models depends on informed use of statistical methods, in combination with prior knowledge of disease. We reviewed published articles to assess reporting and methods used to develop new prognostic models in cancer.

Methods

We developed a systematic search string and identified articles from PubMed. Forty-seven articles were included that satisfied the following inclusion criteria: published in 2005; aiming to predict patient outcome; presenting new prognostic models in cancer with outcome time to an event and including a combination of at least two separate variables; and analysing data using multivariable analysis suitable for time to event data.

Results

In 47 studies, prospective cohort or randomised controlled trial data were used for model development in only 33% (15) of studies. In 30% (14) of the studies insufficient data were available, having fewer than 10 events per variable (EPV) used in model development. EPV could not be calculated in a further 40% (19) of the studies. The coding of candidate variables was only reported in 68% (32) of the studies. Although use of continuous variables was reported in all studies, only one article reported using recommended methods of retaining all these variables as continuous without categorisation. Statistical methods for selection of variables in the multivariate modelling were often flawed. A method that is not recommended, namely, using statistical significance in univariate analysis as a pre-screening test to select variables for inclusion in the multivariate model, was applied in 48% (21) of the studies.

Conclusions

We found that published prognostic models are often characterised by both use of inappropriate methods for development of multivariable models and poor reporting. In addition, models are limited by the lack of studies based on prospective data of sufficient sample size to avoid overfitting. The use of poor methods compromises the reliability of prognostic models developed to provide objective probability estimates to complement clinical intuition of the physician and guidelines.

SUBMITTER: Mallett S

PROVIDER: S-EPMC2856521 | biostudies-literature | 2010 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Reporting methods in studies developing prognostic models in cancer: a review.

Mallett Susan S Royston Patrick P Dutton Susan S Waters Rachel R Altman Douglas G DG

BMC medicine 20100330

<h4>Background</h4>Development of prognostic models enables identification of variables that are influential in predicting patient outcome and the use of these multiple risk factors in a systematic, reproducible way according to evidence based methods. The reliability of models depends on informed use of statistical methods, in combination with prior knowledge of disease. We reviewed published articles to assess reporting and methods used to develop new prognostic models in cancer.<h4>Methods</h ...[more]

PMID: 20353578

Similar Datasets

Project description:BackgroundHIV drug resistance is a global health problem that limits the effectiveness of antiretroviral therapy. Adequate surveillance of HIV drug resistance is challenged by heterogenous and inadequate data reporting, which compromises the accuracy, interpretation, and usability of prevalence estimates. Previous research has found that the quality of reporting in studies of HIV drug resistance prevalence is low, and thus better guidance is needed to ensure complete and uniform reporting.ObjectiveThis paper contributes to the process of developing reporting guidelines for prevalence studies of HIV drug resistance by reporting the methodology used in creating a reporting item checklist and generating key insights on items that are important to report.MethodsWe will conduct a sequential explanatory mixed methods study among authors and users of studies of HIV drug resistance. The two-phase design will include a cross-sectional electronic survey (quantitative phase) followed by a focus group discussion (qualitative phase). Survey participants will rate the essentiality of various reporting items. This data will be analyzed using content validity ratios to determine the items that will be retained for focus group discussions. Participants in these discussions will revise the items and any additionally suggested items and settle on a complete reporting item checklist. We will also conduct a thematic analysis of the group discussions to identify emergent themes regarding the agreement process.ResultsAs of November 2021, data collection for both phases of the study is complete. In July 2021, 51 participants had provided informed consent and completed the electronic survey. In October 2021, focus group discussions were held. Nine participants in total participated in two virtual focus group discussions. As of May 2022, data are being analyzed.ConclusionsThis study supports the development of a reporting checklist for studies of HIV drug resistance by achieving agreement among experts on what items should be reported in these studies. The results of this work will be refined and elaborated on by a writing committee of HIV drug resistance experts and external reviewers to develop finalized reporting guidelines.International registered report identifier (irrid)DERR1-10.2196/35969.

Project description:Increasingly, computer simulation models are used for economic and policy evaluation in cancer prevention and control. A model's predictions of key outcomes, such as screening effectiveness, depend on the values of unobservable natural history parameters. Calibration is the process of determining the values of unobservable parameters by constraining model output to replicate observed data. Because there are many approaches for model calibration and little consensus on best practices, we surveyed the literature to catalogue the use and reporting of these methods in cancer simulation models. We conducted a MEDLINE search (1980 through 2006) for articles on cancer-screening models and supplemented search results with articles from our personal reference databases. For each article, two authors independently abstracted pre-determined items using a standard form. Data items included cancer site, model type, methods used for determination of unobservable parameter values and description of any calibration protocol. All authors reached consensus on items of disagreement. Reviews and non-cancer models were excluded. Articles describing analytical models, which estimate parameters with statistical approaches (e.g. maximum likelihood) were catalogued separately. Models that included unobservable parameters were analysed and classified by whether calibration methods were reported and if so, the methods used. The review process yielded 154 articles that met our inclusion criteria and, of these, we concluded that 131 may have used calibration methods to determine model parameters. Although the term 'calibration' was not always used, descriptions of calibration or 'model fitting' were found in 50% (n = 66) of the articles, with an additional 16% (n = 21) providing a reference to methods. Calibration target data were identified in nearly all of these articles. Other methodological details, such as the goodness-of-fit metric, were discussed in 54% (n = 47 of 87) of the articles reporting calibration methods, while few details were provided on the algorithms used to search the parameter space. Our review shows that the use of cancer simulation modelling is increasing, although thorough descriptions of calibration procedures are rare in the published literature for these models. Calibration is a key component of model development and is central to the validity and credibility of subsequent analyses and inferences drawn from model predictions. To aid peer-review and facilitate discussion of modelling methods, we propose a standardized Calibration Reporting Checklist for model documentation.

Project description:ObjectiveRoutinely collected health data, collected for administrative and clinical purposes, without specific a priori research questions, are increasingly used for observational, comparative effectiveness, health services research, and clinical trials. The rapid evolution and availability of routinely collected data for research has brought to light specific issues not addressed by existing reporting guidelines. The aim of the present project was to determine the priorities of stakeholders in order to guide the development of the REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement.MethodsTwo modified electronic Delphi surveys were sent to stakeholders. The first determined themes deemed important to include in the RECORD statement, and was analyzed using qualitative methods. The second determined quantitative prioritization of the themes based on categorization of manuscript headings. The surveys were followed by a meeting of RECORD working committee, and re-engagement with stakeholders via an online commentary period.ResultsThe qualitative survey (76 responses of 123 surveys sent) generated 10 overarching themes and 13 themes derived from existing STROBE categories. Highest-rated overall items for inclusion were: Disease/exposure identification algorithms; Characteristics of the population included in databases; and Characteristics of the data. In the quantitative survey (71 responses of 135 sent), the importance assigned to each of the compiled themes varied depending on the manuscript section to which they were assigned. Following the working committee meeting, online ranking by stakeholders provided feedback and resulted in revision of the final checklist.ConclusionsThe RECORD statement incorporated the suggestions provided by a large, diverse group of stakeholders to create a reporting checklist specific to observational research using routinely collected health data. Our findings point to unique aspects of studies conducted with routinely collected health data and the perceived need for better reporting of methodological issues.

Project description:BackgroundWith the growing excitement of the potential benefits of using machine learning and artificial intelligence in medicine, the number of published clinical prediction models that use these approaches has increased. However, there is evidence (albeit limited) that suggests that the reporting of machine learning-specific aspects in these studies is poor. Further, there are no reviews assessing the reporting quality or broadly accepted reporting guidelines for these aspects.ObjectiveThis paper presents the protocol for a systematic review that will assess the reporting quality of machine learning-specific aspects in studies that use machine learning to develop clinical prediction models.MethodsWe will include studies that use a supervised machine learning algorithm to develop a prediction model for use in clinical practice (ie, for diagnosis or prognosis of a condition or identification of candidates for health care interventions). We will search MEDLINE for studies published in 2019, pseudorandomly sort the records, and screen until we obtain 100 studies that meet our inclusion criteria. We will assess reporting quality with a novel checklist developed in parallel with this review, which includes content derived from existing reporting guidelines, textbooks, and consultations with experts. The checklist will cover 4 key areas where the reporting of machine learning studies is unique: modelling steps (order and data used for each step), model performance (eg, reporting the performance of each model compared), statistical methods (eg, describing the tuning approach), and presentation of models (eg, specifying the predictors that contributed to the final model).ResultsWe completed data analysis in August 2021 and are writing the manuscript. We expect to submit the results to a peer-reviewed journal in early 2022.ConclusionsThis review will contribute to more standardized and complete reporting in the field by identifying areas where reporting is poor and can be improved.Trial registrationPROSPERO International Prospective Register of Systematic Reviews CRD42020206167; https://www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=206167.International registered report identifier (irrid)RR1-10.2196/30956.

Project description:AIMS:We conducted a systematic review assessing the reporting quality of studies validating models based on machine learning (ML) for clinical diagnosis, with a specific focus on the reporting of information concerning the participants on which the diagnostic task was evaluated on. METHOD:Medline Core Clinical Journals were searched for studies published between July 2015 and July 2018. Two reviewers independently screened the retrieved articles, a third reviewer resolved any discrepancies. An extraction list was developed from the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis guideline. Two reviewers independently extracted the data from the eligible articles. Third and fourth reviewers checked, verified the extracted data as well as resolved any discrepancies between the reviewers. RESULTS:The search results yielded 161 papers, of which 28 conformed to the eligibility criteria. Detail of data source was reported in 24 of the 28 papers. For all of the papers, the set of patients on which the ML-based diagnostic system was evaluated was partitioned from a larger dataset, and the method for deriving such set was always reported. Information on the diagnostic/non-diagnostic classification was reported well (23/28). The least reported items were the use of reporting guideline (0/28), distribution of disease severity (8/28 patient flow diagram (10/28) and distribution of alternative diagnosis (10/28). A large proportion of studies (23/28) had a delay between the conduct of the reference standard and ML tests, while one study did not and four studies were unclear. For 15 studies, it was unclear whether the evaluation group corresponded to the setting in which the ML test will be applied to. CONCLUSION:All studies in this review failed to use reporting guidelines, and a large proportion of them lacked adequate detail on participants, making it difficult to replicate, assess and interpret study findings. PROSPERO REGISTRATION NUMBER:CRD42018099167.

Project description:Globally, chronic kidney disease (CKD) is an emerging public health challenge but accurate data on its true prevalence are scarce, particularly in poorly resourced regions such as sub-Saharan Africa (SSA). Limited funding for population-based studies, poor laboratory infrastructure and the absence of a validated estimating equation for kidney function in Africans are contributing factors. Consequently, most available studies used to estimate population prevalence are hospital-based, with small samples of participants who are at high risk for kidney disease. While serum creatinine is most commonly used to estimate glomerular filtration, there is considerable potential bias in the measurement of creatinine that might lead to inaccurate estimates of kidney disease at individual and population level. To address this, the Laboratory Working Group of the National Kidney Disease Education Program published recommendations in 2006 to standardize the laboratory measurement of creatinine. The primary objective of this review was to appraise implementation of these recommendations in studies conducted in SSA after 2006. Secondary objectives were to assess bias relating to choice of estimating equations for assessing glomerular function in Africans and to evaluate use of recommended diagnostic criteria for CKD. This study was registered with Prospero (CRD42017068151), and using PubMed, African Journals Online and Web of Science, 5845 abstracts were reviewed and 252 full-text articles included for narrative analysis. Overall, two-thirds of studies did not report laboratory methods for creatinine measurement and just over 80% did not report whether their creatinine measurement was isotope dilution mass spectroscopy (IDMS) traceable. For those reporting a method, Jaffe was the most common (93%). The four-variable Modification of Diet in Renal Disease (4-v MDRD) equation was most frequently used (42%), followed by the CKD Epidemiology Collaboration (CKD-EPI) equation for creatinine (26%). For the 4-v MDRD equation and CKD-EPI equations, respectively, one-third to one half of studies clarified use of the coefficient for African-American (AA) ethnicity. When reporting CKD prevalence, <15% of studies fulfilled Kidney Disease: Improving Global Outcomes criteria and even fewer used a population-based sample. Six studies compared performance of estimating equations to measured glomerular filtration rate (GFR) demonstrating that coefficients for AA ethnicity used in the 4-v MDRD and the CKD-EPI equations overestimated GFR in Africans. To improve on reporting in future studies, we propose an 'easy to use' checklist that will standardize reporting of kidney function and improve the quality of studies in the region. This research contributes some understanding of the factors requiring attention to ensure accurate assessment of the burden of kidney disease in SSA. Many of these factors are difficult to address and extend beyond individual researchers to health systems and governmental policy, but understanding the burden of kidney disease is a critical first step to informing an integrated public health response that would provide appropriate screening, prevention and management of kidney disease in countries from SSA. This is particularly relevant as CKD is a common pathway in both infectious and non-communicable diseases, and multimorbidity is now commonplace, and even more so when those living with severe kidney disease have limited or no access to renal replacement therapy.

Dataset Information

Reporting methods in studies developing prognostic models in cancer: a review.

Background

Methods

Results

Conclusions

Publications

Reporting methods in studies developing prognostic models in cancer: a review.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets