The increasing need for systematic reviews of prognosis studies: strategies to facilitate review production and improve quality of primary research
Diagnostic and Prognostic Research volume 3, Article number: 2 (2019)
Personalized, precision, and risk-based medicine are becoming increasingly important in medicine. These involve the use of information about the prognosis of a patient, to make individualized treatment decisions. This has led to an accumulating amount of literature available on prognosis studies. To summarize and evaluate this information overload, high-quality systematic reviews are essential, additionally helping us to facilitate interpretation and usability of prognosis study findings and to identify gaps in literature. Four types of prognosis studies can be identified: overall prognosis, prognostic factors, prognostic models, and predictors of treatment effect. Methodologists have focussed on developing methods and tools for every step of a systematic review for reviews of all four types of prognosis studies, from formulating the review question and writing a protocol to searching for studies, assessing risk of bias, meta-analysing results, and interpretation of results. The growing attention for prognosis research has led to the introduction of the Cochrane Prognosis Methods Group (PMG). Since 2016, reviews of prognosis studies are formally implemented within Cochrane. With these recent methodological developments and tools, and the implementation within Cochrane, it becomes increasingly feasible to perform high-quality reviews of prognosis studies that will have an impact on clinical practice.
Clinical practice roughly consists of diagnosis, prognosis, and treatment. Treatment selection used to be driven mostly by the diagnosis that was made, but prognosis has become increasingly important to inform evidence-based decisions about healthcare. Clinical trials mainly focused on estimating a single overall effect, while decisions are being made on an individual level. Some patients benefit more (or less) than average; therefore, identifying those subgroups with different effects of an intervention has become very important to guide evidence-informed decision making. This personalized, precision, and risk-based medicine all involves the use of prognostic and predictive characteristics to make individualized treatment decisions for patients. The shift to personalized medicine has led to an accumulating amount of evidence available from prognosis studies. Reviews of prognosis studies make this information overload informative and usable. They summarize and evaluate the available evidence and guide the interpretation of results, in order to facilitate optimal use of all existing evidence. Reviews of high quality provide trusted evidence for stakeholders, like clinicians and guideline developers, to help them deciding which prognostic model or factor to use in clinical practice or implement in evidence-based guidelines. In addition, reviews identify gaps and redundant or unnecessary studies in the scientific literature, highlight flaws in conduct and reporting of primary studies, and identify and indicate which further studies are needed [1,2,3,4,5]. Therefore, reviews should serve as the essential starting point for clinical researchers of primary studies when designing a new prognosis study. The aim of this editorial is to provide an overview of the improvements in methods to perform systematic reviews of prognosis studies and freely available tools and templates. In addition, we want to raise awareness amongst clinical researchers of primary prognosis studies that those reviews and tools (e.g. reporting guidelines) are essential to use when a new study is designed, conducted, and reported. Our ultimate goal is to facilitate the production of only necessary, highly relevant, and unbiased reviews, which provide an overview of high-quality and useful primary prognosis studies.
Primary prognosis studies are presented as four types in the PROGnosis RESearch Strategy (PROGRESS) partnership series [6,7,8,9]: overall prognosis, prognostic factors, prognostic models, and predictors of treatment effect (also known as predictive factors, or treatment selection factors). Studies on (1) overall prognosis give insight in the occurrence of certain outcomes in a certain time frame, of a group of individuals with a certain health condition (not necessarily a disease). An example can be to study the overall survival in women with ovarian cancer. Studies on (2) prognostic factors identify variables that are prognostic for a certain outcome in a certain individual within a given timeframe, e.g. the prognostic value of c-reactive protein in predicting the 10-year risk of cardiovascular disease. Prognostic model studies (3) combine prognostic factors in a single model to make personalized predictions for individuals with a certain health condition and study the development and transportability or generalizability of a model to other populations. For example, the Pneumonia Severity Index (PSI) combines predictors like age, comorbidities, physical findings, and laboratory findings to estimate the 30-day mortality rate in patients with community-acquired pneumonia . A validation study of the PSI showed that this model is not suitable to use in people with bacteraemic pneumococcal pneumonia . Studies on predictors of treatment effect (4) aim to identify individuals’ factors that are associated with the effectiveness of a certain treatment, e.g. the presence of the oncogene HER2/neu is predictive of the effectivity of the monoclonal antibody trastuzumab for treating breast cancer . An additional primary study type is where several predictors of treatment effect are combined, to form a predictive model that predicts treatment effect. A model like this can be used to select individuals that benefit most from a certain treatment.
All types of primary prognosis studies can be summarized, evaluated, and interpreted in different types of systematic reviews, following the broad range of aims and objectives of the included prognosis studies. Reviews are, for example, helpful to give an overview of all available prognostic factors or models (e.g. to identify all factors or models for the prediction of heart failure in patients with type 2 diabetes mellitus), to study the prognostic value of a certain externally validated prognostic model (e.g. the predictive (prognostic) performance of the Revised Cardiac Risk Score for cardiac outcomes after noncardiac surgery ), or the added value of one or more predictors on top of an existing model (e.g. the added value of coronary artery calcification to a model for cardiovascular disease prediction ). As usually prognosis studies suffer from extensive heterogeneity in selected populations and the measurement and definition of predictors and outcomes, most systematic reviews also aim to identify sources of this heterogeneity.
Methodological guidance for most steps of conducting a systematic review is currently developed. Amongst others, to facilitate searching and reduce the number of references to be screened, methodological search filters are available [15,16,17,18], and increasingly, data mining tools (like ) are being developed that can identify discriminative words to narrow down search results. To facilitate framing the review question, and data extraction and critical appraisal of prognostic model studies, the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist has been developed (though it can also be adopted to other types of prognosis studies) . Risk of bias assessment of included studies can be challenging, as evidence on the influence of design choices on the performance of a model is limited. However, tools for risk of bias assessment for both prognostic factor studies (QUIPS) and prediction models (PROBAST) are available [21, 22]. It can also be challenging to perform a quantitative synthesis (meta-analysis) of the results of prognosis studies due to heterogeneity in selected populations, measurement and definitions of predictors and outcomes, and reporting of performance measures. Methods to deal with these issues in reviews of prognostic model and prognostic factor studies have been described [23,24,25]. Further, we are working on guidance for presenting and interpreting the results of systematic reviews and guidance for reporting systematic reviews. For primary prognosis studies, the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement has been published in 2015, to guide completeness of reporting of essential elements of primary prediction studies [26, 27]. Hopefully, this will improve the reporting and therefore lead to more informative systematic reviews.
The growing attention for prognosis research and the increasing emphasis on the importance of prognostic information in clinical practice have led to the introduction of the Cochrane Prognosis Methods Group (PMG) in 2007 . Over the years, a growing group of experts in the field of primary prognosis studies and evidence synthesis have joined this group to work together and develop tools and guidance necessary for facilitating reviews of prognosis studies. Since 2016, reviews of prognosis studies are formally adopted and implemented within Cochrane (via the Cochrane PMG ). The first two Cochrane reviews were published in 2018 [30, 31], ten protocols are published in the Cochrane Library, and five titles have been registered. The implementation within Cochrane comes together with the development of tools and templates for conducting a review of prognosis studies. Trainings and webinars are organized by the Cochrane PMG, aiming to give researchers sufficient skills on how to use the tools and templates and up-to-date knowledge on performing a systematic review of prognosis studies (see  for available tools and templates). All tools, templates, and methods developed by researchers involved with Cochrane are also available for authors writing a non-Cochrane review.
In summary, systematic reviews are urgently needed to summarize the growing amount of prognostic evidence, to evaluate the available evidence and guide the interpretation of results, in order to facilitate optimal use of existing evidence for medical practice and policy making. With the recent methodological developments and tools for systematic reviews of prognosis studies, it becomes increasingly feasible to perform these reviews. With the implementation within Cochrane, it is ensured that reviews of high quality will be produced that will have an impact on clinical practice.
Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies
Prognosis Methods Group
Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis
Bouwmeester W, Zuithoff NP, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KG. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):1–12.
Collins GS, Omar O, Shanyinde M, Yu LM. A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods. J Clin Epidemiol. 2013;66(3):268–77.
Damen JA, Hooft L, Schuit E, Debray TP, Collins GS, Tzoulaki I, Lassale CM, Siontis GC, Chiocchia V, Roberts C, et al. Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ. 2016;353:i2416.
Siontis GC, Tzoulaki I, Castaldi PJ, Ioannidis JP. External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol. 2015;68(1):25–34.
Mallett S, Royston P, Dutton S, Waters R, Altman DG. Reporting methods in studies developing prognostic models in cancer: a review. BMC Med. 2010;8:20.
Hemingway H, Croft P, Perel P, Hayden JA, Abrams K, Timmis A, Briggs A, Udumyan R, Moons KG, Steyerberg EW, et al. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ. 2013;346:e5595.
Riley RD, Hayden JA, Steyerberg EW, Moons KG, Abrams K, Kyzas PA, Malats N, Briggs A, Schroter S, Altman DG, et al. Prognosis research strategy (PROGRESS) 2: prognostic factor research. PLoS Med. 2013;10(2):e1001380.
Steyerberg EW, Moons KG, van der Windt DA, Hayden JA, Perel P, Schroter S, Riley RD, Hemingway H, Altman DG. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381.
Hingorani AD, Windt DA, Riley RD, Abrams K, Moons KG, Steyerberg EW, Schroter S, Sauerbrei W, Altman DG, Hemingway H. Prognosis research strategy (PROGRESS) 4: stratified medicine research. BMJ. 2013;346:e5793.
Fine MJ, Auble TE, Yealy DM, Hanusa BH, Weissfeld LA, Singer DE, Coley CM, Marrie TJ, Kapoor WN. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med. 1997;336(4):243–50.
Feldman C, Alanee S, Yu VL, Richards GA, Ortqvist A, Rello J, Chiou CC, Chedid MB, Wagener MM, Klugman KP. Severity of illness scoring systems in patients with bacteraemic pneumococcal pneumonia: implications for the intensive care unit care. Clin Microbiol Infect. 2009;15(9):850–7.
Ahmed S, Sami A, Xiang J. HER2-directed therapy: current treatment options for HER2-positive breast cancer. Breast cancer (Tokyo, Japan). 2015;22(2):101–16.
Ford MK, Beattie WS, Wijeysundera DN. Systematic review: prediction of perioperative cardiac complications and mortality by the revised cardiac risk index. Ann Intern Med. 2010;152(1):26–35.
Peters SA, Bakker M, den Ruijter HM, Bots ML. Added value of CAC in risk stratification for cardiovascular events: a systematic review. Eur J Clin Investig. 2012;42(1):110–6.
Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically sound prognostic studies in MEDLINE: an analytic survey. BMC Med. 2004;2:23.
Wong SS, Wilczynski NL, Haynes RB, Ramkissoonsingh R. Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE. AMIA Annu Symp Proc. 2003;2003:728–32.
Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc. 2001;8(4):391–7.
Geersing GJ, Bouwmeester W, Zuithoff P, Spijker R, Leeflang M, Moons KG. Search filters for finding prognostic and diagnostic prediction studies in Medline to enhance systematic reviews. PLoS One. 2012;7(2):e32844.
Data mining tools. http://cbdm-01.zdv.uni-mainz.de/~jfontain/cms/. Accessed 14 Aug 2018.
Moons KG, de Groot JA, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, Reitsma JB, Collins GS. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. 2014;11(10):e1001744.
Hayden JA, van der Windt DA, Cartwright JL, Cote P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med. 2013;158(4):280–6.
Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S: PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med 2019;170(1):51–58.
Debray TP, Damen JA, Snell KI, Ensor J, Hooft L, Reitsma JB, Riley RD, Moons KG. A guide to systematic review and meta-analysis of prediction model performance. BMJ. 2017;356:i6460.
Debray TP, Damen JA, Riley RD, Snell K, Reitsma JB, Hooft L, Collins GS, Moons KG. A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes. Stat Methods Med Res. 2018:962280218785504. https://journals.sagepub.com/doi/abs/10.1177/0962280218785504#articleCitationDownloadContainer.
Riley RD, Moons KGM, Snell KI, Ensor J, Hooft L, Altman DG, Hayden J, Collins GS, Debray TPA. A guide to systematic review and meta-analysis of prognostic factor studies. BMJ. 2019;364:k4597.
Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015;162(1):55–63.
Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, Vickers AJ, Ransohoff DF, Collins GS. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1–73.
Riley RD, Ridley G, Williams K, Altman DG, Hayden J, de Vet HC. Prognosis research: toward evidence-based results and a Cochrane methods group. J Clin Epidemiol. 2007;60(8):863–5 author reply 865-866.
Cochrane Prognosis Methods Group. www.methods.cochrane.org/prognosis/. Accessed 14 Aug 2018.
Westby MJ, Dumville JC, Stubbs N, Norman G, Wong JK, Cullum N, Riley RD. Protease activity as a prognostic factor for wound healing in venous leg ulcers. Cochrane Database Syst Rev. 2018;9:Cd012841.
Richter B, Hemmingsen B, Metzendorf MI, Takwoingi Y. Development of type 2 diabetes mellitus in people with intermediate hyperglycaemia. Cochrane Database Syst Rev. 2018;10:Cd012661.
JAAGD and LH received funding from the Cochrane Strategic Methods Fund “Trusted methods and Support for Cochrane Reviews of Prognostic Studies”.
Availability of data and materials
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Damen, J.A.A.G., Hooft, L. The increasing need for systematic reviews of prognosis studies: strategies to facilitate review production and improve quality of primary research. Diagn Progn Res 3, 2 (2019). https://doi.org/10.1186/s41512-019-0049-6
- Systematic review