Table 1 Stages in the development and evaluation of clinical prediction rules

Stage 1. Identifying the need for a CPR

• Consider conducting qualitative research with clinicians to determine clinical relevance and credibility of CPR

• Conduct a systematic review of the literature to identify and evaluate existing CPRs developed for the same purpose

• Consider updating, validating or testing the impact of existing CPRs

Stage 2. Derivation of a CPR according to methodological standards

Study design for the derivation of a CPR

• Consider registering the study and publishing a protocol

• Ensure the dataset is representative of the population for whom the CPR is intended

• Conduct a prospective multicentre cohort study

Statistical analysis

• Conduct multivariable regression analysis (logistic for binary outcomes, Cox for long-term prognostic outcomes)

• Identify the model to be used, plus rationale if other methods used

Missing data

• Use multiple imputation

Selection of candidate predictors for inclusion in a multivariable model

• Only include relevant predictors based on evidence in the literature/clinical experience

• Aim for a sample size with a minimum of ten events per predictor, preferably more

• Avoid selection based on univariable significance testing

• Avoid categorising continuous predictors

Selection of predictors during multivariable modelling

• Backward elimination of predictors is preferred

• Avoid data-driven selection and incorporate subject-matter knowledge into the selection process

Definition and assessment of predictor and outcome variables

• Define predictor and outcome variables clearly

• Consider inter-rater reliability of predictor measurement and potential measurement error

• Aim for blind assessment of predictor and outcome variables

Internal validation

• Use cross-validation or bootstrapping and adjust for optimism

• Ensure to repeat each step of model development if using bootstrapping

CPR performance measures

• Assess and report both calibration and discrimination

• Consider decision curve analysis to estimate the clinical utility of the CPR

Presentation of a CPR

• Report the regression coefficients of the final model, including the intercept or baseline hazard

• Consider a clinical calculator if the CPR is complex

Reporting the derivation of a CPR

• Adhere to the TRIPOD guidelines [36]

Stage 3. External validation and refinement of a CPR

Study design for the external validation of a CPR

• Conduct a prospective multicentre cohort study

• Aim for a sample size with a minimum of 100 outcome events, preferably 200

• Consider using a framework of generalisability to enhance the interpretation of the findings [34]

Types of external validation

• Conduct temporal, geographical and domain validation studies to ensure maximum generalisability

• If multiple validations have been performed, conduct a meta-analysis to summarise the overall performance of the CPR, using a published framework [35]

Refinement of a CPR: model updating or adjustment

• Consider updating, adjusting or recalibrating the CPR if poor performance is found in an external validation study

• Consider further external validation of updated CPRs

Comparing the performance of CPRs

• Compare the CPR with other existing CPRs for the same condition

• Ensure the statistical procedures used for comparison are appropriate; consider a decision-analytic approach

Reporting the external validation of a CPR

• Adhere to the TRIPOD guidelines [36]

Stage 4. Impact of a CPR on clinical practice

Study design for an impact analysis

• Consider whether the CPR is ready for implementation

• Conduct a cluster randomised trial with centres as clusters, or a before–after study

• Perform appropriate sample size calculations

• Consider decision-analytic modelling as an intermediate step prior to a formal impact study

Measures of impact of a CPR

• Report the safety and efficacy of the CPR

• Report the impact of the CPR on clinician behaviour if assessed

Acceptability of a CPR

• Evaluate the acceptability of the CPR using the validated OADRI [48], or using qualitative or vignette methods

Comparison of a CPR with unstructured clinical judgement

• Compare the sensitivity and specificity of the CPR with clinicians own predictions/decisions

The four phases of impact analysis for CPRs

• Follow the framework for the impact analysis of CPRs [33]

• Ensure extensive preparatory and feasibility work is conducted prior to a formal impact study

Reporting the impact analysis of a CPR

• There are currently no published reporting guidelines for impact studies of CPRs; this is an area for future research

Stage 5. Cost-effectiveness

• Conduct a formal economic evaluation, with sensitivity analyses to examine the uncertainty of the model projections

Stage 6. Long-term implementation and dissemination

• Devise and evaluate targeted implementation strategies to ensure maximum uptake

Barriers and facilitators to the use of CPRs

• Assess barriers to the use of the CPR and devise strategies to overcome these