Skip to main content

Table 4 Design of the impact study

From: Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

1. Consider a decision analytic study

 If not previously performed, a decision analytic study may link the available evidence to estimate the theoretical impact on decision making and/or patient outcome.

2. Consider studying the effects on both physician behavior and patient outcome

 Changes in process or behavior may not be sufficient to improve patient outcome. Studying the effects on patient outcome typically requires more time and money.

3. Consider additional data collection to improve the understanding of the impact study results

 The impact does not only depend on the prediction model, but also on physician decision making and the effectiveness of subsequent treatment. Without additional data that is collected during the impact study, the effects of the individual components may difficult to disentangle.

4. Compare the use of a prediction model to care-as-usual

 Physicians are not naive in patient selection and making interventional decisions. The impact of a prediction model (assistive or directive) is its value over and above current clinical decision making.

5. Cluster-randomized trial as the optimal design

 Randomization of practices or practitioners aims to prevent learning effects and contamination between study groups. Nonetheless, time and costs to perform a cluster-randomized study should be weighed against its expected informational value.

6. Consider using each study group as its own control

 The balance between study groups may be improved when using a stepped wedge design and including pre-trial observations.

7. The impact of the prediction model will depend on the predicted probabilitya

 Predicted probability should be considered an effect modifier in the statistical analysis, which requires, e.g. stratification or use of its interaction term with ‘study group’ in regression analyses.

8. All predictors should be available for care-as-usual patientsa

 A probability-dependent analysis of the results requires that the predicted probabilities can afterwards also be estimated for the care-as-usual patients (control group). Accordingly, all predictors must be available for care-as-usual patients, even the costly or invasive predictor variables.

  1. aAdditional item, not further explained in the manuscript