From: Comparison of dynamic updating strategies for clinical prediction models
Update interval | Metric | 1 quarter | 2 quarters | 4 quarters | 8 quarters |
---|---|---|---|---|---|
Never update | BS | 0.118 | 0.118 | 0.118 | 0.118 |
AUC | 0.603 | 0.603 | 0.603 | 0.603 | |
H-L | 290.5 | 290.5 | 290.5 | 290.5 | |
Calibration intercept | −0.754 | −0.754 | −0.754 | −0.754 | |
Calibration slope | 0.791 | 0.791 | 0.791 | 0.791 | |
# Updates | 0 | 0 | 0 | 0 | |
Closed testing procedure | BS | 0.116 | 0.114 | 0.114 | 0.114 |
AUC | 0.575 | 0.602 | 0.598 | 0.602 | |
H-L | 118.4 | 54.64 | 37.80 | 82.15 | |
Calibration intercept | −1.658 | −0.561 | −0.570 | −0.887 | |
Calibration slope | 0.118 | 0.754 | 0.724 | 0.577 | |
# Updates | 12 | 5 | 4 | 2 | |
Always recalibrate intercept | BS | 0.113 | 0.113 | 0.114 | 0.114 |
AUC | 0.605 | 0.605 | 0.603 | 0.605 | |
H-L | 29.33 | 28.92 | 39.21 | 69.76 | |
Calibration intercept | −0.472 | −0.453 | −0.525 | −0.595 | |
Calibration slope | 0.758 | 0.781 | 0.760 | 0.755 | |
# Updates | 24 | 12 | 6 | 3 | |
Always recalibrate intercept and slope | BS | 0.113 | 0.113 | 0.114 | 0.114 |
AUC | 0.601 | 0.602 | 0.602 | 0.605 | |
H-L | 18.53 | 27.91 | 35.41 | 59.22 | |
Calibration intercept | −0.451 | −0.353 | −0.443 | −0.517 | |
Calibration slope | 0.771 | 0.839 | 0.809 | 0.802 | |
# Updates | 24 | 12 | 6 | 3 | |
Always refit (revision) | BS | 0.118 | 0.114 | 0.114 | 0.114 |
AUC | 0.570 | 0.582 | 0.592 | 0.601 | |
H-L | 357.8 | 106.6 | 56.25 | 77.01 | |
Calibration intercept | −1.835 | −1.534 | −0.972 | −0.897 | |
Calibration slope | 0.021 | 0.180 | 0.499 | 0.570 | |
# Updates | 24 | 12 | 6 | 3 |