Skip to main content

Table 3 Performance of the CAD models based on the determined sample size for various fixed and adaptive sample size methods. Results are shown as medians (with interquartile ranges) across 500 repetitions

From: Adaptive sample size determination for the development of clinical prediction models

  Sample size Bootstrap-corrected performance
Sample size method N EPP AUC Slope
Basic strategy
 Fixeda: 10 EPP 300 (250–300) 11 (10–11) 0.706 (0.684–0.726) 0.774 (0.751–0.795)
 Fixeda: Riley’s method 700 (700–700) 28 (27–28) 0.713 (0.701–0.726) 0.894 (0.886–0.901)
 Adaptive: stopping rule 1b 850 (750–900) 33 (30–35) 0.717 (0.705–0.727) 0.909 (0.905–0.914)
 Adaptive: stopping rule 2b 1500 (1400–1550) 59 (56–62) 0.719 (0.712–0.725) 0.949 (0.946–0.952)
Restricted cubic splines
 Fixeda: 10 EPP 350 (350–400) 11 (10–11) 0.703 (0.684–0.721) 0.766 (0.745–0.783)
 Fixeda: Riley’s method 900 (900–900) 26 (26–27) 0.712 (0701–0.724) 0.887 (0.877–0.894)
 Adaptive: stopping rule 1b 1100 (1050–1200) 31 (28–33) 0.715 (0.705–0.724) 0.907 (0.904–0.911)
 Adaptive: stopping rule 2b 1850 (1800–1900) 58 (55–60) 0.718 (0.712–0.724) 0.947 (0.945–0.949)
Firth’s correction
 Fixeda: 10 EPP 300 (250–300) 11 (10–11) 0.706 (0.682–0.726) 0.822 (0.797–0.846)
 Fixeda: Riley’s method 700 (700–700) 28 (27–28) 0.712 (0.700–0.726) 0.913 (0.904–0.923)
 Adaptive: stopping rule 1b 750 (700–800) 31 (28–32) 0.713 (0.703–0.727) 0.922 (0.917–0.928)
 Adaptive: stopping rule 2b 1500 (1450–1600) 59 (56–63) 0.718 (0.710–0.726) 0.958 (0.956–0.961)
Including backward selection
 Fixeda: 10 EPP 300 (250–300) 11 (10–11) 0.690 (0.666–0.715) 0.798 (0.772–0.819)
 Fixeda: Riley’s method 700 (700–700) 28 (27–28) 0.708 (0.694–0.722) 0.894 (0.883–0.905)
 Adaptive: stopping rule 1b 800 (750–900) 33 (29–36) 0.712 (0.701–0.723) 0.909 (0.905–0.915)
 Adaptive: stopping rule 2b 1550 (1400–1650) 61 (56–65) 0.716 (0.709–0.724) 0.948 (0.946–0.951)
  1. AUC area under the receiver operating characteristic curve (or c-statistic), slope calibration slope, EPP events per parameter
  2. aThe analysis went in batches of 50 patients, therefore fixed sample sizes were rounded upwards to the next multiple of 50
  3. bStopping rule 1: calibration slope ≥ 0.9 and AUC optimism < = 0.02 at two consecutive assessments. Stopping rule 2: calibration slope ≥ 0.9 and AUC optimism < = 0.01 at two consecutive assessments