Skip to main content

Table 3 PROBAST signalling questions for model development and validation analyses in all 62 studies

From: Risk of bias of prognostic models developed using machine learning: a systematic review in oncology

PROBAST domain and signalling questions Development analysis (152 models) Validation analysis (37 models)
Yes/probably yes No/probably no No information Yes/probably yes No/probably no No information
n (%; 95% CI) n (%; 95% CI) n (%; 95% CI) n (%; 95% CI) n (%; 95% CI) n (%; 95% CI)
1. PARTICIPANTS
  1.1. Were appropriate data sources used, e.g., cohort, randomized controlled trial, or nested case–control study data? 115 (75.7; 68.1,81.9) 19 (12.5; 8.1,18.8) 18 (11.8; 7.6,18.1) 30 (81.1; 64.7,90.9) 2 (5.4; 1.3,20) 5 (13.5; 5.6.29.3)
  1.2. Were all inclusions and exclusions of participants appropriate? 100 (65.8; 57.8,72.9) 13 (8.6; 5,14.2) 39 (25.7; 19.3,33.3) 24 (64.9; 47.9,78.8) - 13 (35.1; 21.2.52.1)
2. PREDICTORS
  2.1. Were predictors defined and assessed in a similar way for all participants? 117 (77; 69.6,83) 14 (9.2; 5.5,15) 21 (13.8; 9.2,20.3) 26 (70.3; 53.3,83.1) - 11 (29.7; 16.9.46.7)
  2.2. Were predictor assessments made without knowledge of outcome data? 73 (48; 40.1,56) 1 (0.7; 0.1,4.6) 78 (51.3; 43.3,59.2) 20 (54.1; 37.6,69.7) - 17 (46; 30.3.62.4)
  2.3. Are all predictors available at the time the model is intended to be used? 91 (59.9; 51.8,67.4) - 61 (40.1; 32.6,48.2) 22 (59.5; 42.7,74.3) - 15 (40.5; 25.7.57.3)
3. OUTCOMES
  3.1. Was the outcome determined appropriately? 130 (85.5; 78.9,90.3) 4 (2.6; 1,6.9) 18 (11.8; 7.6,18.1) 30 (81.1; 64.7,90.9) - 7 (18.9; 9.1.35.3)
 3.2. Was a prespecified or standard outcome definition used? 122 (80.3; 73.1,85.9) 13 (8.6; 5,14.2) 17 (11.2; 7,17.3) 23 (62.2; 45.2,76.6) 7 (18.9; 9.1,35.3) 7 (18.9; 9.1.35.3)
  3.3. Were predictors excluded from the outcome definition? 117 (77; 69.6,83) 6 (4; 1.8,8.6) 29 (19.1; 13.6,26.2) 28 (75.7; 58.9,87.1) - 9 (24.3; 12.9.41.1)
  3.4. Was the outcome defined and determined in a similar way for all participants? 115 (75.7; 68.1,81.9) 11 (7.2; 4,12.6) 26 (17.1; 11.9,24) 35 (94.6; 80,98.7) - 2 (5.4; 1.3.20)
  3.5. Was the outcome determined without knowledge of predictor information? 106 (69.7; 61.9,76.6) 6 (4; 1.8,8.6) 40 (26.3; 19.9,33.9) 28 (75.7; 58.9,87.1) - 9 (24.3; 12.9.41.1)
  3.6. Was the time interval between predictor assessment and outcome determination appropriate? 100 (65.8; 57.8,72.9) 5 (3.3; 1.4,7.7) 47 (30.9; 24,38.8) 21 (56.8; 40.1,72) 5 (13.5; 5.6,29.3) 11 (29.7; 16.9.46.7)
  4. ANALYSIS       
  4.1. Were there a reasonable number of participants with the outcome? 44 (29; 22.2,36.7) 77 (50.7; 42.7,58.6) 31 (20.4; 14.7,27.6) 10 (27; 14.9,44) 16 (43.2; 28,59.9) 11 (29.7; 16.9,46.7)
  4.2. Were continuous and categorical predictors handled appropriately? 30 (19.7; 14.1,26.9) 57 (37.5; 30.1,45.5) 65 (42.8; 35.1,50.8) 19 (51.4; 35.1,67.3) 1 (2.7; 0.4,17.8) 17 (46; 30.3,62.4)
  4.3. Were all enrolled participants included in the analysis? 43 (28.3; 21.7,36) 49 (32.2; 25.2,40.1) 60 (39.5; 32,47.5) 17 (46; 30.3,62.4) 9 (24.3; 12.9,41.1) 11 (29.7; 16.9,46.7)
  4.4. Were participants with missing data handled appropriately? 24 (15.8; 10.8,22.5) 70 (46.1; 38.2,54.1) 58 (38.2; 30.7,46.2) 6 (16.2; 7.3,32.4) 15 (40.5; 25.7,57.3) 16 (43.2; 28,59.9)
  4.5. Was selection of predictors based on univariable analysis avoided? 68 (44.7; 37,52.8) 49 (32.2; 25.2,40.1) 35 (23; 17,30.4) NA
  4.6. Were complexities in the data (e.g., censoring, competing risks, sampling of control participants) accounted for appropriately? 10 (6.6; 3.6,11.8) 28 (18.4; 13,25.5) 114 (75; 67.4,81.3) 2 (5.4; 1.3,20) - 35 (94.6; 80,98.7)
  4.7. Were relevant model performance measures evaluated appropriately? 28 (18.4; 13,25.5) 87 (57.2; 49.2,64.9) 37 (24.3; 18.1,31.9) 10 (27; 14.9,44) 13 (35.1; 21.2,52.1) 14 (37.8; 23.4,54.8)
  4.8. Were model overfitting and optimism in model performance accounted for? 52 (34.2; 27.1,42.2) 84 (55.3; 47.2,63) 16 (10.5; 6.5,16.5) NA
  4.9. Do predictors and their assigned weights in the final model correspond to the results from the reported multivariable analysis? 24 (15.8; 10.8,22.5) 8 (5.3; 2.6,10.2) 120 (79; 71.7,84.7) NA
  1. Y  Yes, PY  Probably yes, N  No, PN  Probably no, NI  No information