From: Risk of bias of prognostic models developed using machine learning: a systematic review in oncology
PROBAST domain and signalling questions | Development analysis (152 models) | Validation analysis (37 models) | ||||
---|---|---|---|---|---|---|
Yes/probably yes | No/probably no | No information | Yes/probably yes | No/probably no | No information | |
n (%; 95% CI) | n (%; 95% CI) | n (%; 95% CI) | n (%; 95% CI) | n (%; 95% CI) | n (%; 95% CI) | |
1. PARTICIPANTS | ||||||
  1.1. Were appropriate data sources used, e.g., cohort, randomized controlled trial, or nested case–control study data? | 115 (75.7; 68.1,81.9) | 19 (12.5; 8.1,18.8) | 18 (11.8; 7.6,18.1) | 30 (81.1; 64.7,90.9) | 2 (5.4; 1.3,20) | 5 (13.5; 5.6.29.3) |
  1.2. Were all inclusions and exclusions of participants appropriate? | 100 (65.8; 57.8,72.9) | 13 (8.6; 5,14.2) | 39 (25.7; 19.3,33.3) | 24 (64.9; 47.9,78.8) | - | 13 (35.1; 21.2.52.1) |
2. PREDICTORS | ||||||
  2.1. Were predictors defined and assessed in a similar way for all participants? | 117 (77; 69.6,83) | 14 (9.2; 5.5,15) | 21 (13.8; 9.2,20.3) | 26 (70.3; 53.3,83.1) | - | 11 (29.7; 16.9.46.7) |
  2.2. Were predictor assessments made without knowledge of outcome data? | 73 (48; 40.1,56) | 1 (0.7; 0.1,4.6) | 78 (51.3; 43.3,59.2) | 20 (54.1; 37.6,69.7) | - | 17 (46; 30.3.62.4) |
  2.3. Are all predictors available at the time the model is intended to be used? | 91 (59.9; 51.8,67.4) | - | 61 (40.1; 32.6,48.2) | 22 (59.5; 42.7,74.3) | - | 15 (40.5; 25.7.57.3) |
3. OUTCOMES | ||||||
  3.1. Was the outcome determined appropriately? | 130 (85.5; 78.9,90.3) | 4 (2.6; 1,6.9) | 18 (11.8; 7.6,18.1) | 30 (81.1; 64.7,90.9) | - | 7 (18.9; 9.1.35.3) |
 3.2. Was a prespecified or standard outcome definition used? | 122 (80.3; 73.1,85.9) | 13 (8.6; 5,14.2) | 17 (11.2; 7,17.3) | 23 (62.2; 45.2,76.6) | 7 (18.9; 9.1,35.3) | 7 (18.9; 9.1.35.3) |
  3.3. Were predictors excluded from the outcome definition? | 117 (77; 69.6,83) | 6 (4; 1.8,8.6) | 29 (19.1; 13.6,26.2) | 28 (75.7; 58.9,87.1) | - | 9 (24.3; 12.9.41.1) |
  3.4. Was the outcome defined and determined in a similar way for all participants? | 115 (75.7; 68.1,81.9) | 11 (7.2; 4,12.6) | 26 (17.1; 11.9,24) | 35 (94.6; 80,98.7) | - | 2 (5.4; 1.3.20) |
  3.5. Was the outcome determined without knowledge of predictor information? | 106 (69.7; 61.9,76.6) | 6 (4; 1.8,8.6) | 40 (26.3; 19.9,33.9) | 28 (75.7; 58.9,87.1) | - | 9 (24.3; 12.9.41.1) |
  3.6. Was the time interval between predictor assessment and outcome determination appropriate? | 100 (65.8; 57.8,72.9) | 5 (3.3; 1.4,7.7) | 47 (30.9; 24,38.8) | 21 (56.8; 40.1,72) | 5 (13.5; 5.6,29.3) | 11 (29.7; 16.9.46.7) |
  4. ANALYSIS |  |  |  |  |  |  |
  4.1. Were there a reasonable number of participants with the outcome? | 44 (29; 22.2,36.7) | 77 (50.7; 42.7,58.6) | 31 (20.4; 14.7,27.6) | 10 (27; 14.9,44) | 16 (43.2; 28,59.9) | 11 (29.7; 16.9,46.7) |
  4.2. Were continuous and categorical predictors handled appropriately? | 30 (19.7; 14.1,26.9) | 57 (37.5; 30.1,45.5) | 65 (42.8; 35.1,50.8) | 19 (51.4; 35.1,67.3) | 1 (2.7; 0.4,17.8) | 17 (46; 30.3,62.4) |
  4.3. Were all enrolled participants included in the analysis? | 43 (28.3; 21.7,36) | 49 (32.2; 25.2,40.1) | 60 (39.5; 32,47.5) | 17 (46; 30.3,62.4) | 9 (24.3; 12.9,41.1) | 11 (29.7; 16.9,46.7) |
  4.4. Were participants with missing data handled appropriately? | 24 (15.8; 10.8,22.5) | 70 (46.1; 38.2,54.1) | 58 (38.2; 30.7,46.2) | 6 (16.2; 7.3,32.4) | 15 (40.5; 25.7,57.3) | 16 (43.2; 28,59.9) |
  4.5. Was selection of predictors based on univariable analysis avoided? | 68 (44.7; 37,52.8) | 49 (32.2; 25.2,40.1) | 35 (23; 17,30.4) | NA | ||
  4.6. Were complexities in the data (e.g., censoring, competing risks, sampling of control participants) accounted for appropriately? | 10 (6.6; 3.6,11.8) | 28 (18.4; 13,25.5) | 114 (75; 67.4,81.3) | 2 (5.4; 1.3,20) | - | 35 (94.6; 80,98.7) |
  4.7. Were relevant model performance measures evaluated appropriately? | 28 (18.4; 13,25.5) | 87 (57.2; 49.2,64.9) | 37 (24.3; 18.1,31.9) | 10 (27; 14.9,44) | 13 (35.1; 21.2,52.1) | 14 (37.8; 23.4,54.8) |
  4.8. Were model overfitting and optimism in model performance accounted for? | 52 (34.2; 27.1,42.2) | 84 (55.3; 47.2,63) | 16 (10.5; 6.5,16.5) | NA | ||
  4.9. Do predictors and their assigned weights in the final model correspond to the results from the reported multivariable analysis? | 24 (15.8; 10.8,22.5) | 8 (5.3; 2.6,10.2) | 120 (79; 71.7,84.7) | NA |