Skip to main content

Table 3 PROBAST signalling questions for model development and validation analyses in all 62 studies

From: Risk of bias of prognostic models developed using machine learning: a systematic review in oncology

PROBAST domain and signalling questions

Development analysis (152 models)

Validation analysis (37 models)

Yes/probably yes

No/probably no

No information

Yes/probably yes

No/probably no

No information

n (%; 95% CI)

n (%; 95% CI)

n (%; 95% CI)

n (%; 95% CI)

n (%; 95% CI)

n (%; 95% CI)

1. PARTICIPANTS

  1.1. Were appropriate data sources used, e.g., cohort, randomized controlled trial, or nested case–control study data?

115 (75.7; 68.1,81.9)

19 (12.5; 8.1,18.8)

18 (11.8; 7.6,18.1)

30 (81.1; 64.7,90.9)

2 (5.4; 1.3,20)

5 (13.5; 5.6.29.3)

  1.2. Were all inclusions and exclusions of participants appropriate?

100 (65.8; 57.8,72.9)

13 (8.6; 5,14.2)

39 (25.7; 19.3,33.3)

24 (64.9; 47.9,78.8)

-

13 (35.1; 21.2.52.1)

2. PREDICTORS

  2.1. Were predictors defined and assessed in a similar way for all participants?

117 (77; 69.6,83)

14 (9.2; 5.5,15)

21 (13.8; 9.2,20.3)

26 (70.3; 53.3,83.1)

-

11 (29.7; 16.9.46.7)

  2.2. Were predictor assessments made without knowledge of outcome data?

73 (48; 40.1,56)

1 (0.7; 0.1,4.6)

78 (51.3; 43.3,59.2)

20 (54.1; 37.6,69.7)

-

17 (46; 30.3.62.4)

  2.3. Are all predictors available at the time the model is intended to be used?

91 (59.9; 51.8,67.4)

-

61 (40.1; 32.6,48.2)

22 (59.5; 42.7,74.3)

-

15 (40.5; 25.7.57.3)

3. OUTCOMES

  3.1. Was the outcome determined appropriately?

130 (85.5; 78.9,90.3)

4 (2.6; 1,6.9)

18 (11.8; 7.6,18.1)

30 (81.1; 64.7,90.9)

-

7 (18.9; 9.1.35.3)

 3.2. Was a prespecified or standard outcome definition used?

122 (80.3; 73.1,85.9)

13 (8.6; 5,14.2)

17 (11.2; 7,17.3)

23 (62.2; 45.2,76.6)

7 (18.9; 9.1,35.3)

7 (18.9; 9.1.35.3)

  3.3. Were predictors excluded from the outcome definition?

117 (77; 69.6,83)

6 (4; 1.8,8.6)

29 (19.1; 13.6,26.2)

28 (75.7; 58.9,87.1)

-

9 (24.3; 12.9.41.1)

  3.4. Was the outcome defined and determined in a similar way for all participants?

115 (75.7; 68.1,81.9)

11 (7.2; 4,12.6)

26 (17.1; 11.9,24)

35 (94.6; 80,98.7)

-

2 (5.4; 1.3.20)

  3.5. Was the outcome determined without knowledge of predictor information?

106 (69.7; 61.9,76.6)

6 (4; 1.8,8.6)

40 (26.3; 19.9,33.9)

28 (75.7; 58.9,87.1)

-

9 (24.3; 12.9.41.1)

  3.6. Was the time interval between predictor assessment and outcome determination appropriate?

100 (65.8; 57.8,72.9)

5 (3.3; 1.4,7.7)

47 (30.9; 24,38.8)

21 (56.8; 40.1,72)

5 (13.5; 5.6,29.3)

11 (29.7; 16.9.46.7)

  4. ANALYSIS

      

  4.1. Were there a reasonable number of participants with the outcome?

44 (29; 22.2,36.7)

77 (50.7; 42.7,58.6)

31 (20.4; 14.7,27.6)

10 (27; 14.9,44)

16 (43.2; 28,59.9)

11 (29.7; 16.9,46.7)

  4.2. Were continuous and categorical predictors handled appropriately?

30 (19.7; 14.1,26.9)

57 (37.5; 30.1,45.5)

65 (42.8; 35.1,50.8)

19 (51.4; 35.1,67.3)

1 (2.7; 0.4,17.8)

17 (46; 30.3,62.4)

  4.3. Were all enrolled participants included in the analysis?

43 (28.3; 21.7,36)

49 (32.2; 25.2,40.1)

60 (39.5; 32,47.5)

17 (46; 30.3,62.4)

9 (24.3; 12.9,41.1)

11 (29.7; 16.9,46.7)

  4.4. Were participants with missing data handled appropriately?

24 (15.8; 10.8,22.5)

70 (46.1; 38.2,54.1)

58 (38.2; 30.7,46.2)

6 (16.2; 7.3,32.4)

15 (40.5; 25.7,57.3)

16 (43.2; 28,59.9)

  4.5. Was selection of predictors based on univariable analysis avoided?

68 (44.7; 37,52.8)

49 (32.2; 25.2,40.1)

35 (23; 17,30.4)

NA

  4.6. Were complexities in the data (e.g., censoring, competing risks, sampling of control participants) accounted for appropriately?

10 (6.6; 3.6,11.8)

28 (18.4; 13,25.5)

114 (75; 67.4,81.3)

2 (5.4; 1.3,20)

-

35 (94.6; 80,98.7)

  4.7. Were relevant model performance measures evaluated appropriately?

28 (18.4; 13,25.5)

87 (57.2; 49.2,64.9)

37 (24.3; 18.1,31.9)

10 (27; 14.9,44)

13 (35.1; 21.2,52.1)

14 (37.8; 23.4,54.8)

  4.8. Were model overfitting and optimism in model performance accounted for?

52 (34.2; 27.1,42.2)

84 (55.3; 47.2,63)

16 (10.5; 6.5,16.5)

NA

  4.9. Do predictors and their assigned weights in the final model correspond to the results from the reported multivariable analysis?

24 (15.8; 10.8,22.5)

8 (5.3; 2.6,10.2)

120 (79; 71.7,84.7)

NA

  1. Y  Yes, PY  Probably yes, N  No, PN  Probably no, NI  No information