Table 4 Measures of model performance in predicting 28-day risk of COVID-19 mortality in the temporal and geographical internal validation, using the “selected” models
Approach; measures of infection prevalence included; model format | Validation period | Region omitted from analysis | C-statistic | Observed mean risk (in validation data) (%) | Predicted mean risk (in validation data) (%) | Calibration | |
|---|---|---|---|---|---|---|---|
Intercept (95% CI) | Slope (95% CI) | ||||||
Temporal internal validation | |||||||
(A), none, Cox | 3 | – | 0.941 | 0.0104 | 0.0037 | 1.03 (0.97, 1.08) | 1.05 (1.00, 1.11) |
(B), modelled estimates, Poisson | 3 | – | 0.943 | 0.0104 | 0.0195 | −0.63 (−0.68, −0.57) | 1.04 (0.98, 1.09) |
Geographical internal validation | |||||||
(A), none, Cox | 1 | East | 0.907 | 0.0036 | 0.0034 | 0.05 (−0.15, 0.25) | 0.85 (0.65, 1.05) |
London | 0.948 | 0.0102 | 0.0024 | 1.46 (1.27, 1.65) | 0.98 (0.78, 1.17) | ||
Midlands | 0.927 | 0.0052 | 0.0038 | 0.31 (0.14, 0.48) | 0.95 (0.78, 1.12) | ||
North East and Yorkshire | 0.896 | 0.0024 | 0.0046 | −0.66 (−0.92, −0.39) | 0.81 (0.55, 1.08) | ||
North West | 0.857 | 0.0021 | 0.0045 | −0.78 (−1.19, −0.36) | 0.72 (0.31, 1.14) | ||
South East | 0.946 | 0.0025 | 0.0035 | −0.33 (−0.78, 0.12) | 1.06 (0.61, 1.51) | ||
South West | 0.933 | 0.0014 | 0.0042 | −1.10 (−1.53, −0.67) | 0.92 (0.50, 1.35) | ||
2 | East | 0.930 | 0.0365 | 0.0034 | 2.39 (2.33, 2.45) | 1.05 (0.99, 1.11) | |
London | 0.932 | 0.0396 | 0.0023 | 2.84 (2.74, 2.94) | 0.90 (0.80, 1.00) | ||
Midlands | 0.931 | 0.0405 | 0.0038 | 2.39 (2.33, 2.45) | 0.99 (0.93, 1.05) | ||
North East and Yorkshire | 0.937 | 0.0420 | 0.0046 | 2.22 (2.16, 2.28) | 1.00 (0.94, 1.07) | ||
North West | 0.931 | 0.0493 | 0.0045 | 2.40 (2.32, 2.49) | 1.02 (0.94, 1.11) | ||
South East | 0.938 | 0.0238 | 0.0035 | 1.92 (1.77, 2.07) | 1.04 (0.89, 1.19) | ||
South West | 0.941 | 0.0235 | 0.0041 | 1.74 (1.64, 1.84) | 1.05 (0.94, 1.15) | ||
3 | East | 0.942 | 0.0106 | 0.0033 | 1.17 (1.05, 1.28) | 1.08 (0.97, 1.20) | |
London | 0.947 | 0.0033 | 0.0023 | 0.37 (0.04, 0.71) | 0.91 (0.57, 1.24) | ||
Midlands | 0.935 | 0.0120 | 0.0037 | 1.18 (1.07, 1.29) | 0.99 (0.88, 1.10) | ||
North East and Yorkshire | 0.945 | 0.0142 | 0.0045 | 1.15 (1.04, 1.26) | 1.04 (0.93, 1.15) | ||
North West | 0.935 | 0.0168 | 0.0044 | 1.33 (1.19, 1.48) | 1.06 (0.92, 1.21) | ||
South East | 0.965 | 0.0052 | 0.0035 | 0.42 (0.10, 0.73) | 1.20 (0.89, 1.52) | ||
South West | 0.920 | 0.0047 | 0.0041 | 0.15 (−0.09, 0.38) | 1.07 (0.83, 1.30) | ||
(B), modelled estimates, Poisson | 1 | East | 0.913 | 0.0036 | 0.0020 | 0.57 (0.37, 0.77) | 0.87 (0.68, 1.07) |
London | 0.952 | 0.0102 | 0.0081 | 0.23 (0.04, 0.42) | 0.99 (0.80, 1.19) | ||
Midlands | 0.924 | 0.0052 | 0.0058 | −0.12 (−0.29, 0.04) | 0.91 (0.75, 1.08) | ||
North East and Yorkshire | 0.907 | 0.0024 | 0.0035 | −0.37 (−0.64, −0.10) | 0.86 (0.59, 1.12) | ||
North West | 0.857 | 0.0021 | 0.0045 | −0.77 (−1.19, −0.36) | 0.71 (0.30, 1.13) | ||
South East | 0.937 | 0.0025 | 0.0048 | −0.63 (−1.08, −0.18) | 1.01 (0.56, 1.45) | ||
South West | 0.931 | 0.0014 | 0.0047 | −1.21 (−1.64, −0.79) | 0.93 (0.50, 1.35) | ||
2 | East | 0.936 | 0.0365 | 0.0356 | 0.03 (−0.04, 0.09) | 1.02 (0.96, 1.09) | |
London | 0.937 | 0.0396 | 0.0429 | −0.08 (−0.18, 0.02) | 0.93 (0.83, 1.02) | ||
Midlands | 0.933 | 0.0405 | 0.0373 | 0.08 (0.02, 0.14) | 0.99 (0.93, 1.05) | ||
North East and Yorkshire | 0.939 | 0.0420 | 0.0352 | 0.18 (0.11, 0.24) | 0.99 (0.93, 1.06) | ||
North West | 0.931 | 0.0493 | 0.0392 | 0.23 (0.15, 0.32) | 0.99 (0.91, 1.08) | ||
South East | 0.943 | 0.0238 | 0.0319 | −0.30 (−0.44, −0.15) | 1.04 (0.89, 1.19) | ||
South West | 0.941 | 0.0235 | 0.0277 | −0.17 (−0.27, −0.06) | 1.02 (0.91, 1.12) | ||
3 | East | 0.943 | 0.0106 | 0.0103 | 0.03 (−0.09, 0.14) | 1.05 (0.93, 1.17) | |
London | 0.954 | 0.0033 | 0.0102 | −1.12 (−1.46, −0.78) | 0.94 (0.60, 1.28) | ||
Midlands | 0.938 | 0.0120 | 0.0153 | −0.24 (−0.35, −0.13) | 0.99 (0.88, 1.10) | ||
North East and Yorkshire | 0.944 | 0.0142 | 0.0140 | 0.01 (−0.10, 0.12) | 1.03 (0.92, 1.14) | ||
North West | 0.935 | 0.0168 | 0.0205 | −0.20 (−0.35, −0.05) | 1.03 (0.88, 1.18) | ||
South East | 0.965 | 0.0052 | 0.0110 | −0.74 (−1.05, −0.42) | 1.16 (0.85, 1.48) | ||
South West | 0.921 | 0.0047 | 0.0071 | −0.41 (−0.64, −0.18) | 1.04 (0.81, 1.27) | ||