Table 1 Summary of the characteristics and pros and cons for different modelling approaches
Models | Characteristics | Advantages | Disadvantages |
|---|---|---|---|
Existing approaches | |||
Fixed model never updated | • Model and coefficients fixed • Never updated | • Cheap (funding available) • Low complexity and easy to communicate | • Can become miscalibrated quickly • Dethroned by new model likely developed in future • Ends up as research waste • Loss of information |
Model with ad hoc updating (e.g. EuroSCORE) | • Updated when opportunity allows • Fixed coefficients between updates | • Easy to maintain • Cheap (funding available) • Low complexity • Little manual labour • Advantageous over developing a completely new model | • Non-responsive to calibration drift • Long data-action latency |
Models that get periodically updated (e.g. QRISK) | • Fixed regular updates • Set time period between updates | • Lower chance of miscalibration than above • Allows predictors to be included/excluded from the model • Relatively low complexity | • Funding required • Can still observe calibration drift between updates • Increased maintenance • Requires more than manual labour to maintain • Uncertainty on length of time needed between updates |
Proposed approaches | |||
Models with discrete updating and continual validation/monitoring (learning prediction system with discrete updating and continual monitoring) | • Updated when opportunity allows • Continuously monitors new data • Updated as a result of the monitoring • Feeds back information to the model on how and when to update | • Monitoring informs updates • Only update when required • Reactive to changes • Transports well across settings and populations | • Funding and infrastructure required • Update does not immediately follow after suggestion from monitoring • Requires some manual labour to maintain |
Complete dynamic system (continual model update with continual validation/monitoring) (learning prediction system with continual updating and monitoring) | • Dynamic model • Continuously monitors new data • Feeds back information to the model | • Efficient • Potential to be more accurate • Provides less miscalibrated results • “Reacts” quicker to change (responsive) • Possible to automate • Less manual labour to maintain • Transports well across settings and populations • Do not need to store the data | • Requires access to an appropriate “living” data source that is linked to the relevant outcomes • Uncertainty on how one should validate dynamic prediction models • Uncertainty on when to include/exclude predictors • Deciding how much to discount historical data • Uncertainty around when to update the model • Lack of software packages • Complexity of approach • Lack of requisite expertise by those developing the model • Lack of transparency • Inconsistent outputs from day to day • Funding |