Skip to main content

Table 1 Summary of the characteristics and pros and cons for different modelling approaches

From: Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems?

Models Characteristics Advantages Disadvantages
Existing approaches
 Fixed model never updated • Model and coefficients fixed
• Never updated
• Cheap (funding available)
• Low complexity and easy to communicate
• Can become miscalibrated quickly
• Dethroned by new model likely developed in future
• Ends up as research waste
• Loss of information
 Model with ad hoc updating (e.g. EuroSCORE) • Updated when opportunity allows
• Fixed coefficients between updates
• Easy to maintain
• Cheap (funding available)
• Low complexity
• Little manual labour
• Advantageous over developing a completely new model
• Non-responsive to calibration drift
• Long data-action latency
 Models that get periodically updated (e.g. QRISK) • Fixed regular updates
• Set time period between updates
• Lower chance of miscalibration than above
• Allows predictors to be included/excluded from the model
• Relatively low complexity
• Funding required
• Can still observe calibration drift between updates
• Increased maintenance
• Requires more than manual labour to maintain
• Uncertainty on length of time needed between updates
Proposed approaches
 Models with discrete updating and continual validation/monitoring (learning prediction system with discrete updating and continual monitoring) • Updated when opportunity allows
• Continuously monitors new data
• Updated as a result of the monitoring
• Feeds back information to the model on how and when to update
• Monitoring informs updates
• Only update when required
• Reactive to changes
• Transports well across settings and populations
• Funding and infrastructure required
• Update does not immediately follow after suggestion from monitoring
• Requires some manual labour to maintain
 Complete dynamic system (continual model update with continual validation/monitoring) (learning prediction system with continual updating and monitoring) • Dynamic model
• Continuously monitors new data
• Feeds back information to the model
• Efficient
• Potential to be more accurate
• Provides less miscalibrated results
• “Reacts” quicker to change (responsive)
• Possible to automate
• Less manual labour to maintain
• Transports well across settings and populations
• Do not need to store the data
• Requires access to an appropriate “living” data source that is linked to the relevant outcomes
• Uncertainty on how one should validate dynamic prediction models
• Uncertainty on when to include/exclude predictors
• Deciding how much to discount historical data
• Uncertainty around when to update the model
• Lack of software packages
• Complexity of approach
• Lack of requisite expertise by those developing the model
• Lack of transparency
• Inconsistent outputs from day to day
• Funding