Skip to main content

Table 1 Summary of the characteristics and pros and cons for different modelling approaches

From: Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems?





Existing approaches

 Fixed model never updated

• Model and coefficients fixed

• Never updated

• Cheap (funding available)

• Low complexity and easy to communicate

• Can become miscalibrated quickly

• Dethroned by new model likely developed in future

• Ends up as research waste

• Loss of information

 Model with ad hoc updating (e.g. EuroSCORE)

• Updated when opportunity allows

• Fixed coefficients between updates

• Easy to maintain

• Cheap (funding available)

• Low complexity

• Little manual labour

• Advantageous over developing a completely new model

• Non-responsive to calibration drift

• Long data-action latency

 Models that get periodically updated (e.g. QRISK)

• Fixed regular updates

• Set time period between updates

• Lower chance of miscalibration than above

• Allows predictors to be included/excluded from the model

• Relatively low complexity

• Funding required

• Can still observe calibration drift between updates

• Increased maintenance

• Requires more than manual labour to maintain

• Uncertainty on length of time needed between updates

Proposed approaches

 Models with discrete updating and continual validation/monitoring (learning prediction system with discrete updating and continual monitoring)

• Updated when opportunity allows

• Continuously monitors new data

• Updated as a result of the monitoring

• Feeds back information to the model on how and when to update

• Monitoring informs updates

• Only update when required

• Reactive to changes

• Transports well across settings and populations

• Funding and infrastructure required

• Update does not immediately follow after suggestion from monitoring

• Requires some manual labour to maintain

 Complete dynamic system (continual model update with continual validation/monitoring) (learning prediction system with continual updating and monitoring)

• Dynamic model

• Continuously monitors new data

• Feeds back information to the model

• Efficient

• Potential to be more accurate

• Provides less miscalibrated results

• “Reacts” quicker to change (responsive)

• Possible to automate

• Less manual labour to maintain

• Transports well across settings and populations

• Do not need to store the data

• Requires access to an appropriate “living” data source that is linked to the relevant outcomes

• Uncertainty on how one should validate dynamic prediction models

• Uncertainty on when to include/exclude predictors

• Deciding how much to discount historical data

• Uncertainty around when to update the model

• Lack of software packages

• Complexity of approach

• Lack of requisite expertise by those developing the model

• Lack of transparency

• Inconsistent outputs from day to day

• Funding