Skip to main content

Development of risk prediction models to predict urine culture growth for adults with suspected urinary tract infection in the emergency department: protocol for an electronic health record study from a single UK university hospital



Urinary tract infection (UTI) is a leading cause of hospital admissions and is diagnosed based on urinary symptoms and microbiological cultures. Due to lags in the availability of culture results of up to 72 h, and the limitations of routine diagnostics, many patients with suspected UTI are started on antibiotic treatment unnecessarily. Predictive models based on routinely collected clinical information may help clinicians to rule out a diagnosis of bacterial UTI in low-risk patients shortly after hospital admission, providing additional evidence to guide antibiotic treatment decisions.


Using electronic hospital records from Queen Elizabeth Hospital Birmingham (QEHB) collected between 2011 and 2017, we aim to develop a series of models that estimate the probability of bacterial UTI at presentation in the emergency department (ED) among individuals with suspected UTI syndromes. Predictions will be made during ED attendance and at different time points after hospital admission to assess whether predictive performance may be improved over time as more information becomes available about patient status. All models will be externally validated for expected future performance using QEHB data from 2018/2019.


Risk prediction models using electronic health records offer a new approach to improve antibiotic prescribing decisions, integrating clinical and demographic data with test results to stratify patients according to their probability of bacterial infection. Used in conjunction with expert opinion, they may help clinicians to identify patients that benefit the most from early antibiotic cessation.

Peer Review reports


Urinary tract infection (UTI) is a leading cause of hospital admissions, accounting for 16% of all avoidable emergency admissions [1]. UTI presents with a clinical spectrum that ranges from urosepsis and pyelonephritis to mild urinary symptoms, each of which merits different durations of antibiotic treatment or potentially no antibiotics at all [2, 3]. The diagnosis of UTI syndromes is based on a combination of symptoms and microbiological culture of urine (bacteriuria) and/or blood (bacteraemia) [4]. Obtaining microbiological results introduces a bottleneck for evidence-based diagnosis, since cultures often take 48–72 h to grow. In the meantime, patients are often treated with antibiotics. Previous studies have found that up to 50% of such antibiotic use is unnecessary [5,6,7]. A wide range of additional information is collected as part of routine hospital care, which may provide an opportunity to reduce the diagnostic uncertainty introduced by the delay in culture results. Stored within electronic health records (EHR), these auxiliary data may help to create risk prediction models that can be used to predict the likely culture result and identify patients who are highly unlikely to have bacterial UTI.

We are aware of very few studies that have looked into using routine health data to predict bacteriuria in emergency department (ED) settings [8, 9]. In a recent study, Taylor et al. predicted bacterial growth in urine sampled from more than 80,000 patients with potential UTI symptoms in four US EDs [8]. Their best performing model achieved an area under the receiver operating characteristic curve (AUROC) of 0.90, with a sensitivity of 61.7% and a specificity of 94.9%. However, there are several reasons why it is difficult to apply this model in an NHS hospital including inclusion of urinalysis results that are not regularly performed in the UK, a relatively broad definition of the population at risk and the exclusion of microbiological culture of blood. In the only other study that we are aware of that attempted to predict bacteriuria in the ED, Wigton et al. achieved a lower AUROC of 0.78 on a sample of 506 patients [9]. Several further studies were performed in primary care settings [10,11,12,13,14] but their generalisability to a generally sicker ED population is questionable.

In this study, we will expand on previously published work [8, 9] and develop a model which aims to judge the probability of bacterial UTI in UK patients who present with suspected UTI in the ED. The models will be developed and tested using data on individuals presenting in the ED at Queen Elizabeth Hospital Birmingham (QEHB). QEHB has EHR which are ideally suited for this purpose, containing high-quality and detailed information on diagnoses, outcomes, investigations, vital signs, drug treatments and diagnostic coding dating back to 2011 [15]. Using these hospital records, our model aims to predict the probability that urinary pathogens will grow in urine and/or blood cultures collected during ED attendance. For admitted patients, additional predictions will be made at specific intervals throughout the first three days of their hospital stay to investigate whether additional information gathered during their inpatient stay, but before availability of culture results, allows to predict culture growth with increased certainty. Finally, we will explore differences in model performance and clinical progression for important subpopulations including the elderly and patients with a recorded alternative infective syndrome (e.g. pneumonia) at arrival or discharge, which do not require antibiotics for UTI but may need them for the treatment of the other infection.

Aims and objectives


To use EHR data from a large UK teaching hospital to predict patients’ probability of bacterial UTI at arrival among individuals with suspected UTI in the ED.


  1. a)

    To develop models that predict bacterial growth in urine and/or blood samples collected during ED attendance based on clinical information recorded in the patient’s medical history and in the ED

  2. b)

    To assess the change in predictive performance at pre-defined times after admission (0, 12, 24, 36, 48, 60 and 72 h) to determine whether additional inpatient data collected up to 72 h after admission to hospital leads to increased predictive certainty

  3. c)

    To compare the predictive performance of each model in different subpopulations, considering sex, age, clinical syndrome (lower UTI, pyelonephritis, urosepsis), final diagnoses (UTI, other infection, non-infective diagnosis) and risk of complications (death, admission to intensive care, length of stay)

  4. d)

    To evaluate the expected performance of our models in temporally and/or geographically independent patient populations


Source of data

QEHB is part of University Hospitals Birmingham NHS Foundation Trust, one of the largest teaching hospitals in England. The trust serves a population of more than 2.2 million patients per year, a large proportion of whom are seen at QEHB [16]. Detailed information on all patients admitted to QEHB is recorded within its electronic patient management system, including clinical diagnoses, observations, assessments and laboratory results [15]. Unlike many other trusts in England, QEHB has also recorded drug prescriptions electronically for more than 10 years, making it an invaluable resource for research linked to antibiotic prescribing.

Development dataset

To develop the predictive models, we will use data from all eligible patients who attended the ED at QEHB between 1 November 2011 and 31 December 2017 (electronic recording of ED diagnosis at QEHB started after a system change at the end of October 2011).

Validation dataset

We will use data collected at QEHB between 1 January 2018 and 31 March 2019 to externally validate the model. Patients who were included in the development dataset due to an earlier attendance will be excluded from the validation dataset. We will undertake external validation of our models in an independent dataset from University College London Hospitals NHS Foundation Trust.


Inclusion and exclusion criteria

All patients who attended the ED at QEHB within the study period and who had a urine sample submitted for microbiological testing within 24 h of arrival are eligible for inclusion in the study. A window of 24 h was chosen to account for discrepancies between when the sample was collected and when the urine sample was recorded in the laboratory system (particularly overnight). Patients enter the study at registration in the ED and exit the study on the earliest of the following dates: date of discharge, date of death, date of transfer to a different hospital or date of urine culture results.

Individuals aged < 18 years, pregnant women, patients who were not admitted via the ED and patients whose urine sample was submitted for culture but was not cultured due to standard laboratory protocols at QEHB (see the “Outcome” section for details) will be excluded from the analysis.


The principal outcome of interest is microbiological growth (≥ 104 colony-forming units/mL). Only urine samples that were eventually cultured will be included in the analysis. Microbiological cultures at QEHB are performed in accordance with standard laboratory procedures (UK Standards for Microbiology Investigations: SMI B41, Investigation of Urine; SMI B37, investigation of blood cultures (for organisms other than Mycobacterium species) [17]. The decision whether to culture a urine sample depends on cell count results performed in the laboratory. Only urines with white blood cell counts and bacteria counts above a threshold value were cultured. At the start of the study, the threshold value for proceeding to culture was white cell counts > 40/μL or bacteria counts > 4000/μL. This was adjusted to white cell counts > 80/μL or bacteria counts > 8000/μL following the introduction of a revised standard operating procedure in the microbiology laboratory in October 2015. Performing cell counts is not possible for urine samples less than 4 mL or for samples too viscous to pass through the instrument. Samples for which cell counts could not be performed are always cultured and included in the analysis. Following the standard procedure at QEHB, (heavy) mixed growth in the urine sample will be considered as contamination, except where E. Coli was present. In addition, samples will be classified as positive if there are < 104 colony-forming units/mL but the same urinary pathogen is identified from a blood culture, implying urosepsis.


We will consider a wide range of candidate predictors relating to characteristics of the urine sample, a patient’s clinical presentation at the start of and throughout the hospital stay, and to risk factors encoded in a patient’s medical history (Table 1). Candidate predictors were chosen based on clinical experience, the frequency with which variables are measured in the clinical context where the model is likely to be applied, and existing literature [8].

Table 1 Candidate predictors of bacteriuria measured (a) once at admission (constant throughout one hospital stay; time independent) and (b) multiple times throughout a patient’s hospital stay (time-dependent)

Sample size

Each year, around 60,000 patients are seen in the ED at QEHB. In 2014, more than 4500 patients were admitted to QEHB and prescribed an antibiotic. Preliminary analysis suggests that 20% of these prescriptions were for suspected UTI syndromes; hence, we expect ~ 5400 admitted patients using data from late 2011 to end of 2017 (6 years) [19]. Based on clinical experience, we expect a similar number of patients with suspected UTI syndromes to be discharged directly from the ED, resulting in an estimated total training sample of ~ 10,800 patients. Assuming a prevalence of bacteriuria of 30% like that reported by Taylor et al. previously, this would imply > 30 events per variable when including all variables defined in Table 1.

Statistical analysis methods

Feature engineering and selection

All continuous predictors will be winsorized at the 1st and 99th percentile to account for outliers and normalised to lie within the range (0, 1). Categorical predictors will be encoded in a full-rank encoding, combining levels with a small number of cases (< 5%). Predictors with zero variance will be excluded before analysis. For highly correlated predictors (correlation coefficient > 0.9 using Spearman’s rank correlation), one predictor will be removed before analysis based on clinical judgement. Similarly, predictors which are found to be largely missing and might thus not be expected to be present when the model will be used in practice at QEHB will be removed from the analysis before fitting the models.

We will consider the use of fractional polynomials (FP) with up to four degrees of freedom (i.e. 2 fractional polynomial terms) for each numerical predictor [20, 21]. We will estimate the optimal number of FPs using the Akaike Information Criterion. Once the best-fitting FPs have been determined, we will consider models with all predictors and parsimonious models selected via backwards feature elimination based on Wald statistics and Rubin’s rules [22]. Since the large number of possible predictors might limit the model’s usability in clinical practice, we follow Taylor et al. and consider a minimal model based on age, sex, urinalysis results and history of UTI [8].

Type of model

Baseline model in the ED

We will first develop a multivariable logistic regression model to predict bacterial growth in the urine and/or blood sample at the end of ED attendance. A prediction will be made for each patient based on the fitted value, which will serve as a baseline comparison for all further models considered.

Landmarking models at distinct time points after hospital admission

Additional measurements taken during the first couple of days in hospital may further improve the predictive power of our risk prediction models. We will develop a set of landmarking logistic regression models [23] that predict the probability of bacterial growth in the ED urine sample at pre-defined times t = {0, 12, 24, 36, 48, 60} hours after the patient has left the ED and was admitted to the hospital ward. In order to do so, we require a value for each included predictor at time t. Since predictors are measured irregularly throughout the patient’s hospital stay, we will first train a multivariate generalized linear mixed model (MGLMM) on all past predictor values up to time t to estimate the most likely value of each predictor at time t (see the “Missing data” section below for details). Values at time t will be estimated using the best linear unbiased predictors from the empirical Bayes posterior distribution of the random effects, conditional on past predictor measurements [23]. The estimated predictor values will then be fed to a logistic regression model that predicts the probability of microbiological growth in the ED sample after having observed the patient for t hours. As a result, patients might have more than one prediction, one for each time t at which they were still part of the at-risk population. Only patients still admitted and without a culture result at time t will be considered at-risk and will be included in the fitting and evaluation of the logistic regression model for time t.

Missing data

In EHR data, information is only recorded when events take place and we cannot distinguish between cases in which a test or diagnosis was not made and cases in which they were made but not recorded. Consequently, if historical variables such as co-morbidities, procedures, admission records, test results and procedures are not recorded (e.g. because they were performed at another hospital), we will have to assume that these events did not take place. For other variables with missing values that should have been obtained during the current visit (particularly vital signs and laboratory measurements), we will examine the pattern of missingness and impute values where appropriate depending on the type of prediction model.

Our baseline model is a logistic regression, which requires a non-missing value for each included predictor. We will use multivariate imputation by chained equations (MICE) based on the assumption that data are missing at random, i.e. whether a variable is missing or not only depends on the values of observed variables [24]. Following standard MICE procedures [25], we will include all predictors as well as the prediction outcome in the imputation procedure and impute 5 datasets with 10 iterations per dataset (Table 2). Depending on computational feasibility, we will aim to impute up to 100 datasets for our final model to ensure that we obtain robust imputations. Model training will be performed on the imputed development dataset. However, we cannot use the same imputation procedure to evaluate our models since we expect predictors to also be missing during model deployment. When used in practice, our model must impute any missing data in real-time before making a prediction, but at this point, no outcome will be available yet to use in the imputation. This will tend to result in suboptimal imputations when the model is used in practice [25]. To obtain an honest estimate of the performance of our models, we will evaluate them on a second set of imputations that were fit without using the outcome in the imputation procedure, emulating the situation in which the model will ultimately be used [26].

Table 2 Conditional models used in the multivariate imputation by chained equations

For our time-dependent models, the nature of missing data slightly differs. Values for each predictor might have been recorded never, once or multiple times before time t and we are interested in estimating the most likely value at time t. To estimate a good approximation for each predictor, we will separately fit a MGLMM at each landmarking time [23]. Each model will include fixed intercepts and slopes for each predictor and a time-dependent covariate indicating concurrent antibiotic treatment. We will consider correlation structures of varying complexity, with uncorrelated and correlated patient-specific random intercepts and/or slopes for each predictor. If the MGLMM is intractable, we will consider a simpler last observation carried forward (LOCF) method to estimate predictor values at time t, or a mixture of LOCF and MGLMM.

Model validation

Clinical diagnosis of bacterial UTI requires the presence of urinary symptoms in addition to microbiological culture. Bacteriuria in the absence of urinary symptoms (called asymptomatic bacteriuria) should not be treated with antibiotics [2]. Prevalence of asymptomatic bacteriuria differs between patient groups and increases for example with age. Whereas a urine sample might be sent for culture in many different patients “just in case”, a clinically usable model to confirm or rule out suspected bacterial UTI needs to perform especially well in patients with urinary symptoms. In our main analysis, we will therefore validate our models in the subgroup of patients with a suspected ED diagnosis of lower UTI or pyelonephritis, and our final model will be chosen based on the performance in this group. This group differs from the training population, which will include all patients irrespective of ED diagnosis to increase sample size and provide our model with enough power to learn general relationships. In a secondary analysis, we will also evaluate the performance of our models in patients without an ED diagnosis of UTI as well as in different age groups, by sex and by outcome (i.e. discharge diagnosis, death, admission to intensive care unit, length of stay). We will further consider training our model using only data from patients with a suspected ED diagnosis of lower UTI or pyelonephritis for training to ensure that a heterogeneous training population is not obscuring important relationships in patients with suspected UTI. Finally, we will perform secondary analyses limited to the first visit of each patient and to data after 2015, assessing the impact of repeated patient visits and the impact of increased culture thresholds on our models.

Internal validation

Model discrimination in each scenario will be assessed via multiple performance metrics: AUROC, Brier score, area under the precision-recall curve (AUPRC), specificity and negative predictive value (NPV). We will estimate each model’s specificity and NPV at a pre-set sensitivity of 95%, which will evaluate the model’s ability to be used as a screening tool to rule out bacterial UTI. We will assess how well predicted and observed probabilities correspond within each predicted decile (model calibration) by creating a calibration plot and estimating the calibration slope. An estimated slope > 1 indicates underfitting, whereas a slope < 1 indicates overfitting.

Evaluating the model only on the development dataset or a single validation dataset leads to optimistic estimations of the true model performance (henceforth called the apparent performance) [27]. To obtain a more reliable estimate of model performance, we will draw at least 100 bootstrap samples of the development dataset. Where computation time allows for it, we will consider up to 1000 bootstrap samples. All preprocessing and analysis steps including missing data imputation, estimation of fractional polynomials, feature selection and model evaluation will be carried out independently within each bootstrapped sample to avoid any data leakage [28]. The result will be one final model per bootstrapped sample. Evaluating each model on the bootstrap sample in which it was developed provides another estimate of the apparent performance, this time within the bootstrap. To estimate the magnitude of optimism in this bootstrapped apparent performance, we will simultaneously evaluate the bootstrapped model in the original development dataset (called test performance). The difference between test performance and bootstrapped apparent performance will be an estimate of model optimism.

Averaging estimates of the optimism across all bootstrapped samples results in a stable estimate of the optimism [27]. The final, optimism-corrected (“true”) estimate of model performance will then be calculated as follows:

$$ \mathrm{performance}=\mathrm{apparen}{\mathrm{t}}_{\mathrm{original}}-\mathrm{mean}\left(\mathrm{optimism}\right)=\mathrm{apparen}{\mathrm{t}}_{\mathrm{original}}-\frac{1}{B}{\sum}_{b=1}^B\left(\mathrm{apparen}{\mathrm{t}}_b-\mathrm{tes}{\mathrm{t}}_{\mathrm{original}}\right) $$

All metrics used in the model evaluation (AUROC, AUPRC, specificity and NPV) will be adjusted for optimism.

External validation

The performance of the model (AUROC, AUPRC, specificity and NPV) in a new dataset will be evaluated using EHRs from patients with suspected UTI who were admitted to QEHB between 1 January 2018 and 31 March 2019. We will summarise average performance and calibration in this temporally independent sample. We will further validate the model in a geographically independent sample of patients from University College London Hospitals NHS Foundation Trust.

All analyses will be performed using the statistical software R [29] including but not necessarily limited to the packages: tidyverse [30], tidymodels [31], mice [32] and mfp [33].


The need to reduce inappropriate antibiotic prescribing in secondary care is widely acknowledged, but progress is thwarted by the lack of rapid and reliable diagnostic tests for bacterial infection. Risk prediction models using data contained within EHR offer a new approach to improve antibiotic prescribing decisions, by integrating clinical and demographic data with test results to stratify patients according to their likelihood of bacterial infection.

However, diagnostic uncertainty represents a major obstacle in the application of risk prediction models for bacterial infection. Clinical infection syndromes often overlap, and diagnoses are often not confirmed by microbial culture. This makes it difficult to reliably distinguish infection from non-infectious conditions, but also to discriminate between clinical infection syndromes.

For these reasons, we have not attempted to develop a model which supports decision around antibiotic initiation in the ED, recognising that few doctors will be willing to withhold antibiotics if patients are unwell and the diagnosis is uncertain. Instead, we have opted for a model that identifies patients who may benefit from early antibiotic cessation since they are actually at low risk of bacterial UTI. Descriptive analyses of patients who have been categorised by the model as low/high risk of bacterial UTI will identify categories of patients who are most likely to be low risk, for example based on age, sex and UTI syndrome at presentation. This will be used in conjunction with expert clinical opinion to define a “low-risk” population of patients who have been treated with antibiotics for suspected UTI but are unlikely to benefit from antibiotic treatment. Individuals from this population sub-group will be asked to participate in a proof of concept trial, and randomised to either stop antibiotics early, or to continue antibiotic as per standard care. The trial will assess the safety and feasibility of early antibiotic cessation in these patients and lay the foundation for a future multi-centre trial. It will also demonstrate the potential use of EHR datasets to guide prescribing decisions.

Availability of data and materials

The data that support the findings of this study are available from University Hospitals Birmingham NHS Foundation Trust, but restrictions apply to the availability of these data, which are not publicly available. Data are however available from the authors upon reasonable request and with permission of University Hospitals Birmingham NHS Foundation Trust. Codelists can be found in the appendix and analytical code used in the study will be made available by the authors.



Area under the precision-recall curve


Area under the receiver operating characteristic curve


Charlson Co-morbidity Index


Defined daily dose


Emergency department


Electronic health records


Fractional polynomials


International Classification of Diseases 10th Revision


Index of Multiple Deprivation




Last observation carried forward


Multivariate generalized linear mixed model


Multivariate imputation by chained equations


National Health Service


Negative predictive value


Classification of Interventions and Procedures version 4


Queen Elizabeth Hospital Birmingham


UK Standards for Microbiology Investigations


Urinary tract infection


  1. 1.

    Blunt I. Focus on preventable admissions: trends in emergency admissions for ambulatory care sensitive conditions, 2001 to 2013. The Health Foundation and The Nuffield Trust; 2013.

  2. 2.

    Urinary tract infection (lower): antimicrobial prescribing. NICE guideline [NG109]. National Institute of Health and Care Excellence. Accessed 28 May 2019.

  3. 3.

    National Institute of Health and Care Excellence. Pyelonephritis - acute. Accessed 10 Jan 2020.

  4. 4.

    National Institute for Health and Care Excellence. Urinary tract infection (lower) - women. Accessed 13 Jan 2020.

  5. 5.

    Woodford HJ, George J. Diagnosis and management of urinary tract infection in hospitalized older people. J Am Geriatr Soc. 2009;57:107–14.

    Article  Google Scholar 

  6. 6.

    Tomas ME, Getman D, Donskey CJ, Hecker MT. Overdiagnosis of Urinary Tract Infection and Underdiagnosis of Sexually Transmitted Infection in Adult Women Presenting to an Emergency Department. J Clin Microbiol. 2015;53:2686–92.

    CAS  Article  Google Scholar 

  7. 7.

    McIsaac WJ, Hunchak CL. Overestimation error and unnecessary antibiotic prescriptions for acute cystitis in adult women. Med Decis Mak. 2011;31:405–11.

    Article  Google Scholar 

  8. 8.

    Taylor RA, Moore CL, Cheung K-H, Brandt C. Predicting urinary tract infections in the emergency department with machine learning. PLoS One. 2018;13:e0194085.

    Article  Google Scholar 

  9. 9.

    Wigton RS, Hoellerich VL, Ornato JP, Leu V, Mazzotta LA, Cheng IH. Use of clinical findings in the diagnosis of urinary tract infection in women. Arch Intern Med. 1985;145:2222–7.

    CAS  Article  Google Scholar 

  10. 10.

    Little P, Turner S, Rumsby K, Warner G, Moore M, Lowes JA, et al. Developing clinical rules to predict urinary tract infection in primary care settings: sensitivity and specificity of near patient tests (dipsticks) and clinical scores. Br J Gen Pract. 2006;56:606–12.

    PubMed  PubMed Central  Google Scholar 

  11. 11.

    McIsaac WJ, Moineddin R, Ross S. Validation of a decision aid to assist physicians in reducing unnecessary antibiotic drug use for acute cystitis. Arch Intern Med. 2007;167:2201–6.

    Article  Google Scholar 

  12. 12.

    Heckerling PS, Canaris GJ, Flach SD, Tape TG, Wigton RS, Gerber BS. Predictors of urinary tract infection based on artificial neural networks and genetic algorithms. Int J Med Inform. 2007;76:289–96.

    Article  Google Scholar 

  13. 13.

    Gadalla AAH, Friberg IM, Kift-Morgan A, Zhang J, Eberl M, Topley N, et al. Identification of clinical and urine biomarkers for uncomplicated urinary tract infection using machine learning algorithms. Sci Rep. 2019;9:19694.

    CAS  Article  Google Scholar 

  14. 14.

    Burton RJ, Albur M, Eberl M, Cuff SM. Using artificial intelligence to reduce diagnostic workload without compromising detection of urinary tract infections. BMC Med Inform Decis Mak. 2019;19:171.

    Article  Google Scholar 

  15. 15.

    Freemantle N, Ray D, Falcaro M, McNulty D, Shallcross L, Wood J, et al. BMI upon discharge from hospital and its relationship with survival: an observational study utilising linked patient records. J R Soc Med. 2016;109:230–8.

    Article  Google Scholar 

  16. 16.

    National Health Service. University Hospitals Birmingham NHS Foundation Trust. 2009. Accessed 17 Dec 2019.

  17. 17.

    Public Health England. Standards for microbiology investigations (UK SMI). GOV.UK. 2014. Accessed 29 Jul 2019.

  18. 18.

    World Health Organisation. WHOCC - ATC/DDD Index. Accessed 17 Dec 2019.

  19. 19.

    Shallcross LJ, Freemantle N, Nisar S, Ray D. A cross-sectional study of blood cultures and antibiotic use in patients admitted from the Emergency Department: missed opportunities for antimicrobial stewardship. BMC Infect Dis. 2016;16:166.

    Article  Google Scholar 

  20. 20.

    Morris TP, White IR, Carpenter JR, Stanworth SJ, Royston P. Combining fractional polynomial model building with multiple imputation. Stat Med. 2015;34:3298–317.

    Article  Google Scholar 

  21. 21.

    Ambler G, Royston P. Fractional polynomial model selection procedures: investigation of type i error rate. J Stat Comput Simul. 2001;69:89–108.

    Article  Google Scholar 

  22. 22.

    Wood AM, White IR, Royston P. How should variable selection be performed with multiply imputed data? Stat Med. 2008;27:3227–46.

    Article  Google Scholar 

  23. 23.

    Paige E, Barrett J, Stevens D, Keogh RH, Sweeting MJ, Nazareth I, et al. Landmark Models for Optimizing the Use of Repeated Measurements of Risk Factors in Electronic Health Records to Predict Future Disease Risk. Am J Epidemiol. 2018;187:1530–8.

    Article  Google Scholar 

  24. 24.

    White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med. 2011;30:377–99.

    Article  Google Scholar 

  25. 25.

    Moons KGM, Donders RART, Stijnen T, Harrell FE Jr. Using the outcome for imputation of missing predictor values was preferred. J Clin Epidemiol. 2006;59:1092–101.

    Article  Google Scholar 

  26. 26.

    Wood AM, Royston P, White IR. The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data. Biom J. 2015;57:614–32.

    Article  Google Scholar 

  27. 27.

    Steyerberg EW, Harrell FE Jr, Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54:774–81.

    CAS  Article  Google Scholar 

  28. 28.

    Smialowski P, Frishman D, Kramer S. Pitfalls of supervised feature selection. Bioinformatics. 2010;26:440–3.

    CAS  Article  Google Scholar 

  29. 29.

    R Core Team. R: A language and environment for statistical computing. 2018.

  30. 30.

    Wickham H, Averick M, Bryan J, Chang W, McGowan L, François R, et al. Welcome to the Tidyverse. JOSS. 2019;4:1686.

    Article  Google Scholar 

  31. 31.

    Kuhn M, Wickham H. tidymodels: Easily Install and Load the “Tidymodels” Packages. 2020.

  32. 32.

    van Buuren S, Groothuis-Oudshoorn K. mice: Multivariate Imputation by Chained Equations in R. J Stat Softw, Articles. 2011;45:1–67.

  33. 33.

    Ambler G, Benner A. mfp: Multivariable Fractional Polynomials. 2015.

Download references


We thank Dr Tim Morris and Prof Ruth Keogh for their invaluable comments on the study design and the protocol draft.


LS is funded by a National Institute for Health Research (NIHR) Clinician Scientist award (CS02016-007) for this research project. This publication presents independent research funded by the National Institute for Health Research (NIHR). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. This work was further supported by a Rosetrees & the Stoneygate Trusts award (M627).

Author information




The study was conceived by PR, MJG, NF and LS. PR and LS developed and wrote the study protocol. DM extracted the data and provided support and guidance on data interpretation. OC provided detailed guidance on missing data methodology. All authors read and approved the final protocol.

Corresponding author

Correspondence to Patrick Rockenschaub.

Ethics declarations

Ethics approval and consent to participate

This study has Health Research Authority (HRA) approval ref: 17/HRA/3427. Ethical approval was not required since the study uses pseudonymised datasets that were collected as part of routine clinical care. The dataset is stored within the UCL Data Safe Haven accessible only by named researchers. All researchers who work on the dataset have undertaken training in information governance and GDPR within the last 12 months.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rockenschaub, P., Gill, M.J., McNulty, D. et al. Development of risk prediction models to predict urine culture growth for adults with suspected urinary tract infection in the emergency department: protocol for an electronic health record study from a single UK university hospital. Diagn Progn Res 4, 15 (2020).

Download citation


  • Protocol
  • Diagnosis
  • Urinary tract infection
  • Prediction models
  • Hospital