Development of a clinical prediction rule for sepsis in primary care: protocol of the TeSD-IT study

Background Early recognition and treatment of sepsis is crucial to prevent detrimental outcomes. General practitioners (GPs) are often the first healthcare providers to encounter seriously-ill patients. The aim of this study is to assess the value of clinical information and additional tests to develop a clinical prediction rule (CPR) to support early diagnosis and management of sepsis by GPs. Methods We will perform a diagnostic study in the setting of out-of-hours home visits in four GP cooperatives in the Netherlands. Acutely ill adult patients suspected of a serious infection are screened for eligibility by the GP. The following candidate predictors are prospectively recorded: 1) age; 2) body temperature; 3) systolic blood pressure; 4) heart rate; 5) respiratory rate; 6) peripheral oxygen saturation; 7) altered mental status; 8) rigors and 9) rapid illness progression. After the clinical assessment of the GP, blood samples are collected in all patients to measure C-reactive protein, lactate and procalcitonin. All patients receive care as usual. The primary outcome is presence or absence of sepsis within 72 hours of inclusion, according to an expert panel. The need for hospital treatment for any indication will be assessed by the expert panel as a secondary outcome. Multivariable logistic regression is used to design an optimal prediction model first, and subsequently derive a simplified CPR that enhances feasibility of using the model in daily clinical practice. Bootstrapping will be performed for internal validation of both the optimal model and simplified CPR. Performance of both models will be compared to existing CPRs for sepsis. Discussion This study will enable us to develop a CPR for the recognition of sepsis in a high-risk primary care setting to aid the decision which patients have to be immediately referred to a hospital and who can be safely treated at home. As clinical signs and blood samples will be retrieved prospectively in all participants, near complete data will be

4 available for analyses. External validation is needed before the CPR is implemented in routine care and to determine in which pre-hospital settings care can be improved using the CPR.
Trial registration The study is registered in the Netherlands Trial Registry (registration number NTR7026).

Background
Sepsis is a life-threatening complication of an infection. Early detection and initiation of adequate treatment is the key factor influencing outcome. 1− 4 It is estimated that annually 49 million people suffer from sepsis worldwide, of which 11 million do not survive. 5 In 2017, the WHO declared sepsis a global healthcare priority and urged member states to improve recognition and treatment of sepsis. 6 Global efforts to reduce mortality and morbidity from sepsis have focused on hospital settings, but patients' early stages of sepsis are often presented in primary care. General practitioners (GPs) are confronted with acutely ill patients with a variety of symptoms, signs and potential diagnoses. Within minutes they have to decide whether patients can be treated at home or have to be referred to a hospital for further assessment.
In the Netherlands, out-of-hours primary care is provided by large GP cooperatives. 7 Patients are only assessed by a GP if the medical complaint cannot wait until the following working day. In contrast to other common time-critical conditions such as stroke and myocardial infarction, patients with sepsis are more likely to contact a GP cooperative instead of an emergency medical service prior to hospital treatment. 8 Data from our preliminary research on patients admitted to an intensive care unit (ICU) due to community acquired sepsis, showed that about half of the patients had prior contact with a GP cooperative. Two thirds of these patients were referred to the hospital after the first 5 contact. 9 The majority of the patients were assessed during a home visit.
In the hospital setting, vital signs are used to screen for sepsis in patient with suspected infections. The Systemic Inflammatory Response Syndrome (SIRS) was introduced in 1992 to define sepsis. 10 Besides the white blood count, the SIRS parameters are heart rate < 90/min, respiratory rate > 20/min, and a body temperature < 36 °C or > 38 °C. As SIRS criteria are often present in patients without serious infections and one in eight patients admitted to the ICU with sepsis were found to lack positive SIRS criteria, a new consensus definition was formulated in 2016. In the "Sepsis-3" definition, the Sequential Organ Failure Assessment (SOFA) score was proposed to diagnose sepsis. 1 As this score is not easy to apply outside the ICU, the quick SOFA (qSOFA) was introduced for rapid bedside assessment. The parameters used in the qSOFA are an altered mental status, systolic blood pressure ≤ 100 mmHg, and a respiratory rate ≥ 22/min. A positive score on two or more parameters predicts an increased risk of mortality. However, the qSOFA is not suitable as a screening tool as it lacks sensitivity. 11− 12 C-reactive protein (CRP), lactate and procalcitonin (PCT) have all shown to increase the sensitivity and overall diagnostic performance of the qSOFA. 13− 15 Both signs and symptoms as well as additional biomarkers like CRP, lactate and PCT are however not evaluated for early detection of sepsis in primary care.
The aim of the TeSD-IT study (Testing for Sepsis in primary care: Diagnostic and prognostic study Investigating the potential benefits of point of care Testing) is to develop a clinical prediction rule (CPR) to improve the detection of sepsis while limiting unnecessary referrals in acutely ill patients presenting at the GP cooperative home visits.
Clinical signs and symptoms as well as blood tests are considered as candidate variables.

Setting, and design
We will perform a prospective diagnostic cohort study in the Netherlands in four GP cooperatives (Ede, Den Bosch, Uden and Oss) for out-of-hours primary care. The cooperatives serve a total of approximately 830,000 inhabitants in a mixed urban, suburban and rural area. The cooperatives are based in or adjacent to regional hospitals.

Patients
Patients will be recruited during out-of-hours home visits by GPs. Patients only receive home visits when they have acute medical complaints that cannot wait until the next working day and they are not able to visit the GP cooperative location for a clinic consultation. This is decided after telephone assessment by a triage nurse based on the Netherlands Triage System (NTS). 16

Candidate predictors
We selected nine clinical features and three blood tests as candidate predictors for the development of the clinical prediction model ( Table 1). Parameters of widely used scoring systems such as the SIRS, qSOFA and National Early Warning Score (NEWS) 17 were considered, as well as clinical features used in guidelines such as the Netherlands Triage Standard (NTS) and NICE Sepsis guideline. 18 Candidate predictors were considered when 7 they were expected to have diagnostic value for sepsis, based on previous studies, and can be easily and objectively measured by GPs.
Candidate blood tests had to be currently used in the hospital setting for the diagnosis and/or prognosis of sepsis, and, preferably, to be available as a point-of-care test for reasons of implementation. CRP and lactate measurement are part of the standard care in patients with suspected sepsis during assessment in the Emergency Department (ED) in the Netherlands. Procalcitonin (PCT) is not routinely measured in most hospitals, but we decided to include PCT as a candidate predictor as PCT might be superior to CRP, 19 and the NICE sepsis guideline specifically recommends to further evaluate PCT in sepsis research.

Outcome measures
The primary outcome measure is sepsis within 72 hours of inclusion. This will be determined by an expert panel using the Sepsis-3 criteria. 1 In this consensus definition, sepsis is defined by an increase of two SOFA-points (Table 2) due to infection.
To limit the workload for the experts, we will install three expert panels, each comprising a GP, an emergency physician, and an internist(-intensivist). Each case will be assessed by one panel. All relevant information from medical records from the GP and the hospital when applicable will be presented to the panel. If there is no consensus on the primary outcome, the case will be discussed in a face-to-face consensus meeting with all three experts to determine the final outcome. Interobserver agreement between the three panels will be assessed in a selection of 10% of the cases that will be assessed by all panels. Besides the dichotomous primary outcome "sepsis within 72 hours", the likelihood of sepsis will be assigned a numerical score between 0 and 10. This gives information on remaining uncertainty regarding sepsis classification, providing insight in the degree of 8 bias that may be introduced when calculating diagnostic accuracy measures using dichotomous sepsis classification. 20 Furthermore, the need for hospital treatment is scored between 0 and 10 by the expert panel as a secondary outcome. An average score above 5 will be regarded as a patient that should best be referred to the hospital immediately by the GP and a score ≤ 5 as a patient that does not have to be referred immediately.
Other outcome measures are hospitalisation (length of stay and type of care: ICU or regular ward), maximum SOFA score in the first 72 hours, 30 day all-cause mortality, final diagnosis, and medical costs.

Study period
The inclusion period is from June 2018 until April 2020. If in April 2020 the required sample size is not reached, the patient inclusion will be prolonged until the minimum required number of events has been reached. End of follow-up is 30 days after inclusion of the last patient.

Procedure during home visit
All patients receive usual care. Patients will be screened for eligibility during home visits by the attending GP of the GP cooperative. Verbal informed consent is obtained from the patient or his legal representative. The GP is (routinely) accompanied by a chauffeur during the home visit. The chauffeurs are used to practically assist the GP during the visit.
Portable monitors (Philips intelliVue MP2 or X2) will be available to record peripheral oxygen saturation, automated blood pressure and heart-and respiratory rate by three lead electrodes on the chest.
The GP records the assessment of the candidate predictors in a case report form. In addition, the GP records if he/she has a gut feeling that "something is wrong" and 9 provides the likelihood of presence of sepsis at inclusion on a scale from 0 to 10.
All study materials will be taken to the patient's home in a study bag. The venous samples will be collected by either the GP or an on-call laboratory assistant or nurse within one hour after inclusion, with a maximum of eight hours. Written informed consent is obtained prior to the collection of the blood samples. In case the patient is referred to the hospital and the blood samples are not collected by the GP, the study bag will be transported with the patient to the ED. Subsequently, the laboratory assistant on call will visit the patient in the hospital and collect the blood samples. Three blood tubes will be collected: 10 ml for serum, 10 ml for EDTA plasma and a 2 ml heparin tube. Lactate will be measured immediately afterwards from a single drop of blood taken from the heparin tube, using the StatStrip Xpress (Nova Biomedical) point-of-care test. The remaining blood samples will be taken to the hospital laboratory and divided into six samples of serum and six samples of EDTA plasma. The aliquots will be temporarily stored at the local laboratory at <-70°C.
Two samples (1 ml serum and 1 ml EDTA plasma) will be transported to the Jeroen Bosch Hospital for CRP and PCT analyses, and the remaining samples (5x 0.5 ml serum and 5x 0.5 ml EDTA plasma) will be stored for 15 years at <-80°C at the UMC Utrecht for potential future testing.

Training and remuneration of personnel
Chauffeurs of the GP cooperatives are trained in using portable monitors for vital sign measurement and other study procedures. At the GP cooperative in Ede the chauffeurs will also be trained in the measurement of POC-lactate, as GPs will collect the venous blood samples themselves occasionally. The laboratory assistants and nurses who will be on call for the collection of the blood samples will also be trained in the POC-lactate measurement and other study procedures including the obtaining of written informed consent. Attending GPs are informed by an information letter by mail and hard copy at the GP cooperative. Leaflets with a summary of the study procedures will also be available.

Follow-up
The total follow-up time is 30 days. Patients will be asked to complete the EQ-5D-5L questionnaire 22 at the end of follow-up (1) at the day of completion of the questionnaire; (2) before the onset of the recent disease episode (i.e. their health status of at least one month ago); and (3) for the worst day they remember from their recent disease episode.
Furthermore, patients will be asked to report on consumption of medical resources during the 30-day follow-up period. Productivity losses are not considered, as most patients are elderly and not doing paid work. In case of no response to the questionnaire after one week, patients are contacted once by telephone as a reminder.

Data-extraction
Relevant medical information will be obtained from the patient's (regular) GP, the GP cooperative, and the hospital. Medication use and comorbidities before inclusion are retrieved from GP electronic records, as well as information on any subsequent contacts.
The medical record of the assessment at the time of inclusion is retrieved from the GP cooperative. The following data from the electronic medical record of the hospital will be collected: full reports from ED and hospital discharge; date and time of ED visit, hospital admittance and discharge (including type of ward); vital signs, EMV score, leucocyte count, thrombocyte count, creatinine, bilirubin, CRP and lactate measured in the first 72 hours after inclusion; cultures taken in the first 72 hours after inclusion; radiodiagnostic procedures in the first seven days after inclusion; antibiotic prescriptions during hospitalisation; intravenous volume therapy in the first 72 hours (defined as more than 1.5 litres of fluids in 24 hours).

Sample size
In total, 12 candidate predictors are chosen for the development of the prediction model (table 1). Using the rule of thumb of 10 events per variable, 23 we need 120 patients reaching the primary outcome "sepsis within 72 hours of inclusion" in the final dataset.
Prior to the start, the prevalence of sepsis based on previous research and literature was estimated to be around 12%. However, preliminary data from the patients included in the study so far, indicates the prevalence of sepsis in the study cohort to be more likely to be around 30-40%. After the first 100 cases are assessed by the expert panel, we will determine the final target sample size.

Descrriptive statistics
We will use a combination of IBM SPSS Statistics and R Statistical Software for all analyses. We will start with descriptive analyses on baseline characteristics (age, sex, comorbidities, vital sign measurements and other clinical features, blood tests results, baseline EQ-5D score), final diagnosis, hospital admission, ICU admission, length of stay, EQ-5D compared to baseline, and 30-day mortality. Results will be stratified based on whether patients do or do not meet the primary outcome sepsis.

Data cleaning
Range and distribution of all continuous variables will be graphically inspected, and any outliers (more than three standard deviations from the mean) will be discussed and dealt with accordingly. Any missing data on clinical features or blood tests, will be accounted for by applying multiple imputation techniques. Prediction model development and performance will be analysed using the imputed datasets.

Development of the prediction model
A multivariable penalized logistic regression model will be developed, based on the variables listed in table 1, for predicting the primary outcome (sepsis within 72 hours after inclusion). We will use a two-stepped approach entering and selecting clinical features first, and blood tests second. In both steps the selection of predictors will be based on a stepwise backward selection, using change in Akaike information criterion (AIC) for selecting the preferred model. 24 The goal is to generate an efficient model by eliminating

Performance of the prediction model
The performance of all three models will be determined based on their discrimination and calibration. Discrimination will be evaluated based on the area under the receiveroperator characteristic (AUROC). Calibration will be assessed by plotting observed and expected probabilities and inspecting this plot graphically. Measures of calibration will 13 include calibration slope, calibration in the large, observed/expected (O/E) ratio, and the Brier score. 25 We will perform internal validation for all three models by using a bootstrap simulation. The resulting distribution will reflect optimism and the degree of overfitting. 26 The SIRS criteria, NEWS score, and qSOFA score will be calculated for all individuals in the TeSD-IT study. Diagnostic performance of the existing models will be determined by calculating the same measures of discrimination and calibration as described in the sections above, and comparing these with the three models that were developed.
To assess the added value of the prediction models on top of usual care, other outcomes than sepsis will be considered. This is crucial for gaining insight in the net benefit of using the CPRs in daily practice. For example, when a patient is predicted as non-sepsis by the model, but the patient was referred by the GP, improvement compared to care as usual is only the case if hospital treatment was not needed according to the expert panel. To assess the added value, the proportion of reclassifications within the original contingency tables will be presented for the following outcomes: 1) the gut feeling of the visiting GP that "something is wrong" 2) the assessment of the visiting GP for the likelihood of sepsis, and 3) the decision of the visiting GP whether or not to refer the patient to the hospital.

Cost-effectiveness and budget impact analysis
We will measure costs from a societal perspective, including health care costs and patient costs within and outside the hospital. Productivity costs will be ignored as the average age of patients participating will exceed the age of pensioning in the Netherlands. The patient questionnaire as well as follow-up data from hospital and GP medical records is used for the calculation of total and per patient costs. The EQ-5D scores retrieved from the questionnaire are used to calculate QALYs. Our patient outcome analysis will generate QALYs for different health states that will be used in health economic modelling, such as a complicated sepsis case (including ICU admission), hospital admittance for a suspicion of sepsis, and an infectious disease episode without hospital admission. Different scenarios with different levels of implementation of point-of-care testing (POCT) for sepsis in general practice will be analysed and compared to standard of care: 100% use of the best performing testing strategy; 70%, 30% and 0% use of POCT for suspicion of sepsis (the latter representing usual care). The budget impact will be assessed using the health economic model that will be build for the economic evaluation and results will be analysed in a probabilistic way.

Discussion
The TeSD-IT study is a diagnostic and prognostic study, designed for the development of a clinical prediction rule for the early recognition of sepsis in primary care. A limited number of nine clinical parameters and three blood tests were selected. This will enable us to construct the model using multivariable logistic regression techniques with 120 events included in the dataset. We have chosen to recruit patients in the setting of out-ofhours home visits performed by a GP. In this setting GPs frequently encounter seriously ill patients in whom they instantly have to decide whether or not to refer the patient (immediately) to the hospital. A methodological advantage is that the required sample size is substantially lower than in other primary care settings with a lower incidence of sepsis. However, external validation in other settings and populations will be needed before implementing the CPR more broadly.
The diagnosis of sepsis is not straightforward. In 2016, new consensus definitions for sepsis were published, which we try to implement are good as possible. As both the presence of infection as well as organ failure can be equivocal, we use expert panels to determine the final outcome. The outcome should be clinically relevant for the GP. The rationale of the timeframe of 72 hours is that patients who are found septic within this 15 period after GP assessment would likely benefit from immediate hospital referral. Not all patients with organ failure need hospital treatment to recover, and not all patients with severe infections that are treated with intravenous antibiotic therapy have signs of organ failure. However, we believe the diagnosis of sepsis based on the Sepsis-3 definitions is the most relevant endpoint for GPs to differentiate between patients who are likely to benefit from immediate hospital treatment and patients who might be treated at home successfully. The expert panel will also rate the need for hospital treatment for every patient, regardless of the diagnosis. This will enable us to evaluate the effect of the new CPR on medically unnecessary referrals.
The expert panels will be instructed to use the SOFA score (increase of ≥ 2 points from baseline due to infection) to define the primary outcome "sepsis within 72 hours of inclusion". This is consistent with the Sepsis-3 consensus definition and leads to an objective and reproducible endpoint in absence of a gold standard. However, this approach introduces the risk of incorporation bias. Parameters included in the SOFA score are more likely to be predictive of sepsis in our model. However, the blood tests (CRP, lactate and procalcitonin) are not included in the SOFA score, which limits the risk of incorporation bias for these tests. Furthermore, vital signs measured in the first 72 hours are used to calculate the SOFA score and not only at the time of inclusion.
To compare the performance of the new CPR with usual care, not only the decision to refer the patient to the ED, but also the assessment of the GP of the likelihood of sepsis on a scale from 0-10 and the presence of a "gut-feeling something is wrong" is used. We will examine if the CPR will outperform those assessments of the GP in order to likely improve the usual care.
Only three candidate blood tests were selected for the development of the CPR. Various other biomarkers have promising diagnostic and/or prognostic properties in patients with 16 suspected sepsis. 27− 28 At the start of this study, sufficient evidence of the additional diagnostic and/or prognostic value above lactate, CRP and PCT was lacking. However, sufficient blood samples are stored for retroactive testing of multiple additional biomarkers.
Although to our best knowledge no CPRs for sepsis in primary care were previously developed, several sepsis screening tools were published for the ambulance setting. 29 However, none of those have adequate inclusion criteria, data collection and clinically relevant endpoints for use in the primary care setting. The study was approved by the METC Utrecht, and is registered under number 18/169.
Written informed consent will be obtained before the collection of blood samples and after initial verbal informed consent. In mentally incapacitated patients, consent will be provided by the legal representative. In case written informed consent is not feasible prior to the collection of blood samples due to an acute life-threatening condition, the written consent will be asked as soon as possible afterwards. All collected data -including the blood results -will be stored without identifying information under a study number.
Personal details of the patient which are needed for the data collection will be store in an secured online environment until the data collection is completed. The data and excess blood samples will be stored for 15 years afterwards. Additional test of stored blood samples will only be done in patients who opted in in the written informed consent.

Availability of data and materials
Not applicable.

Funding
The study was financially supported by ZonMw, grant number 843001811. In kind support for material costs were provided by ThermoFischer (procalcitonin analyses), Nova Biomedical (POC lactate measurement) and Philips (portable monitoring devices). Neither funding party had a role in the design of this study, interpretation of the data, or decision to submit results.

Authors' contributions
FL contributed to the study conceptualisation and design, and drafted the manuscript. RH contributed to the study conceptualisation and design and revised the manuscript critically for important intellectual content. KJ and MS contributed to the design of the statistical analyses and co-drafted the manuscript. GF, AP, AB, JJO and AZ contributed to the study design and revised the manuscript critically for important intellectual content.TV supervised the study, contributed to the study design and revised the manuscript critically for important intellectual content. All authors read and approved the final manuscript and are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.