- Open Access
- Open Peer Review
Development of a clinical decision support tool for diagnostic imaging use in patients with low back pain: a study protocol
Diagnostic and Prognostic Researchvolume 3, Article number: 1 (2019)
Low back pain is one of the most common and disabling health problems in Canada and internationally. In most cases, low back pain is a benign, self-limiting condition that can be managed with little diagnostic investigation or treatment. Yet contrary to clinical practice guideline recommendations, diagnostic imaging (here meaning X-ray, MRI, CT) is commonly used in the assessment of low back pain. Diagnostic imaging is of limited value in most cases, exposing patients to unnecessary radiation and leading to increased health services use and worse patient health outcomes. The Choosing Wisely campaign has highlighted the need to reduce diagnostic imaging for low back pain; however, no clinical decision rules are available.
This project will develop a clinical decision support tool for appropriate use of diagnostic imaging for patients with low back pain in the emergency department. We will conduct a prospective cohort study at five Canadian emergency departments. The study will follow recommendations for prediction model development and testing. The study population will be 4000 patients presenting to the emergency department with low back pain. We will assess potential clinical indications of emergent-cause (i.e., “red flag” items), including clinical characteristics and past history. Our outcome, emergent-cause for low back pain such as fracture, cancer, infection, or cauda equina syndrome, will be assessed at discharge and at 1-, 3-, and 12-month follow-up periods using information from self-report and health administrative data. We will construct and assess the performance of a multivariable prediction model that has strong measurement properties, presented as a clinical decision support tool acceptable to knowledge users.
Practice guidelines describe “red flags” for which diagnostic imaging is likely appropriate. However, recommendations across guidelines are discordant, and few studies have evaluated these criteria to determine which characteristics best predict emergent etiology that warrant diagnostic imaging. A clinical decision support tool, that recommends diagnostic imaging where appropriate, has the potential to improve clinical care and patient outcomes and reduce costs associated with managing low back pain patients.
Low back pain is common and costly. Almost every Canadian suffers from low back pain at some point in their lives . It is a leading cause of disability nationally and globally (the number one cause in 2015 ) and results in enormous direct health care and lost productivity costs [3,4,5,6]. In Canada, it is one of the most common reasons for seeking health care: back pain is the third most common presenting complaint at emergency departments for individuals aged 20–64 years . There are approximately 360,000 visits to Canadian emergency departments for low back pain each year.
In most cases, low back pain is non-specific, that is, it is a benign, self-limiting condition that can be managed with little diagnostic investigation or treatment. Numerous systematic reviews and practice guidelines support the limited use of diagnostic imaging for the investigation and treatment of non-specific low back pain [8,9,10,11], recommending plain-film X-ray, MRI, or CT only in special circumstances for patients with low back pain when there are “red flag” indications . Red flags are clinical characteristics suggestive of underlying emergent cause, including potential fracture, cancer, infection, or cauda equina syndrome. These diagnoses are rare, occurring in less than 10% of low back pain cases [12, 13].
Overuse of diagnostic imaging for low back pain (here meaning X-ray, MRI, CT) results in wasted resources and harms. Contrary to recommendations from clinical practice guidelines, diagnostic imaging for non-specific low back pain remains high [14, 15]. A recent study by our team in a Nova Scotia emergency department found that 30% of patients presenting to the emergency department, diagnosed with non-specific low back pain, had received diagnostic imaging , much higher than the expected rate of 10%. Emery et al. reported that only 44% of requests for lumbar spine MRI were considered appropriate . Harms of inappropriate diagnostic imaging include exposing patients to unnecessary radiation, with lumbar spine X-ray having an exposure 75 times higher than that of a chest X-ray . Furthermore, diagnostic imaging for low back pain has been shown to result in more subsequent testing and increased use of health services , including surgery , with no improvement in patient outcomes. This is potentially related to high false positive rates for several recommended red flags [21, 22] and “coincidental” radiological findings [23, 24]. Importantly, patients with non-specific low back pain who receive diagnostic imaging have worse long-term health outcomes [25, 26]. While the overuse of diagnostic imaging is recognized as problematic, there are also potential harms, for example, a delayed emergent diagnosis and management that may be associated with not using diagnostic imaging when appropriate.
Clinical prediction models, or clinical decision support tools, can improve practice and patient outcomes. Clinical support tools, built from multivariable prediction models, statistically combine information about patient and disease characteristics to assist providers in making patient-specific and evidence-informed decisions at the point of care. They can be used to shift practice patterns and to educate providers and patients. Clinical decision support tools have been used successfully in many fields [27, 28], including informing the appropriate use of radiographs for ankle sprain (Ottawa Ankle Rules [29,30,31]), and the necessity for immobilization and diagnostic imaging in trauma patients with neck pain (Canadian C-Spine Rule [32, 33]). The Ottawa Ankle Rules have led to decreased use of ankle radiography, wait times, and healthcare costs .
Few studies have adequately investigated the use of red flags as diagnostic criteria for emergent-cause low back pain. There are discordant recommendations for diagnostic imaging across clinical practice guidelines, high false positive rates reported for several recommended red flags , and uncertainty about how characteristics should be combined to inform decision-making. Although diagnostic accuracy of red flags for emergent disease in conditions such as low back pain is a challenging research topic due to small event rate, particularly in primary care, the evaluation of the performance of combinations of red flags is both necessary and promising . Two earlier studies have attempted to develop decision tools for low back pain [21, 36]. These studies were conducted in primary care settings, limited by insufficient sample size, and a low number of emergent cases were identified. Moreover, they did not examine all emergent causes of low back pain that may warrant diagnostic imaging, instead only focusing on vertebral fractures.
The primary objective of this study is to develop a clinical decision support tool for the appropriate use of diagnostic imaging for patients with low back pain in the emergency department setting. The tool, once externally validated in a future study, will stratify patients by risk and accurately predict those patients presenting to the emergency department who are likely to have an emergent cause for their low back pain and therefore require imaging. Secondary study objectives are to provide prevalence estimates of low back pain with emergent-cause in the emergency department setting, and predictive accuracy data about currently recommended individual red flags and combinations of red flag screening items that should be recommended in clinical guidelines.
This cohort study of patients who present to the emergency department with a primary complaint of low back pain will prospectively collect primary data, develop and test a prediction model, and develop a clinical decision support tool through established methods [37,38,39]. We will select potential variables, develop the prediction model, and assess the model performance (overall goodness-of-fit, calibration, discrimination, and clinical usefulness). We will present the clinical decision support tool as a simple index. Future studies will test external validity and include impact assessment.
We will follow guidelines for the development of clinical decision support tools [40,41,42] and the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) checklist .
Study sites and participants
Data for this study will be collected from three urban tertiary care centers and two community emergency medicine sites located in Nova Scotia and Ontario, Canada, all of which provide 24-h, 7 days/week care. The studied emergency departments vary by density and patient volume, university affiliation, and the presence of a rapid assessment unit. Data collection from five different research-oriented locations will enhance geographic generalizability while ensuring feasibility. To assess generalizability, we will compare basic demographic characteristics of our eligible study population with that of the eligible population from the Canadian Institute for Health Information, National Ambulatory Care Reporting System, which represents approximately 63% of all emergency department visits in Canada.
The target population for our study is adults who present to the emergency department for low back pain. The sample will include all patients aged 18 and over with a valid provincial health plan number who present to one of the study sites’ emergency departments during the data collection period from approximately January 2019 to July 2020 with a primary complaint of low back pain (emergency department intake complaint codes: “back pain” or “traumatic back/spine injury”; we have previously observed both of these triage codes used for non-specific low back pain diagnoses); pain in the region bound by T12 and the lower gluteal fold, with or without pain referred beyond this region. We will exclude patients who are pregnant, those unable to understand English, and those with severe cognitive impairment that would negate consent.
The target recruitment for participants in our study is 4000 consecutive eligible patients, enrolled over a period of 15–18 months (approximately 50% from each province). Electronic notification at registration will alert triage staff to potentially eligible patients. Triage staff will subsequently give the patient the participant self-report survey and insert a physician data form into the patient chart for completion by the treating physician. Study procedures have been developed to optimize patient recruitment and minimize missing data, including integration of recruitment and data collection with the care process, raising healthcare team awareness of the study, training, and monitoring.
Candidate predictor variables
We will assess potential red flag characteristics suggestive of cancer, fracture, infection, or cauda equina syndrome, which are all emergent causes that may present as low back pain and warrant diagnostic imaging. We will consider as candidate predictor variable characteristics that have evidence of predictive accuracy (positive likelihood ratio > 5.0), are recommended by the American College of Physicians clinical practice guideline  or the American College of Radiology , and/or identified by clinical experts. Several recent systematic reviews and studies have guided our selection and measurement of potential items [13, 22, 45]. Potential items, specific to at least one of the four emergent diagnoses investigated, will include age, sex, medical history, clinical characteristics, and the results of immediately available tests (Additional file 1).
Data sources and collection
To develop a prediction model, we will collect participant characteristics (potential red flags and descriptive characteristics at the time of presentation to the emergency department (index visit)) and health outcomes (reference standard assessed over a 12-month follow-up period). To do this, we will use data from four sources: (1) physician data form (index visit), (2) emergency department clinical records (index visit), (3) participant surveys (index visit and follow-up), and (4) provincial administrative health “billing” data (follow-up) (Fig. 1).
Data collection at index visit (potential red flags and descriptive characteristics)
We will collect participant information using a simple healthcare provider survey (physician data form), emergency department clinical records, and a participant survey. The physician data form will be a single page, self-administered hard copy in patients’ charts and will collect data about the patient’s red flag characteristics as well as an item asking for the attending physician’s assessment of the likelihood that the patient’s low back pain has an emergent cause. Physicians will be trained to complete the standardized assessment prior to the start of the study. We will conduct a blinded inter-rater reliability testing of items in a subset of 100 participants assessed by two independent physicians (sample adequate for agreement ≥ 0.5; 0.2 relative error).
Emergency department clinical record data for all eligible participants will be retrieved from the Emergency Department Information System, used by all study emergency departments, and from related medical records (charts). We will use a pre-tested MS Access database to obtain Emergency Department Information System data and confirm and complete as necessary with patient clinical records. These data will enable us to characterize the participant population (“Descriptive Characteristics”, Fig. 1). The participant self-report survey will be completed in the emergency department during the index visit. The survey will collect information about baseline characteristics unavailable in the Emergency Department Information System (including back pain history and episode/health characteristics) for complete description of the study population and to allow comparison with other low back pain cohorts (Additional file 2).
Data collection at follow-up
Our primary outcome is confirmed by the diagnosis of emergent-cause low back pain, which is a composite outcome defined as a confirmed diagnosis of fracture, cancer, infection, or cauda equina syndrome. It is clinically relevant to assess this composite outcome since all component diagnoses are equally important to the emergency medicine clinician to identify a patient who has a cause for their low back pain that requires different management. We expect that fracture will be the most common component diagnosis (approximately 70% of emergent-cause low back pain diagnoses).
We will determine the presence of emergent-cause low back pain (fracture, cancer, infection, or cauda equina syndrome) in two ways to provide robustness to our results: (1) using combined information from emergency department clinical records and participant self-report, and (2) through provincial administrative health data, each with 12 months of follow-up (Additional file 3).
First, we will collect information about physician-assigned diagnosis at emergency department discharge and any records of return visits from emergency department clinical records and from assessor-blinded participant self-report survey administered by telephone at 3 and 12 months. For participant follow-up, we will use a computerized tracking system with service and support from the Canadian Longitudinal Study of Health and Aging (Halifax), which has a computer-assisted telephone interview site. Following Henschke et al. , participants will be asked, “Low back pain is occasionally the result of a fracture, infection or cancer. Has a healthcare provider said that your back pain is caused by one of these rare diseases?” and prompted for additional details about their diagnosis and investigations received. Multiple points of follow-up will limit recall bias, and a 12-month follow-up will allow sufficient time for emergence of potential emergent causes of low back pain. All participants diagnosed or reporting a new diagnosis of emergent-cause low back pain will have their diagnosis confirmed through their patient record (if consent is granted) or be referred to a study doctor who will confirm their diagnosis.
Second, we will use provincial administrative health data (physician billings) to identify new diagnoses in the 12 months following the index visit that are likely associated with the low back pain episode. In Nova Scotia, Health Data Nova Scotia, and in Ontario, Institute for Clinical Evaluative Sciences, maintain copies of provincial medical services insurance databases, which capture all information on billing claims for health services provided by practicing physicians in the province, which are linked to individual participants’ provincial health card numbers. Relevant diagnoses associated with emergent causes of low back pain will be defined using diagnostic codes. We will explore the robustness of our results with this administrative reference standard. A strength of this secondary analysis is that it will allow us to test and refine a reliable administrative definition of emergent-cause low back pain diagnosis by comparing with available participant self-report survey data, facilitating future external validation studies in other settings.
We will use three strategies to address potential for missing data: (1) study design and implementation methods to avoid missing data, (2) multiple sources of data to fill in missing data and to allow analyses to explore potential reasons and patterns of missing data, (3) describe missingness and testing assumptions about missingness to determine appropriate analyses, including complete case, and multiple imputation for key variables (red flags in the prediction model), with sensitivity analysis to test the impact of different assumptions about why data are missing for analyses.
All data collection instruments have been designed to reduce missingness in responses, both through the brevity of questions and the layout of the instruments. To ensure quality of data, we will train the emergency medicine physicians and other clinical staff about the study procedures, including patient and physician questionnaires. A challenge that we have anticipated is the completion of the physician checklist; we do not expect a substantial amount of missing data for individual red flag variables (since they are part of standard care), but rather fully non-completed physician checklists. Our methods will aim to limit this; however, in these cases, we do not expect that the reasons for missing red flag items will be related to the outcome (i.e., missing at random; limiting precision, but not introducing bias). In this situation, we will cautiously use a multiple imputation approach and compare these results to results in a complete case analysis. If multiple imputation and complete case analyses give different results, we will explore potential reasons for this .
We will follow steps recommended by Stiell and Wells for the development of a clinical decision/prediction rule , further informed by literature from the field of prediction model development [37, 42, 47] and clinical decision support systems . Our purpose is to develop an accurate and clinically useful multivariable model that can inform a clinical decision support tool.
Our preliminary selection of items for the model has been guided by available systematic reviews and clinical practice guidelines. We will prioritize variables that are easy to collect, have good face validity and clinical acceptance, and acceptable inter-rater reliability for inclusion in the model. To judge the most promising variables, we will consider univariate associations, correlation between variables, extent and patterns of missing data, inter-rater reliability (K > 0.8), and clinically acceptable combinations of items. In model development, we will consider continuous variables (as recommended to not lose information). We do not plan a priori to examine interaction terms.
We will build the prediction model, accounting for non-independence of data from our five study sites, by constructing a multivariable logistic regression model including the most promising variables. We will compare models that are theoretically driven (a priori fully specified) to those generated using backward selection (simplified models). We will cautiously estimate model coefficients using bootstrap sampling and lasso penalized regression methods. The final prediction model may be modified based on findings from the internal validation. We aim to develop a model that has strong measurement properties, that it is sensitive and sufficiently specific, and is clinically acceptable, logical, and easy to use.
Assess model performance
We will assess the overall performance of the resulting prediction models by calculating discrimination (c-index and receiver operating curves) and calibration (predicted vs. observed plots) [42, 49]. We will assess the ability of the models to classify patients into risk groups across a range of thresholds: less than 1%, 1 to < 5%, 5 to < 10%, and greater than 10%. We will quantify the clinical usefulness of the prediction models using the net benefit approach to balance benefits and harms, proposed by Vickers at al. .
All statistical analyses will be conducted using Stata and R.
Study sample size
The target sample of 4000 patients was calculated to have appropriate statistical power based on 10 events per variable for the outcome of interest, emergent-cause low back pain [42, 47], assuming 3–6% of participants experience the outcome event [35, 51, 52], and an analysis of 10 independent variables in our prediction model, accounting for 20% attrition at 12 months. Our expected 128 events will provide adequate sample size for internal validation .
Clinical decision support tool presentation
The final step of model development will be translation into a practical risk stratification approach or tool for potential use by healthcare providers . We will develop the clinical decision support tool from the prediction model, simplifying the regression coefficients to an easily calculated score. The practical tool will be presented along with expected probabilities of outcome and recommendations for diagnostic imaging and with sensitivity/specificity for low- versus high-risk groups. We will incorporate design considerations raised in the clinical decision support literature, in our team’s on-going related studies, and in stakeholder consultations.
Our methodological approaches will mitigate challenges specific to studying emergent-cause low back pain and in general for the development of prediction models/tools. Our prospective cohort design will allow inclusion of relevant predictors (credible, reliable, and measured appropriately) and limit missing values. We will include a large sample from a clinically relevant setting with high outcome prevalence to ensure sufficient sample size for adequate model development. Selection and attrition biases common to prospective studies will be mitigated using established recruitment protocols and support from emergency departments’ administration and staff for consecutive participation and expertise from the Halifax Canadian Longitudinal Study of Health and Aging site for complete follow-up; potential participation bias will be assessed using administrative data. A limitation of our study includes the lack of a single standard approach to identify our target condition (emergent-cause low back pain); however, we will use multiple approaches and compare physician diagnosis/patient self-report, with an administrative data defined measure.
This study will provide much needed evidence on the predictive value of individuals and combined red flags and will produce an internally valid, reliable, and acceptable clinical decision support tool to inform appropriate decisions on diagnostic imaging for patients presenting with low back pain (note that the tool is not intended to support decisions about which diagnostic imaging approach is indicated). Future research will test external validity of the tool in other settings, evaluate implementation, and study the impact of the tool on patient and health services outcomes; this is planned by our team in collaboration with national and international networks. The clear need for a clinical decision support tool and engagement of end-users throughout our study will ensure its usefulness and positive impact on clinical practice.
This project will result in new knowledge about the prevalence of emergent-cause of low back pain and the predictive accuracy of clinical characteristics to inform appropriate diagnostic imaging. The resulting clinical decision support tool will establish a foundation to improve the delivery of appropriate care for patients by enabling evidence-based decision-making, while seizing the opportunity to reduce unnecessary costs and harms and improving patient outcomes for this common health condition.
Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis
Balague F, Mannion AF, Pellise F, Cedraschi C. Non-specific low back pain. Lancet. 2012;379(9814):482–91.
GBD 2015 Disease and Injury Incidence and Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 310 diseases and injuries, 1990–2015: a systematic analysis for the Global Burden of Disease Study 2015. Lancet. 2016;388:1545–602.
Dagenais S, Caro J, Haldeman S. A systematic review of low back pain cost of illness studies in the United States and internationally. Spine J. 2008;8(1):8–20.
Hayden JA, Chou R, Hogg-Johnson S, Bombardier C. Systematic reviews of low back pain prognosis had variable methods and results: guidance for future prognosis reviews. J Clin Epidemiol. 2009;62(8):781–96.e1.
Lim SS, Vos T, Flaxman AD, Danaei G, Shibuya K, Adair-Rohani H, et al. A comparative risk assessment of burden of disease and injury attributable to 67 risk factors and risk factor clusters in 21 regions, 1990-2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet. 2012;380(9859):2224–60.
Martin BI, Deyo RA, Mirza SK, Turner JA, Comstock BA, Hollingworth W, et al. Expenditures and health status among adults with back and neck problems. JAMA. 2008;299(6):656–64.
Canadian Institute for Health Information. Emergency Department Highlights in 2014–2015: Information Sheet (Oct. 2015); 2015; Available from: https://secure.cihi.ca/free_products/NACRS_ED_QuickStats_Infosheet_2014-15_ENweb.pdf. Accessed 26 Sept 2016
Canadian Association of Radiologists. Referral Guidelines, Section C. Spine; 2012. Available from: https://car.ca/wp-content/uploads/Spine.pdf. Accessed 11 Oct 2016
Chou R, Qaseem A, Snow V, Casey D, Cross JT Jr, Shekelle P, et al. Diagnosis and treatment of low back pain: a joint clinical practice guideline from the American College of Physicians and the American Pain Society. Ann Intern Med. 2007;147(7):478–91.
Jarvik JG, Deyo RA. Diagnostic evaluation of low back pain with emphasis on imaging. Ann Intern Med. 2002;137(7):586–97.
Delitto A, George SZ, Van Dillen LR, Whitman JM, Sowa G, Shekelle P, et al. Low back pain. J Orthop Sports Phys Ther. 2012;42(4):A1–57.
Henschke N, Maher CG, Refshauge KM. Screening for malignancy in low back pain patients: a systematic review. Eur Spine J. 2007;16(10):1673–9.
Williams CM, Henschke N, Maher CG, van Tulder MW, Koes BW, Macaskill P, et al. Red flags to screen for vertebral fracture in patients presenting with low-back pain. Cochrane Database Syst Rev. 2013;1:CD008643.
Friedman BW, Chilstrom M, Bijur PE, Gallagher EJ. Diagnostic testing and treatment of low back pain in United States emergency departments: a national perspective. Spine. 2010;35(24):E1406–11.
McCaughey EJ, Li L, Georgiou A, Golding MH, Westbrook JI. Imaging for patients presenting to an emergency department with back pain: impact on patient pathway. Emerg Med Australas. 2016;28(4):412–8.
Nunn M, Edwards J, Hayden J, Kiberd J, Magee K. Prevalence & management practices for patients presenting to the emergency department with low back pain. Buxton: Presentation International Back & Neck Pain Forum; 2016.
Emery DJ, Shojania KG, Forster AJ, Mojaverian N, Feasby TE. Overuse of magnetic resonance imaging. JAMA Intern Med. 2013;173(9):823–5.
Fazel R, Krumholz HM, Wang Y, Ross JS, Chen J, Ting HH, et al. Exposure to low-dose ionizing radiation from medical imaging procedures. N Engl J Med. 2009;361(9):849–57.
Kendrick D, Fielding K, Bentley E, Miller P, Kerslake R, Pringle M. The role of radiography in primary care patients with low back pain of at least 6 weeks duration: a randomised (unblinded) controlled trial. Health Technol Assess. 2001;5(30):1–69.
Jarvik JG, Hollingworth W, Martin B, Emerson SS, Gray DT, Overman S, et al. Rapid magnetic resonance imaging vs radiographs for patients with low back pain: a randomized controlled trial. JAMA. 2003;289(21):2810–8.
Henschke N, Maher CG, Refshauge KM, Herbert RD, Cumming RG, Bleasel J, et al. Prevalence of and screening for serious spinal pathology in patients presenting to primary care settings with acute low back pain. Arthritis Rheum. 2009;60(10):3072–80.
Downie A, Williams CM, Henschke N, Hancock MJ, Ostelo RW, de Vet HC, et al. Red flags to screen for malignancy and fracture in patients with low back pain: systematic review. BMJ. 2013;347:f7095.
Jensen MC, Brant-Zawadzki MN, Obuchowski N, Modic MT, Malkasian D, Ross JS. Magnetic resonance imaging of the lumbar spine in people without back pain. N Engl J Med. 1994;331(2):69–73.
You JJ, Purdy I, Rothwell DM, Przybysz R, Fang J, Laupacis A. Indications for and results of outpatient computed tomography and magnetic resonance imaging in Ontario. Can Assoc Radiol J. 2008;59(3):135–43.
Ash LM, Modic MT, Obuchowski NA, Ross JS, Brant-Zawadzki MN, Grooff PN. Effects of diagnostic information, per se, on patient outcomes in acute radiculopathy and low back pain. AJNR Am J Neuroradiol. 2008;29(6):1098–103.
Deyo RA. Real help and red herrings in spinal imaging. N Engl J Med. 2013;368(11):1056–8.
Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012;157(1):29–43.
Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765.
Stiell I, Wells G, Laupacis A, Brison R, Verbeek R, Vandemheen K, et al. Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries. Multicentre Ankle Rule Study Group. BMJ. 1995;311(7005):594–7.
Stiell IG, Greenberg GH, McKnight RD, Nair RC, McDowell I, Reardon M, et al. Decision rules for the use of radiography in acute ankle injuries. Refinement and prospective validation. JAMA. 1993;269(9):1127–32.
Stiell IG, McKnight RD, Greenberg GH, McDowell I, Nair RC, Wells GA, et al. Implementation of the Ottawa ankle rules. JAMA. 1994;271(11):827–32.
Stiell IG, Clement CM, Grimshaw J, Brison RJ, Rowe BH, Schull MJ, et al. Implementation of the Canadian C-Spine Rule: prospective 12 centre cluster randomised trial. BMJ. 2009;339:b4146.
Stiell IG, Wells GA, Vandemheen KL, Clement CM, Lesiuk H, De Maio VJ, et al. The Canadian C-spine rule for radiography in alert and stable trauma patients. JAMA. 2001;286(15):1841–8.
Anis AH, Stiell IG, Stewart DG, Laupacis A. Cost-effectiveness analysis of the Ottawa ankle rules. Ann Emerg Med. 1995;26(4):422–8.
de Schepper EI, Koes BW, Oei EH, Bierma-Zeinstra SM, Luijsterburg PA. The added prognostic value of MRI findings for recovery in patients with low back pain in primary care: a 1-year follow-up cohort study. Eur Spine J. 2016;25(4):1234–41.
Roman M, Brown C, Richardson W, Isaacs R, Howes C, Cook C. The development of a clinical decision making algorithm for detection of osteoporotic vertebral compression fracture or wedge deformity. J Man Manip Ther. 2010;18(1):44–9.
Steyerberg EW, Moons KGM, van der Windt DA, Hayden JA, Perel P, Schroter S, et al. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381.
Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014;35(29):1925–31.
Riley RD, Hayden JA, Steyerberg EW, Moons KG, Abrams K, Kyzas PA, et al. Prognosis Research Strategy (PROGRESS) 2: prognostic factor research. PLoS Med. 2013;10(2):e1001380.
Stiell IG, Wells GA. Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med. 1999;33(4):437–47.
Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med. 2006;144(3):201–9.
Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–38.
Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD). Ann Intern Med. 2015;162(10):735–6.
Patel ND, Broderick DF, Burns J, Deshmukh TK, Fries IB, Harvey HB, et al. ACR Appropriateness Criteria Low Back Pain. J Am Coll Radiol. 2016;13(9):1069–78.
Henschke N, Maher CG, Ostelo RW, de Vet HC, Macaskill P, Irwig L. Red flags to screen for malignancy in patients with low-back pain. Cochrane Database Syst Rev. 2013;2:CD008686.
Sterne JA, White I, Carlin JB, Spratt M, Royston P, Kenward MG, Wood AM, Carpenter JR. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338:b2393.
Royston P, Ambler G, Sauerbrei W. The use of fractional polynomials to model continuous risk variables in epidemiology. Int J Epidemiol. 1999;28(5):964–74.
Khorasani R, Hentel K, Darer J, Langlotz C, Ip IK, Manaker S, et al. Ten commandments for effective clinical decision support for imaging: enabling evidence-based practice to improve quality and reduce waste. AJR Am J Roentgenol. 2014;203(5):945–51.
Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol. 2016;74:167–76.
Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ. 2016;352:i6.
Enthoven WT, Geuze J, Scheele J, Bierma-Zeinstra SM, Bueving HJ, Bohnen AM, et al. Prevalence and “red flags” regarding specified causes of back pain in older adults presenting in general practice. Phys Ther. 2016;96(3):305–12.
Edwards J, Hayden J, Gregoire B, Asbridge M, Magee K. The prevalence of low back pain in emergency settings: a systematic review and meta-analysis. BMC Musculoskelet Disord. 2017;18:143.
Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. 2016;35(2):214–26.
Agency for Healthcare Research and Quality - AHRQ. Health Information Technology. Best practices transforming quality, safety, and efficiency: clinical decision support (CDS); c2015; Available from: https://healthit.ahrq.gov/ahrq-funded-projects/clinical-decision-support-cds. Accessed 26 Sept 2016
The authors would like to thank Ms. Andrea Smith for her contributions to the development of the study protocol.
Funding for this research has been provided by the Canadian Institutes of Health Research (CIHR) through their Project Grant competition (#374823).
Availability of data and materials
Handling and peer review of study protocols
This protocol has undergone review by our funder (Canadian Institutes of Health Research) and the Nova Scotia Health Authority Research Ethics Board.
Ethics approval and consent to participate
Ethical approval has been obtained by the Nova Scotia Health Authority (NSHA) Research Ethics Board (ROMEO File # 1023090).
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.