A generic nomogram for multinomial prediction models: theory and guidance for construction
 Maarten van Smeden^{1}Email authorView ORCID ID profile,
 Joris AH de Groot^{1},
 Stavros Nikolakopoulos^{1},
 Loes CM Bertens^{2},
 Karel GM Moons^{1} and
 Johannes B. Reitsma^{1}
DOI: 10.1186/s4151201700105
© The Author(s) 2017
Received: 9 September 2016
Accepted: 13 January 2017
Published: 10 April 2017
Abstract
Background
The use of multinomial logistic regression models is advocated for modeling the associations of covariates with three or more mutually exclusive outcome categories. As compared to a binary logistic regression analysis, the simultaneous modeling of multiple outcome categories using a multinomial model often better resembles the clinical setting, where a physician typically must distinguish between more than two possible diagnoses or outcome events for an individual patient (e.g., the differential diagnosis). A disadvantage of the multinomial logistic model is that the interpretation of its results is often complex. In particular, the calculation of predicted probabilities for the various outcomes requires a series of careful calculations. Nomograms are widely used in studies reporting binary logistic regression models to facilitate the interpretation of the results and allow the calculation of the predicted probability for individuals.
Methods and results
In this paper we outline an approach for deriving a generic nomogram for multinomial logistic regression models and an accompanying scoring chart that can further simplify the calculation of predicted multinomial probabilities. We illustrate the use of the nomogram and scoring chart and their interpretation using a clinical example.
Conclusions
The generic multinomial nomogram and scoring chart can be used irrespective of the number of outcome categories that are present.
Keywords
Prediction Nomogram Graphical presentation Multinomial outcomes Logistic model Scoring chartBackground
The use of multinomial logistic regression modeling has been encouraged to study multiple unordered outcome categories (and possibly their combination) simultaneously [1, 2]. The multinomial logistic model can be considered to be an extension of the popular binary logistic regression model, which is often used in the presence of two mutually exclusive outcome categories. More specifically, multinomial logistic regression analysis can be viewed as a series of binary logistic regression analyses where one of the outcome categories is the reference category in each binary submodel.
Researchers often collapse multiple outcome categories into a single binary (composite) outcome and use binary logistic regression models to analyze their data. Consequently, potentially important information about the different outcome categories is lost. It can also be argued that a binary outcome often does not accurately reflect clinical practice, where physicians commonly have to make decisions while considering more than two relevant choices. For instance, physicians often consider the presence of differential diagnoses (and prognoses) for an individual patient simultaneously [1].
One of the key reasons why researchers might refrain from multinomial logistic regression analysis is that the results from these models are more complex to interpret and more elaborate than results from a binary logistic regression analysis. In the multinomial context, regression coefficients are estimated for each binary submodel reflecting the relation of covariates to one outcome category relative to the reference category. The number of estimated parameters quickly increases with additional outcome categories considered. The large amount of information from multinomial models can easily overwhelm researchers and clinicians. In addition, when using the multinomial logistic model for estimating probabilities for individual patients, the computation involved for the various outcomes requires a series of careful calculations.
Nomograms found many applications in the reporting of binary logistic regression models (for recent examples, see [3, 4]). A nomogram can not only improve insights of clinicians into the results of a logistic model, it can also be used to arrive at a predicted probability of outcome(s) of interest that is (are) tailored to the profile of an individual patient in a graphical manner. Nomograms can thus facilitate clinical decision making during clinical encounters. So far, nomograms have been used primarily for improving the reporting of models with only two outcome categories. We are aware of one recent paper that reported on the construction and one on the application of a nomogram for multinomial models. However, the focus of these papers was limited to constructing a nomogram for a limited number of outcome categories [5] and the reporting of results of one specific dataset [6].
In this manuscript, we present how to construct, interpret, and use a generic nomogram for a multinomial logistic prediction model. We will first specify the multinomial regression model and then present a general approach for deriving the nomogram and accompanying scoring chart for such models irrespective of the number of outcome categories that are present. We will illustrate the use of the nomogram and its interpretation using a clinical example on the risk of operative delivery [7].
Multinomial logistic model
Methods: Constructing the multinomial nomogram
To simplify notation, in the following we let o _{ ik }= lnπ _{ k }(x _{ i }), l _{ ik }=lp_{ k }(x _{ i }) and \(s_{i} =  \ln \left [ {1+\sum _{p=1}^{K1}}\exp \left \{\text {lp}_{p}(\boldsymbol {x}_{i})\right \}\right ]\).
The parallel scale nomogram makes use of the relation: o _{ ik }=l _{ ik }+s _{ i }. Each of these three elements in this relation corresponds to one of the three vertical axes of the nomogram. The axes are denoted by L (left axis), O (middle axis), and S (right axis). Axis L is a scaled function of linear predictor k,m _{1} l _{ ik }, where m _{1} is the scaling factor. Axis O corresponds to the probability of observing category k,m _{2} o _{ ik }. Lastly, axis S is a scaled function of the sum of exponentiated linear predictors, m _{3} s _{ i }. Axes O and S are on the natural log scale.
Step 1: placing the outer axes (L and S)
To obtain an adequately sized nomogram, determine the desired common height (h) for the outer two axes (L and S) and the horizontal distance between them (d). The two parallel axes are placed in the vertical direction. The values for h and d are assigned at the discretion of the researcher in a common metric (e.g., centimeters or inches). Larger values for h and d will allow for more precise reading of values.
Step 2: determine the scaling factors m _{ 1 }, m _{ 2 }, m _{ 3 }
For determining the scaling factors for the outer axes (m _{1} and m _{3}), the relevant ranges of l _{ ik } and s _{ i } need to be considered. The limits of these ranges (l ^{low},l ^{up},s ^{low},s ^{up}) may be determined by the range observed in the data set where the model was developed, e.g., \(l^{\text {low}} = \text {min}(\hat {l}_{ik})\) and \(l^{\text {up}} = \text {max}(\hat {l}_{ik})\). These limits define the corresponding limits of the axes. Once these ranges are chosen, the scaling factors m _{1} and m _{3} are computed by m _{1}=h/(l ^{up}−l ^{low}) and m _{3}=h/(s ^{up}−s ^{low}). The remaining scaling factor is given by m _{2}=m _{1} m _{3}/(m _{1}+m _{3}).
Step 3: placing the middle axis (O)
The O axis is placed parallel to the outer axes. The horizontal distance between the axes L and O is given by d _{ LO }=d−d/(m1/m3+1).
Step 4: placing tick marks and labels
For the outer axes L and S, two sequences of values for the tick marks and corresponding labels are defined: l _{ T }=(l ^{low},…,l _{ t },…,l ^{up}) and s _{ T }=(s ^{low},…,s ^{ t },…,s ^{up}). Tick mark t on the axis should be placed relative to the lower end of that axis at a distance of m _{1}×(l _{ t }−l ^{low}) for axis L and m _{3}×(s _{ t }−l ^{low}) for axis S. For axis O, we define: o ^{low}= exp{l ^{low}+s ^{low}}. Because the axis O represents a log transformed probability scale, first a sequence of arithmetic probabilities \(o^{*}_{T}\) is defined with values between exp{o ^{low}} and 1. Then, tick mark t for this sequence may be placed at \(m_{2} \times (\ln (o^{*}_{t})  f(o^{\text {low}}))\) labeled by \(o^{*}_{T}\). Axis S is labeled by \(\sum _{p=1}^{K1}\exp \{\text {lp}_{p}(\boldsymbol {x}_{i})\}\).
Methods: Constructing the scoring chart
The use of the nomogram by health professionals can be improved by additionally presenting a scoring chart. This scoring chart provides a graphical approach to arriving at the values for the two outer axes L and S of the nomogram for any relevant combination of values on the covariates (x _{ i }). For brevity, in this section we only consider the case of a multinomial logistic regression model with first order main effects. The scoring chart (and nomogram) can be extended to accommodate situations where higher order and interaction effects are present.
To make the scoring chart userfriendly, the individual effects of covariate j, (\(\hat {\beta }_{jk}x_{ij}\)), that make up the linear predictor k are rescaled to a “standardized” score. The sum over these individual effects together with a baseline score make up a “standardized” total score. This total score is a linear transformation of l _{ ik }. To facilitate the applicability of this standardized total score approach to the nomogram, the scaling of axis L should be adjusted accordingly. Below we detail the threestep procedure to arrive at the scoring system that makes up the scoring chart.
Step 5: standardized covariate effects: points
The estimated multinomial logistic regression coefficients, \(\hat {\beta }_{jk}\), are rescaled relative to the largest (conditional) covariate effect on a scale that has a minimum of 0 and a maximum of 100. First, the relevant ranges for each of the covariate variables are considered. Let the boundaries of these relevant ranges be denoted: \({x}^{\text {low}}_{j}\) and \({x}^{\text {up}}_{j}\). The rescaling factor and rescaled coefficients are then computed by \(r = 100/{\smash {\max _{j,k}}}( \hat {\beta }_{jk} {x}^{\text {up}}_{j}  \hat {\beta }_{jk} {x}^{\text {low}}_{j})\) and \(\hat {\beta }^{*}_{jk} = r \times \hat {\beta }_{jk}\). The covariate effects are “standardized” by \(\text {Points}_{jk}(x_{ij}) = \hat {\beta }^{*}_{jk}x_{ij}  \min (\hat {\beta }^{*}_{jk}{x}^{\text {up}}_{j},\hat {\beta }^{*}_{jk}{x}^{\text {low}}_{j})\).
Step 6: standardized total effect: total points
A baseline score for each category (except the reference category) is defined that takes into account the standardization that has been performed at step 5. The baseline score is computed by \(bl_{k} = r \times \hat {\alpha }_{k} + \sum _{j} \min (\hat {\beta }^{*}_{jk}{x}^{\text {up}}_{j},\hat {\beta }^{*}_{jk}{x}^{\text {low}}_{j})\). To also obtain a “standardized” baseline score such that the minimum rescaled baseline score is zero, we subtract the minimum baseline score, \(bl_{k}^{*}= bl_{k}  {\smash {\min _{k}}}(bl_{k})\). The standardized total effect for category k given covariate values is then given by Total\(_{k} = bl_{k}^{*} + \sum _{j} \text {Points}_{jk}(x_{ij})\). Notice that \(l_{ik}=\text {lp}_{k}(\boldsymbol {x}_{i}) =({Total}_{k} + {\smash {\min _{k}}}(bl_{k}))/r\).
Step 7: connecting the standardized total effects to the S axis
A horizontal axes representing Total _{ k } is placed near the lower end of the scoring chart. Another parallel horizontal axis is placed: the values on this axis are related to the former axis by \(\exp \{({Total}_{k} + {\smash {\min _{k}}}(bl_{k}))/r\}\). Taking the sum over the values that can be read off from the axis for all categories (except the reference category) is all the information necessary for determining the value on the S axis.
Result and discussion
Empirical example: predicting the risk of operative delivery
To illustrate the suggested scoring chart and nomogram for the reporting of multinomial logistic models, we use a previously published model on predicting the risk of operative delivery [7]. We detail their use by considering a specific hypothetical subject (described below). The prediction model used in this illustration was developed using data from a randomized clinical trial conducted in the Netherlands [9].
Multivariable associations for multinomial antepartum prediction model, predicting the risk of operative delivery
IVDFD vs spont.  CSFD vs spont.  IVDFTP vs spont.  CSFTP vs spont.  

\(\hat {\beta }_{jk}\)  OR(95% CI)  \(\hat {\beta }_{jk}\)  OR(95% CI)  \(\hat {\beta }_{jk}\)  OR(95% CI)  \(\hat {\beta }_{jk}\)  OR(95% CI)  
Intercept  −13.1  −15.6  −11.1  −15.4  
Maternal age, years  0.029  1.03 (1.01, 1.05)  0.052  1.05 (1.02, 1.09)  0.054  1.06 (1.03, 1.08)  0.056  1.06 (1.04, 1.08) 
Gestational age, weeks  0.26  1.29 (1.18, 1.41)  0.32  1.38 (1.22, 1.56)  0.038  1.04 (0.95, 1.13)  0.13  1.14 (1.05, 1.24) 
Nulliparous  2.05  7.79 (5.26, 11.5)  1.13  3.09 (2.09–4.55)  3.39  29.7 (17.2–51.1)  2.65  14.1 (9.78–20.3) 
Previous caesarean delivery  1.77  5.87 (3.70, 9.32)  1.06  2.88 (1.74, 4.76)  2.39  10.9 (5.92, 20.1)  2.23  9.34 (6.17, 1.41) 
Neonatal female gender  −0.19  0.83 (0.67, 1.03)  −0.5  0.61 (0.45, 0.83)  −0.25  0.78 (0.63, 0.96)  −0.013  0.99 (0.81, 1.20) 
Birthweight, 100g increments  −0.059  0.94 (0.92, 0.97)  −0.079  0.92 (0.89, 0.96)  0.083  1.09 (1.06, 1.11)  0.12  1.12 (1.10, 1.15) 
Maternal diabetes mellitus  0.32  1.37 (0.65, 2.91)  0.99  2.69 (1.29, 5.60)  −0.24  0.79 (0.35, 1.76)  0.87  2.38 (1.44, 3.95) 
Using the scoring chart (Fig. 2), each subject characteristic can be converted into a score for a particular outcome category (except the reference category). We have illustrated this procedure by drawing vertical dashed lines between the axes only for the subject characteristic “maternal age” (32 years) to the Points axis (upper end scoring chart) in the figure. We leave it to the interested reader to draw the lines for the remaining characteristics. By adding the points of the separate predictors to the baseline points for each of the chosen outcomes, one obtains the total points for this particular outcome. Each of these total points should be marked at the line indicated by Total Points (lower end scoring chart). For the considered subject, the total points are 159 (IVDFD), 140 (CSFD), 149 (IVDFTP), and 189 (CSFTP). Further, by drawing vertical lines from the marked points on Total Points axis to the Exp(l _{ ik }) axis (illustrated using dashed lines), one can readoff the exponentiated linear predictor for each of the outcome categories. This information is then put in the boxes right under the Exp(l _{ ik }) axis. Finally, by taking the sum over these values (registered under box “S”) one obtains all information needed to use the nomogram (Fig. 3).
To use the nomogram (Fig. 3), one first marks on the Total Points axis (left axis) the total points for each outcome category except the reference category, as derived from the scoring chart. One than marks the value registered under box “S” in the scoring chart on the S axis of (Fig. 3). The points marked on the Total points axis and point on the S axis are then connected using straight lines. From the middle “O” axis, one can now readoff the predicted probabilities for each of the four outcomes for this particular case. Finally, by subtracting these four probabilities from one, the probability of the reference category is obtained, corresponding to the probability of a “spontaneous delivery”.
Conclusion
Nomograms are not likely to be used very often in contemporary clinical practice for calculating probabilities for individual patients given the considerable burden placed on the user and also because its accuracy is limited by the precision with which physical markings can be drawn, reproduced, viewed, and aligned. For this purpose, electronic implementation of a prediction model by means of a website calculator, mobile app, or an algorithm implemented into the electronic medical file of the patients to directly calculate the relevant probabilities may be considered. However, we still think that nomograms remain quite useful, even today, as they do offer great visual insight a (multinomial) prediction model, which is often perceived as a blackbox (for a further discussion on the usefulness of nomograms we refer to [10, 11]). Therefore, we do expect that our general approach to construct and report scoring charts and nomograms for multinomial logistic regression models will facilitate the interpretation and use of such models to a wider audience. Our approach is flexible and generalizable and can be used irrespective of the number of outcome categories and types of covariates present. Rcode for developing the nomogram is available from http://mvansmeden.net/software/multinomialnomogram.
Abbreviations
 CS:

Caesarean section
 FD:

Fetal distress
 FTP:

Failure to progress
 IVD:

Instrumental vaginal delivery
Declarations
Acknowledgements
Not applicable.
Funding
We gratefully acknowledge financial contribution from the Netherlands Organisation for Scientific Research (project 918.10.615).
Availability of data and materials
Not applicable.
Authors’ contributions
MvS, JdG, JR, LB, and KM drafted the concept for the current paper. MvS wrote the initial version of the paper. SN contributed to the statistical notation. All authors commented on drafts of the article and approved the manuscript.
Competing interests
The authors declare that they have no competing interests.
Consent for publication
Not applicable.
Ethics approval and consent to participate
Not applicable.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License(http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Authors’ Affiliations
References
 Biesheuvel CJ, Vergouwe Y, Steyerberg EW, Grobbee DE, Moons KGM. Polytomous logistic regression analysis could be applied more often in diagnostic research. J Clin Epidemiol. 2008; 61(2):125–34.View ArticlePubMedGoogle Scholar
 Begg CB, Gray R. Calculation of Polychotomous Logistic Regression Parameters Using Individualized Regressions. Biometrika. 1984; 71(1):11.View ArticleGoogle Scholar
 Ogura K, Fujiwara T, Yasunaga H, Matsui H, Jeon DG, Cho WH, Hiraga H, Ishii T, Yonemoto T, Kamoda H, et al. Development and external validation of nomograms predicting distant metastases and overall survival after neoadjuvant chemotherapy and surgery for patients with nonmetastatic osteosarcoma: A multiinstitutional study. Cancer. 2015; 121(21):3844–52.View ArticlePubMedPubMed CentralGoogle Scholar
 Aviram G, Shmueli H, Adam SZ, Bendet A, ZivBaran T, Steinvil A, Berliner AS, Nesher N, BenGal Y, Topilsky Y. Pulmonary Hypertension: A Nomogram Based on CT Pulmonary Angiographic Data for Prediction in Patients without Pulmonary Embolism. Radiology. 2015; 277(1):236–46.View ArticlePubMedGoogle Scholar
 Ardoino I, Lanzoni M, Marano G, Boracchi P, Sagrini E, Gianstefani A, Piscaglia F, Biganzoli EM. Widen NomoGram for multinomial logistic regression: an application to staging liver fibrosis in chronic hepatitis C patients. Stat Methods Med Res. 2014. [Epub].
 Bertens LCM, Moons KGM, Rutten FH, van Mourik Y, Hoes AW, Reitsma JB. A nomogram was developed to enhance the use of multinomial logistic regression modeling in diagnostic research. J Clin Epidemiol. 2016; 71:51–71.View ArticlePubMedGoogle Scholar
 Schuit E, Kwee A, Westerhuis M, Van Dessel H, Graziosi G, Van Lith J, Nijhuis J, Oei S, Oosterbaan H, Schuitemaker N, et al. A clinical prediction model to assess the risk of operative delivery. BJOG. 2012; 119(8):915–23.View ArticlePubMedGoogle Scholar
 Doerfler R. On JargonThe Lost Art of Nomography. UMAP J. 2009; 30(4):457.Google Scholar
 Westerhuis MEMH, Visser GHA, Moons KGM, van Beek E, Benders MJ, Bijvoet SM, van Dessel HJHM, Drogtrop AP, van Geijn HP, Graziosi GC, et al. Cardiotocography Plus ST Analysis of Fetal Electrocardiogram Compared With Cardiotocography Only for Intrapartum Monitoring. Obstet Gynecol. 2010; 115(6):1173–80.View ArticlePubMedGoogle Scholar
 Marshall RJ. Are regression “nomograms” useful?. J Clin Epidemiol. 2016; 78:4–6.View ArticlePubMedGoogle Scholar
 Reitsma JB, van Smeden M, de Groot JAH, Bertens LCM, Moons KGM. Response to the commentary on “A nomogram was developed to enhance the use of multinomial logistic regression modelling in diagnostic research”. J Clin Epidemiol. 2016; 78:7–9.View ArticlePubMedGoogle Scholar