Development and internal validation of a model predicting patient-reported shoulder function after arthroscopic rotator cuff repair in a Swiss setting
Diagnostic and Prognostic Research volume 7, Article number: 21 (2023)
Prediction models for outcomes after orthopedic surgery provide patients with evidence-based postoperative outcome expectations. Our objectives were (1) to identify prognostic factors associated with the postoperative shoulder function outcome (the Oxford Shoulder Score (OSS)) and (2) to develop and validate a prediction model for postoperative OSS.
Patients undergoing arthroscopic rotator cuff repair (ARCR) were prospectively documented at a Swiss orthopedic tertiary care center. The first primary ARCR in adult patients with a partial or complete rotator cuff tear were included between October 2013 and June 2021. Thirty-two potential prognostic factors were used for prediction model development. Two sets of factors identified using the knowledge from three experienced surgeons (Set 1) and Bayesian projection predictive variable selection (Set 2) were compared in terms of model performance using R squared and root-mean-squared error (RMSE) across 45 multiple imputed data sets using chained equations and complete case data.
Multiple imputation using data from 1510 patients was performed. Set 2 retained the following factors: American Society of Anesthesiologists (ASA) classification, baseline level of depression and anxiety, baseline OSS, operation duration, tear severity, and biceps status and treatment. Apparent model performance was R-squared = 0.174 and RMSE = 7.514, dropping to R-squared = 0.156, and RMSE = 7.603 after correction for optimism.
A prediction model for patients undergoing ARCR was developed using solely baseline and operative data in order to provide patients and surgeons with individualized expectations for postoperative shoulder function outcomes. Yet, model performance should be improved before being used in clinical routine.
Rotator cuff tear is one of the most frequent orthopedic disorders . For patients undergoing arthroscopic rotator cuff repair (ARCR), improvement of shoulder function is one of the main reasons to undergo surgery alongside reduction of shoulder pain or return to sports activities .
Outcomes prediction after medical interventions has become a topic of growing interest with the use of prediction models . Indeed, such prediction models can provide patients and surgeons with individualized and evidence-based predictions of surgery success (e.g., by predicting the postoperative shoulder function status or the probability of occurrence of adverse events), supporting healthcare in the decision-making process. Patients may have strong expectations, which should best be tailored to their own health profile and injury characteristics . Such prediction models require prospective and representative outcome data of high-quality and parsimonious development .
Despite the rising interest, it is still unclear which prognostic factors are associated with perceived shoulder function outcomes after ARCR [6,7,8,9,10,11,12]. The lack of both prospectively collected data and proper methodology to develop and report multivariable prediction models have dramatically impaired the strength of the underlying evidence . The implementation of a local register at a tertiary care clinic in 2013 was the first step towards better documentation and understanding of health outcome data after ARCR . This register has laid the foundation for carrying out a large national ARCR study initiated in 2020 . Both initiatives are set to predict key health outcome data after ARCR such as that recently established for the occurrence of highly prevalent adverse events, such as postoperative shoulder stiffness .
Our objectives are (1) to identify potential prognostic factors associated with the postoperative Oxford Shoulder Score (OSS) and (2) to develop and validate a prediction model for postoperative OSS.
The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement was used as a reporting guideline .
Source of data
From January 2010, patients undergoing ARCR in a Swiss tertiary orthopedic clinic were prospectively documented in a register . At the clinic, patient follow-up at 6-month post-surgery comprised various objective and subjective patient-reported outcomes including the Constant-Murley score, Oxford Shoulder Score (OSS), and subjective shoulder value (SSV). The scales were filled out on paper form or via e-mail the week after the clinical examination. An additional 2- to 4-year postoperative evaluation of patient-reported outcomes and level of satisfaction was made by postal questionnaire. All parameters were collected in clinical report forms after the baseline clinical examination or immediately after surgery. Quality checks and data management were done using a Research Electronic Data Capture (REDCap) database .
Adult patients were included if (1) they had a partial or complete rotator cuff tear that was assessed by magnetic resonance imaging (MRI) and confirmed intraoperatively and (2) underwent ARCR between October 2013 and June 2021. Revision surgeries were excluded as well as contralateral ARCRs in patients with bilateral injuries.
Treatment and rehabilitation
Shoulder arthroscopy was performed according to internationally standardized procedures with patients in a beach-chair position under general anesthesia . All patients followed a standard 3-phase postoperative physical therapy protocol involving the following: (1) 6 weeks of passive mobilization with an abduction brace (DonJoy UltraSling ER; ORMED GmbH, Freiburg, Germany), (2) 4 to 6 weeks of active-assisted mobilization and coordination training, and (3) specific progressive resistance exercises for the operated shoulder.
For prediction modeling of 6-month outcome, we focused on the OSS, which is a 12-item patient-reported outcome assessing daily functional activities in relation to use of the shoulder . The OSS is a condition-specific questionnaire developed for patients with a degenerative or inflammatory shoulder condition including rotator cuff injuries. The twelve items or questions are answered by the patient independently and address the degree of pain and possible handicaps experienced during activities of daily living within the last 4 postoperative weeks. There are five response categories for each question corresponding to a score ranging from 0 to 4. All scores are then combined to produce a final score ranging from 0 (worst outcome) to 48 (best outcome).
A list of 37 prognostic factors was generated by the primary and senior authors (T. S., L. A.) based on previous systematic reviews [6,7,8,9,10,11,12]. Three experienced orthopedic shoulder surgeons (A. M., M. F., M. S.) were asked to independently assess the importance of each prognostic factor using a scale ranging from 0 (not important) to 5 (very important) for the prediction of the 6-month OSS. All 37 factors were documented and available in the local register. We excluded five potential prognostic factors describing redundant information (baseline pain-related question from the Constant-Murley score, general health status, rotator cuff tear pattern, the extent of the rotator cuff tear, and the baseline pain-related question from a visual analog scale).
Thirty-two potential prognostic factors were finally retained for development of the prediction model including the following: 18 patients and disease-related parameters collected at baseline (age at surgery, sex, body mass index, American Society of Anesthesiologists (ASA) physical status classification, dominance of affected side, smoking status at surgery, three preoperative treatment variables (medication, physiotherapy, and steroid infiltrations), traumatic onset, symptom duration, level of depression and anxiety using the European Quality-of-Life 5 Dimensions 5 Level (EQ-5D-5L) scale  and baseline functional scales (OSS, Constant-Murley score, range of parameters (flexion, abduction, external rotation)), and muscle strength) and 14 operative findings and details collected during the surgery (supraspinatus tear, subscapularis tear, infraspinatus tear, tear severity, level of fatty infiltration, tendon degeneration, tendon delamination, operation duration, number of anchors used, number of threads used, acromioclavicular joint resection, acromioplasty, capsulotomy, and biceps tendon status and treatment).
Post hoc sample size calculation
An R package developed by Riley et al.  was used to estimate the necessary sample size for the development of a multivariable prediction model for continuous outcomes (pmsampsize package). The estimation required an expected R-squared value for the future prediction model of 0.2, a shrinkage factor of 0.9, a multiplicative margin of error of 1.1, the number of parameters to be assessed during the multivariable prediction model development of 32, and an average outcome value in the population of interest (i.e., intercept) of 40 with its standard deviation (SD) of 8. The necessary sample size estimated at 1199 assumed a 0.05 acceptable difference in apparent and adjusted R-squared.
Considering our data set showed variable missing data rates and that these missing values were missing at random, multiple imputation of missing data was performed using chained equations  with 45 datasets based on all the available information (including the date of surgery).
All analyses were performed using R .
Type of model used
Linear regression models were fitted using ordinary least square estimation. Univariable regression coefficients and their 95% confidence intervals (CI) were reported and compared between multiple imputed datasets and complete case data.
Prognostic factor handling
Following the ten principles to strengthen prognosis research , continuous prognostic factors were kept continuous as far as possible. Based on univariable regression models, second-order polynomial transformations of continuous predictors were tested to account for the nonlinear association with postoperative OSS. For categorical variables, an attempt was made to avoid sparse categories.
Model building procedures
Two sets of factors were compared: “Set 1” regrouped the eleven factors estimated by surgeons as having the best predictive ability ( including the baseline OSS value), and “Set 2” was composed of the variables identified using Bayesian projection predictive variable selection using the projpred package . Five-fold cross validation was performed for variable selection based on the complete case data. The set of variables with the best predictive ability (in terms of root-mean-square error (RMSE)) was then identified.
The predictive performance of the two sets of factors were compared in terms of R-squared and RMSE. Internal bootstrap validation was performed to estimate optimism-corrected model performance using 500 repetitions . Regression coefficients were presented in the main text for the model with the best optimism-corrected predictive ability.
Patient and public involvement in research statement
No patient or member of the public was involved in the design of this study. Patients directly filled out the Oxford Shoulder Score based on their own experience after the surgery.
Overall, 1555 patients were assessed for eligibility between October 2013 and June 2021. After application of our eligibility criteria, 45 patients were excluded (1 patient owed to the group IV of ASA classification; 44 were scheduled for a revision surgery) leading to a set of 1510 patients. Complete-case data were available for 712 patients. The percentage of missing values for specific variables ranged from 0 to 45.4% (Table 1).
Comparisons between thirty-two prognostic factor distributions across average multiple imputed data and available complete-case data are summarized in Table 1. Univariable associations between each candidate prognostic factor and the 6-month OSS are also reported. No prognostic factors showed nonlinear univariable associations with the postoperative outcome, even after multiple imputation.
After multiple imputation, the median baseline OSS was 29 (interquartile range (IQR) 23, 35) for the whole population and increased to 43 points (IQR 37, 48) 6 months after the surgery (Table 1). The 6-month OSS values were recorded between 4 and 11 months after the surgery. Overall, 181 patients achieved the maximum OSS value at 6 months (12%). For 167 patients (11%), the change between baseline and the 6-month postoperative OSS was below or equal to 0.
Model specification and comparison
Set 1 and Set 2 were composed of 11 and 7 prognostic factors, respectively (see Additional file 1 and Table 3). After correction for optimism, Set 2 had better model performance than any other model after multiple imputation with R-squared (0.156) and RMSE (7.603) (Table 2).
For Set 2, adjusted regression coefficients were similar across multiple imputed and complete-case data (Table 3). This model included data collected during baseline examinations and operative details.
We developed a prediction model for shoulder function 6 months after an ARCR. Lack of important prognostic factors might explain the improvable model performance. We used robust techniques for the variables selection. Seven patients, disease, and operative factors were retained in the model showing the best model performance after multiple imputation. Our model development highlights the importance to take into account baseline and operative details when the aim is to predict postoperative course of functional status after an orthopedic intervention. Yet, the model performance suggests a model update before implementation in clinical practice, which is planned with the implementation of the ARCR_Pred data .
With sound model performance, such models might be useful for clinicians and policymakers with a primary focus on patient-centered healthcare. Alongside individual expectations of shoulder function outcomes held by the surgeon, such models provide useful insight into the associations between prognostic factors and patient-reported outcomes. This modeled evidence might then be discussed with patients during clinical examinations to forecast their possibilities of improvements after surgery based on informed consideration of a set of pre-established factors. The limitations encountered in this study can be addressed by stronger national support in the development of clinical registers, which form the basis for value-based health care in orthopedic surgery [15, 16].
In prognostic research studies, associations between a given factor and an outcome have to be replicated across different studies in order to classify the factor as one with potential prognostic importance . In our recent systematic review, we identified that low preoperative functional status was associated with greater improvement in PROMs, which seems logical given that patients with poorer outcome scores are expected to have a greater change in outcome . In this case, the type of modeled outcome has substantial importance on the direction in which the associations occur. In the present work, patients with a higher preoperative OSS had better a 6-month score. Optimization of preoperative functional scores might therefore be important in order to expect better postoperative outcomes. This could be achieved by implementing a better selection of the patients undergoing an ARCR (i.e., patients below a certain score could first be optimized with preoperative non-interventional treatment).
Frangiamore et al. recently developed models for various postoperative outcomes also impacted by substantial ceiling and floor effects, which they addressed with the use of proportional odds ordinal logistic regression . Their set comprised twelve factors based on clinical rationale (age at surgery, sex, worker’s compensation status, previous cuff repair, tear size, critical shoulder angle, baseline outcome values, length of follow-up, fatty infiltration, tear shape, tendon stump length, and multiple tendon involvement). Unfortunately, this working group did not report a model performance indicator, which leaves us with a lack of information for comparison. In a recent review, we identified the need for researchers to use adequate reporting guidelines when reporting multivariable prediction models .
This study followed the most recent reporting guidelines along with the availability of the code, which makes this study reproducible and transparent. We included various patient, disease, and procedure-related factors and asked experienced surgeons to assess their predictive ability. We also implemented robust variable selection techniques, such as the Bayesian projection predictive variable selection.
The lack of other important predictors (including postoperative management or peri-operative factors) in our model development may be partially responsible for our inability to accurately make forecasts for patients with extreme postoperative values. For instance, the occurrence of adverse events such as repair failure, neurological injury, persistent pain, or shoulder stiffness might dramatically affect our predictions. It might therefore be important to accurately identify and monitor patients with postoperative adverse events. This study also suffered from the absence of proper external validation.
Before being implemented in clinical practice, our prediction model should show better model performance. We believe the identification of a new set of variables using the ARCR_Pred data might help in that regard by using a variety of new parameters encompassing those of preoperative and/or peri-operative management .
Availability of data and materials
The datasets and code used during the current study are available from the corresponding author on reasonable request.
Akaike’s information criterion
Arthroscopic rotator cuff repair
Oxford Shoulder Score
Urwin M, Symmons D, Allison T, Brammah T, Busby H, Roxby M, Simmons A, Williams G. Estimating the burden of musculoskeletal disorders in the community: the comparative prevalence of symptoms at different anatomical sites, and the relation to social deprivation. Ann Rheum Dis. 1998;57(11):649–55.
Warth RJ, Briggs KK, Dornan GJ, Horan MP, Millett PJ. Patient expectations before arthroscopic shoulder surgery: correlation with patients’ reasons for seeking treatment. J Shoulder Elbow Surg. 2013;22(12):1676–81.
Hemingway H, Croft P, Perel P, Hayden JA, Abrams K, Timmis A, Briggs A, Udumyan R, Moons KG, Steyerberg EW, et al. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ. 2013;346:e5595.
Riley RD, and others (eds). Prognosis Research in Healthcare: Concepts, Methods, and Impact (Oxford, 2019; online edn, Oxford Academic, 1 Feb. 2019). https://doi.org/10.1093/med/9780198796619.001.0001. Accessed 15 Sept 2023.
Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014;35(29):1925–31.
Fermont AJ, Wolterbeek N, Wessel RN, Baeyens JP, de Bie RA. Prognostic factors for successful recovery after arthroscopic rotator cuff repair: a systematic literature review. J Orthop Sports Phys Ther. 2014;44(3):153–63.
Khair MM, Lehman J, Tsouris N, Gulotta LV. A systematic review of preoperative fatty infiltration and rotator cuff outcomes. HSS J. 2016;12(2):170–6.
Lambers Heerspink FO, Dorrestijn O, van Raay JJ, Diercks RL. Specific patient-related prognostic factors for rotator cuff repair: a systematic review. J Shoulder Elbow Surg. 2014;23(7):1073–80.
McElvany MD, McGoldrick E, Gee AO, Neradilek MB, Matsen FA 3rd. Rotator cuff repair: published evidence on factors associated with repair integrity and clinical outcome. Am J Sports Med. 2015;43(2):491–500.
Muller AM, Flury M, Alsayed HN, Audige L. Influence of patient and diagnostic parameters on reported retear rates after arthroscopic rotator cuff repair. Knee Surg Sports Traumatol Arthrosc. 2017;25(7):2089–99.
Raman J, Walton D, MacDermid JC, Athwal GS. Predictors of outcomes after rotator cuff repair-a meta-analysis. J Hand Ther. 2017;30(3):276–92.
Saccomanno MF, Sircana G, Cazzato G, Donati F, Randelli P, Milano G. Prognostic factors influencing the outcome of rotator cuff repair: a systematic review. Knee Surg Sports Traumatol Arthrosc. 2016;24(12):3809–19.
Baum C, Müller A, Audigé L, Stojanov T. Prognostische Faktoren der arthroskopischen Rotatorenmanschettenrekonstruktion. Arthroskopie. 2021;34(3):179–84.
Flury M, Kolling C, Grobet C, Kunz SN, Audigé L. Implementation of a local outcome register for arthroscopic rotator cuff tear repair. Obere Extremität. 2015;10(1):33–40.
Audige L, Bucher HCC, Aghlmandi S, Stojanov T, Schwappach D, Hunziker S, Candrian C, Cunningham G, Durchholz H, Eid K, et al. Swiss-wide multicentre evaluation and prediction of core outcomes in arthroscopic rotator cuff repair: protocol for the ARCR_Pred cohort study. BMJ Open. 2021;11(4):e045702.
Audigé L, Aghlmandi S, Grobet C, Stojanov T, Müller AM, Felsch Q, Gleich J, Flury M, Scheibel M. Prediction of shoulder stiffness after arthroscopic rotator cuff repair. Am J Sports Med. 2021;49(11):3030–9.
Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): the TRIPOD statement. Br J Surg. 2015;102(3):148–58.
Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81.
Imhoff AB, Ticker JB, Fu FH. Atlas of Shoulder Arthroscopy (1st ed.). CRC Press; 2003. https://doi.org/10.1201/9780367804053.
Dawson J, Rogers K, Fitzpatrick R, Carr A. The Oxford Shoulder Score revisited. Arch Orthop Trauma Surg. 2009;129(1):119–23.
Herdman M, Gudex C, Lloyd A, Janssen M, Kind P, Parkin D, Bonsel G, Badia X. Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual Life Res. 2011;20(10):1727–36.
Riley RD, Snell KIE, Ensor J, Burke DL, Harrell FE Jr, Moons KGM, Collins GS. Minimum sample size for developing a multivariable prediction model: part I–continuous outcomes. Stat Med. 2019;38(7):1262–75.
Van Buuren S, Groothuis-Oudshoorn K. Mice: multivariate imputation by chained equations in R. J Stat Softw. 2011;45:1–67.
Team RC. R: a language and environment for statistical computing. 2013.
Riley RD, van der Windt D, Croft P, Moons KG. Ten principles to strengthen prognosis research. In: Riley RD, van der, editors. Prognosis research in healthcare: concepts, methods, and impact. Windt D, Croft P, Moons KG: Oxford University Press; 2019. p. 69–84.
Piironen J, Paasiniemi M, Vehtari A. Projective inference in high-dimensional problems: prediction and feature selection. 2020.
Harrell FE, Jr. Regression modeling strategies. Springer International Publishing; 2016. https://doi.org/10.1007/978-3-319-19425-7.
Huguet A, Hayden JA, Stinson J, McGrath PJ, Chambers CT, Tougas ME, Wozney L. Judging the quality of evidence in reviews of prognostic factor research: adapting the GRADE framework. Syst Rev. 2013;2(1):71.
Stojanov T, Laurent A, Modler L, Aghlmandi S, Appenzeller-Herzog C, Loucas R, Loucas M, Müller AM. Prognostic factors for improvement of shoulder function after arthroscopic rotator cuff repair: a systematic review. JSES Int. 2022;7:50–7.
Frangiamore S, Dornan GJ, Horan MP, Mannava S, Fritz EM, Hussain ZB, Moatshe G, Godin JA, Pogorzelski J, Millett PJ. Predictive modeling to determine functional outcomes after arthroscopic rotator cuff repair. Am J Sports Med. 2020;48(7):1559–67.
Stojanov T, Modler L, Muller AM, Aghlmandi S, Appenzeller-Herzog C, Loucas R, Loucas M, Audige L. Prognostic factors for the occurrence of post-operative shoulder stiffness after arthroscopic rotator cuff repair: a systematic review. BMC Musculoskelet Disord. 2022;23(1):99.
The authors would like to acknowledge the support of Dr. Melissa Wilhelmi, medical writer at Schulthess Klinik, Zurich, Switzerland, for manuscript proofreading.
Open access funding provided by University of Basel This project was supported by a Swiss National Science Foundation grant (grant number: 184959) and by Schulthess Klinik and the University Hospital of Basel with regard to respective personal salaries.
Ethics approval and consent to participate
The rotator cuff register implementation and analysis were approved by the cantonal ethics committee of Zürich (KEK-ZH no. 2014–0253). All procedures performed in studies involving human participants were in accordance with the ethical standards dictated by the local cantonal research committee. Informed consent was obtained from all individual participants included in the study.
Consent for publication
One or more of the authors have declared the following potential conflicts of interest or sources of funding: M. F. and M. S. have received consulting fees from Arthrex.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Stojanov, T., Aghlmandi, S., Müller, A.M. et al. Development and internal validation of a model predicting patient-reported shoulder function after arthroscopic rotator cuff repair in a Swiss setting. Diagn Progn Res 7, 21 (2023). https://doi.org/10.1186/s41512-023-00156-y