Skip to main content
Log in

A System for Automated Determination of Perioperative Patient Acuity

  • Patient Facing Systems
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

The widely used American Society of Anesthesiologists Physical Status (ASA PS) classification is subjective, requires manual clinician review to score, and has limited granularity. Our objective was to develop a system that automatically generates an ASA PS with finer granularity by creating a continuous ASA PS score. Supervised machine learning methods were used to create a model that predicts a patient’s ASA PS on a continuous scale using the patient’s home medications and comorbidities. Three different types of predictive models were trained: regression models, ordinal models, and classification models. The performance and agreement of each model to anesthesiologists were compared by calculating the mean squared error (MSE), rounded MSE and Cohen’s Kappa on a holdout set. To assess model performance on continuous ASA PS, model rankings were compared to two anesthesiologists on a subset of ASA PS 3 case pairs. The random forest regression model achieved the best MSE and rounded MSE. A model consisting of three random forest classifiers (split model) achieved the best Cohen’s Kappa. The model’s agreement with our anesthesiologists on the ASA PS 3 case pairs yielded fair to moderate Kappa values. The results suggest that the random forest split classification model can predict ASA PS with agreement similar to that of anesthesiologists reported in literature and produce a continuous score in which agreement in accurately judging granularity is fair to moderate.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Dripps, R. D., New classification of physical status (editorial). Anesthesiology 24:111, 1963.

    Google Scholar 

  2. Daabiss, M., American Society of Anesthesiologists physical status classification. Ind. J. Anaesth. 55:111–115, 2011.

    Article  Google Scholar 

  3. Ridgeway, S., Wilson, J., Charlet, A., Pearson, A., and Coello, R., Infection of the surgical site after arthroplasty of the hip. J. Bone Joint Surg. Br. 87:844–850, 2005.

    Article  PubMed  CAS  Google Scholar 

  4. Tang, R., Chen, H. H., Wang, Y. L., Changchien, C. R., Chen, J.-S., Hsu, K.-C., Chiang, J.-M., and Wang, J.-Y., Risk factors for surgical site infection after elective resection of the colon and rectum: A single-center prospective study of 2,809 consecutive patients. Ann. Surg. 234:181–189, 2001.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  5. Sauvanet, A., Mariette, C., Thomas, P., Lozac’h, P., Segol, P., and Tiret, E., Mortality and Morbidity after resection for adenocarcinoma of the gastroesophageal junction: predictive factors. J. Am. Coll. Surg. 201:253–262, 2005.

    Article  PubMed  Google Scholar 

  6. Prause, G., Offner, A., Ratzenhofer-Komenda, B., Vicenzi, M., Smolle, J., and Smolle-Juttner, F., Comparison of two preoperative indices to predict perioperative mortality in non-cardiac thoracic surgery. Eur. J. Cardiothorac. Surg. 11:670–675, 1997.

    Article  PubMed  CAS  Google Scholar 

  7. Carey, M. S., Victory, R., Stitt, L., and Tsang, N., Factors that influence length of stay for in-patient gynecology surgery: Is the case mix group (CMG) or type of procedure more important? J. Obstet. Gynaecol. Can. 28:149–155, 2006.

    Article  PubMed  Google Scholar 

  8. Riley, R., Holman, C., and Fletcher, D., Inter-rater reliability of the ASA PS physical status classification in a sample of anaesthetists in Western Australia. Anaesth. Intens. Care 42(5):614–618, 2014.

    Google Scholar 

  9. Ringdal, K. G., Skaga, N. O., Steen, P. A., Hestnes, M., Laake, P., Jones, J. M., and Lossius, H. M., Classification of comorbidity in trauma: the reliability of pre-injury ASA physical status classification. Injury 44:29–35, 2013.

    Article  PubMed  Google Scholar 

  10. Ihejirika, R. C., Thakore, R. V., Sathiyakumar, V., Ehrenfeld, J. M., Obremskey, W. T., and Sethi, M. K., An assessment of the inter-rater reliability of the ASA PS physical status score in the orthopaedic trauma population. Injury 46:542–546, 2015.

    Article  PubMed  Google Scholar 

  11. Sankar, A., Johnson, S. R., Beattie, W. S., Tait, G., and Wijeysundera, D. N., Reliability of the American Society of Anesthesiologists physical status scale in clinical practice. Br. J. Anaesth. 113(3):424–432, 2014.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  12. Karpagavalli, S., Jamuna, K. S., and Vijaya, M. S., Machine learning approach for preoperative anaesthetic risk prediction. Int. J. Recent Trends Eng. 1:19–22, 2009.

    Google Scholar 

  13. Lazouni, M., Daho, M., Settouti, N., Chikh, M., and Mahmoudi, S., In: Amine, A., Otmane, A. M., Belleatreche, L. (Eds), Machine Learning Tool for Automatic ASA Detection, Modeling Approaches and Algorithms for Advanced Computer Applications, volume 488. Switzerland: Springer, Cham, 2013, 9–16.

    Google Scholar 

  14. Buitinck, L., Louppe, G., Blonde, M., Pedregosa, F., Muller, A.C., Grisel, O., Niculae, V., Prettenhofer, P., Gramfort, A., Grobler, J., Layton, R., Vanderplas, J., Joly, A., Holt, B., and Varoquaux, G., API design for machine learning software: Experiences from the scikit-learn project. ECML PKDD workshop: Languages for data mining and Mach. Learn.108–122, 2013.

  15. Dieleman, S., Schlüter, J., Raffel, C., Olson, E., Sønderby, S. K., Nouri, D., Maturana, D., Thoma, M., Battenberg, E., Kelly, J., and De Fauw, J., Lasagne: First Release. Geneva, Switzerland: Zenodo, 2015.

    Google Scholar 

  16. Bergstra, J., Yamins, D., and Cox, D., Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. Int. Conf. Mach. Learn. 115–23, 2013.

  17. Maclurs, M., and Willett, W., Misinterpretation and misuse of the kappa statistic. Am. J. Epidemiol. 126:161–169, 1987.

    Article  Google Scholar 

  18. Enhanced therapeutic classification system. Available at http://www.fdbhealth.com/fdb-medknowledge-foundations/. Accessed August 7, 2015.

  19. Denny, J. C., Bastarache, L., Ritchie, M. D., Carroll, R. J., Zink, R., Mosley, J. D., Field, J. R., Pulley, J. M., Ramirez, A. H., Bowton, E., Basford, M. A., Carrell, D. S., Peissig, P. L., Kho, A. N., Pacheco, J. A., Rasmussen, L. V., Crosslin, D. R., Crane, P. K., Pathak, J., Bielinski, S. J., Pendergrass, S. A., Xu, H., Hindorff, L. A., Li, R., Manolio, T. A., Chute, C. G., Chisholm, R. L., Larson, E. B., Jarvik, G. P., Brilliant, M. H., Mccarty, C. A., Kullo, I. J., Haines, J. L., Crawford, D. C., Masys, D. R., and Roden, D. M., Systematic comparison of phenome-wide association study of electronic medical record data and genome-wide association study data. Nat. Biotechnol. 31:1102–1110, 2013.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  20. Singh, A., Nadkarni, G., Guttag, J., and Bottinger, E., Leveraging hierarchy in medical codes for predictive modeling. Proc. 5th ACM Conf. Bioinform., Comput. Biol. Health Info.. 96–103, 2014.

  21. Schmidhuber, J., Deep learning in neural networks: an overview. Neural Netw. 61:85–117, 2015.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors acknowledge the Vanderbilt Anesthesiology and Perioperative Informatics Research division for their assistance with data access.

Funding

This work was supported by the NIH/NIBIB Grant R01EB020666 and the NLM Training Grant 4T15LM 7450–15 from the National Institutes of Health located in Bethesda, MD.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Linda Zhang.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

This article is part of the Topical Collection on Patient Facing Systems

Appendices

Appendix A. Feature classes

The feature classes were:

  • Age (integer): Age of the patient.

  • BMI (continuous decimal): Body mass index of the patient.

  • Surgery service (binary): The primary surgical service performing the procedure (70 total).

  • Previous surgery (binary): A value indicating whether the patient has previously had any surgery at our institution.

  • Preoperative medications (count): Each medication is encoded into the 21-category top-level hierarchy from the First Databank (FDB) Enhanced Therapeutic Classification [18]. For example, clopidogrel (Plavix) would be represented in the “Blood” category.

  • Inpatient ICD-9-CM codes (count): ICD-9-CM codes received while in the hospital. In creating these features, we tested numerical counts for: raw codes, parent codes, ICD-9-CM chapters, PheWAS (Phenome-Wide Association Study classification) codes [19], ICD-9-CM hierarchy information and temporally structured ICD-9-CM chapters. Details for ICD-9-CM feature structure are described below.

  • Outpatient ICD-9-CM codes (count): ICD-9-CM codes received while not admitted to the hospital. In creating these features, we tested numerical counts for: raw codes, parent codes, ICD-9-CM chapters, PheWAS codes, ICD-9-CM hierarchy and temporally structured ICD-9-CM chapters.

We tested different combinations of feature classes by making every possible combination of the feature classes and running the model using 5-fold cross validation. We then evaluated the resulting model by calculating the mean receiver operating characteristic curve (ROC AUC) and compared it to other combinations.

For the features from ICD-9-CM codes, we tested: raw codes, parent codes, ICD-9-CM chapters, PheWAS codes, and ICD-9-CM hierarchy and temporally structured ICD-9-CM chapters. The raw ICD-9-CM codes resulted in approximately 4000 features, while both the parent codes and PheWAS codes resulted in approximately 1000 features. The ICD-9-CM chapters result in 20 features, the ICD-9-CM hierarchy results in 20 features, and the temporally structured ICD-9-CM chapters result in 240 features (20 chapters * 12 months).

Incorporating specific ICD hierarchy information can sometimes improve prediction quality by leveraging the relationships between correlated ICD codes and their parents [20]. We tested these hierarchy-based variables as potential predictors.

Additionally, temporal ICD features were constructed by dividing the year before surgery into month-long windows for each ICD chapter, where the value of a month’s window is the number of each of the ICD-9-CM chapter codes occurring in that month.

Appendix B. Neural network description, architecture, and optimization

Deep learning for temporal data

In the past decade, neural networks have become very popular. A standard neural network consists of a number of simple, connected processors called neurons, each of which performs a simple regression. The regression weights of all neurons are iteratively adjusted during model training to maximize the accuracy of prediction. Deep neural networks have many layers, and the output of neurons in one layer provide the input to the neurons in the next layer [21]. Deep learning models scale well to large data sets but require a large training data set.

Convolutional neural networks are deep neural networks that emphasize smaller local patterns that may show up in different locations in the data space. In this project, we sought data patterns that would capture information about short-range temporal proximity. In other words, the convolutional model identified patterns that are local to a small span of, say, two or three months, regardless of when in absolute time those months were positioned.

Hybrid network architecture

The hybrid neural network uses both the deep and convolutional networks to learn from the data. The deep neural network operates on the non-temporal data (age, BMI, etc) while the convolutional neural network takes temporal data as its input. To combine the two types of data, we concatenated the output hidden units from the convolutional neural network with the hidden units from the last hidden layer of the deep neural network. The resulting layer is fed into an output layer, which uses a softmax as the activation function.

Binary classification optimized parameters

The final parameters from hyperopt that we found were logistic classification with an elastic net penalty, k-nearest neighbors with k = 4, random forests using 79 estimators and a depth of 19, deep neural networks with 0.1 hidden drop, a depth of 3 and width of 400, and the hybrid network with the same deep network parameters plus convolutional network parameters with 20 filters and a window size of 3. The hybrid neural network architecture and final parameters can be seen in Table 6.

Table 6 Hybrid neural network architecture

Regression optimized parameters

The final parameters found from hyperopt for regression models were random forests using 95 estimators and a depth of 19, deep neural networks with 0.2 hidden drop, a depth of 2 and width of 500 (Table 7). For linear regression and ordinal regression, we used an elastic net to evaluate the importance of the features, but found that removing features caused a decrease in MSE, and in the final models used all the features. For the split classifiers, we used the parameters from the binary classification models.

Table 7 Deep neural network regression model architecture

Appendix C. Individual feature class performance

Table 8 The mean area under the receiver operating characteristic curve (ROC AUC) scores for each individual feature class, and combined classifier measured using 5-fold cross validation

Appendix D. Model performance on original data

Table 9 Holdout mean squared error (MSE) for continuous models trained on original training data using linear, random forest, deep neural network (DNN) and ordinal regression, and random forest and DNN split classification

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, L., Fabbri, D., Lasko, T.A. et al. A System for Automated Determination of Perioperative Patient Acuity. J Med Syst 42, 123 (2018). https://doi.org/10.1007/s10916-018-0977-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10916-018-0977-7

Keywords

Navigation