Development and validation of a machine learning ASA-score to identify candidates for comprehensive preoperative screening and risk stratification

Objective: The ASA physical status (ASA-PS) is determined by an anesthesia provider or surgeon to communicate co-morbidities relevant to perioperative risk. Assigning an ASA-PS is a clinical decision and there is substantial provider-dependent variability. We developed and externally validated a machine learning-derived algorithm to determine ASA-PS (ML-PS) based on data available in the medical record. Design: Retrospective multicenter hospital registry study. Setting: University-affiliated hospital networks. Patients: Patients who received anesthesia at Beth Israel Deaconess Medical Center (Boston, MA, training [n = 361,602] and internal validation cohorts [n = 90,400]) and Montefiore Medical Center (Bronx, NY, external validation cohort [n = 254,412]). Measurements: The ML-PS was created using a supervised random forest model with 35 preoperatively available variables. Its predictive ability for 30-day mortality, postoperative ICU admission, and adverse discharge were determined by logistic regression. Main results: The anesthesiologist ASA-PS and ML-PS were in agreement in 57.2% of the cases (moderate inter-rater agreement). Compared with anesthesiologist rating, ML-PS assigned more patients into extreme ASA-PS (I and IV), (p < 0.01), and less patients in ASA II and III (p < 0.01). ML-PS and anesthesiologist ASA-PS had excellent predictive values for 30-day mortality, and good predictive values for postoperative ICU admission and adverse discharge. Among the 3594 patients who died within 30 days after surgery, net reclassification improvement analysis revealed that using the ML-PS, 1281 (35.6%) patients were reclassified into the higher clinical risk category compared with anesthesiologist rating. However, in a subgroup of multiple co-morbidity patients, anesthesiologist ASA-PS had a better predictive accuracy than ML-PS. Conclusions: We created and validated a machine learning physical status based on preoperatively available data. The ability to identify patients at high risk early in the preoperative process independent of the provider's decision is a part of the process we use to standardize the stratified preoperative evaluation of patients scheduled for ambulatory surgery.