Prognostic Accuracy of the SOFA Score, SIRS Criteria, and qSOFA Score for In-Hospital Mortality Among Adults With Suspected Infection Admitted to the Intensive Care Unit

Key Points

Question  Does an increase of 2 or more points in Sequential [Sepsis-related] Organ Failure Assessment (SOFA) score have greater prognostic accuracy in patients who are critically ill with suspected infection than 2 or more systemic inflammatory response syndrome (SIRS) criteria or quick SOFA (qSOFA) score points?

Findings  In this retrospective cohort analysis that included 184 875 adults, SOFA (area under the receiver operating characteristic curve [AUROC], 0.753) demonstrated significantly greater discrimination for in-hospital mortality than SIRS criteria (AUROC, 0.589) or qSOFA (AUROC, 0.607).

Meaning  Among patients admitted to the intensive care unit with suspected infection, defining sepsis by an increase in SOFA score provided greater prognostic accuracy for in-hospital mortality than either SIRS criteria or qSOFA.

Importance  The Sepsis-3 Criteria emphasized the value of a change of 2 or more points in the Sequential [Sepsis-related] Organ Failure Assessment (SOFA) score, introduced quick SOFA (qSOFA), and removed the systemic inflammatory response syndrome (SIRS) criteria from the sepsis definition.

Objective  Externally validate and assess the discriminatory capacities of an increase in SOFA score by 2 or more points, 2 or more SIRS criteria, or a qSOFA score of 2 or more points for outcomes among patients who are critically ill with suspected infection.

Design, Setting, and Participants  Retrospective cohort analysis of 184 875 patients with an infection-related primary admission diagnosis in 182 Australian and New Zealand intensive care units (ICUs) from 2000 through 2015.

Exposures  SOFA, qSOFA, and SIRS criteria applied to data collected within 24 hours of ICU admission.

Main Outcomes and Measures  The primary outcome was in-hospital mortality. In-hospital mortality or ICU length of stay (LOS) of 3 days or more was a composite secondary outcome. Discrimination was assessed using the area under the receiver operating characteristic curve (AUROC). Adjusted analyses were performed using a model of baseline risk determined using variables independent of the scoring systems.

Results  Among 184 875 patients (mean age, 62.9 years [SD, 17.4]; women, 82 540 [44.6%]; most common diagnosis bacterial pneumonia, 32 634 [17.7%]), a total of 34 578 patients (18.7%) died in the hospital, and 102 976 patients (55.7%) died or experienced an ICU LOS of 3 days or more. SOFA score increased by 2 or more points in 90.1%; 86.7% manifested 2 or more SIRS criteria, and 54.4% had a qSOFA score of 2 or more points. SOFA demonstrated significantly greater discrimination for in-hospital mortality (crude AUROC, 0.753 [99% CI, 0.750-0.757]) than SIRS criteria (crude AUROC, 0.589 [99% CI, 0.585-0.593]) or qSOFA (crude AUROC, 0.607 [99% CI, 0.603-0.611]). Incremental improvements were 0.164 (99% CI, 0.159-0.169) for SOFA vs SIRS criteria and 0.146 (99% CI, 0.142-0.151) for SOFA vs qSOFA (P <.001). SOFA (AUROC, 0.736 [99% CI, 0.733-0.739]) outperformed the other scores for the secondary end point (SIRS criteria: AUROC, 0.609 [99% CI, 0.606-0.612]; qSOFA: AUROC, 0.606 [99% CI, 0.602-0.609]). Incremental improvements were 0.127 (99% CI, 0.123-0.131) for SOFA vs SIRS criteria and 0.131 (99% CI, 0.127-0.134) for SOFA vs qSOFA (P <.001). Findings were consistent for both outcomes in multiple sensitivity analyses.

Conclusions and Relevance  Among adults with suspected infection admitted to an ICU, an increase in SOFA score of 2 or more had greater prognostic accuracy for in-hospital mortality than SIRS criteria or the qSOFA score. These findings suggest that SIRS criteria and qSOFA may have limited utility for predicting mortality in an ICU setting.

Sepsis remains difficult to define^1-4 but represents a significant burden of disease. A recent meta-analysis estimated its annual global incidence at 31.5 million cases, with 19.4 million cases of severe sepsis, resulting in 5.3 million deaths.⁵ It has been recognized that survival following sepsis is associated with long-term physical, cognitive, and psychosocial morbidity,⁶ and an increased mortality rate up to 2 years after an event.⁷

Accurate diagnostic criteria and consensus definitions have an important role in adult intensive care medicine, providing tools for research, benchmarking, performance monitoring, and accreditation.⁸ Seymour and colleagues⁹ published data concerning the validity of a 2 or more–point change in the Sequential [Sepsis-related] Organ Failure Assessment (SOFA) score as a means of identifying sepsis among patients who are critically ill with suspected infection, assuming a SOFA of 0 for patients not known to have preexisting organ dysfunction. In addition, the concept of the quick SOFA (qSOFA) score was introduced as a possible predictive tool among encounters with suspected infection outside the intensive care unit (ICU). These data were drawn from North American cohorts and a single German cohort⁹ and have not been validated externally.

The primary aims of this study were to (1) assess the effect of an increase in SOFA score of 2 or more points, 2 or more systemic inflammatory response syndrome (SIRS) criteria, and a qSOFA score of 2 or more points measured within the first 24 hours of admission in discriminating in-hospital mortality or prolonged length of stay among patients with suspected infection admitted to ICUs throughout Australia and New Zealand and (2) to validate the use of qSOFA in this setting.^1,2,10

The Alfred Hospital Human Research Ethics Committee in Melbourne, Australia, approved this study with a waiver of informed consent.

A retrospective cohort study was performed using young and older adult (aged ≥17 years) admissions with suspected infection in the Australian and New Zealand Intensive Care Society (ANZICS) Adult Patient Database. All patient records between 2000 and 2015 were screened for study inclusion. Repeat ICU admissions from the same hospital episode and patients transferred to another ICU facility whose eventual outcome was unknown were excluded. Infection or suspected infection was then inferred from infection-related ICU admission diagnoses according to the ANZICS modification of the Acute Physiology and Chronic Health Evaluation (APACHE) III scoring system,¹¹ as used in previous work.^10,12

The following data were extracted: demographic information, indigenous status, ICU and hospital length of stay, vital status at ICU and hospital discharge, physiological and laboratory variables collected for calculation of APACHE II and III scores, and the Australian and New Zealand Risk of Death (ANZROD) prediction model.^13,14 Serum lactate levels were not available. SOFA scores (range, 0 [best] to 24 [worst] points),^4,15 SIRS status (range, 0 [best] to 4 [worst] criteria),¹ and qSOFA scores (range, 0 [best] to 3 [worst] points)^4,9 were calculated using physiological and laboratory parameters recorded from within the first 24 hours of ICU admission. Standard criteria were applied, with a threshold of 2 or more points being used with each scoring system. In keeping with the Sepsis-3 consensus statement,⁴ a Glasgow Coma Scale Score (range, 3 [worst] to 15 [best] points) of less than 15 was used for estimation of the qSOFA score. In the primary analysis, a baseline SOFA score of 0 was assumed for all patients.⁴ Where individual components of SIRS criteria, SOFA, and qSOFA were missing, no contribution was made to the total score (eg, equivalent to assigning 0 or imputing a normal value). Where all components were unknown, the patient was assigned a missing score, and excluded from the primary analysis. In subsequent sensitivity analyses, 2 additional approaches were employed: first, patients with known chronic renal, liver, or respiratory dysfunction (defined per APACHE II and III comorbidity codes¹³) were excluded, and second, a baseline SOFA score of 4 was assigned for chronic renal or hepatic dysfunction and a baseline score of 2 for chronic respiratory impairment.

In accordance with Seymour et al,⁹ the primary study outcome was in-hospital mortality with a composite secondary outcome of in-hospital mortality or an ICU length of stay of 3 days or longer.

All analyses were performed using SAS (SAS Institute), version 9.4. Group comparisons were conducted using χ² tests for equal proportions, t tests for normally distributed data, and Wilcoxon rank sum tests otherwise. Discriminatory power was determined by comparing the area under the receiver operating characteristic curve (AUROC) for each score individually (unadjusted analysis) and in conjunction with a baseline risk model (adjusted analysis). Specific AUROC (99% CI) values were generated, along with incremental improvement between the scores (99% CI).

Baseline risk models for in-hospital mortality and in-hospital mortality or ICU stay of 3 or more days were created using hierarchical logistic regression with patients nested within sites and sites treated as a random variable. Both models were constructed using all available information at the time of ICU admission including factors relating to the ICU (size, type, location, and admission source), admission time (month, day, and hour) and patient (age, sex, comorbidities, pregnancy, diabetes, indigenous status, and treatment limitations) (eTables 1-2 in the Supplement). Predictive validity was determined by dividing baseline risk into deciles and comparing outcomes among patients with an increase of 2 or more points (SOFA) or qSOFA and SIRS criteria of 2 or more against patients with an increase of less than 2 for SOFA score or qSOFA and SIRS criteria of less than 2 at each decile of risk.

Sensitivity analyses were performed for ventilation status, multiple imputation to account for missing data, qSOFA calculated using Glasgow Coma Scale score less than 14 for altered mentation, the presence of chronic comorbidities, admission source, and improved baseline models that sequentially include diagnosis and physiological variables (separate to those used to calculate each of the candidate scores) collected over the first 24 hours in the ICU. Agreement between the SOFA score, SIRS criteria, and qSOFA score was determined using Cronbach α with 99% CIs generated with bootstrapping (100 samples). To increase the robustness of the analysis, a 2-sided P value of .01 was used to indicate statistical significance. Further details of the statistical analysis can be found in the eAppendix of the Supplement.

Data pertaining to 1 499 753 adult admissions were recorded in the ANZICS Adult Patient Database for the period of 2000-2015, drawn from 182 ICUs across Australia and New Zealand. Following restriction to completed index ICU admissions and selection of infection-related diagnoses,^10,12 a final cohort of 184 875 cases were identified (eFigure 1 and eTable 3 in the Supplement). As shown in Table 1 (also in eTable 3 of the Supplement), mean age was 62.9 years (SD, 17.4), 55.4% (n = 102 335) were male, and 44.6% (n = 82 540) were female. Bacterial pneumonia was listed as the most common diagnosis among 32 634 patients (17.7%). There were 34 578 patients (18.7%) who died in the hospital. The secondary outcome of in-hospital mortality or length of stay in the ICU of 3 days or more occurred in 102 976 patients (55.7%).

Of the study cohort, 165 103 patients (90.1%) had an increase of SOFA score from baseline of 2 or more; 158 710 patients (86.7%) manifested 2 or more SIRS criteria, and 99 611 patients (54.4%) had a qSOFA score of 2 or more. The distributions of each score and their relationship with in-hospital mortality are presented in Figure 1 and Figure 2.

The SOFA score could not be calculated in 1544 patients (0.8%); SIRS status, 1901 patients (1.0%); qSOFA score, 1797 patients (1.0%). In-hospital mortality was 28.6%, 27.2% and 27.2% for each of these groups, respectively, for which complete data were missing (eFigure 2 in the Supplement).

Discrimination of in-hospital mortality (Table 2) was significantly higher using SOFA (AUROC, 0.753 [99% CI, 0.750-0.757]) than either SIRS criteria (AUROC, 0.589 [99% CI, 0.585-0.593]) or qSOFA (AUROC, 0.607 [99% CI, 0.603-0.611]) with all incremental differences being statistically significant (between-group difference: SOFA vs qSOFA, 0.146 [99% CI, 0.142-0.151]; SOFA vs SIRS criteria, 0.164 [99% CI, 0.159-0.169]; qSOFA vs SIRS criteria, 0.018 [99% CI, 0.013-0.023]; all P < .001) (Figure 3A). Similarly, when considered in conjunction with baseline prediction of mortality, SOFA (AUROC, 0.815 [99% CI, 0.811-0.818]) outperformed both SIRS criteria (AUROC, 0.755 [99% CI, 0.752-0.759]) and qSOFA (AUROC, 0.762 [99% CI, 0.758-0.765]) for discrimination of hospital mortality (Figure 3B), with all incremental differences being statistically significant (between-group difference: SOFA vs SIRS criteria, 0.047 [99% CI, 0.044-0.049]; SOFA vs qSOFA, 0.042 [99% CI, 0.040-0.044]; qSOFA vs SIRS criteria, 0.005 [99% CI, 0.003-0.006]); all P < .001).

The superior discriminatory performance of SOFA over both SIRS criteria and qSOFA was further maintained when considering the secondary outcome of hospital mortality or ICU length of stay of 3 or more days when considered in isolation or in conjunction with the baseline prediction (Table 2, Figure 3C, and Figure 3D).

Additional data showing calibration plots for observed and expected outcomes are demonstrated in eFigure 3 and 4 of the Supplement. Full baseline risk models for in-hospital mortality and for the secondary outcome of hospital mortality or ICU length of stay of 3 or more days are also provided in eTables 1 and 2 of the Supplement.

The in-hospital mortality of patients who had an increase from baseline in SOFA score of 2 or more was 20.2% (33 365 of 165 103 patients) vs 4.4% (793 of 18 228 patients), P < .001 (between-group difference, 15.9% [99% CI, 15.4%-16.3%]); for those who manifested 2 or more SIRS criteria, mortality was 19.9% (31 648 of 158 710 patients) vs 9.8% (2387 of 24 264 patients), P < .001 (between-group difference, 10.1% [99% CI, 9.5%-10.7%]); and for those who had a qSOFA score of 2 or more, mortality was 22.8% (22 758 of 99 611 patients) vs 13.6% (11 332 of 83 457 patients), P < .001 (between-group difference, 9.3% [99% CI, 8.8%-9.7%]); when compared with those with less than 2 points or criteria on each respective score.

Patients with an increase from baseline in SOFA score of 2 or more had a greater incremental increase in mortality across all deciles of baseline risk, than patients with 2 or more SIRS criteria, or those with a qSOFA score of 2 or more (Table 3 and Figure 4A). The incremental increase in mortality when a patient had an increase in SOFA score of 2 or more ranged from an odds ratio (OR) of 7.46 (99% CI, 4.01-13.89) among those in the lowest baseline risk decile to an OR of 2.70 (99% CI, 2.14-3.42) among those in the highest decile.

This analysis was repeated with the composite secondary outcome (Table 3 and Figure 4B). Those with an increase of SOFA score of 2 or more had a significantly greater incremental increase in risk of the composite outcome of in-hospital mortality or length of stay in the ICU of 3 or more days, at all deciles of baseline risk, than patients who manifested 2 or more SIRS criteria or those with a qSOFA score of 2 or more points. The incremental increase in risk of the composite secondary outcome when a patient had an increase in SOFA score of 2 or more varied from an OR of 3.59 (99% CI, 3.15-4.10) among those in the lowest baseline risk decile to an OR of 4.13 (99% CI, 3.38-5.03) among those in the highest decile.

A number of sensitivity analyses were performed, wherein the AUROC over and above the baseline level for the primary outcome (using each scoring system) was calculated in a range of scenarios. These included (1) mechanically ventilated vs nonventilated patients, (2) using multiple imputation to account for missing data, (3) using a Glasgow Coma Scale score less than 14 vs less than 15 for calculation of qSOFA, (4) having patients with chronic organ dysfunction either excluded or assigned a given baseline SOFA score, (5) differing the ICU admission source, and (6) using improved baseline models. In each case, SOFA demonstrated superior discrimination compared with SIRS criteria or qSOFA (eTable 4 in the Supplement).

In comparison with SIRS criteria or a qSOFA score of 2 or more, an increase in SOFA score of 2 or more within 24 hours of ICU admission demonstrated superior prognostic accuracy for in-hospital mortality and the composite outcome of in-hospital mortality or ICU length of stay of 3 days or more among patients with suspected infection admitted to Australian and New Zealand ICUs.

This study examined and compared the discriminatory capacity of SOFA, SIRS criteria, and qSOFA in a large ICU population of patients outside of the cohort used to develop the new definition of sepsis. Kaukonen and colleagues¹⁰ have previously demonstrated that SIRS criteria are an imperfect predictor of ICU mortality, inappropriately excluding otherwise similar patients with infection, organ failure, and increased mortality. Furthermore, use of SIRS criteria failed to define a transition point in risk of death, despite adjustment for baseline characteristics.¹⁰

Conversely, SOFA scores have been demonstrated to be a useful predictor of ICU mortality.¹⁵ Rivera-Fernández and colleagues¹⁶ demonstrated that 28-day mortality was related to mean and maximum daily SOFA scores in a cohort of patients who were critically ill with an AUROC of 0.95.¹⁶ The relationship between SOFA scores and risk of death has been confirmed in a variety of subgroups^14,16-18 including sepsis.^9,15 In the sepsis consensus definition, the ability of SOFA, SIRS criteria, and qSOFA scores to discriminate in-hospital mortality or prolonged ICU length of stay was determined in a cohort of both ICU and non-ICU encounters with suspected infection.⁹ SOFA demonstrated significantly greater capacity compared with qSOFA and SIRS criteria in the ICU cohort.⁹ This study demonstrated findings consistent with these reports.

The mean age (63 years) in this study was similar to that reported in the ICU validation cohort for the definition of sepsis by Seymour et al,⁹ as was the proportion of males (approximately 55%). In-hospital mortality in this study was marginally higher (19% in this study vs 16% in Seymour et al), although roughly equal fractions manifested an increase in SOFA score of 2 or more (90% in this study vs 91% in Seymour et al). Importantly, this cohort was more than 11 times larger than the total number of ICU encounters used by Seymour et al,⁹ and was derived from multiple ICUs dispersed among 2 countries. The adjusted AUROCs for in-hospital mortality for SOFA score, SIRS criteria, and qSOFA score in this ICU cohort were substantially higher than previously reported,⁹ which was, in part, likely a reflection of the strength of the baseline model used in analysis.

Using a large binational database independent of the sepsis consensus development cohorts, this study confirmed that an increase in SOFA score of 2 or more points within the first 24 hours of ICU admission had superior prognostic accuracy for mortality or ICU length of stay of 3 days or more compared with SIRS criteria or qSOFA. It established that the use of an increase in SOFA score of 2 or more points to define sepsis was an appropriate data-based starting point, and would likely have broad external validity in ICU patients from developed countries. This study’s findings confirmed that the SIRS criteria provided no additional predictive use for mortality or prolonged ICU length of stay beyond that achieved with SOFA. Finally, this study confirmed that, as already suggested in the consensus statement, the qSOFA score had little additional predictive value over the SIRS criteria among patients admitted to the ICU with suspected infection, and that among these patients, the use of qSOFA may be low.

This study had a number of strengths. It was based on one of the largest complete ICU data sets, with a cohort of 184 875 infection-related admissions over a 16-year period, encompassing a wide geographical area. Moreover, due to the use of a prospective, quality-surveillance data collection process, these results were unlikely to be biased. This study used the primary diagnosis leading to ICU admission for patient selection recorded in the ANZICS Adult Patient Database. This use of a clinical diagnosis, rather than administrative coding data increased the generalizability and applicability of the study’s findings. The reported patients had similar characteristics, sex, age, and outcomes to those used by Seymour et al,⁹ for which they provide an important external validation. This study was further strengthened by the use of a reproducible identification process for infection,^10,12 and the use of the same methodology as in the consensus paper.⁹ This study considered in-hospital mortality as a primary end point, but also measured a composite of in-hospital mortality or ICU length of stay of 3 days or more (as applied to the consensus definition), thus assessing the discrimination for both of these outcomes for the 3 measures. Finally, these data had excellent internal validity, as evidenced by an in-hospital mortality rate (18.7%) that is highly consistent with a recent multicenter study exploring early goal-directed therapy for early septic shock conducted in Australia and New Zealand.¹⁷

The data used for this analysis were not primarily collected for such study purposes. SOFA, SIRS criteria, and qSOFA could only be studied for the first 24 hours in ICU. Biochemical and physiological values could have come from any time within the first 24 hours of ICU admission and, as a result, could not be more accurately linked to the timing of the diagnosis of infection. The SOFA score used should be considered a modification of the original because the cardiovascular component was estimated without knowledge of inotrope or vasopressor dose. The incidence of nosocomial infections and of infections in patients admitted with another diagnosis were unknown.¹⁸ However, although this study could not comment specifically on the applicability of these scores to patients who develop infections later during their ICU stay, the consistency of findings across multiple sensitivity analyses suggests that these scores might have similar performance in all ICU patients. A population at risk for sepsis could not be studied to assess the diagnostic value of all 3 measures. Finally, this study did not examine the non-ICU population. As a result, although the advantage of the qSOFA score is its brevity and use of clinical variables only, its applicability to patients with suspected infection in the rest of the hospital (on the wards or in the Emergency Department) could not be determined by this study.^9,19-21

Among adults with suspected infection admitted to an ICU, an increase in SOFA score of 2 or more points had greater prognostic accuracy for in-hospital mortality than SIRS criteria or qSOFA. These findings suggest that SIRS and qSOFA may have limited use for predicting mortality in an ICU setting.

Corresponding Author: David V. Pilcher, MBBS, FRACP, FCICM, Department of Intensive Care and Hyperbaric Medicine, Alfred Hospital, Commercial Road, Prahran, Melbourne VIC 3181, Australia (d.pilcher@alfred.org.au).

Author Contributions: Drs Pilcher and Bailey had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Raith, McGloughlin, MacIsaac, Bellomo, Pilcher.
Acquisition, analysis, or interpretation of data: Raith, Udy, Bailey, Bellomo, Pilcher.
Drafting of the manuscript: Raith, Udy, McGloughlin, Bellomo, Pilcher.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Bailey, Pilcher.
Administrative, technical, or material support: Raith, Udy, Bellomo, Pilcher.
Supervision: Udy, McGloughlin, Bellomo, Pilcher.
No additional contributions: MacIsaac.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Udy receives salary support from Monash University as an Alfred ICU Monash University practitioner fellow and receives support from the National Health and Medical Research Council of Australia (Early Career Fellowship). Dr Bailey receives salary support from the School of Public Health, Monash University. Dr Pilcher receives salary support from Monash University as an Alfred ICU Monash University practitioner fellow.

Additional Information: This work was performed at the Centre for Outcome and Resource Evaluation of the Australian and New Zealand Intensive Care Society (ANZICS), Melbourne, Australia, and at the Department of Intensive Care and Hyperbaric Medicine of Alfred Hospital in Prahran, Melbourne, Australia. We thank ANZICS Centre for Outcomes and Resource Evaluation (CORE) in completing this research and the work of data collectors and clinicians who contributed to the CORE registries throughout Australia and New Zealand. The ANZICS CORE registry database is funded by local jurisdictions. No one received compensation for their contribution.