- Journal List
- Crit Care Explor
- v.5(7); 2023 Jul
- PMC10317482
As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsem*nt of, or agreement with, the contents by NLM or the National Institutes of Health.
Learn more: PMC Disclaimer | PMC Copyright Notice
Crit Care Explor. 2023 Jul; 5(7): e0941.
Published online 2023 Jun 30. doi:10.1097/CCE.0000000000000941
PMCID: PMC10317482
PMID: 37405252
John Cull, MD,
Robert Brevetta, DO, Jeff Gerac, MD, Shanu Kothari, MD, and Dawn Blackhurst, DrPH
Author information Copyright and License information PMC Disclaimer
Associated Data
- Supplementary Materials
OBJECTIVES:
Earlier treatment of sepsis leads to decreased mortality. Epic is an electronic medical record providing a predictive alert system for sepsis, the Epic Sepsis Model (ESM) Inpatient Predictive Analytic Tool. External validation of this system is lacking. This study aims to evaluate the ESM as a sepsis screening tool and determine whether an association exists between ESM alert system implementation and subsequent sepsis-related mortality.
DESIGN:
Before-and-after study comparing baseline and intervention period.
SETTING:
Urban 746-bed academic level 1 trauma center.
PATIENTS:
Adult acute care inpatients discharged between January 12, 2018, and July 31, 2019.
INTERVENTIONS:
During the before period, ESM was turned on in the background, but nurses and providers were not alerted of results. The system was then activated to alert providers of scores greater than or equal to 5, a set point determined using receiver operating characteristic curve analysis (area under the curve, 0.834; p < 0.001).
MEASUREMENTS AND MAIN RESULTS:
Primary outcome was mortality during hospitalization; secondary outcomes were sepsis order set utilization, length of stay, and timing of administration of sepsis-appropriate antibiotics. Of the 11,512 inpatient encounters assessed by ESM, 10.2% (1,171) had sepsis based on diagnosis codes. As a screening test, the ESM had sensitivity, specificity, positive predictive value, and negative predictive value rates of 86.0%, 80.8%, 33.8%, and 98.11%, respectively. After ESM implementation, unadjusted mortality rates in patients with ESM score greater than or equal to 5 and who had not yet received sepsis-appropriate antibiotics declined from 24.3% to 15.9%; multivariable analysis yielded an odds ratio of sepsis-related mortality (95% CI) of 0.56 (0.39–0.80).
CONCLUSIONS:
In this single-center before-and-after study, utilization of the ESM score as a screening test was associated with a 44% reduction in the odds of sepsis-related mortality. Due to wide utilization of Epic, this is a potentially promising tool to improve sepsis mortality in the United States. This study is hypothesis generating, and further work with more rigorous study design is needed.
Keywords: early warning score, hospital mortality, predictive value of tests, sepsis, validation study
KEY POINTS
Question: Is the Epic Sepsis Model (ESM) a useful screening tool for identifying patients at risk of a sepsis diagnosis during hospitalization and can implementation of the alert system reduce sepsis mortality?
Findings: At ESM set point greater than or equal to 5, overall discrimination (area under the receiver operating characteristic curve) was 83.4% (p < 0.001) with 86.0% sensitivity, 80.8% specificity, and positive and negative predictive values of 33.8% and 98.1%, respectively. Implementation of the ESM as a screening tool alert system was associated with a reduction in mortality rate among patients with a score greater than or equal to 5 and who had not yet received sepsis-related antibiotics (odds ratio, 0.56; 95% CI, 0.39–0.80).
Meaning: The Epic sepsis alert system is a potentially promising tool for improving sepsis mortality throughout the United States; however, more rigorous controlled studies are needed.
Sepsis is life-threatening organ dysfunction caused by a dysregulated host response to infection (1). The impact of sepsis on healthcare is substantial. The frequency of sepsis in hospitalized patients is 2–6% with a mortality rate of 15–40% (2–4) and costs 23.6 billion dollars (about $73 per person in the United States) per year, equivalent to 6.2% of the U.S. healthcare budget (5, 6).
Several studies found that early identification and treatment of sepsis leads to decreased mortality (7–10). The Updated 2018 Surviving Sepsis Campaign has made recommendations to bundle care into a “hour-1 bundle” with the intention to begin resuscitation and management of sepsis at time of presentation (11). To improve outcomes and to comply with this new “hour-1 bundle” recommendation, early identification systems and notification systems are needed to identify sepsis sooner.
Multiple early detection methods of degree of illness exist such as the modified Early Warning Score and the National Early Warning Score (12–15); however, early detection alert systems specifically for sepsis detection have not been widely adopted. Barriers to implementing these detection methods include system processes issues, and difficulty integrating these various early alert systems into electronic health records. Some hospitals have invested in developing or purchasing alert systems, which have been employed with varying success (16–20). However, these disparate alert systems inhibit widespread dissemination of standardized practice.
Epic Systems is the largest provider of information health technology in the United States. According to Epic, the hospitals and healthcare groups using their electronic medical record (EMR) serve 54% of patients in the United States (21). In 2018, Epic developed a predictive model for sepsis using 500,000 patient encounters, and 80 demographic and clinical variables called the Epic Sepsis Model (ESM) Inpatient Predictive Analytic Tool. The benefit of this alert system is that it is easily integrated into the electronic health record and is generalizable to many healthcare systems.
Our health system implemented Epic as our EMR provider in 2016 and was one of the early adopters of this sepsis alert model. Epic reports its hospital level performance as an area under the receiver operating characteristic curve (AUC) at 0.76–0.83 (22). There has been one other study that externally validated the ESM, and reported an AUC of 0.63, which is substantially worse than the performance reported by Epic (23). Widespread external validation of this tool is important, owing to the high mortality rate and high cost of sepsis. In this study, we set out to externally validate ESM at our institution. The purposes of this study were to determine the best score cut-point for utilization as a screening tool for potential sepsis in our population, and to then assess the ESM as an alert system to improve mortality in patients with sepsis.
MATERIALS AND METHODS
Sample and Setting
The study was conducted at a 746-bed academic level 1 trauma center, Prisma Health, in Greenville, SC. The study began as a quality improvement project to validate the new Epic sepsis analytic tool in our acute-care population. The ESM is a proprietary tool developed to assess the risk of sepsis in patients. It uses a set of approximately 80 data elements, encompassing various clinical parameters and patient characteristics. These elements include vital signs, laboratory results, comorbidities, and demographic factors. Each data element is weighted according to its relative contribution in determining the risk of sepsis and a numerical score is generated that reflects this risk. One feature of the ESM tool is its ability to identify the specific data elements that are contributing most significantly to an elevated score.
This study was conducted as a before-and-after study. The Prisma Health Institutional Review Board reviewed and approved this study (protocol number Pro00081382) with a waiver for informed consent.
Determination of Cut-Point Score
All adult inpatients were assessed using the ESM from January 12, 2018, to April 30, 2018. During this timeframe, the ESM was turned on in the background; however, the data were not visible to staff. There was a total of 11,512 inpatient encounters with ESM data available for analysis. For the “gold standard” definition of sepsis, we used the International Classification of Diseases, 10th Revision sepsis diagnosis codes listed in Appendix A (http://links.lww.com/CCX/B217). Thus, any patient with an identified sepsis diagnosis code, in any sequence, was counted as sepsis positive. The ESM screening test “cut point” was determined using receiver operating characteristic (ROC) curve analysis. The ESM is continually updated and the maximum ESM score during the hospitalization was used in the ROC analysis.
Healthcare Worker Training
Next, we educated physicians and nurses on the ESM. Education for the physicians was performed using a required completion of a Computer Based Training module. Physicians were educated on how the tool was developed with the goal of earlier recognition and treatment of sepsis. The nurses were provided sepsis alert system education during staff meetings and explained the rationale, importance, and objectives of the tool. An ESM score above the threshold was to be treated as a “critical value,” whereby the nurses were to call physicians, provide physicians with the pertinent clinical information that led to the elevated score, and receive physician instruction regarding ongoing patient care.
Evaluating Model As Alert System
After clinical team instruction was completed, the ESM score alert system was turned on and implemented in the acute care settings. We then compared patient characteristics and outcomes between the baseline period (e.g., pre-implementation when the score was not visible to staff from January 12, 2018, to October 21, 2018) and the intervention period (e.g., post-implementation, when the ESM education had been completed, the score was made visible to staff, and there was an expectation of staff to act on the ESM score from October 22, 2018, to July 31, 2019). The study population for analysis in both time frames consisted of those patients with an ESM score above the optimal cutoff; these patients were deemed “at risk” for a sepsis diagnosis during the hospitalization. Patients who received antibiotics for suspected sepsis before arrival to the hospital were excluded from the study analysis (n = 371). The primary outcome was mortality during the hospitalization. Secondary outcomes were sepsis order set utilization, hospital and ICU length of stay (LOS), and timing of administration of sepsis-appropriate antibiotics.
Statistical Analysis
Bivariate analyses of the differences in factors between the two time periods were conducted using the chi-square test for categorical data and the Wilcoxon rank-sum test for continuous data. Multivariable logistic regression analysis was used to assess independent predictive factors for the outcome of mortality. Factors with p values of less than 0.20 in bivariate analyses were selected for inclusion in the logistic regression model. Factors with a clinical association with mortality (i.e., patient age and need for intensive care), or an observed association with the study time periods were included in the logistic models. p values of less than 0.05 and nonoverlapping Fisher exact 95% CIs were used to assess statistical significance. SAS Enterprise Guide 8.3 software was used for all statistical analyses (SAS Institute, Cary, NC).
RESULTS
Determination of Cut-Point Score
From January 12, 2018, to April 30, 2018, a total of 11,512 inpatient encounters were assessed by the ESM. Of these 11,512 encounters, 10.2% (1,171) were identified as having sepsis based on diagnosis codes. For identifying the ESM “cut-point” screening value, we used the maximum score during the hospitalization. The average maximum score of patient encounters with a diagnosis of sepsis was 14.9 (range, 1–64) and the average maximum score of patient encounters with no diagnosis of sepsis was 3.2 (range, 0–67). Of the 1,171 patients with a sepsis diagnosis, 1,007 (86.0%) had a maximum ESM score greater than or equal to 5 (i.e., sensitivity); of the 10,341 patients without a sepsis diagnosis, 8,359 (80.8%) had a maximum ESM score less than 5 (i.e., specificity). Positive and negative predictive values were 33.8% and 98.1%, respectively. An ESM score of 5 or greater yielded the best result to serve as a screening tool for sepsis diagnosis during hospitalization (Supplemental Fig. 1, http://links.lww.com/CCX/B218; AUC, 0.834; p < 0.001).
Evaluating Model As Alert System
Results of bivariate analyses between the two time periods are provided for three subgroups of patients in order of clinical specificity: 1) patients with ESM score greater than or equal to 5, 2) patients with score greater than or equal to 5 and a diagnosis of sepsis, and 3) patients with score greater than or equal to 5, a diagnosis of sepsis, and no receipt of antibiotics before ESM score greater than or equal to 5 (Tables Tables11–3). Tables Tables44 and 55 provide multivariable logistic regression analysis results in the latter two populations.
TABLE 1.
Characteristics and Outcomes of Patients With Epic Sepsis Model Score Greater Than or Equal to 5
| Characteristic | Hospitalizations, n (%) | p | |
|---|---|---|---|
| Baseline Period | Intervention Period | ||
| n | 4,397 | 5,159 | — |
| Age, median (IQR), yr | 65 (54–76) | 65 (54–76) | 0.689 |
| Gender | |||
| Female | 2,065 (47.0) | 2,491 (48.3) | 0.198 |
| Male | 2,332 (53.0) | 2,668 (51.7) | |
| Race/ethnicity | |||
| Caucasian | 3,232 (73.5) | 3,762 (72.9) | 0.600 |
| African American | 976 (22.2) | 1,172 (22.7) | |
| Hispanic | 104 (2.4) | 137 (2.7) | |
| Other | 85 (1.9) | 88 (1.7) | |
| Admitted through emergency department | 3,234 (73.6) | 3,995 (77.4) | < 0.001 |
| Sepsis score (first score ≥ 5), median (IQR) | 6 (5–7) | 6 (5–7) | 0.001 |
| Sepsis order set used | 907 (20.6) | 1,049 (20.3) | 0.722 |
| Diagnosis of sepsis | 1,477 (33.6) | 1,511 (29.3) | < 0.001 |
| Hospital length of stay, median (IQR), d | 6.3 (3.7–11.8) | 6.0 (3.3–10.8) | < 0.001 |
| Admitted to ICU during hospitalization | 1,722 (39.2) | 2,194 (42.5) | < 0.001 |
| ICU length of stay, median (IQR), d | 2.8 (1.5–6.0) | 2.5 (1.3–4.8) | < 0.001 |
| Mortality | 524 (11.9) | 519 (10.1) | 0.004 |
Open in a separate window
IQR = interquartile range.
TABLE 3.
Characteristics and Outcomes of Patients With Epic Sepsis Model Score Greater Than or Equal to 5, Diagnosis of Sepsis and No Sepsis-Related IV Antibiotic Administered Prior to Epic Sepsis Model Greater Than or Equal to 5
| Characteristic | Hospitalizations, n (%) | p | |
|---|---|---|---|
| Baseline Period | Intervention Period | ||
| n | 304 | 628 | — |
| Age, median (IQR), yr | 69 (57–79) | 66 (53–76) | 0.023 |
| Gender | |||
| Female | 136 (44.7) | 316 (50.3) | 0.110 |
| Male | 168 (55.3) | 312 (49.7) | |
| Race/ethnicity | |||
| Caucasian | 219 (72.0) | 450 (71.7) | 0.250 |
| African American | 66 (21.7) | 147 (23.4) | |
| Hispanic | 15 (4.9) | 17 (2.7) | |
| Other | 4 (1.3) | 14 (2.2) | |
| Admitted through emergency department | 229 (75.3) | 497 (79.1) | 0.189 |
| Sepsis score (first score ≥ 5), median (IQR) | 6 (5–7) | 6 (5–7) | 0.338 |
| Sepsis order set used | 105 (34.5) | 271 (43.2) | 0.012 |
| Hospital length of stay, median (IQR), d | 6.1 (3.8–13.0) | 7.2 (3.9–12.9) | 0.380 |
| Admitted to ICU during hospitalization | 115 (37.8) | 313 (49.8) | < 0.001 |
| ICU length of stay, median (IQR), d | 3.2 (1.7–8.3) | 2.8 (1.3–5.2) | 0.020 |
| Time from ESM ≥ 5 to antibiotic administration, median (IQR), min | 150 (45–900) | 90 (33–255) | < 0.001 |
| Time from ESM ≥ 5 to antibiotic administration ≤ 180 min (3 hr) | 169 (55.6) | 438 (69.8) | < 0.001 |
| Mortality | 74 (24.3) | 100 (15.9) | 0.002 |
Open in a separate window
ESM = Epic Sepsis Model, IQR = interquartile range.
TABLE 4.
Multivariable Logistic Regression Analysis for Predictors of Hospital Mortality in Patients With a Sepsis Diagnosis and Epic Sepsis Model Score Greater Than or Equal to 5 (n = 2,988)
| Predictive Factor | OR (95% CI) | p |
|---|---|---|
| Patient Age (10-yr intervals) | 1.21 (1.13–1.29) | < 0.001 |
| Admitted through the emergency department | ||
| No | Referent | |
| Yes | 0.78 (0.61–1.00) | 0.051 |
| Admitted to ICU during hospitalization | ||
| No | Referent | |
| Yes | 2.62 (2.13–3.22) | < 0.001 |
| Order set used | ||
| No | Referent | |
| Yes | 0.70 (0.57–0.87) | < 0.001 |
| Study time period | ||
| Baseline | Referent | |
| Intervention | 0.71 (0.58–0.87) | < 0.001 |
Open in a separate window
OR = odds ratio.
TABLE 5.
Multivariable Logistic Regression Analysis for Predictors of Hospital Mortality in Patients With a Sepsis Diagnosis and No Sepsis-Related IV Antibiotic Administered Prior to Epic Sepsis Model Score Greater Than or Equal to 5 (n = 932)
| Predictive Factor | OR (95% CI) | p |
|---|---|---|
| Patient Age (10-yr intervals) | 1.18 (1.06–1.32) | 0.003 |
| Admitted through the emergency department | ||
| No | Referent | |
| Yes | 0.60 (0.41–0.89) | 0.012 |
| Admitted to ICU during hospitalization | ||
| No | Referent | |
| Yes | 2.46 (1.73–3.49) | < 0.001 |
| Order set used | ||
| No | Referent | |
| Yes | 0.85 (0.58–1.25) | 0.412 |
| Time from Epic Sepsis Model ≥ 5 to antibiotic administration | ||
| > 3 hr | Referent | |
| ≤ 3 hr | 0.83 (0.57–1.21) | 0.328 |
| Study time period | ||
| Baseline | Referent | |
| Intervention | 0.56 (0.39–0.80) | 0.001 |
Open in a separate window
OR = odds ratio.
Observed mortality rates and 95% CIs for baseline and intervention time periods are presented for all inpatients, subgroups of patients with and without a sepsis diagnoses, and ESM scores greater than or equal to 5 and less than 5 (Fig. Fig.11). Statistically significant differences were noted between the baseline and intervention periods in the ESM score greater than or equal to 5 population (all patients and patients with a sepsis diagnosis); however, no differences were discerned in the overall population or in those with ESM scores less than 5.
Figure 1.
Observed mortality rates by time period, Epic Sepsis Model (ESM) group, and presence or absence of sepsis diagnosis. Asterisk indicates a statistically significant difference in mortality rates between baseline and intervention time periods.
In the sepsis “alert” population (i.e., ESM score ≥ 5), statistically significant differences between the two time periods were noted in the proportions of patients with a diagnosis of sepsis (Table Table11), the proportion being admitted through the emergency department (ED) (Tables (Tables11 and 22), and the proportion having an ICU admission during the hospital stay (Tables (Tables11–3). Median differences occurred in sepsis score (Tables (Tables11 and and2),2), hospital LOS (Table (Table1),1), and ICU LOS (Tables (Tables11–3). Mortality rates were statistically significantly lower during the intervention period compared with the baseline period in all three populations—10.1% versus 11.9% (Table (Table1;1; p = 0.004), 14.5% versus 18.5% (Table (Table2;2; p = 0.003), and 15.9% versus 24.3% (Table Table33; p = 0.002).
TABLE 2.
Characteristics and Outcomes of Patients With Epic Sepsis Model Score Greater Than or Equal to 5 and Diagnosis of Sepsis
| Characteristic | Hospitalizations, n (%) | p | |
|---|---|---|---|
| Baseline Period | Intervention Period | ||
| n | 1,477 | 1,511 | — |
| Age, median (IQR), yr | 65 (53–76) | 64 (53–75) | 0.282 |
| Gender | |||
| Female | 715 (48.4) | 710 (47.0) | 0.437 |
| Male | 762 (51.6) | 801 (53.0) | |
| Race/ethnicity | |||
| Caucasian | 1,114 (75.4) | 1,111 (73.5) | 0.477 |
| African American | 289 (19.6) | 330 (21.8) | |
| Hispanic | 42 (2.8) | 41 (2.7) | |
| Other | 32 (2.2) | 29 (1.9) | |
| Admitted through emergency department | 1,172 (79.4) | 1,258 (83.3) | 0.006 |
| Sepsis score (first score ≥ 5), median (IQR) | 6 (5–8) | 6 (5–7) | 0.018 |
| Sepsis order set used | 680 (46.0) | 731 (48.4) | 0.200 |
| Hospital length of stay, median (IQR), d | 6.8 (4.0–12.8) | 7.0 (3.8–12.5) | 0.841 |
| Admitted to ICU during hospitalization | 644 (43.6) | 768 (50.8) | < 0.001 |
| ICU length of stay, median (IQR), d | 2.9 (1.5–6.1) | 2.7 (1.3–5.2) | 0.025 |
| Mortality | 273 (18.5) | 219 (14.5) | 0.003 |
Open in a separate window
IQR = interquartile range.
In the population of patients who had not received antibiotics prior to ESM score greater than or equal to 5 (Table (Table3),3), there was increased utilization of the sepsis order set during the intervention period (43.2% vs 34.5%; p = 0.012) and decreased time to administration of antibiotics after the ESM score greater than or equal to 5 alert (median 90 vs 150 min; p < 0.001). Receipt of antibiotics within three hours of the ESM alert increased from 55.6% to 69.8% during the baseline and intervention periods, respectively (p < 0.001).
Multivariable logistic regression analyses results for predictors of hospital mortality are provided in Tables Tables44 and and5.5. In patients with ESM Scores greater than or equal to 5 and a sepsis diagnosis (Table (Table4),4), increasing age (10-yr intervals) and admission to the ICU were independently associated with increased risk of mortality (odds ratios [ORs], 1.21 and 2.62, respectively). Conversely, order set utilization and interventional time period were associated with decreased risk of mortality (OR, 0.70 and 0.71, respectively). In patients with ESM scores greater than or equal to 5, a diagnosis of sepsis, and no IV antibiotics prior to ESM greater than or equal to 5 (Table (Table5);5); patient age and admission to the ICU were independently associated with increased risk of mortality (ORs, 1.18 and 2.46, respectively). Admission through the ED and the interventional time period were associated with decreased risk of mortality (ORs, 0.60 and 0.56, respectively); however, shorter time to antibiotic administration and order set utilization did not reach statistical significance in this smaller subset of patients.
DISCUSSION
This is one of the first studies attempting to externally validate the ESM Inpatient Predictive Analytic Tool and demonstrate its effect on mortality. Using this screening tool, we observed an absolute unadjusted mortality rate reduction of 4% in all patients with ESM greater than or equal to 5 and a sepsis diagnosis, and of 8% in those who had not received antibiotics before ESM score was greater than or equal to 5, respectively. In addition, the corresponding risk-adjusted ORs were 0.71 and 0.56 in the two populations comparing the intervention time period to the baseline time period. While a causal relationship cannot be definitively identified in this uncontrolled before-and-after study, the decrease in mortality rate observed was accompanied by differences in process measures that lend support to a causal effect. These include a decreased time to antibiotic administration, which was seen in bivariate analyses (Table (Table3)3) but did not reach statistical significance in multivariate analyses (Table (Table5).5). Similarly, sepsis order set usage increased in the intervention time period and was an independent predictor of decreased mortality in the ESM greater than or equal to 5 with sepsis diagnosis population (Table (Table4).4). However, this did not reach statistical significance in the smaller subset of patients who had not yet received antibiotics (Table (Table55).
Several other studies have also demonstrated the effectiveness of a sepsis early warning system (18–20, 23–27). These studies have also shown an improvement in early intervention for antibiotic administration and escalation (18, 26), diagnostic test (18–20), improvement of LOS (19, 28), and mortality (19, 22, 23, 25, 26). Another potential cause for this decrease mortality is the early admission of septic patients to the ICU. Although we can speculate on the reasons behind the increased and early admissions to the ICU observed in the intervention group, we cannot definitively establish a causal relationship based on the data obtained from our study.
However, not all studies have found electronic sepsis alert systems to improve patient outcomes (17, 28). A recent study from the University of Michigan by Wong et al (23) looked at 27,697 patients and concluded that the ESM had “poor discrimination and calibration in predicting the onset of sepsis.” There are several limitations of the study by Wong et al (23) that could explain the differing results. First, in the study by Wong et al (23), the score threshold for alerting the physicians for the possibility of sepsis was arbitrarily chosen at a score threshold of 6 or higher without validating the scoring system at their institution. Alternatively, our study used a cut point of 5 after an extensive validation period at our institution. Second, Wong et al (23) found that only 7% of patients with sepsis who were missed by a clinician based on timely administration of antibiotics prior to an ESM score of greater than 6, while 31.2% of patients in our study with sepsis were not on sepsis-appropriate antibiotics prior to the ESM score of greater than or equal to 5. This large discrepancy of already being on the appropriate IV antibiotics between the study by Wong et al (23) and the study reported here might be explained by the lower cut point of 5 we used, or different prescribing patterns of antibiotics between hospitals. While the study by Wong et al (23) was a retrospective review after the alert system was already implemented, the strength of our study is that we were able to compare sepsis-related outcomes using ESM scores prior to and after making them available for clinical use.
Another recent trial by Downing et al (17) failed to find a difference in sepsis-related outcomes including mortality, antibiotics given, lactic acid order, or rate of ICU admission/LOS with implementation of a sepsis alert system. The study by Downing et al (17) was a randomized controlled trial of 1,123 patients using a sepsis alert based on the definition of systemic inflammatory response syndrome (SIRS) plus a source of infection. Clinicians were alerted whether the patients were on antibiotics or not. A substantial portion of these patients (66%) were already on antibiotics when the alert system paged; therefore, less than 200 patients in each group did not have antibiotics when the sepsis alert fired. With a false positive rate of 40% in their study, approximately 100 patients in each group had sepsis without antibiotics prior to alert; therefore, the study was not powered to demonstrate a difference in mortality (17).
Sepsis alert systems that use clinical information not exclusively based on SIRS criteria tend to demonstrate improvement of more meaningful patient centered outcomes such as mortality (19, 24, 25, 27) compared with studies that had the presence of SIRS as an indication for alert (17, 20, 28). As of this article, Guirgis et al (29) is the only study, which demonstrated a mortality benefit for septic patients that used SIRS criteria for sepsis for an alert system.
Alert fatigue is also a well-recognized weakness of sepsis warning systems. An alert system becomes less efficacious in changing clinical practice as the frequency of the alerts increases or as the time the alert system has been in place increases. A study by Austrian et al (30) examined a sepsis alert system in the ED with 2/4 SIRS criteria being met. In that study, the positive predictive value (PPV) of the alert was only 13.0%, and there were 97,216 sepsis alerts over a 2-year period (2). The Sepsis alert system that we implemented using the ESM had a better PPV of sepsis at 33.8%. While a low PPV may raise concerns about the accuracy of positive test results, this trade-off is justified in certain clinical scenarios such as sepsis where there is a high disease prevalence in the hospital and a low PPV is still accompanied by a reasonable absolute number of true positives. By setting a cut point for “screening” that maximized sensitivity we ensured that a greater number of potential sepsis patients would receive the alert and the possibility of earlier intervention. With a PPV of 33.8%, roughly one in every three patients evaluated for sepsis would have a diagnosis of sepsis. Approximately 6–7% of the inpatient population in our hospital had an ESM score alert, which is roughly 45–50 patients per day. Physicians would be alerted only for patients who were not already being treated for sepsis. This less frequent firing of the sepsis alert system may mitigate alert fatigue as a possible reason for our improved results.
The present study has several limitations that should be acknowledged. First, this study used billing diagnosis codes to identify patients with sepsis. To study the outcomes of sepsis, an accurate and consistent way to retrospectively identify large groups of septic patients is necessary; however, no gold standard exists (31). This project began as a quality improvement effort, which relies on billing codes to identify sepsis. Although these billing codes have poor construct validity, they have wide availability and application to other institutions (32–34). A detailed human review for a more specific definition of sepsis was not feasible due to the measurement burden on this large of a scale.
Second, it is important to note that this investigation employed an uncontrolled before-and-after study design, which is inherently susceptible to bias. The primary vulnerability associated with before-and-after studies is the potential bias introduced by temporal trends or other longitudinal changes that occur between the before and after periods, thereby influencing the study outcomes. While efforts were made to minimize confounding factors, such as the absence of institution-wide changes in sepsis treatment during the study period, it should be recognized that the physicians and nurses involved in the study were educated on the ESM. This education may have contributed to an increased overall awareness of sepsis among the healthcare providers, potentially leading to a reduced time to administer antibiotics. Consequently, this could have influenced the observed outcomes.
Furthermore, it is important to acknowledge that this study focused on a specific single institutional setting, which may limit the generalizability of the findings to other contexts. Factors such as variations in healthcare practices, patient demographics, or resource availability could affect results in different healthcare settings.
Critics of sepsis early warning systems may postulate that the over-reliance of early warning systems for sepsis may increase the use of inappropriate antibiotics, testing, and costs in patients who had a high ESM score but did not eventually have sepsis. Clinical judgment remains essential, and further efforts are needed to improve the specificity of the ESM.
CONCLUSIONS
In this uncontrolled before-and-after study, we observed a 44% reduction in the odds of sepsis-related mortality after implementation of the ESM alert (OR, 0.56; 95% CI, 0.39–0.80). Due to the wide utilization of Epic for EMRs and the ease of integration into the EMR, this is a promising tool for an immediate impact on the improvement of sepsis mortality throughout the United States. Our study is hypothesis generating, and further work should use a controlled cohort or interrupted time series design to minimize bias and strengthen conclusions regarding the potential impact of the ESM alert system on sepsis mortality.
Supplementary Material
Click here to view.(197K, pdf)
Click here to view.(18K, pdf)
Footnotes
Dr. Cull involved in conceptualization, methodology, data curation, writing—original draft, and writing—review & editing. Drs. Brevetta, Gerac, and Kothari involved in conceptualization, methodology, writing—review & editing. Dr. Blackhurst involved in conceptualization, methodology, data curation, formal analysis, visualization, and writing—review & editing.
The authors have disclosed that they do not have any potential conflicts of interest.
This study was reviewed and approved by the Prisma Health Institutional Review Board on October 1, 2018 (No. Pro00081382).
Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s website (http://journals.lww.com/ccejournal).
REFERENCES
1. Opal SM, Rubenfeld GD, Poll TVD, et al.: The third international consensus definitions for sepsis and septic shock (Sepsis-3).JAMA2016; 315:801–810 [PMC free article] [PubMed] [Google Scholar]
2. Rhee C, Dantes R, Epstein L, et al.: Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014.JAMA2017; 02215:1241–1249 [PMC free article] [PubMed] [Google Scholar]
3. Kadri SS, Rhee C, Strich JR, et al.: Estimating ten-year trends in septic shock incidence and mortality in United States academic medical centers using clinical data.Chest2017; 151:278–285 [PMC free article] [PubMed] [Google Scholar]
4. Hatfield KM, Dantes RB, Baggs J, et al.: Assessing variability in hospital-level mortality among U.S. Medicare beneficiaries with hospitalizations for severe sepsis and septic shock.Crit Care Med2018; 46:1753–1760 [PMC free article] [PubMed] [Google Scholar]
5. Chalupka AN, Talmor D: The economics of sepsis.Crit Care Clin2019; 28:57–76, vi [PubMed] [Google Scholar]
6. Torio CM, Moore BJ: National inpatient hospital costs: The most expensive conditions by payer, 2013: Statistical Brief #204.In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville, MD, Agency for Healthcare Research and Quality (US), 2006, p 6. [PubMed] [Google Scholar]
7. Leisman DE, Doerfler ME, Ward MF, et al.: Survival benefit and cost savings from compliance with a simplified 3-hour sepsis bundle in a series of prospective, multisite, observational cohorts.Crit Care Med2017; 45:395–406 [PubMed] [Google Scholar]
8. Seymour CW, Gesten F, Prescott HC, et al.: Time to treatment and mortality during mandated emergency care for sepsis.N Engl J Med2017; 376:2235–2244 [PMC free article] [PubMed] [Google Scholar]
9. Ferrer R, Martin-Loeches I, Phillips G, et al.: Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour.Crit Care Med2014; 42:1749–1755 [PubMed] [Google Scholar]
10. Kumar A, Roberts D, Wood KE, et al.: Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock*.Crit Care Med2006; 34:1589–1596 [PubMed] [Google Scholar]
11. Levy MM, Evans LE, Rhodes A: The surviving sepsis campaign bundle: 2018 update.Intensive Care Med2018; 44:925–928 [PubMed] [Google Scholar]
12. Churpek MM, Snyder A, Han X, et al.: Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside theintensive care unit.Am J Respir Crit Care Med2017; 195:906–911 [PMC free article] [PubMed] [Google Scholar]
13. Smith GB, Prytherch DR, Meredith P, et al.: The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.Resuscitation2019; 84:465–470 [PubMed] [Google Scholar]
14. Subbe CP, Kruger M, Rutherford P, et al.: Original papers QJM validation of a modified early warning score in medical admissions.Q J Med2001; 94:521–526 [PubMed] [Google Scholar]
15. Pullyblank A, Tavare A, Little H, et al.: Implementation of the national early warning score in patients with suspicion of sepsis: Evaluation of a system-wide quality improvement project.Br J Gen Pract2020; 70:e381–e388 [PMC free article] [PubMed] [Google Scholar]
16. Nguyen SQ, Mwakalindile E, Booth JS, et al.: Automated electronic medical record sepsis detection in the emergency department.PeerJ2014; 2:e343. [PMC free article] [PubMed] [Google Scholar]
17. Downing NL, Rolnick J, Poole SF, et al.: Electronic health record-based clinical decision support alert for severe sepsis: A randomised evaluation.BMJ Qual Saf2019; 28:762–768 [PMC free article] [PubMed] [Google Scholar]
18. Sawyer AM, Deal EN, Labelle AJ, et al.: Implementation of a real-time computerized sepsis alert in nonintensive care unit patients*.SCCM2011; 39:469–473 [PubMed] [Google Scholar]
19. Shimabukuro DW, Barton CW, Feldman MD, et al.: Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: A randomised clinical trial.BMJ Open Resp Res2017; 880:1–8 [PMC free article] [PubMed] [Google Scholar]
20. Nelson JL, Smith BL, Jared JD, et al.: Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis.YMEM2019; 57:500–504 [PubMed] [Google Scholar]
21. Glaze J: Epic Systems Draws on Literature Greats for Its Next Expansion. Wisconsin State Journal, 2015. Available at: https://madison.com/news/local/govt-and-politics/epic-systems-draws-on-literature-greats-for-its-next-expansion/article_4d1cf67c-2abf-5cfd-8ce1-2da60ed84194.html. Accessed June 1, 2022
22. Bennett TD, Russell S, King J, et al.: Accuracy of the epic sepsis prediction model in a regional health system.arXiv Preprint posted online February 19, 2019. doi:10.48550/arXiv.1902.07276
23. Wong A, Otles E, Donnelly JP, et al.: External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients.JAMA Intern Med2021; 181:1065–1070 [PMC free article] [PubMed] [Google Scholar]
24. Manaktala S, Claypool SR: Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality.J Am Med Inform Assoc2017; 24:88–95 [PMC free article] [PubMed] [Google Scholar]
25. Honeyford K, Cooke GS, Kinderlerer A, et al.: Evaluating a digital sepsis alert in a London multisite hospital network: A natural experiment using electronic health record data.J Am Med Inform Assoc2020; 27:274–283 [PMC free article] [PubMed] [Google Scholar]
26. Shah T, Sterk E, Rech MA: Emergency department sepsis screening tool decreases time to antibiotics in patients with sepsis.Am J Emerg Med2018; 36:1745–1748 [PubMed] [Google Scholar]
27. Westra BL, Landman S, Yadav P, et al.: Secondary analysis of an electronic surveillance system combined with multi-focal interventions for early detection of sepsis.Appl Clin Inform2017; 8:47–66 [PMC free article] [PubMed] [Google Scholar]
28. Embi PJ, Leonard AC: Evaluating alert fatigue over time to EHR-based clinical trial alerts: Findings from a randomized controlled study.J Am Med Inform Assoc2012; 19:145–148 [PMC free article] [PubMed] [Google Scholar]
29. Guirgis FW, Jones L, Esma R, et al.: Managing sepsis: Electronic recognition, rapid response teams, ad standardized care saves lives.J Crit Care2017; 40:296–302 [PMC free article] [PubMed] [Google Scholar]
30. Austrian JS, Jamin CT, Doty GR, et al.: Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay.J Am Med Inform Assoc2018; 25:523–529 [PMC free article] [PubMed] [Google Scholar]
31. Saria S, Henry KE: Too many definitions of sepsis: Can machine learning leverage the electronic health record to increase accuracy and bring consensus?Crit Care Med2020; 48:137–141 [PubMed] [Google Scholar]
32. Jolley RJ, Quan H, Jette N, et al.: Validation and optimisation of an ICD-10-coded case definition for sepsis using administrative health data.BMJ Open2015; 5:e009487–e009410 [PMC free article] [PubMed] [Google Scholar]
33. Jolley RJ, Sawka KJ, Yergens DW, et al.: Validity of administrative data in recording sepsis: A systematic review.Crit Care2015; 19:139. [PMC free article] [PubMed] [Google Scholar]
34. Fleischmann-Struzek C, Thomas-Rüddel DO, Schettler A, et al.: Comparing the validity of different ICD coding abstraction strategies for sepsis case identification in German claims data.PLoS One2018; 13:e0198847. [PMC free article] [PubMed] [Google Scholar]
Articles from Critical Care Explorations are provided here courtesy of Wolters Kluwer Health
