-
+Figure 1. Total and Variant Case Counts with variant changepoints on April 24, 2021; June 19, 2021; December 18, 2021; March 5, 2022; June 18, 2022.
@@ -51,9 +49,9 @@
Establishing National Ground Truth Changepoints
Methods
Next, we needed to identify how these ground truth changepoints compared to changepoints in indicators (using the same three algorithms). We used all available data streams during our selected time range from Delphi’s COVIDCast database, which corresponds to 60 indicators (including the Facebook survey, Google Search, claims, hospitalization, and mortality indicators), each at the state/territory and national tier. When we aggregated the calculated changepoints per indicator per week, as Fig. 2 shows, we identified that some weeks have many more changepoints than others. To match the number of ground truth changepoints, we filter to the top 5 weeks per indicator (major indicator changepoints) to directly compare with the ground truth changepoints.
-
+Figure 2. The aggregated count of changepoints per week in all fifty two regions by all three algorithms for estimated COVID related outpatient doctor visits
Then, we categorized indicators as early, on-time, or late using the following definitions:
@@ -117,23 +115,23 @@Results and Analysis
Early Indicators: Early indicators are the most important in identifying changing health dynamics. For example, the majority (6/8) of indicators from Google Symptoms are early indicators, especially indicators related to fevers. In Fig 3. we see one example of Google Symptoms early indicator (which includes searches for Fever, Hyperthermia, Chills, Shivering, and Low grade fever), where the major indicator changepoints lead ground truth changepoints by approximately one week.
-
+Figure 3. This early indicator has major indicator changepoints (blue bars) which occur a week or more before the ground truth changepoints (black line). This data is sourced from Google Symptoms and shows search data for keywords “Fever, Hyperthermia, Chills, Shivering, Low grade fever”
On-time Indicators: On-time indicators (Fig. 4) most frequently correspond to data involving insurance claims, suspected COVID cases among hospital admissions, and, unsurprisingly, other COVID-19 incidence data reports from JHU-CSSE. Other notable early or on-time indicators include: the number of doctors visits for cases with COVID or COVID-like symptoms, Data Strategy and Execution Workgroup Community Profile Report (CPR) data on the number of people who were fully vaccinated.
-
+Figure 4. This on-time indicator has major indicator changepoints (blue bars) which occur within a two week window around the ground truth changepoints (black).
Variable Indicators: Variable indicators are perhaps the most interesting set of indicators. For most of the available range, the JHU-CSSE new confirmed COVID-19 cases daily indicator was early, but the last changepoint was late. Many other JHU-CSSE indicators follow this pattern. Another example is the Google Symptom Search data related to smell and taste loss, specifically, “Anosmia, Dysgeusia, Ageusia.” In the figure below, we see the relationship between the indicator’s major changepoints and the ground truth change from early in the first ground truth changepoint to late in the middle of the pandemic, and then early again for the rest of the ground truth changepoints.
-
+Figure 5. This variable indicator has changepoints (blue bars) which occur early and late relative to the critical changepoints (black) established using CDC data.
@@ -144,6 +142,7 @@
Limitations and Discussion
Due to state level reporting inconsistencies, we could only analyze 60 out of 79 indicators available in the timespan investigated. Many of these indicators were missing data, like that the number of confirmed COVID-19 patients admitted to all hospitals in the state of Alaska was reported only five times within the 595 day search window. We also recognize that not all regions will be impacted by emerging variants at the same time in the same way and that a further detailed analysis which takes into account different impacts per region is an important avenue for future work.
Overall, Changepoint detection is a powerful tool to identify early indicators. Of Delphi’s sixty indicators, we identified several on time and early indicators of emerging variants from the data available. We also found out that for many of the indicators, the number of days they led or lagged disease phenomena changed over time. Still, if these public health indicators continue to receive high quality data, tracking these indicators closely can help us identify changing health dynamics.
The python notebook used for this analysis can be found on Github here.
+This work was supported by the Centers for Disease Control and Prevention of the U.S. Department of Health and Human Services (HHS) as part of a cooperative agreement funded solely by CDC/HHS under federal award identification number U01IP001121, “Delphi Influenza Forecasting Center of Excellence”; and by CDC funded contract number 75D30123C15907, “Digital Public Health Surveillance for the 21st Century”. The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement by, CDC/HHS or the U.S. Government. Additionally, this material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE1745016 and DGE2140739. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
diff --git a/content/blog/2024-01-01-flash-intro.Rmd b/content/blog/2024-01-01-flash-intro.Rmd index 6f34dc739..a101d3957 100644 --- a/content/blog/2024-01-01-flash-intro.Rmd +++ b/content/blog/2024-01-01-flash-intro.Rmd @@ -18,7 +18,7 @@ output: toc: true --- -Delphi publishes millions of public-health-related data points per day, including the total number of daily influenza cases, hospitalizations, and deaths per county and state in the United States (US). This data helps public health practitioners, data professionals, and members of the public make important, informed decisions relating to health and well-being. +Delphi publishes millions of public-health-related data points per day, such as the total number of daily COVID-19 cases, hospitalizations, and deaths per county and state in the United States (US). This data helps public health practitioners, data professionals, and members of the public make important, informed decisions relating to health and well-being. Yet, as data volumes continue to grow quickly (Delphi's data volume expanded 1000x in just 3 years), it is infeasible for data reviewers to inspect every one of these data points for subtle changes in diff --git a/content/blog/2024-01-01-flash-intro.html b/content/blog/2024-01-01-flash-intro.html index c0da0036a..ffc7ac290 100644 --- a/content/blog/2024-01-01-flash-intro.html +++ b/content/blog/2024-01-01-flash-intro.html @@ -26,10 +26,10 @@
These issues, if undetected, can have critical downstream ramifications on data users (as shown by the example in Fig 1).
-
-These issues, if undetected, can have critical downstream ramifications for data users (as shown by the example in Fig 1).
+
+Fig 1. Data quality changes in case counts, shown by the large spikes in March and July 2022, when cases were trending down, resulted in similar spikes for predicted counts (red) from multiple forecasts that were then sent to the US CDC. A weekly forecast per state, for cases, hospitalizations, and deaths, up to 4 weeks in the future means that modeling teams would have to review 600 forecasts per week and may not have been able to catch the upstream data issue.
We care about finding data issues like these so that we can alert downstream data users accordingly. That is why our goal in the FlaSH team (Flagging Anomalies in Streams related to public Health) is to quickly identify data points that warrant human inspection and create tools to support data review. Towards this goal, our team of researchers, engineers, and data reviewers iterate on our deployed interdisciplinary approach. In this blog series, we will cover the different methods and perspectives of the FlaSH project.
Members: Ananya Joshi, Nolan Gormley, Richa Gadgil, Tina Townes
diff --git a/content/blog/2024-01-30-flash-framework.Rmd b/content/blog/2024-01-30-flash-framework.Rmd new file mode 100644 index 000000000..b2664051e --- /dev/null +++ b/content/blog/2024-01-30-flash-framework.Rmd @@ -0,0 +1,63 @@ +--- +title: Alerting Systems are Incompatible with Modern Public Health Data Monitoring +author: Ananya Joshi +date: 2024-01-01 +tags: + - flash +authors: + - ajoshi +heroImage: blog_alerts_hero.png +heroImageThumb: blog_alerts_thumb.png +output: + blogdown::html_page: + toc: true +acknowledgements: Thank you to George Haff, Carlyn Van Dyke, and Ron Lunde for editing this blog post. +--- +Insights from public health data can keep communities safe. However, identifying these insights in large volumes of modern public health data can be laborious^[Rosen, George. A history of public health. JHU Press, 2015.]. As a result, over the past few decades, public health agencies have built monitoring systems, like [ESSENCE](https://www.cdc.gov/nssp/new-users.html) (CDC), [EIOS](https://www.who.int/initiatives/eios) (WHO), and [DHIS2](https://dhis2.org/) (WHO), where users can set custom statistical alerts and then investigate these alerts using data visualizations^[Chen, Hsinchun, Daniel Zeng, and Ping Yan. Infectious disease informatics: syndromic surveillance for public health and biodefense. Vol. 21. New York: Springer, 2010.]. These alerting systems largely follow the following formula^[Murphy, Sean Patrick, and Howard Burkom. "Recombinant temporal aberration detection algorithms for enhanced biosurveillance." Journal of the American Medical Informatics Association 15.1 (2008): 77-86.] as shown in Fig 1.: + + ++"The current disconnect among algorithm developers, implementers, and [experts] … foster[s] distrust in statistical monitoring and in biosurveillance itself." - Galit Shmeuli & Howard Burkom^[Shmueli, Galit, and Howard Burkom. "Statistical challenges facing early outbreak detection in biosurveillance." Technometrics 52.1 (2010): 39-51.] ++ +Interestingly, through our prior system, I had the opportunity to experience all three roles, from developing the outlier detection methods, implementing and deploying the methods over Delphi's data, and then analyzing the outputs of the methods. These three roles came with different constraints and priorities. These experiences set the foundation for devising an interdisciplinary approach. + +
Insights from public health data can keep communities safe. However, identifying these insights in large volumes of modern public health data can be laborious1. As a result, over the past few decades, public health agencies have built monitoring systems, like ESSENCE (CDC), EIOS (WHO), and DHIS2 (WHO), where users can set custom statistical alerts and then investigate these alerts using data visualizations2. These alerting systems largely follow the following formula3 as shown in Fig 1.:
+
+-
+
- Predict a value that you expect the data to have +
- Calculate the difference between your observed and predicted values +
- Contextualize the difference (e.g., to account for variability in the data) +
- Send alerts when this value passes a threshold +
Using this formula, if we assume that our data follows the bell curve of a normal distribution, we can calculate the difference between our prediction (which might be the average of the data) and what we observe, and divide by the standard deviation. If this value exceeds a specific threshold (here, we use 3), then the probability that we would observe that data point in that bell curve would small (here, it’s less than 1%), which means that it is quite surprising. By setting this threshold high, only a small percentage of points are identified as outliers and thus result in alerts.
+This was the same intuition we used while initially designing Delphi’s data monitoring system at the pandemic’s start. This monitoring system was particularly important because Delphi’s stakeholders wanted to be aware of any events in the data - like changes in data quality or disease dynamics. First, we split these events into data validation (e.g., negative cases) and events that needed a human-facing statistical alert (using the above formula) so that an expert could ultimately interpret and classify the event.
+However, we quickly realized this formula did not work with Delphi’s large data volume. To illustrate: creating alerts for 1% of the data over millions of data points, like those analyzed by Delphi, results in 10,000+ alerts - far too many for us at Delphi to inspect and investigate! Standard correction approaches were also incompatible with public health data. Our first attempt was reducing the percentage of alerts (e.g., changing thresholds that would identify the data as alerts from 1% to 0.01%). However, because modeling public health data is difficult, defining thresholds (e.g., 1 in 10000) when data streams only have a few hundred historical data points can be somewhat arbitrary. Additionally, when applying the same standard threshold-generating function across all the data (adaptive thresholds), we missed some important events from signals that were not well-calibrated to the function. Tuning these parameters by hand across hundreds of signals was tedious, and Delphi turned off the outlier alerting system.
+The historical alerting framework cannot keep up with monitoring the increasingly large volumes of heterogeneous public health data that organizations like Delphi process. Delphi has increased its volume by 1000x in the past three years alone. And, although Delphi processes data from several geographical regions, even local public health agencies who use systems (that focus on one geography) under the existing alerting framework have challenged its usefulness (e.g. “What Can You Really Do with 35,000 Statistical Alerts a Week Anyways?” (2019)). We anticipate this failure mode for alerting systems will become more prevalent as data volumes grow.
+Bridging the gap with FlaSH
+Identifying and communicating noteworthy events from public health data is a critical step for future pandemic prevention4. But how should we redesign our event detection process? +After reflecting on our prior system, we realized the reason why existing monitoring fails is best summarized by the following quote:
++“The current disconnect among algorithm developers, implementers, and [experts] … foster[s] distrust in statistical monitoring and in biosurveillance itself.” - Galit Shmeuli & Howard Burkom5 ++
Interestingly, through our prior system, I had the opportunity to experience all three roles, from developing the outlier detection methods, implementing and deploying the methods over Delphi’s data, and then analyzing the outputs of the methods. These three roles came with different constraints and priorities. These experiences set the foundation for devising an interdisciplinary approach.
+
+Combining the methods, engineering, and quality assurance perspectives for event detection from large-scale public health streams, we developed a human-in-the-loop approach to address some fundamental challenges in existing alerting systems. Namely, we rank Delphi’s recently updated data by how much it warrants reviewer attention. Reviewers can then decide how many data points they want to investigate, which prioritizes a reviewer’s time and does not expect them to review a certain number of alerts flagged via some arbitrary threshold.
+Here’s a quick illustration of why the ranking approach works. The most limited resource in this setting is an expert reviewer’s time and attention. We want to show this reviewer as many potential events as possible so that they can interpret the data with their known context.
+
+Now imagine each data point per signal was given an outlier score, and there is an arbitrary threshold where scores higher than that threshold are considered outliers. If we develop granular enough rankings, we can directly compare outlier points between different signals without arbitrary thresholds. Then, our data reviewer can focus on data points in their order of importance and stop the review at any time.
+Developing this framework required new methods, visualizations, and evaluations. Over the past few years, our goal as part of the FlaSH team was to implement and iterate on this framework. We’re excited to share insights from these perspectives as we’ve moved FlaSH from a sandboxed research project to a deployed part of Delphi’s systems that helps our users. For the next post in this series, I’ll describe some of the novel methods we use to rank events in large volumes of public health data. Then, Richa Gadgil and Nolan Gormley will share their thoughts on moving these research prototypes into a production environment. Third, Tina Townes and Catalina Vajiac will describe the data visualization and quality assurance processes and considerations they encountered while facilitating event detection investigation. Beyond that, we’ll share updates on how we work towards tightening the human-in-the-loop process!
+This blog post includes material from my thesis proposal, “Identifying Anomalous Events in Modern +Public Health Data Streams”
+This work was supported by the Centers for Disease Control and Prevention of the U.S. Department of Health and Human Services (HHS) as part of a cooperative agreement funded solely by CDC/HHS under federal award identification number U01IP001121, “Delphi Influenza Forecasting Center of Excellence”; and by CDC funded contract number 75D30123C15907, “Digital Public Health Surveillance for the 21st Century”. The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement by, CDC/HHS or the U.S. Government. Additionally, this material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE1745016 and DGE2140739. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
++
-
+
Rosen, George. A history of public health. JHU Press, 2015.↩︎
+Chen, Hsinchun, Daniel Zeng, and Ping Yan. Infectious disease informatics: syndromic surveillance for public health and biodefense. Vol. 21. New York: Springer, 2010.↩︎
+Murphy, Sean Patrick, and Howard Burkom. “Recombinant temporal aberration detection algorithms for enhanced biosurveillance.” Journal of the American Medical Informatics Association 15.1 (2008): 77-86.↩︎
+“Rapidly Detecting and Responding to Health Emergencies.” World Health Organization, https://www.who.int/activities/rapidly-detecting-and-responding-to-health-emergencies. Accessed 29 Jan. 2024.↩︎
+Shmueli, Galit, and Howard Burkom. “Statistical challenges facing early outbreak detection in biosurveillance.” Technometrics 52.1 (2010): 39-51.↩︎
+