National and state-level google-symptoms signals are on different scales

[ryantibs]

The COVID Symptom Searches on Google signal appears to be on the scale of roughly 0-100 at the national level and 0-1 at the state level. This results in very weird looking plots on the dashboard (when looking at a state, the national curve is overlaid, and makes the state curve look like it's basically at zero for all time):

@krivard @nmdefries @jingjtang: Is this just due to the way we're constructing the national signal (a bug)? Assuming so we should fix this ASAP because it's hurting the dashboard visualizations.

(Assuming not, we need to figure out a different way to visualize the national and state signals in tandem, and then I would transfer this issue over on the www-covidcast repo.).

[krivard]

This is expected behavior: google symptoms is, roughly speaking, a count not a proportion. Nation is the sum of all states; each state is the sum of its counties, etc.

[ryantibs]

It's not a count, because each time series is self-normalized. From their documentation:

---

The data shows the relative popularity of symptoms in searches within a geographical region. To normalize and scale the daily and the weekly time series (processed separately), we do the following for each region:

1. First, we count the number of searches for each symptom in that region for that day/week.
2. Next, we divide this count by the total number of Search users in the region for that day/week to calculate relative popularity (which can be interpreted as the probability that a user in this region will search for the given symptom on that day/week). We refer to this ratio as the normalized popularity of a symptom.
3. We then find the maximum value of the normalized popularity across the entire published time range for that region, over all symptoms using the chosen time resolution (day/week). We scale this maximum value to 100. All the other values are mapped to proportionally smaller values (linear scaling) in the range 0-100.
4. Finally, we store the scaling factor and use it to scale values (for the same region and time resolution) in subsequent releases. In future updates, when a symptom popularity exceeds the previously-observed maximum value (found in step 3), the new scaled values may be larger than 100.

---

So, given this, it doesn't make sense to sum them up. And in doing so, we make the visualizations like the ones I was showing above meaningless. An average or population-weighted average would make more sense. Agreed?

And a question: is this only a relevant consideration for the national signal? Am I correct in assuming this is the only place in which we're doing summing on our side? (It looks like they publish county and state values, so we're just grabbing these and storing them? We're not doing any summing to calculate state values, right?)

[krivard]

We use county and state values as-is.

We use a population-weighted average for msa and hrr.

We sum to get hhs and nation; this was confirmed by Jingjing in code review.

[jingjtang]

Ryan is right, the number is in an arbitrary unit that is defined by Google and we are provided with state/county level data saparately. The aggregation between different geo levels shouldn't happen. I had a vague memory to this PR, will double check after a meeting this afternoon.

But intuitively, the current strategy seems incorrect to me. It's likely that when I did a couple of code review for those add hhs and national PRs, I didn't notice the one for Google Symptom. Sorry about that.

[krivard]

@jingjtang can you take a look at correcting this?

[jingjtang]

@jingjtang can you take a look at correcting this?

Sure

[krivard]

Just so we have it written down here: the fix for this is to

• drop msa, hrr, and hhs aggregations
• revise nation aggregation to be a weighted average
• added: revise hhs aggregation to be a weighted average

Once the pipeline code is fixed, we'll need to take the following actions to fully implement them for covidcast3:

• update the API documentation for google-symptoms
• drop msa, hrr, and hhs data for google-symptoms in the database
• run the google-symptoms pipeline once for the full calendar of reference dates and upload
• added: drop bad old data for google-symptoms hhs and nation in the database

We can decide later whether to regenerate the version history for google-symptoms. iirc the source data doesn't do any backfill or other kinds of revisions, so we can do a straight mapping from reference date to nominal publish date.

[krivard]

Fixed!