summarize() with multi-row returns

[krlmlr]

As of dplyr 1.0.0, summarize() will create multiple rows per group, according to the length of the return value of the summary function. This new feature leads to unintended behavior if the vector return is accidental, and also can lead to data loss.

library(conflicted)
library(dplyr)

my_custom_summary_function <- function(n) {
  # Should return a scalar, but I accidentally return a vector
  rep(n, n)
}

tibble(n = 2:0) %>% 
  group_by(n) %>% 
  summarize(out = my_custom_summary_function(n), .groups = "drop") %>% 
  ungroup()
#> # A tibble: 3 × 2
#>       n   out
#>   <int> <int>
#> 1     1     1
#> 2     2     2
#> 3     2     2

^{Created on 2022-08-01 by the reprex package (v2.0.1)}

Should we introduce a .multi = c("allow", "require", "fail") argument that supports the pre-1.0.0 strict mode of operation? Should .multi = "fail" even be the default?

library(conflicted)
library(dplyr)

my_custom_summary_function <- function(n) {
  # Should return a scalar, but I accidentally return a vector
  rep(n, n)
}

tibble(n = 2:0) %>% 
  group_by(n) %>% 
  summarize(out = my_custom_summary_function(n), .groups = "drop", .multi = "fail") %>% 
  ungroup()
## Error: `out` has length != 1 in groups 1, 3, use `.multi = "allow"` if this is intended

^{Imagined on 2022-08-01 by the reprex package (v2.0.1)}

[hadley]

I'm not too worried about this; in general any misspecified summary function could corrupt data.

[DavisVaughan]

See also https://twitter.com/drob/status/1563198515626770432?s=20&t=iTFWSCPNOGWalIrpXHx2qg

[kcarnold]

I have occasionally written multi-row summarize pipelines intentionally, but when I do that I need to very carefully document what that code is doing. When I teach summarize, the working model I use is "one row per group"; otherwise it gets confused with mutate. Yes, any bugs in summary functions could corrupt data, but this behavior should be opt-in because (1) R's recycling rules make this sort of behavior easy to trigger accidentally, (2) it's hard to notice and then diagnose when it does happen, and (3) since one-row-per-group is the expected behavior, code that does something different should look like it's doing something different.

[hadley]

Our plan is to deprecate this behaviour in summarise() and instead introduce a new function specifically for this purpose. We just need a name. Ideas so far:

• morph()
• transmogrify()
• multisummarise()
• abridge()
• remodel()
• remould()
• renovate()
• revamp()
• abridge()
• shorten()
• contract()
• lessen()
• condense()
• synopsize()

I think ideally the word would be closer to summarise() than mutate(), i.e. starting later in the alphabet or ending in ise (although then we'd need UK/US variants, which isn't ideal). I think it's ok if the verb implies an unconditional shrinking, even though some uses might increase the number of rows; we also say that [ subsets a vector.

[DavisVaughan]

I like morph() the most out of all of these. It implies some kind of stretching/shrinking of the data without implying any direction. And it is fairly short.

[romainfrancois]

I like morph() too. What will happen with morph(<grouped_df>) ?

Are we considering being specific about shinking/growing ? i.e. we could have shrink() and grow() or something.

[DavisVaughan]

morph(<grouped_df>) would have to work like summarise(<grouped_df>) currently works, I think. i.e. each group computation can return any number of rows, and we recycle the per group results "rowwise" across the resulting columns. And we'd add .by support for morph()

I'm somewhat confident we don't need to care about the direction, mainly it's:

• summarise() has the guarantee of 1 row per group. More predictable for users. Harder to make a mistake. Easier data base translations.
• morph() just relaxes that guarantee, but otherwise works similarly. But when you see morph() in code it should be a clear signal that something is happening that isn't a pure summary, which is pretty nice