Invalid .internal.selfref detected when a data.table is set as an attribute with setattr

[d-sci]

If I have an object (data.table or not) and I add a data.table attribute to it via attr<-, the address of the OBJECT changes, but the address of the ATTRIBUTE does not. I can later extract the data.table attribute, and modifications to it will also modify the attribute.

x <- 'x'
dt <- data.table(a = 1)
tracemem(dt)
#> [1] "<0x39b9bd0>"
tracemem(x)
#> [1] "<0x3203148>"
attr(x, 'dt') <- dt
#> tracemem[0x3203148 -> 0x224b840]: eval
dt2 <- attr(x, 'dt')
dt2[, b := 1]
dt2[]
#>    a b
#> 1: 1 1
attr(x, 'dt')
#>    a b
#> 1: 1 1

But if I add a data.table attribute to it via setattr, the address of the OBJECT does not change, whereas the address of the ATTRIBUTE does. If I later extract the attribute, it will be de-referenced:

x <- 'x'
dt <- data.table(a = 1)
tracemem(dt)
#> [1] "<0x46d3dc0>"
tracemem(x)
#> [1] "<0x3207768>"
setattr(x, 'dt', dt)
#> tracemem[0x46d3dc0 -> 0x432dfc0]: setattr
dt2 <- attr(x, 'dt')
dt2[, b := 1]
#> Warning in `[.data.table`(dt2, , `:=`(b, 1)): Invalid .internal.selfref
#> detected and fixed by taking a (shallow) copy of the data.table so
#> that := can add this new column by reference. At an earlier point, this
#> data.table has been copied by R (or was created manually using structure()
#> or similar). Avoid names<- and attr<- which in R currently (and oddly)
#> may copy the whole data.table. Use set* syntax instead to avoid copying: ?
#> set, ?setnames and ?setattr. If this message doesn't help, please report
#> your use case to the data.table issue tracker so the root cause can be
#> fixed or this message improved.
dt2[]
#>    a b
#> 1: 1 1
attr(x, 'dt')
#>    a
#> 1: 1

Note that in this second example I used setattr, not attr<-, and yet seemingly R copied the data.table somewhere. The message didn't help, so here I am.

sessionInfo:

R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.5 LTS

Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.12.2

loaded via a namespace (and not attached):
[1] compiler_3.5.2 tools_3.5.2    yaml_2.2.0 

[tdeenes]

I think this relates to #1218 (an issue I opened in 2015...). The point is that setattr makes a copy of ANY object which is given as the attribute.

[d-sci]

Indeed the same issue. Interesting to know that bit::setattr has a different (no-copy) behaviour.

[jangorecki]

Note that in this second example I used setattr, not attr<-, and yet seemingly R copied the data.table somewhere. The message didn't help, so here I am.

It doesn't address the issue, but to clarify...
I believe that data (columns) have not been copied, but only list (data.table) been copied. That is fortunately a very cheap copy, and happens every time when selfref is invalid (after using names<- or attr<-).

[JB304245]

Hi,

I got the following:

> dataset[,unique_row_id := c(1:nrow(dataset))]
Warning message:
In `[.data.table`(dataset, , `:=`(unique_row_id, c(1:nrow(dataset)))) :
  Invalid .internal.selfref detected and fixed by taking a (shallow) copy of the data.table so that := can add this new column by reference. At an earlier point, this data.table has been copied by R (or was created manually using structure() or similar). Avoid names<- and attr<- which in R currently (and oddly) may copy the whole data.table. Use set* syntax instead to avoid copying: ?set, ?setnames and ?setattr. If this message doesn't help, please report your use case to the data.table issue tracker so the root cause can be fixed or this message improved.

The step in my code that triggers this is the following:

dataset[[paste0('prediction_', model_version)]] = predict(csgo_live_round_winner,
                                                          prediction_input_matrix)

Where csgo_live_round_winner is an xgboost model object and model_version is a string.

If I move dataset[,unique_row_id := c(1:nrow(dataset))] somewhere above the predict call in my script, i dont get the warning message, so I did that to avoid any potential problems. The order isnt really important.

If I change my code to use data.table syntax in that spot, like such:

prediction_col_name = paste0('prediction_', model_version)
dataset[, c(prediction_col_name) := predict(csgo_live_round_winner,
                                                          prediction_input_matrix)]

There is also no warning message.

I was pretty confused by the warning message, so I followed the instructions and posted here :). Please note im only an intermediate R user and relatively new to data.table

[MichaelChirico]

Hi @JB304245 for this error, reproducible examples are essential for debugging, so anything you could do to make an MRE of your issue would go a long way to it getting solved. Thanks!

[ColeMiller1]

Here is a MRE. It seems related to [[<-.data.table

library(data.table)

dt = data.table(x = 1)
dt[["x"]] = 2L
dt[, y:= 3L]

#> Warning in `[.data.table`(dt, , `:=`(y, 3L)): Invalid .internal.selfref detected
#> and fixed by taking a (shallow) copy of the data.table so that := can add this
#> new column by reference. At an earlier point, this data.table has been copied
#> by R (or was created manually using structure() or similar). Avoid names<- and
#> attr<- which in R currently (and oddly) may copy the whole data.table. Use set*
#> syntax instead to avoid copying: ?set, ?setnames and ?setattr. If this message
#> doesn't help, please report your use case to the data.table issue tracker so the
#> root cause can be fixed or this message improved.

It seems like the warning message should either add that dt$ <- can cause the issue. Or perhaps we could just remove the warning. I tried names(dt) = ... but I could not trigger the warning.

library(data.table)
dt = data.table(x = 1)
names(dt) <- "y"
dt[, z:= 3]
names(dt) <- letters[1:2]
dt[, d:= 4][]
#>        a     b     d
#>    <num> <num> <num>
#> 1:     1     3     4

[MichaelChirico]

Agreed about changing the warning. Not sure we can drop about names (keep in mind it would have to be true for R 3.1.0 as well)...

The error is indeed because of [[<-:

dt = data.table(x = 1)
tracemem(dt)
# [1] "<0x378cb30>"
dt[["x"]] = 2L
# tracemem[0x378cb30 -> 0x3a4f9b8]: 
# tracemem[0x3a4f9b8 -> 0x3a61a28]: [[<-.data.frame [[<-

i.e. a copy is silently made by [[<- and the allocation gets out of sync. setDT after [[<- also fixes it.

In any case I don't think that's related to the current issue & AFAIK is known behavior. @JB304245 if the warning message was confusing for you, could you please open a separate issue for that? It's an intimidating message, probably it's worth some effort to making that message more accessible.

[ColeMiller1]

@MichaelChirico I agree that this sidebar regarding [[<- isn't about directly about this issue. At the same time, what is the resolution to this issue? Addressing #1218 and improving the warning message?