Add API to unfold object-columns into properties via reflection

[holgerbrandl]

As with unfold() in krangl (see https://holgerbrandl.github.io/krangl/data_model/#to-type-or-not-to-type), it would be great if kotlin-df could provide a similar API. Clearly, this could be done manually using df.add(), but with many attributes this is very tedious. And in krangl we've seen that this can be done very efficiently via reflection.

Example (from krangl but yet missing in kdf):

data class City(val name:String, val code:Int)
data class Person(val name:String, val address:City)

val persons : List<Person> = listOf(
    Person("Max", City("Dresden", 12309)),
    Person("Anna", City("Berlin", 10115))
)

val personsDF: DataFrame = persons.asDataFrame() // <- Also sems missing API in kdf?

// unfold City attributes into different columns
personsDF.unfold<City>("address")  // <- Missing API in kdf

For impl see https://github.com/holgerbrandl/krangl/blob/49418b2ca6ee6ae165c034e56e4da77e4707f7ad/src/main/kotlin/krangl/Builder.kt#L115

[koperagen]

persons.asDataFrame() -- i believe persons.toDataFrame() does the trick https://kotlin.github.io/dataframe/createdataframe.html#todataframe

[holgerbrandl]

@koperagen Thanks for the pointer.

I'd still think that the key aspect of this ticket is still valid: Once we have objects of type E in a column of a data-frame we can not conveniently expand E's properties into columns (without writing it out attribute by attribute using add()). That's what unfold was/is doing in krangl and I used it quite frequently.

[koperagen]

@koperagen Thanks for the pointer.

I'd still think that the key aspect of this ticket is still valid: Once we have objects of type E in a column of a data-frame we can not conveniently expand E's properties into columns (without writing it out attribute by attribute using add()). That's what unfold was/is doing in krangl and I used it quite frequently.

Indeed. We'll introduce this API. I also made an example of how it can be done now.

class RepositoryInfo(val data: Any)

fun download(url: String) = RepositoryInfo("fancy response from the API")

val interestingRepos = dataFrameOf("name", "url")(
    "dataframe", "/dataframe",
    "kotlin", "/kotlin",
)

val initialData = interestingRepos
    .add("response") { download("url"()) }

class WebScrappingInitialData(val name: String, val url: String, val response: RepositoryInfo)

val df = initialData.cast<WebScrappingInitialData>(verify = true)

val response by column<RepositoryInfo>()

df
    .replace(response)
    .with { it.asIterable().toDataFrame().asColumnGroup(it.name()) }
    //.ungroup(response)
    .print()

df
    .replace("response")
    .with { (it as DataColumn<RepositoryInfo>)
        .asIterable()
        .toDataFrame {
            // todo a better example for this dsl
            properties(maxDepth = 1) {
                
            }
        }
        .asColumnGroup(it.name())
    }
    .rename { columnGroup("response").children() }.into { "response_" + it.name() }
    .ungroup("response")
    .schema().print()

[nikitinas]

Actually, there is non-documented API for that. I haven't found a good name for that, so it's currently called read:

data class City(val name:String, val code:Int)
data class Person(val name:String, val address:City)

val persons : List<Person> = listOf(
    Person("Max", City("Dresden", 12309)),
    Person("Anna", City("Berlin", 10115))
)

val personsDF = persons.toDataFrame()

personsDF.read(Person::address)

Thank you, @holgerbrandl, unfold seems to be a good name for this operation. Did you take it from some other dataframe library?

[nikitinas]

And also List -> DataFrame conversion is called toDataFrame instead of asDataFrame, because in Kotlin Stdlib as means wrapping and to means copying: asSequence, but toList.

[holgerbrandl]

The name was inspired from the terminology section in https://tidyr.tidyverse.org/. Altough in tidyr they ended up using unnest_wider (see https://tidyr.tidyverse.org/reference/hoist.html). unnest is also a nice candidate imho.

Regarding to vs as, indeed naming was/is wrong in krangl. I'm a slow learner so it took me forever to memorize the semantics here. :-)

[Jolanrensen]

added as unfold() :)