Improve list.chunked() + List<List<T>>.toDataFrame for parsing .srt and similar text files

[koperagen]

Consider file structured as this:

header1
data1

header2
data2

I believe it's popular, one example is srt, but i personally had to deal with it a lot

It can be parsed into dataframe:

File("data").readText().lines().chunked(3).map { it.dropLast(1) }.toDataFrame()

Surprisingly here toDataFrame is not generic Iterable.toDataFrame, but completely another function.
Problem: In current shape it's not helpful.
I end up with something very close to what i want, but required change to code is somewhat non-trivial.

I'd either have to:

val data = File("data").readText().lines().chunked(3).map { it.dropLast(1) }
val df = buildList { 
  add(listOf("col1", "col2"))
  addAll(data)
}.toDataFrame()

Or switch to completely different route:

val df = File("data").readText().lines().chunked(3).map { it.dropLast(1) }.toDataFrame("data").split("data").cast<List<*>>().into("col1", "col2")

with compiler plugin

val df = File("data").readText().lines().chunked(3).map { it.dropLast(1) }.toDataFrame("data").split { data }.into("col1", "col2")

With this API change:

val df = File("data").readText().lines().chunked(3).map { it.dropLast(1) }.toDataFrame(header = listOf("col1", "col2"))

Plugin will understand resulting schema too

[zaleslaw]

I agree that this a popular use-case, I also faced with the same and handled with File API parsing

Great, if tiny example, saying, with subtitles will be also added

[Jolanrensen]

this is a breaking change, can we keep the old function and deprecate it?

[koperagen]

Deprecated and moved from io to api package

[Jolanrensen]

Seems like a useful addition :) it's common in excel/csv's too to take the first row as headers unless headers are supplied explicitly, so it makes sense here too I guess

[koperagen]

@zaleslaw

Great, if tiny example, saying, with subtitles will be also added

Updated documentation.
In the process, updated page with headers to be able to refer to individual operations
Before:

After:

[Jolanrensen]

cool! :D

[Jolanrensen]

useful feature and even better example :)

You still have a failing test, but other than that it looks good