Skip to content

Conversation

Kopilov
Copy link
Contributor

@Kopilov Kopilov commented Jul 15, 2022

Arrow reading test with more data examples (generation is presented at https://github.com/Kopilov/arrow_example).

Read UInt-s without overflow, read date and time as date and time.

NullabilityOptions pattern.

@Kopilov Kopilov mentioned this pull request Jul 15, 2022
@Kopilov Kopilov changed the title Expanding Arrow reading support Expand Arrow reading support Jul 15, 2022
@koperagen
Copy link
Collaborator

This test fails. Can you check? We use it as a sample for docs
https://github.com/Kotlin/dataframe/blob/master/tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/Read.kt#L79

Also, in public class ArrowFeather : SupportedFormat readDataFrame functions should set NullabilityOptions.Widening as an argument

@Kopilov
Copy link
Contributor Author

Kopilov commented Jul 18, 2022

This test fails. Can you check? We use it as a sample for docs
https://github.com/Kotlin/dataframe/blob/master/tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/Read.kt#L79

This is because I removed file ./tests/src/test/resources/data-arrow_2.0.0_uncompressed.feather (that was originally copied to ./dataframe-arrow/src/test/resources/data-arrow_2.0.0_uncompressed.feather). Is using only one of them possible or we can only restore the copy?

@Kopilov
Copy link
Contributor Author

Kopilov commented Jul 18, 2022

Also, in public class ArrowFeather : SupportedFormat readDataFrame functions should set NullabilityOptions.Widening as an argument

Do I understand correctly that we should use Widening in SupportedFormat readDataFrame and Infer (formerly Keeping) as default in DataFrame.Companion.readArrowFeather methods?

@koperagen
Copy link
Collaborator

Also, in public class ArrowFeather : SupportedFormat readDataFrame functions should set NullabilityOptions.Widening as an argument

Do I understand correctly that we should use Widening in SupportedFormat readDataFrame and Infer (formerly Keeping) as default in DataFrame.Companion.readArrowFeather methods?

Yes

@koperagen
Copy link
Collaborator

This test fails. Can you check? We use it as a sample for docs
https://github.com/Kotlin/dataframe/blob/master/tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/Read.kt#L79

This is because I removed file ./tests/src/test/resources/data-arrow_2.0.0_uncompressed.feather (that was originally copied to ./dataframe-arrow/src/test/resources/data-arrow_2.0.0_uncompressed.feather). Is using only one of them possible or we can only restore the copy?

I think a copy won't harm

@koperagen koperagen merged commit a746109 into Kotlin:master Jul 18, 2022
@koperagen
Copy link
Collaborator

@Kopilov Thank you!

@Kopilov
Copy link
Contributor Author

Kopilov commented Jul 19, 2022

@koperagen you are welcome!

Our next step is to make saving :-)

@Kopilov Kopilov deleted the arrow branch July 19, 2022 06:12
@Jolanrensen Jolanrensen added this to the 0.9.0 milestone Dec 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants