FIX: itemsize wrong for date32[day][pyarrow] dtype #57948 #62657
Conversation
|
How to address test fails not due to commit code changes? Will be glad to receive some guidance. |
|
@jbrockmendel could you provide some insights on this issue? My local CI pipeline has passed all tests with the last commit but not on the pandas repo. |
|
Those failures are unrelated and affecting all PRs at the moment. We all need to wait a little bit for an upstream fix. |
| ("decimal256", 32), | ||
| ], | ||
| ) | ||
| def test_arrow_dtype_itemsize_fixed_width(pa, type_name, expected_size): |
There was a problem hiding this comment.
Can you move this test to pandas/tests/extension/test_arrow.py?
There was a problem hiding this comment.
Only this particular test test_arrow_dtype_itemsize_fixed_width or all 3? I guess all 3 as they test the same functionality?
There was a problem hiding this comment.
All the new tests in this file
| if pa.types.is_boolean(self.pyarrow_dtype): | ||
| return self.numpy_dtype.itemsize |
There was a problem hiding this comment.
Can you move this outside the try except?
There was a problem hiding this comment.
Sure thing. Do you think it would be better to also catch NotImplementedError in the except block?
There was a problem hiding this comment.
catch NotImplementedError in the except block?
Yes or experiment what exceptions .bit_width could raise on the pyarrow side
There was a problem hiding this comment.
I have done the corections and checked running the CI routine on my fork, works as expected.
Waiting for the CI routine to be fixed on this repo.
3 participants
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.Fixes #57948
Summary
ArrowDtype.itemsizewas incorrectly returning8bytes fordate32[day]and otherfixed-widthPyArrow types because it always fell back tonumpy_dtype.itemsize. This PR uses PyArrow'sbit_widthforfixed-widthtypes and gracefully falls back tonumpyforvariable-widthtypes andbooleans.Changes
ArrowDtype.itemsizeto usepyarrow_dtype.bit_widthwhen availableExample
Before:
ArrowDtype(pa.date32()).itemsize == 8After:
ArrowDtype(pa.date32()).itemsize == 4Local Environment
NOTE : If additional information required on the local environment, would be happy to provide.