BUG: when np.datetime64[ns] is a type in a MultiIndex, "NotImplementedError" when trying to return the df from a joblib.delayed

[RongkunWang]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import numpy as np
import pandas as pd
import joblib

def job():
	df = pd.DataFrame({
		'date' : [np.datetime64("20110101", 'ns')], 
		'id' : [1], 
		'val' : [1]
		}).set_index(["date", "id"])
	return df

joblib.Parallel(n_jobs = 32)(
	[joblib.delayed(job)()]
	)

Issue Description

joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
File "/lxhome/wangrong/equity_ls_mf_events/first_project/venvpy312/lib/python3.12/site-packages/joblib/externals/loky/process_executor.py", line 688, in wait_result_broken_or_wakeup
result_item = result_reader.recv()
^^^^^^^^^^^^^^^^^^^^
File "/opt/third/python/3.12/root/lib/python3.12/multiprocessing/connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pandas/_libs/arrays.pyx", line 85, in pandas._libs.arrays.NDArrayBacked.setstate
File "pandas/_libs/arrays.pyx", line 103, in pandas._libs.arrays.NDArrayBacked.setstate
NotImplementedError: (dtype('<M8[ns]'), array(['2026-04-05T16:07:45.133961216'], dtype='datetime64[ns]'))
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/lxhome/wangrong/equity_ls_mf_events/first_project/bugs/pickle_datetime.py", line 18, in
)
^^
File "/lxhome/wangrong/equity_ls_mf_events/first_project/venvpy312/lib/python3.12/site-packages/joblib/parallel.py", line 2072, in call
return output if self.return_generator else list(output)
^^^^^^^^^^^^
File "/lxhome/wangrong/equity_ls_mf_events/first_project/venvpy312/lib/python3.12/site-packages/joblib/parallel.py", line 1682, in _get_outputs
yield from self._retrieve()
File "/lxhome/wangrong/equity_ls_mf_events/first_project/venvpy312/lib/python3.12/site-packages/joblib/parallel.py", line 1784, in _retrieve
self._raise_error_fast()
File "/lxhome/wangrong/equity_ls_mf_events/first_project/venvpy312/lib/python3.12/site-packages/joblib/parallel.py", line 1859, in _raise_error_fast
error_job.get_result(self.timeout)
File "/lxhome/wangrong/equity_ls_mf_events/first_project/venvpy312/lib/python3.12/site-packages/joblib/parallel.py", line 758, in get_result
return self._return_or_raise()
^^^^^^^^^^^^^^^^^^^^^^^
File "/lxhome/wangrong/equity_ls_mf_events/first_project/venvpy312/lib/python3.12/site-packages/joblib/parallel.py", line 773, in _return_or_raise
raise self._result
joblib.externals.loky.process_executor.BrokenProcessPool: A result has failed to un-serialize. Please ensure that the objects returned by the function are always picklable.

Expected Behavior

No error. I observed no error in pandas 2.2.3. It seems there's no error when it's not MultiIndex.

Installed Versions

INSTALLED VERSIONS

commit : 9c8bc3e
python : 3.12.6
python-bits : 64
OS : Linux
OS-release : 3.10.0-1160.119.1.el7.x86_64
Version : #1 SMP Tue Jun 4 14:43:51 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.3.3
numpy : 2.3.4
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 25.3
Cython : None
sphinx : None
IPython : 9.7.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.14.2
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : 6.0.2
matplotlib : 3.10.0
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 20.0.0
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.15.3
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2025.2
qtpy : None
pyqt5 : None

[jbrockmendel]

Can you reproduce this without joblib? Im guessing something pickle-related?

[RongkunWang]

The error does read like pickle related. Without the joblib it is fine. But even with joblib and pickle the older version of pandas is fine too. Cheers, Rongkun.

…

________________________________ From: jbrockmendel ***@***.***> Sent: Tuesday, November 11, 2025 8:01:52 PM To: pandas-dev/pandas ***@***.***> Cc: Rongkun Wang ***@***.***>; Author ***@***.***> Subject: Re: [pandas-dev/pandas] BUG: when np.datetime64[ns] is a type in a MultiIndex, "NotImplementedError" when trying to return the df from a joblib.delayed (Issue #63078) [https://avatars.githubusercontent.com/u/8078968?s=20&v=4]jbrockmendel left a comment (pandas-dev/pandas#63078)<#63078 (comment)> Can you reproduce this without joblib? Im guessing something pickle-related? — Reply to this email directly, view it on GitHub<#63078 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEEGKLZQAS3M5YYUB6PIPHD34IXCBAVCNFSM6AAAAACLZFINYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKMJYGM3DINBWGM>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[threexc]

I am working with the RISE Project, and we have also been encountering this while trying to build riscv64 wheels for pandas v2.3.2 and v2.3.3. We seem to be hitting the first NotImplementedError in pandas/_libs/arrays.pyx on line 107.

Interestingly, it only seems to occur if we let cibuildwheel pull in the latest Cython (3.2.0) - if we force the build to use the same version as our v2.3.1 builds used (3.1.2), everything passes OK.

Compare:

failing pipeline with Cython 3.2.0
passing pipeline with Cython 3.1.2

Note that the early log output from the failing build shows us modifying the pyproject.toml to pin Cython==3.1.2, but that didn't actually take effect for the build because of another workaround we had in the pipeline - you can see the actual version being installed by cibuildwheel is 3.2.0.

It seems like even basic pickling is failing (here using a Python 3.12 wheel we tested in a manylinux container):

(venv) bash-5.2# python3
Python 3.12.9 (main, Aug 14 2025, 00:00:00) [GCC 14.2.1 20250110 (Red Hat 14.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import pandas as pd
>>> print(f'Byte order: {sys.byteorder}')
Byte order: little
>>> print(f'Pandas version: {pd.__version__}')
Pandas version: 2.3.3
>>> cat = pd.Categorical(['bar', 'baz', 'foo'])
>>> print(f'Categorical works: {cat}')
Categorical works: ['bar', 'baz', 'foo']
Categories (3, object): ['bar', 'baz', 'foo']
>>> import pickle
>>> pickled = pickle.dumps(cat)
>>> unpickled = pickle.loads(pickled)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/venv/lib64/python3.12/site-packages/pandas/core/arrays/categorical.py", line 1726, in __setstate__
    return super().__setstate__(state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas/_libs/arrays.pyx", line 85, in pandas._libs.arrays.NDArrayBacked.__setstate__
  File "pandas/_libs/arrays.pyx", line 103, in pandas._libs.arrays.NDArrayBacked.__setstate__
NotImplementedError: (CategoricalDtype(categories=['bar', 'baz', 'foo'], ordered=False, categories_dtype=object), array([0, 1, 2], dtype=int8))