AutoSklearn2Classifier issues being pickle-ed

[23pointsNorth]

Describe the bug

I've been testing the AutoSklearnClassifier and AutoSklearn2Classifier classifiers.

Using the v1, I get expected results in training the model. However, on saving, it crashes with the following details:

/home/bot/.local/lib/python3.6/site-packages/sklearn/linear_model/_stochastic_gradient.py:557: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.
ConvergenceWarning)
/home/bot/.local/lib/python3.6/site-packages/sklearn/base.py:197: FutureWarning: From version 0.24, get_params will raise an AttributeError if a parameter cannot be retrieved as an instance attribute. Previously it would return None.
FutureWarning)
Traceback (most recent call last):
File "/usr/lib/python3.6/pickle.py", line 918, in save_global
obj2, parent = _getattribute(module, name)
File "/usr/lib/python3.6/pickle.py", line 266, in _getattribute
.format(name, obj))
AttributeError: Can't get local attribute 'get_smac_object_callback..get_smac_object' on <function get_smac_object_callback at 0x7ff884638378>
During handling of the above exception, another exception occurred:
joblib.dump(model, osp.join(base_path, model_type + "_model.pkl"))
_pickle.PicklingError: Can't pickle <function get_smac_object_callback..get_smac_object at 0x7ff8427e7f28>: it's not found as autosklearn.experimental.askl2.get_smac_object_callback..get_smac_object

From what I've gathered, the model cannot be pickled.

To Reproduce

// Make a classifier using:
from autosklearn.classification import AutoSklearnClassifier
import joblib
model = AutoSklearnClassifier()
model.fit(X,y)
joblib.dump(model, "model.pkl")

Expected behavior

Both v1 and v2 classifiers to be able to fit, save and re-load.

Actual behavior, stacktrace or logfile

See above

Environment and installation:

Please give details about your installation:
Versions in docker container:
sklearn: 0.22.2.post1
auto-sklearn: 0.8.0
python: 3.6

[Atharex]

AutoSK2 Classifier promptly throws an error after starting. Guess it is the same problem as the OP...

Seems like the tool is trying to pickle a selector into a system library folder, instead of into my current folder
(it also ignores the "output_folder" parameter in the Class constructor)

Example code:

from autosklearn.experimental.askl2 import AutoSklearn2Classifier
model = AutoSklearn2Classifier( 
    include_estimators=["random_forest", ], exclude_estimators=None,
    include_preprocessors=["no_preprocessing", ], exclude_preprocessors=None,
    n_jobs=8,
    output_folder='.')    

Error output:

---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
<ipython-input-84-90b488fce84f> in <module>
      3 import sklearn.metrics
      4 
----> 5 from autosklearn.experimental.askl2 import AutoSklearn2Classifier
      6 model = AutoSklearn2Classifier( #)
      7     include_estimators=["random_forest", ], exclude_estimators=None,

/usr/local/lib/python3.8/site-packages/autosklearn/experimental/askl2.py in <module>
     40         maxima=maxima_for_methods,
     41     )
---> 42     with open(selector_file, 'wb') as fh:
     43         pickle.dump(selector, fh)
     44 

PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.8/site-packages/autosklearn/experimental/selector.pkl'

[mfeurer]

Thanks a lot @23pointsNorth and @Atharex for reporting these issues. I can confirm both issues and we'll work on fixing these.

[budiryan]

@mfeurer,

The error encountered by @23pointsNorth was due to python's standard pickle or joblib not being able to serialise functions. Switching to dill resolves this issue.

However, just switching library will not be enough. Another error will come up: Runtime Error: Lock objects should only be shared between processes through inheritance.

The fix to that subsequent error is in #955, by making the lock object to be global, together with CALLBACK_COUNTER.

[mfeurer]

Closing this as it is fixed by #976.