Error creating dummy predictions: {'error': 'Result queue is empty', 'configuration_origin': 'DUMMY'}

[iXanthos]

Greetings,

I am trying to run autosklearn 0.6.0 on an HPC infrastructure. I am using the kr-vs-kp dataset (https://www.openml.org/d/3) and I have made all the values of the dataset numeric. The error I get is:

Unhandled exception in thread started by libgcc_s.so.1 must be installed for pthread_cancel to work Traceback (most recent call last): File "main_script.py", line 41, in <module> autosklearn_script.autosklearn_func(train_data,train_target,test_data,test_target,dataset_info_dict,task,time_threshold) File "/users/pr004/ixanthos/test_files/autosklearn_script.py", line 31, in autosklearn_func automl.fit(train_data, train_target, metric=autosklearn.metrics.roc_auc) File "/apps/applications/pytorch/1.3.1/lib/python3.6/site-packages/autosklearn/estimators.py", line 664, in fit dataset_name=dataset_name, File "/apps/applications/pytorch/1.3.1/lib/python3.6/site-packages/autosklearn/estimators.py", line 337, in fit self._automl[0].fit(**kwargs) File "/apps/applications/pytorch/1.3.1/lib/python3.6/site-packages/autosklearn/automl.py", line 996, in fit load_models=load_models, File "/apps/applications/pytorch/1.3.1/lib/python3.6/site-packages/autosklearn/automl.py", line 208, in fit only_return_configuration_space=only_return_configuration_space, File "/apps/applications/pytorch/1.3.1/lib/python3.6/site-packages/autosklearn/automl.py", line 384, in _fit num_run = self._do_dummy_prediction(datamanager, num_run) File "/apps/applications/pytorch/1.3.1/lib/python3.6/site-packages/autosklearn/automl.py", line 313, in _do_dummy_prediction raise ValueError("Dummy prediction failed: %s " % str(additional_info)) ValueError: Dummy prediction failed: {'error': 'Result queue is empty', 'configuration_origin': 'DUMMY'}

Any ideas?

Thank you in advance.

[willeppy]

I'm having a similar issue. I'm running this from a Jupyter Notebook and it seems to arise when I try to create a second classifier. For example, if I make a new classifier from autosklearn.classification.AutoSklearnClassifier() it works fine the first time but if I try to make another classifier later in the same notebook I am getting this error.

[iXanthos]

I'm having a similar issue. I'm running this from a Jupyter Notebook and it seems to arise when I try to create a second classifier. For example, if I make a new classifier from autosklearn.classification.AutoSklearnClassifier() it works fine the first time but if I try to make another classifier later in the same notebook I am getting this error.

Is there a way to replicate this? Because I cannot in my Jupyter environment. I have trouble replicating the error I came across myself, so I am trying to understand when it occurs.

[mfeurer]

@iXanthos does this issue still appear with the latest release of Auto-sklearn (0.7.0)?

[arnaldog12]

yes, @mfeurer. I have just tested.

[mfeurer]

@arnaldog12 do you also get the error message:

Unhandled exception in thread started by libgcc_s.so.1 must be installed for pthread_cancel to work

[rabsr]

I am also having similar issue. However I don't get the following error message:

Unhandled exception in thread started by libgcc_s.so.1 must be installed for pthread_cancel to work

[rabsr]

I was getting this error due to insufficient memory. The problem solved for me once I increased ml_memory_limt.

[mfeurer]

That's interesting, thanks for the pointer @rabsr

Does the error look like, i.e. does it contain the threading memory error?

Unhandled exception in thread started by <bound method Thread._bootstrap of <Thread(QueueFeederThread, initial daemon)>>
Traceback (most recent call last):
  File "/home/feurerm/miniconda/3-4.5.4/envs/autosklearn/lib/python3.7/threading.py", line 885, in _bootstrap
MemoryError
[ERROR] [2020-08-14 11:15:37,969:AutoML(1):breast_cancer] Error creating dummy predictions: {'error': 'Result queue is empty', 'configuration_origin': 'DUMMY'} 
Traceback (most recent call last):
  File "/home/feurerm/sync_dir/projects/automl_competition_2015/auto-sklearn/examples/example_holdout.py", line 45, in <module>
    automl.fit(X_train, y_train, dataset_name='breast_cancer')
  File "/home/feurerm/sync_dir/projects/automl_competition_2015/auto-sklearn/autosklearn/estimators.py", line 686, in fit
    dataset_name=dataset_name,
  File "/home/feurerm/sync_dir/projects/automl_competition_2015/auto-sklearn/autosklearn/estimators.py", line 374, in fit
    self._automl[0].fit(**kwargs)
  File "/home/feurerm/sync_dir/projects/automl_competition_2015/auto-sklearn/autosklearn/automl.py", line 1072, in fit
    load_models=load_models,
  File "/home/feurerm/sync_dir/projects/automl_competition_2015/auto-sklearn/autosklearn/automl.py", line 398, in fit
    num_run = self._do_dummy_prediction(datamanager, num_run)
  File "/home/feurerm/sync_dir/projects/automl_competition_2015/auto-sklearn/autosklearn/automl.py", line 277, in _do_dummy_prediction
    % (str(status), str(additional_info)))
ValueError: Dummy prediction failed with run state StatusType.CRASHED and additional output: {'error': 'Result queue is empty', 'configuration_origin': 'DUMMY'}.

Process finished with exit code 1

[rabsr]

@mfeurer I was running in Jupyter Notebook and was using Spark for parallel execution.
The error thrown by spark job:

  File "<ipython-input-6-0b9e1875cdb0>", line 24, in get_spawn_classifier
  File “/tmp/spark/userFiles/autosklearn/estimators.py", line 681, in fit
    dataset_name=dataset_name,
  File "/autosklearn/estimators.py", line 351, in fit
    self._automl[0].fit(**kwargs)
  File "/tmp/spark/user-files/autosklearn/automl.py", line 1026, in fit
    load_models=load_models,
  File "/tmp/spark/user-files/autosklearn/automl.py", line 213, in fit
    only_return_configuration_space=only_return_configuration_space,
  File "/tmp/spark/user-files/autosklearn/automl.py", line 397, in _fit
    num_run = self._do_dummy_prediction(datamanager, num_run)
  File "/tmp/spark/user-files/autosklearn/automl.py", line 318, in _do_dummy_prediction
    raise ValueError("Dummy prediction failed: %s " % str(additional_info))
ValueError: Dummy prediction failed: {'configuration_origin': 'DUMMY', 'error': 'Result queue is empty'} ```

[mfeurer]

Thanks @rabsr. Is this stdout/stderr or what is given in a log file? Unfortunately, we currently don't have a nice way to figuring out what goes wrong with the pynisher here, so we need to work on that. I'll ask around with my colleagues who had interest in improving the pynisher as well and see if we can get something started.

[rabsr]

@mfeurer This is from stdout/stderr.

[mfeurer]

Dear users who had this issue, since Auto-sklearn 0.11.1 the output of a failed DummyClassifier should be way more informative. Is one of you able to reproduce this issue and can post the error message here? Thanks a lot!

[MaxHeuillet]

Hello I am also having this issue...

I run Autosklearn2 on a DGX on a small sized file, so it can not be linked to a memory error.

[ERROR] [2021-02-04 15:00:26,097:Client-AutoML(1):d8cda2dc35a43ca16e484b4f0d0ed92b] Dummy prediction failed with run state StatusType.CRASHED and additional output: {'error': 'Result queue is empty', 'exit_status': "<class 'pynisher.limit_function_call.AnythingException'>", 'subprocess_stdout': '', 'subprocess_stderr': '', 'exitcode': 0, 'configuration_origin': 'DUMMY'}.

The parameters I have are:

automl = autosklearn.classification.AutoSklearnClassifier( ensemble_size=1, time_left_for_this_task = 30 ) automl.fit(train_x, train_y)

What I don't understand is that it works on my local laptop but not on the DGX.

Please help, I need your framework 2.0 working for my experiments.

Thanks !