Missing multiclass-multioutput support

[mitar]

I see that multiclass-multioutput support is missing? Couldn't in that case code just split learning into one model per output?

[mfeurer]

Yes, that is correct. The feature is not implemented since there is no metric for that in scikit-learn, no one of the contributors so far needed that feature and we do not have any data to test that. Do you have any reference for multiclass-multioutput?

[mitar]

I think my issue was in fact #293. A continuous-multioutput was misclassified as multiclass-multioutput so this is why I thought I need it.

I see that there is an active pull requests to add a metric though: scikit-learn/scikit-learn#3681 (addressing scikit-learn/scikit-learn#3453) But it looks stale.

[mfeurer]

Okay, I see. However, continuous-multioutput is not supported by auto-sklearn. This type of class is only supported by the tree-based models in scikit-learn, right? What would be an application of this kind of data?

[mitar]

No, it seems it is supported by any regressor:

Multioutput regression support can be added to any regressor with MultiOutputRegressor. This strategy consists of fitting one regressor per target. Since each target is represented by exactly one regressor it is possible to gain knowledge about the target by inspecting its corresponding regressor. As MultiOutputRegressor fits one regressor per target it can not take advantage of correlations between targets.

BTW, similar can be done for classifiers as well, using MultiOutputClassifier.

I think it would be a good first start, just to have something.

What would be an application of this kind of data?

I mean, application is that I can run auto-sklearn on any dataset I get, no matter the task it has. ;-)

[mfeurer]

The reason why we didn't implement anything together with the MultiOutputClassifier and MultiOutputRegressor is that they didn't nicely support partial_fit and models with warmstarts. As partial_fit seems to be added in sklearn==0.19 we can hopefully add this feature then.

[BrechtBa]

Is there any progress on this issue?
I also have several multi-output regressions to which I would like to apply auto-sklearn.

[mfeurer]

No, there's no one working on this. Do you want to contribute this feature?

[berisfu]

If the y label is (longitude,latitude),which means i wanna predict a location, auto-sklearn can support? I think there are numerous case about geo location，for example to predict where the user will drop off or pickup for Uber.

[mfeurer]

Auto-sklearn currently does not implement this feature. While we think that this feature would be good to have, I doubt that anyone from our team will implement this in the near future. Therefore, any contribution in getting this feature into Auto-sklearn would be greatly appreciated.

[Skylion007]

What would be needed to get this feature in? This library is missing crucial functionality without supporting these two problem types. I'd be willing to look into this if I can get an idea of how much effort it would take.

[mfeurer]

From the top of my head:

• making sure that the metrics work with this kind of data
• making sure that all the models are able to train on this kind of data
• making sure that ensembles work on this kind of data
  And then doing an integration check.

Point 2 is somewhat optional if only having unsupervised preprocessing and random forests for classification/regression are fine.

[Skylion007]

I mean theoretically every regressor can support it via this cludge? Of course, that's not optimal, but it will work since it cannot correlate data between output but it will give some functionality instead of just erroring. There is a multiclassification class that does the same thing.

For metrics, the following metrics are supported for multioutput according to the doc:

mean_squared_error, mean_absolute_error, explained_variance_score and r2_score.

Following classifiers support multilabel without the cludge.

It's difficult to find a list of regressors that support multioutput but it looks DecisionTrees and forms of Linear regressions along with their variants support it out of the box without the multioutput cludge.

That seems to be what I can find in the latest about which metrics and models work. Haven't found much about ensemble for multioutput regression but at least some of them support multiclass models. Given the list of regressors and models that support it, what would need to be done?

[Skylion007]

Alternatively, RegressorChain will be in the next version of Sklearn so that might be easier to work with: https://github.com/scikit-learn/scikit-learn/pull/9257/files

We already have ClassiferChain after all.

[mfeurer]

Yes, that's what I meant that wrapper which you'd need to plug around all kinds of classifiers - but I think that's secondary to get some basic functionality. Also, I don't think that ClassifierChain and RegressorChain are easily applicable in Auto-sklearn as it would be unclear how to tune their hyperparameters in a fast way.

Regarding the ensemble: Auto-sklearn uses an ensemble to post-hoc combine the models into an ensemble - ideally that one still works afterwards.

I think it would be easiest to start by adding multilabel regression (if that's of interest to you) by:

• adding a new flag handles_multilabel_regression to all models which support it in this directory: https://github.com/automl/auto-sklearn/tree/master/autosklearn/pipeline/components/regression
• add a new task type to the constants: https://github.com/automl/auto-sklearn/blob/master/autosklearn/constants.py
• check for that task type in this util file: https://github.com/automl/auto-sklearn/blob/master/autosklearn/util/pipeline.py
• Make the abstract regression component aware of multilabel regression, similar to the abstract classification component.
• Add a check for multilabel regression here as done for multiclass and multilabel
• Make the estimators and underlying base class aware of multilabel regression.

Please excuse that this is rather complicated and not all in one place, but we didn't desgin Auto-sklearn to be extensible for different tasks.

[charlesfu4]

My thesis topic is quite related to this issue, which is forecasting multi-output electricity load. I wonder if the team is working on multi-output auto regressor. If not, I will be willing to try that.