Data preprocessing configuration

[rabsr]

Hi,

I want to add a custom data preprocessor and configure AutoSklearnClassifier just to include my custom data preprocessing algorithm and exclude other available algorithms.
So basically I have two doubts:

1. How can I add custom data preprocessing tools?
2. How can I exclude data preprocessing step same as we can do for feature_preprocessing using 'no_preprocessing'?

[mfeurer]

Hi,

1. what kind of preprocessing tool do you want to use?
2. Kind of, we do not have any programmatic way for doing so yet, you'd have to change the pipeline in code: https://github.com/automl/auto-sklearn/blob/master/autosklearn/pipeline/classification.py#L277

[rabsr]

Hi @mfeurer

Thanks for reply.

1. We have some modified data preprocessor tools on premise as per our requirements.
2. Are there any plans for making data preprocessing step configurable in coming releases?

I have few other questions regarding Pipeline execution

1. Auto-sklearn uses predefined pipeline steps and does not provide capability to use custom pipelines. Are there any plans for this to include in later releases?
2. We want to add new classification algorithms which need execution on spark. Does autosklearn allows spark execution of its existing and newly added algorithms?
3. Autosklearn allows parallel mode using several computational resources. Can we run each autosklearn pipeline in spark and use hdfs or s3 for shared output directory?

I can work on pipeline extensibility and data preprocessing configuration part.

[mfeurer]

We have some modified data preprocessor tools on premise as per our requirements.

Then I'm afraid you need to change the code of Auto-sklearn to include it.

Are there any plans for making data preprocessing step configurable in coming releases?

Yes, but there's no timeline.

Auto-sklearn uses predefined pipeline steps and does not provide capability to use custom pipelines. Are there any plans for this to include in later releases?

Yes, same here.

We want to add new classification algorithms which need execution on spark. Does autosklearn allows spark execution of its existing and newly added algorithms?

We do not support any framework for loading off workload right now. However, adding the algorithm might still work. Unfortunately I can't say anything about that because I don't know enough about Spark.

Autosklearn allows parallel mode using several computational resources. Can we run each autosklearn pipeline in spark and use hdfs or s3 for shared output directory?

Sorry, I've never tried any of those.

I can work on pipeline extensibility and data preprocessing configuration part.

That would be very much appreciated. Being able to pass custom pipelines to Auto-sklearn would be a great feature to have. Do you have any concrete suggestions on how to approach this?

[rabsr]

@mfeurer I have raised a PR #977 for making data preprocessing step configurable and allow users to skip the step if required. Please review. I haven't fixed test cases/examples or added new test cases required. I will do it once the approach looks fine.

[mfeurer]

Thanks a lot for this PR @rabsr

Reviewing this is on my stack, and I hope that I'll get to this during this week.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs for the next 7 days. Thank you for your contributions.

[rabsr]

Closing as feature merged via #977