Parameter in AutoSklearnEstimator to accept smac callback

[bitsbuffer]

Hi,
The smac3 writes a file at smac3-output/runhistory.json which gives a detailed summary for trials and their results. This file is written when the smac3 completes the black-box optimization. However, apart from logs, there is no way to write the trial information in between. However, by providing an option to the user to pass a callback function, this functionality can be achieved. I have noticed that in smbo.py at line 522, autosklearn is already registering a callback for ensemble learning. Likewise, a user-provided callback can be registered which then can be executed by optimizer/smbo.py _incorporate_run_results

[mfeurer]

Hi @bitsbuffer, thanks a lot for the suggestion. Just checking, do you want general callbacks or would you like to have access to the output of SMAC during the run? I think both could be achieved. Would you like to work on that?

[bitsbuffer]

Hi @mfeurer, I would like if callback gets the results of a trial. This way trial's results can be accessed as soon as it is finished rather than waiting for intensifier to complete. I would really like to take this task up, would submit a PR by the weekend. Hope that works.

[mfeurer]

I would really like to take this task up, would submit a PR by the weekend. Hope that works.

Sorry for the slow reply, that would definitely be appreciated!

[bitsbuffer]

@mfeurer I have created a PR for this functionality, please check #1140

[bitsbuffer]

@mfeurer Any update on this ?

[mfeurer]

I'm currently going through PRs and hope to get to yours today; sorry for the delay!

[bitsbuffer]

@mfeurer Could you review the #1140 changes ?

[mfeurer]

This has been added via #1140 and will be available in the next major release. Thanks a lot for your contribution.