Should we create a `dask.ml` namespace?

[mrocklin]

For dask.distributed we created a namespace in the dask so that people stopped calling import distributed. This was also nice because it makes things feel a bit more integrated

from dask.distributed import Client  # <--- actually imports from a separate distributed module

Should we do the same for other libraries? Dask_ml seems like a good candidate to test this.

This is the relevant file in the dask library: https://github.com/dask/dask/blob/master/dask/distributed.py

[stsievert]

I don't have a strong opinion. I decided to look for some more context:

• Every library listed under "documentation" on https://dask.org has a dask.foo namespace except Dask-ML (array, dataframe, custom applications/delayed, real time/distributed).

I'm not seeing the downside to having a dask.ml namespace if Dask-ML is featured pretty prominently on the documentation. I think consistency is improved with a separate namespace, but that might be outweighed by the downsides.

Documentation	Imports	URL
	import dask.array as da import dask.dataframe as dd import dask_ml import dask.delayed from dask.distributed import Client	https://dask.org/

The "algorithms" heading under https://docs.dask.org adds Dask Bags and XArray, which are importable with import dask.bag and import xarray as xr. The "setup" heading mentions the following methods to setup Dask Distributed:

• Single machine: import dask.distributed
• SSHCluster: from dask.distributed.deploy.ssh import SSHCluster
• Job-Queue: from dask_jobqueue import PBSCluster
• Kubernetes: from dask_kubernetes import KubeCluster
• Yarn: from dask_yarn import YarnCluster

---

More context:

• The 2019 Dask user survey says this many respondents use the following APIs:
  • Dask-ML, XArray: ~25%
  • Futures: ~35%
  • Array: ~55%
  • Delayed: ~60%
  • DataFrame: 75%
• There are 500 SO questions on "dask array," "dask dataframe" and "xarray", and 127 SO questions on "dask-ml" (example query).

[mrocklin]

I tried doing this as a RFC PR, but then I found that it was hard because users will probably want to import across the library boundary like this:

from dask.ml.model_selection import GridSearchCV

And our current approach of from dask_ml import * won't work with this. (current approach below)

try:
    from dask_ml import *
except ImportError as e:
    msg = (
        "Dask ML is not installed.\n\n"
        "Please either conda or pip install dask_ml:\n\n"
        "See https://ml.dask.org/install.html for more information"
    )
    raise ImportError(msg) from e

cc @Carreau and @jakirkham both of whom seem like they might have tricks here.

[Carreau]

# dask/ml.py
from IPython.utils.shimmodule import ShimModule, ShimWarning

sys.modules['dask.ml'] = ShimModule(src='dask.ml', mirror='dask_ml')

$ ipython
Python 3.8.2 | packaged by conda-forge | (default, Apr 24 2020, 07:56:27)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from dask.ml.model_selection import GridSearchCV

In [2]: GridSearchCV.__module__
Out[2]: 'dask_ml.model_selection._search'

Credit to Thomas Kluyver and MinRK who wrote it for IPython.

[Carreau]

Happy to pull that as a separate package if we want to avoid code duplication.

We can likely go way simpler on 3.7+ as modules now support __getattr__ (https://www.python.org/dev/peps/pep-0562/), One reason to go with NEP 29 :-P

[Carreau]

# dask/ml.py
def __getattr__(value):
    import dask_ml
    return getattr(dask_ml, value)

Works on 3.7+. If you put a warning in distributed __init__.py, you can likely catch warning to display a "please import from dask.ml" when users use the direct dask_ml import and not the dask.ml,

You can even likely use entrypoints to avoid doing the try_import, not sure it is worth it, or use __getattr__ in __init__.py itself with if value == 'dask_ml': return dask_ml. [EDIT last part about __init__ is wrong]

[mrocklin]

Wow. Those are both slick implementations.

I think that we can safely progress beyond the question of "if we can do this" to "should we do this"

[Carreau]

"should we do this"

If you do it, then I think for consistency it would be nice to try to discourage the dask_ml import.

Also one thing I'm unsure about is how that affects pickle-ability. When using the shim-importer.

[TomAugspurger]

Don't have strong thoughts here. As you note, the additional complexity here is that dask-ml has a more nested public API than distributed.

[mrocklin]

Matthias' tricks seem to get us past the nested API problem. I'm inclined to try this out.

…

On Mon, Jun 29, 2020 at 4:52 AM Tom Augspurger ***@***.***> wrote: Don't have strong thoughts here. As you note, the additional complexity here is that dask-ml has a more nested public API than distributed. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTAVT427DZEWLN6CC3DRZB57PANCNFSM4OKDEXXA> .

[TomAugspurger]

Sounds good to me. On Mon, Jun 29, 2020 at 8:48 AM Matthew Rocklin <notifications@github.com> wrote:

…

Matthias' tricks seem to get us past the nested API problem. I'm inclined to try this out. On Mon, Jun 29, 2020 at 4:52 AM Tom Augspurger ***@***.***> wrote: > Don't have strong thoughts here. As you note, the additional complexity > here is that dask-ml has a more nested public API than distributed. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#687 (comment)>, or > unsubscribe > < https://github.com/notifications/unsubscribe-auth/AACKZTAVT427DZEWLN6CC3DRZB57PANCNFSM4OKDEXXA > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIST4ACYTYQ366BPIPTRZCLSRANCNFSM4OKDEXXA> .

[mrocklin]

@lesteve any interest? On Mon, Jun 29, 2020 at 2:50 PM Tom Augspurger <notifications@github.com> wrote:

…

Sounds good to me. On Mon, Jun 29, 2020 at 8:48 AM Matthew Rocklin ***@***.***> wrote: > Matthias' tricks seem to get us past the nested API problem. I'm inclined > to try this out. > > On Mon, Jun 29, 2020 at 4:52 AM Tom Augspurger ***@***.*** > > wrote: > > > Don't have strong thoughts here. As you note, the additional complexity > > here is that dask-ml has a more nested public API than distributed. > > > > — > > You are receiving this because you authored the thread. > > Reply to this email directly, view it on GitHub > > <#687 (comment)>, or > > unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/AACKZTAVT427DZEWLN6CC3DRZB57PANCNFSM4OKDEXXA > > > > . > > > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#687 (comment)>, or > unsubscribe > < https://github.com/notifications/unsubscribe-auth/AAKAOIST4ACYTYQ366BPIPTRZCLSRANCNFSM4OKDEXXA > > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTDLHY4MQ2X4VBSIGLDRZEEBLANCNFSM4OKDEXXA> .

[mrocklin]

This is up and ready: dask/dask#7866

[mrocklin]

Whoops. I meant dask/dask#6384