Update Docker image to `v4`

[michaelosthege]

As discussed in the meeting earlier today, it is now time to update the Docker image: https://hub.docker.com/u/pymc

Short-term goals

• Update the Dockerfile to a fresh base image (done in Update dockerfile to PyMC v4 #5881)
• Create a GitHub Action to build & push the Docker image. (Create GH Action to build & update the Docker image #5939)
• Install dependencies to NOT get the NumPy C-API warning... (done in Update dockerfile to PyMC v4 #5881)
• Install graphviz/pydot such that pm.model_to_graphviz() doesn't raise errors. (done in Update dockerfile to PyMC v4 #5881)
• (optional, b/c difficult) Install graphviz dependencies such that it can render to a file 👇

with pymc.Model() as pmodel:
    pymc.Normal()
fp = pathlib.Path("model.pdf")
graph = pymc.model_to_graphviz(pmodel)
graph.render(format="pdf", filename=str(fp))
assert fp.exists()

Simplifications/Constraints

• Python minor versions don't matter much these days, shall we build it for just 3.9 for now?
• We'll want a "latest" build and one per release? For 4.0.0b1 that'll be a manually built one, but for upcoming ones we can automate it?

Tagging @canyon289 @AustinRochford @bwengals who have relevant insight.

I can also contribute a (stripped-down) environment.yml to start from 👇

Known limitations:

• Suffers from NumPy C-API warning
• Graphviz/pydot install to the degree that pm.model_to_graphviz() doesn't raise, but rendering to fixes crashes.

channels:
- defaults
- conda-forge
dependencies:
- conda-build
- libblas
- mkl-service
- numpy>=1.15.0
- pip
# To fix pydot/graphviz (?)
- pango
- pydot
# Limit graphviz upper version because of https://stackoverflow.com/a/69997168/4473230
- python-graphviz<=0.18
- scipy>=1.2.0
# To be discussed:
- jupyterlab
- statsmodels
- watermark
- pip:
  - matplotlib>=3.*
  - pandas>=1.*
  # PyMC installs more reliably with pip
  # as long as important dependencies are installed with conda already
  - git+https://github.com/pymc-devs/pymc@9d4691c4ceaa18dd50081db32f38fd1310359d48
  # To be discussed:
  - scikit-learn
  - openpyxl

When this issue is dealt with we can also close #3160 + #4386 as fixed and wontfix respectively.

[ricardoV94]

The second issue looks wrong

[michaelosthege]

The second issue looks wrong

A typo. Thanks.

Also forgot to mention why this issue is high-priority: We want to re-run a ton of pymc-examples notebooks and doing it in Docker is the easiest to automate.

[canyon289]

Using @bwengals docker image the model graph to png saving is work. Its not minimal but at least shows it can work!

FROM jupyter/minimal-notebook:python-3.9.5

USER root
RUN apt update && apt upgrade -y \
  && apt -y install --no-install-recommends -qq \
     curl ca-certificates make g++ git vim-tiny openssh-client \
     unzip tzdata htop \
  && apt clean && rm -rf /var/lib/apt/lists/*

USER ${NB_UID}
RUN mamba install -c conda-forge -y \
        compilers \
        filelock \
        cython \
        numpy \
        scipy \
        sympy \
        pypolyagamma \
        mkl \
        mkl-service \
        "libblas=*=*mkl" \
        numba \
        numba-scipy \
        "arviz>=0.11.2" \
        "cachetools>=4.2.1" \
        cloudpickle \
        "fastprogress>=0.2.0" \
        "h5py>=2.7" \
        "ipython>=7.16" \
        "libblas=*=*mkl" \
        mkl-service \
        myst-nb \
        "nbsphinx>=0.4" \
        "numpy>=1.15.0" \
        "numpydoc>=0.9" \
        "pandas>=0.24.0" \
        "pre-commit>=2.8.0" \
        pydata-sphinx-theme \
        "pytest-cov>=2.5" \
        "pytest>=3.0" \
        python-graphviz \
        "recommonmark>=0.4" \
        "scipy>1.4.1" \
        "sphinx-autobuild>=0.7" \
        sphinx-panels \
        "sphinx>=1.5" \
        "typing-extensions>=3.7.4" \
        watermark \
        voila \
    && conda clean --all -f -y \
    && fix-permissions "${CONDA_DIR}" \
    && fix-permissions "/home/${NB_USER}"

RUN mamba install -c conda-forge -y \
        scikit-learn linearmodels \
        matplotlib plotly bokeh ipywidgets pre-commit \
        xarray psycopg2 \
        fastapi toml \
    && conda clean --all -f -y \
    && fix-permissions "${CONDA_DIR}" \
    && fix-permissions "/home/${NB_USER}"

RUN mamba install -c conda-forge pymc3 -y \
    && conda clean --all -f -y \
    && fix-permissions "${CONDA_DIR}" \
    && fix-permissions "/home/${NB_USER}"


## other environment variables
ENV JUPYTER_ENABLE_LAB=1
WORKDIR /home/$NB_USER
USER ${NB_UID}

COPY ./model_test.py .

RUN python model_test.py
RUN ls

model.py

import pymc3 as pm

with pm.Model() as model:
    a = pm.Uniform("a", 1, 2)
    b = pm.Normal("b", 1 ,a)

model_diagram = pm.model_to_graphviz(model)
model_diagram.render("model_graph", format="png", cleanup=True)
print("Finished")

[michaelosthege]

@canyon289 do you want to add this Dockerfile alongside a Github action? (https://github.com/marketplace/actions/build-and-push-docker-images#usage)

Should we include the extra dependencies from pymc-examples? That would make it easier to re-run notebooks.

[canyon289]

I wont be able to do this anytime soon, my current focus is on time series.

[nvasilevv]

Hey guys, I would be glad to take over. I have some experience with pymc, docker and github actions. Let me know if you are ok with that.

[michaelosthege]

Hey guys, I would be glad to take over. I have some experience with pymc, docker and github actions. Let me know if you are ok with that.

Sure, go ahead!

[nvasilevv]

Hi all, I tried building the above docker image. It failed for me with mamba not being able to find mkl and mkl-services among other things. The errors made sense, because I ran this on my Mac (m1 chip). Will try to resolve those before continuing. Having said that, I am not sure what of a delay this might introduce.

Even if I was to drop the mkl packages, I still cannot conda-install pymc.