Implement solution for sharing common content used in Dockerfiles

[mthalman]

The Dockerfiles in this repo have A LOT of content that is common between them, or at least a subset of them. Just a couple of examples:

• The Debian packages that get installed in the Helix Dockerfiles are all the same between the different Debian versions.
• Creation of the helixbot user between the Dockerfiles is (roughly) the same.

Having Dockerfiles that all contain their own implementation leads to the danger of inconsistency amongst the whole set. It's also just extra work on behalf of the contributor to have to apply the changes across Dockerfiles (this change had to be applied the same across 21 Dockerfiles: #1390).

We should consider options to solve this problem. There are several things to be considered:

• Use of ARG to parameterize the Dockerfiles: Consider utilizing parameterized Dockerfiles as a means to reduce duplication #231. This solves things only partially and is really only good for simple value replacement like a version number, not for something like a set of commands to be executed. One problem is that image caching doesn't support the use of ARG: Image caching doesn't account for build args docker-tools#1505.
• Encapsulating common logic in scripts that can be executed by multiple Dockerfiles that need that logic. This is nice, but there other elements of Dockerfiles that are common besides just script logic such as setting of ENV vars or just common multi-stage patterns. These script files would also not be supported by image caching: Image caching support for copied local files docker-tools#1270. In order to use a script by multiple Dockerfiles, you'd also need a way to set the build context to copy in the script which requires Add a buildContext option to Platform in Manifest docker-tools#1592.
• Implement Dockerfile templates. Image Builder supports this already and is used by dotnet-docker. But there is a desire to migrate to a different templating language: Migrate off of Cottle templating engine docker-tools#1610. Templates solve the shortcomings of ARG and scripts.

[am11]

The authentic Docker's approach is to extend the syntax using the # syntax directive: https://docs.docker.com/build/buildkit/frontend/. Create a gRPC service, and publish to docker and use # syntax <container URN>.

There is an INCLUDE directive support provided by this third party syntax parser using that mechanism: https://codeberg.org/devthefuture/dockerfile-x, but it ignores everything before the INCLUDE line, which defeats its purpose given our use-cases.

The recommended language platform for these parsers is Go, but we can implement it in .NET; dotnet new grpc project with this protobuf spec for LLB (low level build, which is Dockerfile IR) https://github.com/moby/buildkit/blob/36b0458ff396aef565849e8ec112e7b12088d83d/solver/pb/ops.proto and the gRPC endpoints which BuildKit calls.

The situation quickly becomes complex. Our goal is simply to handle a single instruction, but the BuildKit extension point doesn’t support partial overrides to extend the existing syntax. Also, it only permits one syntax parser to be used at a time. As the default dockerfile parser is versatile and evolving all the time (new instructions, depredations etc.), switching to a custom parser for just one directive would require us to implement the entire set of directives from scratch. This is the same challenge faced by the devthefuture/dockerfile-x team, who must manage all valid syntax to support a single instruction. Their repository contains a significant amount of code to handle various defaults, as it can't simply redirect “everything else” to the default parser.

---

Meanwhile, here is a (poor man's) way with arcade machinery to go about it: am11@a017290. A simple msbuild target supporting DOTNET_DOCKER_IMPORT <path>, run it with e.g. dotnet msbuild eng/MakeDockerfile.targets -t:MakeDockerFiles -p:TemplateFile=$(pwd)/src/azurelinux/3.0/net10.0/cross/riscv64/Dockerfile and it will generate artifacts/obj/src/azurelinux/3.0/net10.0/cross/riscv64/Dockerfile.gen with content replaced.

[lbussell]

Yeah, I briefly considered the idea of writing an alternative frontend in .NET. Even went so far as to look at the spec. That'd be a big project. Given the only motivation here is code-sharing, I don't think we're gonna need that.

I think @mthalman's 2nd bullet point is long-term the lowest-cost solution, taking into consideration both the developer effort to implement it and also the amount of effort required to write new things using that pattern.

It does mean that we won't have a way to share other Dockerfile patterns or instructions besides scripts though. Templating would help us ensure that higher level things like Dockerfile structure remain consistent across all Dockerfiles. So there are pros and cons to each option.

[am11]

We switched from shell scripts approach to everything-in-Dockerfile, it'd be a backward move.

[dougbu]

I think @mthalman's 2nd bullet point is long-term the lowest-cost solution, …

another big upside of shared scripts would be for cases where we need the same commands in different parts of different Dockerfiles. one example is venv setup in Helix images e.g.

dotnet-buildtools-prereqs-docker/src/almalinux/8/helix/amd64/Dockerfile

Lines 39 to 42 in 27aca93

	RUN python -m venv $VIRTUAL_ENV && \
	${VIRTUAL_ENV}/bin/pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
	${VIRTUAL_ENV}/bin/pip install ./helix_scripts-*-py3-none-any.whl && \
	rm ./helix_scripts-*-py3-none-any.whl

(done as helixbot) versus

dotnet-buildtools-prereqs-docker/src/alpine/3.18/helix/Dockerfile

Lines 10 to 14 in 27aca93

	RUN python3 -m venv /venv && \
	source /venv/bin/activate && \
	pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
	pip install ./helix_scripts-*-py3-none-any.whl && \
	rm ./helix_scripts-*-py3-none-any.whl

(done as root in a separate stage). ignore the slight differences in the pipelines — they accomplish the same thing

We switched from shell scripts approach to everything-in-Dockerfile, it'd be a backward move.

we have new information and more common bits, don't we❓ I'm not sure of the history here but really like the idea of not having to review 4 identical changes to 4 almost-identical Dockerfiles to confirm they really are identical

[mthalman]

I'm not sure of the history here but really like the idea of not having to review 4 identical changes to 4 almost-identical Dockerfiles to confirm they really are identical

In the dotnet-docker repo, we make use of templates. So there's a lot of duplicated content across the Dockerfiles. This is validated by a CI test which verifies that all of the Dockerfiles are in sync with the template that produces them. So this does ease the burden of reviewers so they are not forced to review every single updated Dockerfile when template changes are made.

[mthalman]

[Triage] - The templating approach seems to be the least-cost approach right now as it doesn't require new infrastructure changes, specifically related to image caching like the other two options. Templating also provides a lot of clarity as you can see directly in the Dockerfile what its logic is rather than needing to traverse across various script files within the repo. The templating engine of Cottle is a question mark but we are in control of how in depth we make the templates to avoid going to far that would make things difficult in order to migrate to a different engine.