"make docker-buildx" does not work as expected for Helm-based operator

[kurokobo]

Bug Report

What did you do?

• Create new Helm-based operator by following quickstart guide
• Invoke make docker-buildx

# Create new operator
$ mkdir nginx-operator
$ cd nginx-operator
$ operator-sdk init --domain example.com --plugins helm
$ operator-sdk create api --group demo --version v1alpha1 --kind Nginx

# Add test target to make docker-buildx target work
$ echo ".PHONY: test" >> Makefile

# Build cross-platform images
$ IMG="registry.example.com/operator/nginx-operator:v0.0.1" PLATFORMS="linux/arm64,linux/amd64" make docker-buildx

What did you expect to see?

make docker-buildx builds the images for linux/amd64 and linux/arm64.

What did you see instead? Under which circumstances?

make docker-buildx builds the image for linux/amd64 (the platform where the make has been invoked) only.

$ IMG="registry.example.com/operator/nginx-operator:v0.0.1" PLATFORMS="linux/arm64,linux/amd64" make docker-buildx
# copy existing Dockerfile and insert --platform= into Dockerfile.cross, and preserve the original Dockerfile
sed -e '1 s/\(^FROM\)/FROM --platform=\$\{BUILDPLATFORM\}/; t' -e ' 1,// s//FROM --platform=\$\{BUILDPLATFORM\}/' Dockerfile > Dockerfile.cross
docker buildx create --name project-v3-builder
project-v3-builder
docker buildx use project-v3-builder
docker buildx build --push --platform=linux/arm64,linux/amd64 --tag registry.example.com/operator/nginx-operator:v0.0.1 -f Dockerfile.cross .
[+] Building 18.8s (11/11) FINISHED                                                                                                          
 => [internal] booting buildkit                                                                                                         3.3s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                      2.8s
 => => creating container buildx_buildkit_project-v3-builder0                                                                           0.5s
 => [internal] load build definition from Dockerfile.cross                                                                              0.0s
 => => transferring dockerfile: 326B                                                                                                    0.0s
 => [linux/amd64 internal] load metadata for quay.io/operator-framework/helm-operator:v1.33.0                                           2.7s
 => [internal] load .dockerignore                                                                                                       0.0s
 => => transferring context: 2B                                                                                                         0.0s
 => [linux/amd64 1/4] FROM quay.io/operator-framework/helm-operator:v1.33.0@sha256:0556040e2de6bd7f6bb4c9447fb3ced7978eff45c25164cf7b  11.8s
 => => resolve quay.io/operator-framework/helm-operator:v1.33.0@sha256:0556040e2de6bd7f6bb4c9447fb3ced7978eff45c25164cf7b54c1a0b5bd4f1  0.0s
 => => sha256:e07f29d3373eac652b1191f42fcce9ab3383f5b1bc9f58ed1de4c741c8c2d243 35.12MB / 35.12MB                                        2.8s
 => => sha256:d306b23b8556abc754d2bc22290a2072e037d022f813c989bd8f293eeed1bc2d 113B / 113B                                              6.9s
 => => sha256:b58184c5c8d88ce2617d05f127d3befa248e3b06e2bfb27f565f85698af80a02 392B / 392B                                              6.4s
 => => sha256:f4a3c904e5565efee16dbcb82838bbb64a39f8261e4531648bc591c528236c24 39.34MB / 39.34MB                                       10.9s
 => => extracting sha256:f4a3c904e5565efee16dbcb82838bbb64a39f8261e4531648bc591c528236c24                                               0.6s
 => => extracting sha256:b58184c5c8d88ce2617d05f127d3befa248e3b06e2bfb27f565f85698af80a02                                               0.0s
 => => extracting sha256:d306b23b8556abc754d2bc22290a2072e037d022f813c989bd8f293eeed1bc2d                                               0.0s
 => => extracting sha256:e07f29d3373eac652b1191f42fcce9ab3383f5b1bc9f58ed1de4c741c8c2d243                                               0.3s
 => [internal] load build context                                                                                                       0.0s
 => => transferring context: 13.89kB                                                                                                    0.0s
 => [linux/amd64 2/4] COPY watches.yaml /opt/helm/watches.yaml                                                                          0.4s
 => [linux/amd64 3/4] COPY helm-charts  /opt/helm/helm-charts                                                                           0.0s
 => [linux/amd64 4/4] WORKDIR /opt/helm                                                                                                 0.0s
 => exporting to image                                                                                                                  0.4s
 => => exporting layers                                                                                                                 0.0s
 => => exporting manifest sha256:bc55504dd24cb2f5d839b0961fe0e444a23f40e9187ddf1db3315cb6ebae5ff4                                       0.0s
 => => exporting config sha256:40dd737f3f3ae7e4e3975831fd8f31d25a7dfc9ec1dd961d246d992d875719b7                                         0.0s
 => => exporting manifest sha256:b21c3eb522e2e562d15b1013be1cd30a53bd9a95252ab0741b956e6c85d37d86                                       0.0s
 => => exporting config sha256:e3dc10ef192eb2f5fd7dd78db7dcff42053f9923b33832b1408881d86f96a43e                                         0.0s
 => => exporting manifest list sha256:a1e99605dbbd8c911612e2dd805ba3c19e782c3532aa54c29dd1520f24aa27b7                                  0.0s
 => => pushing layers                                                                                                                   0.4s
 => => pushing manifest for registry.example.com/operator/nginx-operator:v0.0.1@sha256:a1e99605dbbd8c911612e2dd805ba3c19e782c3532aa54c  0.0s
 => [auth] sharing credentials for registry.example.com                                                                                 0.0s
docker buildx rm project-v3-builder
rm Dockerfile.cross

Environment

Operator type:

/language helm

Kubernetes cluster type:

$ kubectl version
Client Version: v1.28.5+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.5+k3s1

$ operator-sdk version

$ operator-sdk version
operator-sdk version: "v1.33.0", commit: "542966812906456a8d67cf7284fc6410b104e118", kubernetes version: "1.27.0", go version: "go1.21.5", GOOS: "linux", GOARCH: "amd64"

$ go version (if language is Go)

$ kubectl version

$ kubectl version
Client Version: v1.28.5+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.5+k3s1

Possible Solution

The current docker-buildx target in Makefile for Helm-based operator adds --platform=${BUILDPLATFORM} to the first FROM in Dockerfile:

operator-sdk/internal/plugins/helm/v1/scaffolds/internal/templates/makefile.go

Lines 113 to 115 in ce3a767

	docker-buildx: test ## Build and push docker image for the manager for cross-platform support
	# copy existing Dockerfile and insert --platform=${BUILDPLATFORM} into Dockerfile.cross, and preserve the original Dockerfile
	sed -e '1 s/\(^FROM\)/FROM --platform=\$$\{BUILDPLATFORM\}/; t' -e ' 1,// s//FROM --platform=\$$\{BUILDPLATFORM\}/' Dockerfile > Dockerfile.cross

This is a correct configuration for fast cross-compilable with multi-stage builds for Go-based operator, but Helm-based operator does not use multi-stage builds, so the first FROM image must be for TARGETPLATFORM, instead of BUILDPLATFORM: https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/

My suggested changes:

• Specifying TARGETPLATFORM as --platform is default behavior of BuildKit, so we can remove whole sed command.
• Also, since there is no test target by default, referring test causes error. We should remove test target from docker-buildx.

  $ make docker-buildx IMG="example.com/nginx-operator:v0.0.1" PLATFORMS=linux/arm64,linux/amd64
  make: *** No rule to make target 'test', needed by 'docker-buildx'.  Stop.

Diff:

diff --git a/internal/plugins/helm/v1/scaffolds/internal/templates/makefile.go b/internal/plugins/helm/v1/scaffolds/internal/templates/makefile.go
index 631eb32f..dccaf36a 100644
--- a/internal/plugins/helm/v1/scaffolds/internal/templates/makefile.go
+++ b/internal/plugins/helm/v1/scaffolds/internal/templates/makefile.go
@@ -110,14 +110,11 @@ docker-push: ## Push docker image with the manager.
 # To properly provided solutions that supports more than one platform you should use this option.
 PLATFORMS ?= linux/arm64,linux/amd64,linux/s390x,linux/ppc64le
 .PHONY: docker-buildx
-docker-buildx: test ## Build and push docker image for the manager for cross-platform support
-       # copy existing Dockerfile and insert --platform=${BUILDPLATFORM} into Dockerfile.cross, and preserve the original Dockerfile
-       sed -e '1 s/\(^FROM\)/FROM --platform=\$$\{BUILDPLATFORM\}/; t' -e ' 1,// s//FROM --platform=\$$\{BUILDPLATFORM\}/' Dockerfile > Dockerfile.cross
+docker-buildx: ## Build and push docker image for the manager for cross-platform support
        - docker buildx create --name project-v3-builder
        docker buildx use project-v3-builder
-       - docker buildx build --push --platform=$(PLATFORMS) --tag ${IMG} -f Dockerfile.cross .
+       - docker buildx build --push --platform=$(PLATFORMS) --tag ${IMG} -f Dockerfile .
        - docker buildx rm project-v3-builder
-       rm Dockerfile.cross
 
 ##@ Deployment

Applying above changes can build the images for multi-platforms:

$ IMG="registry.example.com/operator/nginx-operator:v0.0.1" PLATFORMS="linux/arm64,linux/amd64" make docker-buildx
docker buildx create --name project-v3-builder
project-v3-builder
docker buildx use project-v3-builder
docker buildx build --push --platform=linux/arm64,linux/amd64 --tag registry.example.com/operator/nginx-operator:v0.0.1 -f Dockerfile .
[+] Building 11.8s (10/14)                                                                                                                   
 => [internal] booting buildkit                                                                                                         2.7s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                      2.3s
 => => creating container buildx_buildkit_project-v3-builder0                                                                           0.5s
 => [internal] load build definition from Dockerfile                                                                                    0.0s
 => => transferring dockerfile: 292B                                                                                                    0.0s
 => [linux/amd64 internal] load metadata for quay.io/operator-framework/helm-operator:v1.33.0                                           2.9s
 => [linux/arm64 internal] load metadata for quay.io/operator-framework/helm-operator:v1.33.0                                           3.5s
...

Additional context

The same issue can be applied for Ansible plugin. It's already reported:

• Cross Platform docker-buildx target does not work ansible-operator-plugins#48
• Remove broken work-around for specifying build platforms for the operator image ansible-operator-plugins#49