Operator-sdk 1.28.1 uses non-existing storage digest because the new release procedure rebuilds storage image

[tkrishtop]

Bug Report

We created this pull request to restore the previously working situation with the hardcoded digests that allowed us to use the DCI-Preflight integration in disconnected environments.

Unfortunately, the release procedure deleted the digest we hardcoded by pushing scorecard-storage to the same tag.

We really need to fix the release procedure. I propose excluding scorecard-storage from the rebuild entirely because it's a stable wrapper of BusyBox and not intended to change often:

https://github.com/operator-framework/operator-sdk/blob/af14062577914e4c79c77d8a31d2cf808e5ed76b/images/scorecard-storage/Dockerfile#L1C1-L12

Here is the whole story:

• Back in [Disconnected env] Could we replace tag for hardcoded busybox image by digest to make it work in disconnected env?  #5285 it was reported the problem of having tags in images in a disconnected environment, also known as air-gapped. To solve that issue a couple of flags were introduced [Bugfix for #5285] Expose flags for storage and untar images #5306 to allow scorecard to use custom images. The default of those images allowed disconnected environments to work out of the box, i.e. no need to set a custom image.

• This PR (scorecard): Add new images for default untar and storage options #6335 introduced back tags instead of the digest of those images and prevented operator-sdk from being used in the disconnected environments.

• This PR Use digest instead of tags in scorecard images #6393 tried to fix the situation by replacing tags with digests and was released as the patch release operator-sdk 1.28.1. Unfortunately, the release process of operator-sdk was changed as well, and now both storage and untar images are rebuilt during the release. That release procedure erased the hardcoded digests:

$ podman pull quay.io/operator-framework/scorecard-storage@sha256:f7bd62664a0b91034acb977a8bb4ebb76bc98a6e8bdb943eb84c8e364828f056
Trying to pull quay.io/operator-framework/scorecard-storage@sha256:f7bd62664a0b91034acb977a8bb4ebb76bc98a6e8bdb943eb84c8e364828f056...
Error: initializing source docker://quay.io/operator-framework/scorecard-storage@sha256:f7bd62664a0b91034acb977a8bb4ebb76bc98a6e8bdb943eb84c8e364828f056: 
reading manifest sha256:f7bd62664a0b91034acb977a8bb4ebb76bc98a6e8bdb943eb84c8e364828f056 in quay.io/operator-framework/scorecard-storage: 
manifest unknown

Possible Solution

We should probably exclude scorecard-storage from the GitHub action to rebuild all images:

operator-sdk/.github/workflows/freshen-images.yml

Line 7 in af14062

description: Force rebuild of all images.

cc: @tonyskapunk @bcrochet @everettraven

[everettraven]

@tkrishtop Thanks for raising this issue! I agree that it is likely unnecessary to rebuild this image. This issue should be brought up in the community meeting on Monday (May 8th, 2023) for discussion as part of the issue triage process.

[varshaprasad96]

@tkrishtop Would you mind creating a PR to fix the release process to not build scorecard-storage image? We can provide more inputs on where/how the release artifacts are built. Thanks!

[acornett21]

Is this actually happening in the freshen images, since neither the ``untarorstorage` images are referenced in this file? Or is it happening in the release itself?

[tkrishtop]

Hello @everettraven and @varshaprasad96, I'm sorry for missing the discussion. I was off from May 4th to May 8th.

Thank you, @acornett21, for starting the PR. I have proposed some changes to add.

FYI, two years from now, the struggles with mirroring by digests should be over thanks to mirroring by tags coming soon in the new 4.13 release.