operator-sdk bundle validate should have an option to turn warnings into errors

[wallrj]

Feature Request

operator-sdk bundle validate --permissive=false

The documentation advises bundle authors to check their bundle using operator-sdk bundle validate and that:

Any warnings here might turn into failing pipeline tests here. Please correct all issues displayed. A list of fields that are scanned can also be reviewed with this list.

-- https://github.com/operator-framework/community-operators/blob/1591b5e18d47129ed14e83ac2f9832d6cb41bb40/docs/testing-operators.md#L234-L250

So it would be nice if there was a way to run the tool in CI pipelines and have it fail if there are warnings,
so that the problems can be fixed proactively.

And indeed there is a comment in the code to this effect:

operator-sdk/internal/cmd/operator-sdk/bundle/validate/validate.go

Lines 71 to 72 in 836a088

	// TODO: add a "permissive" flag to toggle whether warnings also cause a non-zero
	// exit code to be returned (true by default).

There is some discussion about this in Slack here: https://kubernetes.slack.com/archives/C017UU45SHL/p1644400918179439

/cc @estroz @camilamacedo86

[camilamacedo86]

Hi @wallrj,

So it would be nice if there was a way to run the tool in CI pipelines and have it fail if there are warnings,
so that the problems can be fixed proactively.

I think your suggestion is very interesting. The only cons that I can see here is that WARNINGs might only be raised to let you know that you should ensure something that might not be required be fixed or in your specific use case would not make sense apply any change.

Let's see what others think about that.

[camilamacedo86]

@jmrodri @tlwu2013 wdyt?

[jberkhahn]

This sounds reasonable, would you be interested in submitting this via pull request? An optional flag would be an easy way to do this.

[wallrj]

In operator-framework/api#172 (comment) #5542

I thought we had pluggable test suites these days. If so, then making the test fail should not be an issue as long as the user can decide whether or not they want to execute these tests.

Which makes me think that perhaps there should instead be:

$ operator-sdk bundle validate ./bundle --select-optional suite=operatorframework

vs

$ operator-sdk bundle validate ./bundle --select-required suite=operatorframework

I'll try and create a PR with the simple --permissive=false flag as a starting point for discussion.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale