fix: K8s API does not accept unauthorized requests

[bouskaJ]

Summary

Fixes #2110

Release Note

• The cluster token is now automatically passed to the authentication header for the kubernetes.default.svc OIDC provider

[haydentherapper]

Hey, thanks for the PR. Do you have more information about this issue? The PR linked in the issue was from almost 10 years ago so I'm uncertain why this is now coming up.

I also am not too familiar with Kubernetes setups, so can you confirm if k8sTokenFile is always present? Is whether or not the token is passed something that a deployer can configure?

[codecov]

Codecov Report

❌ Patch coverage is 67.24138% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.55%. Comparing base (cf238ac) to head (d143757).
⚠️ Report is 444 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/config/config.go	67.24%	14 Missing and 5 partials ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2111       +/-   ##
===========================================
- Coverage   57.93%   43.55%   -14.38%     
===========================================
  Files          50       72       +22     
  Lines        3119     5705     +2586     
===========================================
+ Hits         1807     2485      +678     
- Misses       1154     2989     +1835     
- Partials      158      231       +73     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bouskaJ]

Hi @haydentherapper thank you for the review. I can confirm that the token file is always present (as you can see in the Kubernetes doc https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/#directly-accessing-the-rest-api).

For more information about this issue, you can read https://www.redhat.com/en/blog/what-you-need-to-know-red-hat-openshift-416
Especially following paragraph is important for this context

Reduce unauthenticated user or group access
Beginning with OpenShift 4.16, there's a limit on the permissions given to the system:anonymous user and system:unauthenticated group. Only two roles are allowed by default:

system:openshift:public-info-viewer: required for OIDC workflows
system:public-info-viewer: read-only access to non-sensitive information about the cluster
This applies to new clusters only. Clusters that are upgraded are not affected.

For use cases where anonymous access is needed, the cluster administrator must explicitly add those permissions. For example, if you're using webhooks for BuildConfigs from an external system (like GitHub) that don't support sending auth tokens over HTTP, then you'd need to explicitly add system:webhook permissions. We strongly recommend using local rolebindings instead of clusterrole bindings in such scenarios. A cluster administrator can add back a ClusterRoleBinding for the anonymous user as needed, after evaluating the risks and benefits associated with it

[haydentherapper]

If this is specific to OpenShift, can we make this configurable whether or not to attach a token? I'd like to avoid the risk breaking any existing deployments.

[bouskaJ]

@haydentherapper yes, I agree that we should be careful with the backward compatibility. I do not want to add another variable to the https://pkg.go.dev/github.com/sigstore/fulcio/pkg/config#OIDCIssuer that is specific only to Kubernetes issuer, so we have limited options on how to make it configurable.

I added the if file exists check before adding the AUTH header instead. According to the k8s DOC if the file is present, it should always be valid because we follow the documented way on how to connect to the k8s API server. https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/#without-using-a-proxy

[bouskaJ]

@haydentherapper could you please re-review this PR?

[cmurphy]

The link to the Kubernetes PR was for a 1.5 release, but it looks like in just the next release they changed its behavior - https://kubernetes.io/docs/reference/access-authn-authz/authentication/#anonymous-requests - so it doesn't seem like that PR is telling the full story.

It looks like Kubernetes supports adding certain endpoints to an AuthenticationConfiguration list to allow anonymous access, could OpenShift consider doing that?

Could access to this endpoint be granted to the system:openshift:public-info-viewer role, which looks like it is already assigned to the anonymous user and is intended for OIDC workflows?

[cmurphy]

Can you explain more why the k8s issuer would be present in the config but not already have its provider/verifier created in the loop above? This check doesn't look necessary to me.

[bouskaJ]

The first cycle traverses over static OIDCIssuers. The for cycle covers the following configuration:

OIDCIssuers: map[string]OIDCIssuer{
					k8sIssuerURL: {
						IssuerURL: k8sIssuerURL,
						ClientID:  "sigstore",
						Type:      IssuerTypeKubernetes,
					},
				},

(test https://github.com/bouskaJ/fulcio/blob/fix_2110/pkg/config/config_test.go#L866-L873)

For MetaIssuer the GetVerifier function will not get any static verifier without adding it explicitly (see https://github.com/bouskaJ/fulcio/blob/fix_2110/pkg/config/config.go#L197). The function would create a new verifier dynamically in that case. This new client will use http.Client without the authorization.

I am creating the additional static verifier in case the k8s issuer can be resolved by https://github.com/bouskaJ/fulcio/blob/fix_2110/pkg/config/config.go#L156 but it is not present yet.

This behaviour is tested by https://github.com/bouskaJ/fulcio/blob/fix_2110/pkg/config/config_test.go#L875C3-L887C5

[cmurphy]

These could all be consts.

[bouskaJ]

Yes, they could, but I make them vars because it allows me to change the value for testing purposes. https://github.com/bouskaJ/fulcio/blob/fix_2110/pkg/config/config_test.go#L905

[bouskaJ]

@cmurphy Thank you for your comments.

The link to the Kubernetes PR was for a 1.5 release, but it looks like in just the next release they changed its behavior - https://kubernetes.io/docs/reference/access-authn-authz/authentication/#anonymous-requests - so it doesn't seem like that PR is telling the full story.

yes, I linked the very old PR that might not be relevant to recent changes but the https://www.redhat.com/en/blog/what-you-need-to-know-red-hat-openshift-416 is current and comprehensive source.

It looks like Kubernetes supports adding certain endpoints to an AuthenticationConfiguration list to allow anonymous access, could OpenShift consider doing that?

Could access to this endpoint be granted to the system:openshift:public-info-viewer role, which looks like it is already assigned to the anonymous user and is intended for OIDC workflows?

I agree that both Kubernetes and OpenShift allow for role modifications. However, I don't think it's a good practice to lower the default security on the cluster. We should instead focus on finding a way to work within the cluster's default configuration.