Skip to content

Conversation

@joschi36
Copy link
Contributor

@joschi36 joschi36 commented Jun 13, 2025

What this PR does / why we need it:

The current Helm chart for the Loki gateway does not provide a way to inject Nginx configuration directives inside individual location blocks. This makes it impossible to implement common and important authentication or header manipulation schemes that rely on directives like proxy_set_header without overriding the entire Nginx configuration file.

A key use case is mTLS-based multi-tenancy, where an X-Scope-OrgID header must be set based on client certificate details. Due to Nginx's directive inheritance rules, setting this header in serverSnippet is overridden by any location block that defines its own proxy_set_header (e.g., for X-Query-Tags in the /loki/api/v1/ location or WebSocket upgrades in the /loki/api/v1/tail location). This results in failed authentication for some endpoints but not others.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

This change is fully backward-compatible. If locationSnippet is not set, the template renders exactly as it did before, resulting in no change for existing users.

This was tested by implementing the mTLS multi-tenancy scheme described above, which was previously only possible via complex workarounds like overriding the entire nginx.conf file or using a post-renderer. With this change, the configuration becomes trivial and is managed cleanly through values.yaml. The snippet has been added to all location blocks for consistency and to ensure all API endpoints served by the gateway are covered.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@joschi36 joschi36 requested a review from a team as a code owner June 13, 2025 16:37
@CLAassistant
Copy link

CLAassistant commented Jun 13, 2025

CLA assistant check
All committers have signed the CLA.

@joschi36 joschi36 changed the title add location snippet to nginx config feat(helm): add location snippet to nginx config Jun 14, 2025
@joschi36
Copy link
Contributor Author

Seems to be like a duplicate of #11348

Can we go forward with either one?
Customizing the clientMaxBodySize would benefit us as well

Copy link
Contributor

@jkroepke jkroepke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other PR is closed.

Customizing the clientMaxBodySize would benefit us as well

Let do that in a distinct PR as well.

Could you also please provide an use-case for this?

@joschi36
Copy link
Contributor Author

joschi36 commented Jul 9, 2025

Hi @jkroepke

Could you also please provide an use-case for this?

I have provided a use case in the PR description. Do you mean adding it in the values.yaml as an example use case?

A key use case is mTLS-based multi-tenancy, where an X-Scope-OrgID header must be set based on client certificate details. Due to Nginx's directive inheritance rules, setting this header in serverSnippet is overridden by any location block that defines its own proxy_set_header (e.g., for X-Query-Tags in the /loki/api/v1/ location or WebSocket upgrades in the /loki/api/v1/tail location). This results in failed authentication for some endpoints but not others.

Or do you mean regarding clientMaxBodySize. I basically have the same as @panzouh has in his initial PR #11348

Adjusting client_max_body_size in Nginx could allow me fixing nginx errors client intended to send too large body.

But I didn't know until now that this was already solved: 809a024

@jkroepke
Copy link
Contributor

jkroepke commented Jul 9, 2025

LGTM

Thanks for your answers.

Copy link
Contributor

@jkroepke jkroepke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Thanks for your work. Please take note of the merge conflicts and add a note to changelog, including PR at the end of the note.

please take note of #18414 - I guess httpSnippet would work for you as well, since the inherited from the server block issue was resolved there.

@joschi36
Copy link
Contributor Author

add a note to changelog, including PR at the end of the note.

yes just noticed #18414 aswell. This would indeed solve my issue.
do we still want to merge this one?

nevertheless I've updated the branch, implemented you proposed change and added the changelog

@jkroepke
Copy link
Contributor

LGTM.

I didnt have merge power yet, but one of the internal loki team will merge this soon.

@jkroepke
Copy link
Contributor

@joschi36 please check CI looks like make helm-docs is missing

@pull-request-size pull-request-size bot added size/L and removed size/M labels Jul 21, 2025
@nicolevanderhoeven nicolevanderhoeven self-assigned this Jul 21, 2025
Signed-off-by: nicolevanderhoeven <[email protected]>
@joschi36
Copy link
Contributor Author

thx @nicolevanderhoeven

@nicolevanderhoeven
Copy link
Contributor

Generated the reference doc to unblock this and resolved a merge conflict with Changelog. :) Just waiting for final checks, and otherwise LGTM and I'll merge!

Copy link
Contributor

@nicolevanderhoeven nicolevanderhoeven left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GTG!

@nicolevanderhoeven nicolevanderhoeven merged commit 5737751 into grafana:main Jul 22, 2025
75 checks passed
@joschi36 joschi36 deleted the patch-2 branch July 29, 2025 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants