From 7f495c6459ab47db53f65885442bf5ff7bcf7411 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Wed, 30 Jul 2025 10:28:02 +0100 Subject: [PATCH 1/8] added intstructions for how to upgrade zone-aware ingesters --- .../setup/upgrade/upgrade-to-6x/index.md | 36 +++++++++++++++++++ production/helm/loki/CHANGELOG.md | 5 ++- 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/docs/sources/setup/upgrade/upgrade-to-6x/index.md b/docs/sources/setup/upgrade/upgrade-to-6x/index.md index 59622bd8fcf16..0d9159486c270 100644 --- a/docs/sources/setup/upgrade/upgrade-to-6x/index.md +++ b/docs/sources/setup/upgrade/upgrade-to-6x/index.md @@ -88,6 +88,42 @@ resultsCache: With these caches disabled, Loki will return to defaults which enables an in-memory results and chunks cache, so you will still get some caching. +#### BREAKING: Zone-aware ingester StatefulSet serviceName fix (6.34.0+) + +**Affected users**: Only deployments using zone-aware ingester replication (`ingester.zoneAwareReplication.enabled: true`) + +In Helm chart version 6.34.0, [PR #18558](https://github.com/grafana/loki/pull/18558) fixed the `serviceName` field in zone-aware ingester StatefulSets to correctly reference headless services. However, since `serviceName` is an immutable field in Kubernetes StatefulSets, upgrading to 6.34.0 requires manual intervention. + +**Required action before upgrading to 6.34.0**: + +1. **Check if you're affected**: + ```bash + helm get values | grep -A5 zoneAwareReplication + ``` + If `enabled: true` appears, you need to follow these steps. + +2. **Delete the StatefulSets** (data will be preserved): + ```bash + kubectl delete statefulset \ + -ingester-zone-a \ + -ingester-zone-b \ + -ingester-zone-c \ + --cascade=orphan + ``` + +3. **Proceed with the Helm upgrade**: + ```bash + helm upgrade grafana/loki --version 6.34.0 + ``` + +**What happens**: +- PersistentVolumeClaims and data are preserved +- New StatefulSets will be created with correct service references +- Pods will restart and reattach to existing storage + +**Why this change was necessary**: +The previous configuration caused ingester scaling operations to fail because the rollout-operator couldn't find the correct headless services for the `/ingester/prepare-downscale` endpoint. + #### Distributed mode This chart introduces the ability to run Loki in distributed, or [microservices mode](https://grafana.com/docs/loki//get-started/deployment-modes/#microservices-mode). Separate instructions on how to enable this as well as how to migrate from the existing community chart will be coming shortly. diff --git a/production/helm/loki/CHANGELOG.md b/production/helm/loki/CHANGELOG.md index 07fe2001903e8..1092b67ece9b2 100644 --- a/production/helm/loki/CHANGELOG.md +++ b/production/helm/loki/CHANGELOG.md @@ -17,6 +17,7 @@ Entries should include a reference to the pull request that introduced the chang ## 6.34.0 - [CHANGE] Updated version of Grafana Loki to 3.5.3 +- [CHANGE] **BREAKING** Fixed serviceName in zone-aware ingester StatefulSets to reference correct headless services [#18558](https://github.com/grafana/loki/pull/18558). **Upgrade Impact**: Users with zone-aware ingesters (`ingester.zoneAwareReplication.enabled: true`) must manually delete the ingester StatefulSets before upgrading: `kubectl delete statefulset -ingester-zone-a -ingester-zone-b -ingester-zone-c --cascade=orphan`. PersistentVolumeClaims and data will be preserved. **See**: [Helm chart upgrade guide](https://grafana.com/docs/loki/latest/setup/upgrade/upgrade-to-6x/#breaking-zone-aware-ingester-statefulset-servicename-fix-6340) - [FEATURE] Expose labels on volumeClaimTemplates [#13317](https://github.com/grafana/loki/pull/13317) - [FEATURE] Allow setting custom labels for ingester statefulsets [#18536](https://github.com/grafana/loki/pull/18536) - [FEATURE] Added support for chunk-cache-l2 [#17556](https://github.com/grafana/loki/pull/17556) @@ -25,9 +26,7 @@ Entries should include a reference to the pull request that introduced the chang - [ENHANCEMENT] Add configurable `startupProbe` to the loki-sc-rules sidecar container in the backend pods. [#18547](https://github.com/grafana/loki/pull/18547) - [BUGFIX] Add validation for loki.storage.bucketNames [#13781](https://github.com/grafana/loki/pull/13781) - [BUGFIX] Create missing RBAC for the rules sidecar when RBAC is namespaced [#16776](https://github.com/grafana/loki/pull/16776) -- [BUGFIX] Ensure the scc is actually allowing to use hostPath volumes when the rbac.sccAllowHostDirVolumePlugin is set to true [#17680](https://github.com/grafana/loki/pull/17680) -- [BUGFIX] Fix serviceName in zone-aware ingester statefulsets [#18558](https://github.com/grafana/loki/pull/18558) - +- [BUGFIX] Ensure the scc is actually allowing to use hostPath volumes when the rbac.sccAllowHostDirVolumePlugin is set to true [#17680](https://github.com/grafana/loki/pull/17680) ## 6.33.0 - [FEATURE] Allow passing tenant password hash instead of password. [#17049](https://github.com/grafana/loki/pull/17049) From 4004a2deaeb3c847b881a59ce8416fe90e962162 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Wed, 30 Jul 2025 10:32:33 +0100 Subject: [PATCH 2/8] added to _index --- docs/sources/setup/upgrade/_index.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/sources/setup/upgrade/_index.md b/docs/sources/setup/upgrade/_index.md index d6f51f8de9753..6f21937440d47 100644 --- a/docs/sources/setup/upgrade/_index.md +++ b/docs/sources/setup/upgrade/_index.md @@ -43,6 +43,24 @@ The next Loki release introduces a new configuration option (i.e. `-distibutor.m Supported clients should check the configuration options for max send message size if applicable. +## Helm Chart Upgrades + +### Helm Chart 6.34.0 - Zone-aware Ingester Breaking Change + +{{< admonition type="warning" >}} +Helm chart version 6.34.0 introduces a **breaking change** that affects users with zone-aware ingester replication enabled. +{{< /admonition >}} + +If you are using zone-aware ingesters (`ingester.zoneAwareReplication.enabled: true`), upgrading to Helm chart 6.34.0 requires manual StatefulSet deletion before the upgrade. This is due to a fix for the `serviceName` field in zone-aware ingester StatefulSets, which is an immutable field in Kubernetes. + +**For detailed upgrade instructions, see**: [Helm Chart 6.x Upgrade Guide - Zone-aware Ingester Breaking Change](https://grafana.com/docs/loki/latest/setup/upgrade/upgrade-to-6x/#breaking-zone-aware-ingester-statefulset-servicename-fix-6340) + +Key points: +- Only affects deployments with `ingester.zoneAwareReplication.enabled: true` +- Requires manual StatefulSet deletion with `--cascade=orphan` +- **No data loss** - PersistentVolumeClaims and data are preserved +- New StatefulSets will be created with correct service references + ## 3.4.0 ### Loki 3.4.0 From aca651479cd68aa8b042f50dcc20ab9607ae7340 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Wed, 30 Jul 2025 10:48:40 +0100 Subject: [PATCH 3/8] added bucket name to test --- .../helm/loki/scenarios/default-distributed-values.yaml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/production/helm/loki/scenarios/default-distributed-values.yaml b/production/helm/loki/scenarios/default-distributed-values.yaml index 78a1f111cd246..4852562fdb635 100644 --- a/production/helm/loki/scenarios/default-distributed-values.yaml +++ b/production/helm/loki/scenarios/default-distributed-values.yaml @@ -16,7 +16,11 @@ loki: querier: # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing max_concurrent: 4 - + storage: + bucketNames: + chunks: chunks + ruler: ruler + admin: admin #gateway: # ingress: # enabled: true From 7362cc3fc5060f11f02246f96cef4590930a2a3d Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Fri, 1 Aug 2025 10:21:27 +0100 Subject: [PATCH 4/8] feat(helm): added feat and issue templates --- .github/ISSUE_TEMPLATE/helm_feature.md | 75 +++++++++++++++ .github/ISSUE_TEMPLATE/helm_issue.md | 125 +++++++++++++++++++++++++ 2 files changed, 200 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/helm_feature.md create mode 100644 .github/ISSUE_TEMPLATE/helm_issue.md diff --git a/.github/ISSUE_TEMPLATE/helm_feature.md b/.github/ISSUE_TEMPLATE/helm_feature.md new file mode 100644 index 0000000000000..daa6c4b958081 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/helm_feature.md @@ -0,0 +1,75 @@ +name: Loki Helm Chart Feature Request +description: Suggest an idea for the Loki Helm chart +title: "[loki] Helm Feature Request Title" +labels: [enhancement, area/helm] +assignees: + - +body: + - type: markdown + attributes: + value: | + Thanks for taking the time to fill out this feature request for the Loki Helm chart! + + For general Loki usage questions, please use the [Grafana Community Slack](https://slack.grafana.com/). This issue tracker is for feature requests specific to the Loki Helm chart. + + - type: input + id: chart-name + attributes: + label: Which chart? + description: Enter the name of the chart for this feature request (e.g., loki, loki-stack, loki-distributed). Note we only `officially` support the `loki` chart. + placeholder: loki + validations: + required: true + + - type: dropdown + id: deployment-mode + attributes: + label: What's your Loki deployment mode? + description: Select the deployment mode this feature request relates to + options: + - Single Binary (monolithic) + - Simple Scalable (read, write, backend components) + - Microservices (all components separated) + - All deployment modes + - Not applicable + validations: + required: true + + - type: textarea + id: desc + attributes: + label: Is your feature request related to a problem ? + description: Give a clear and concise description of what the problem is. + placeholder: ex. I'm always frustrated when [...] + validations: + required: true + + - type: textarea + id: prop-solution + attributes: + label: Describe the solution you'd like. + description: A clear and concise description of what you want to happen. + validations: + required: true + + - type: textarea + id: alternatives + attributes: + label: Describe alternatives you've considered. + description: A clear and concise description of any alternative solutions or features you've considered. If nothing, please enter `NONE` + validations: + required: true + + - type: textarea + id: additional-ctxt + attributes: + label: Additional context. + description: | + Add any other context about the feature request here, such as: + - Use case details or user stories + - Configuration examples or desired values.yaml changes + - Related Loki features or components + - Links to relevant documentation or discussions + - Screenshots or mockups if applicable + validations: + required: false \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/helm_issue.md b/.github/ISSUE_TEMPLATE/helm_issue.md new file mode 100644 index 0000000000000..037a4a7cc3662 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/helm_issue.md @@ -0,0 +1,125 @@ +name: Loki Helm Chart Bug Report +description: Create a report to help us improve +title: "[loki] Helm Issue Request Title" +labels: [bug, area/helm] +assignees: + - +body: + - type: markdown + attributes: + value: | + Thanks for taking the time to fill out this bug report for the Loki Helm chart! Please be cautious with the sensitive information/logs while filing the issue. + + For general Loki usage questions, please use the [Grafana Community Slack](https://slack.grafana.com/). This issue tracker is for bugs specific to the Loki Helm chart. + - type: textarea + id: desc + attributes: + label: Describe the bug a clear and concise description of what the bug is. + validations: + required: true + + - type: input + id: helm-version + attributes: + label: What's your helm version? + description: Enter the output of `$ helm version` + placeholder: Copy paste the entire output of the above + validations: + required: true + - type: input + id: kubectl-version + attributes: + label: What's your kubectl version? + description: Enter the output of `$ kubectl version` + validations: + required: true + + - type: input + id: chart-name + attributes: + label: Which chart? + description: Enter the name of the chart where you encountered this bug (e.g., loki, loki-stack, loki-distributed). Note we only `officially` support the `loki` chart. + placeholder: loki + validations: + required: true + + - type: input + id: chart-version + attributes: + label: What's the chart version? + description: Enter the version of the chart that you encountered this bug. + validations: + required: true + + - type: dropdown + id: deployment-mode + attributes: + label: What's your Loki deployment mode? + description: Select the deployment mode you're using + options: + - Single Binary (monolithic) + - Simple Scalable (read, write, backend components) + - Microservices (all components separated) + validations: + required: true + + - type: textarea + id: what-happened + attributes: + label: What happened? + description: Enter exactly what happened. + validations: + required: false + + - type: textarea + id: what-expected + attributes: + label: What you expected to happen? + description: Enter what you expected to happen. + validations: + required: false + + - type: textarea + id: how-to-reproduce + attributes: + label: How to reproduce it? + description: As minimally and precisely as possible. + validations: + required: false + + - type: textarea + id: changed-values + attributes: + label: Enter the changed values of values.yaml? + description: Please enter only values which differ from the defaults. Enter `NONE` if nothing's changed. Include any Loki configuration, storage, authentication, or resource settings you've modified. + placeholder: | + loki: + auth_enabled: true + minio: + enabled: true + gateway: + enabled: true + validations: + required: false + + - type: textarea + id: helm-command + attributes: + label: Enter the command that you execute and failing/misfunctioning. + description: Enter the command as-is as how you executed. + placeholder: helm install my-loki grafana/loki --version 6.34.0 --values values.yaml + validations: + required: true + + - type: textarea + id: anything-else + attributes: + label: Anything else we need to know? + description: | + Include any relevant information such as: + - Kubernetes cluster details (version, provider) + - Storage backend being used (filesystem, S3, GCS, etc.) + - Log volume or specific error messages + - Related components (Grafana, Promtail, etc.) + validations: + required: false \ No newline at end of file From fb646f5e65ade9ef1b2a248c0b39c6fd896e6efa Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Fri, 1 Aug 2025 10:37:26 +0100 Subject: [PATCH 5/8] Update .github/ISSUE_TEMPLATE/helm_issue.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Jan-Otto Kröpke Signed-off-by: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/helm_issue.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/ISSUE_TEMPLATE/helm_issue.md b/.github/ISSUE_TEMPLATE/helm_issue.md index 037a4a7cc3662..1809041f9f4fc 100644 --- a/.github/ISSUE_TEMPLATE/helm_issue.md +++ b/.github/ISSUE_TEMPLATE/helm_issue.md @@ -92,6 +92,7 @@ body: attributes: label: Enter the changed values of values.yaml? description: Please enter only values which differ from the defaults. Enter `NONE` if nothing's changed. Include any Loki configuration, storage, authentication, or resource settings you've modified. + render: yaml placeholder: | loki: auth_enabled: true From 9c408c202c8e35d76829ddf473910fc0272d427c Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Fri, 1 Aug 2025 10:37:34 +0100 Subject: [PATCH 6/8] Update .github/ISSUE_TEMPLATE/helm_issue.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Jan-Otto Kröpke Signed-off-by: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/helm_issue.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/ISSUE_TEMPLATE/helm_issue.md b/.github/ISSUE_TEMPLATE/helm_issue.md index 1809041f9f4fc..afa8690a4175a 100644 --- a/.github/ISSUE_TEMPLATE/helm_issue.md +++ b/.github/ISSUE_TEMPLATE/helm_issue.md @@ -109,6 +109,7 @@ body: label: Enter the command that you execute and failing/misfunctioning. description: Enter the command as-is as how you executed. placeholder: helm install my-loki grafana/loki --version 6.34.0 --values values.yaml + render: shell validations: required: true From 1ec55177f4a38bc8cb4730b39fbac470f13ddb77 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Fri, 1 Aug 2025 10:38:43 +0100 Subject: [PATCH 7/8] fixed naming and slack channel --- .github/ISSUE_TEMPLATE/helm_feature.md | 4 ++-- .github/ISSUE_TEMPLATE/helm_issue.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/helm_feature.md b/.github/ISSUE_TEMPLATE/helm_feature.md index daa6c4b958081..6a80a120e08bb 100644 --- a/.github/ISSUE_TEMPLATE/helm_feature.md +++ b/.github/ISSUE_TEMPLATE/helm_feature.md @@ -10,7 +10,7 @@ body: value: | Thanks for taking the time to fill out this feature request for the Loki Helm chart! - For general Loki usage questions, please use the [Grafana Community Slack](https://slack.grafana.com/). This issue tracker is for feature requests specific to the Loki Helm chart. + For general Loki usage questions, please use the [Grafana Community Slack](https://slack.grafana.com/) and join the `#loki` channel. This issue tracker is for feature requests specific to the Loki Helm chart. - type: input id: chart-name @@ -29,7 +29,7 @@ body: options: - Single Binary (monolithic) - Simple Scalable (read, write, backend components) - - Microservices (all components separated) + - Distributed (all components separated) - All deployment modes - Not applicable validations: diff --git a/.github/ISSUE_TEMPLATE/helm_issue.md b/.github/ISSUE_TEMPLATE/helm_issue.md index 037a4a7cc3662..6c517fc4ad422 100644 --- a/.github/ISSUE_TEMPLATE/helm_issue.md +++ b/.github/ISSUE_TEMPLATE/helm_issue.md @@ -10,7 +10,7 @@ body: value: | Thanks for taking the time to fill out this bug report for the Loki Helm chart! Please be cautious with the sensitive information/logs while filing the issue. - For general Loki usage questions, please use the [Grafana Community Slack](https://slack.grafana.com/). This issue tracker is for bugs specific to the Loki Helm chart. + For general Loki usage questions, please use the [Grafana Community Slack](https://slack.grafana.com/) and join the `#loki` channel. This issue tracker is for bugs specific to the Loki Helm chart. - type: textarea id: desc attributes: @@ -59,7 +59,7 @@ body: options: - Single Binary (monolithic) - Simple Scalable (read, write, backend components) - - Microservices (all components separated) + - Distributed (all components separated) validations: required: true From 23082ca5a90a7ea649b98bc28d791bb31e397705 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Mon, 4 Aug 2025 13:59:47 +0100 Subject: [PATCH 8/8] Apply suggestions from code review Co-authored-by: J Stickler Signed-off-by: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/helm_feature.md | 8 ++++---- .github/ISSUE_TEMPLATE/helm_issue.md | 20 ++++++++++---------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/helm_feature.md b/.github/ISSUE_TEMPLATE/helm_feature.md index 6a80a120e08bb..29a787cfa4016 100644 --- a/.github/ISSUE_TEMPLATE/helm_feature.md +++ b/.github/ISSUE_TEMPLATE/helm_feature.md @@ -10,7 +10,7 @@ body: value: | Thanks for taking the time to fill out this feature request for the Loki Helm chart! - For general Loki usage questions, please use the [Grafana Community Slack](https://slack.grafana.com/) and join the `#loki` channel. This issue tracker is for feature requests specific to the Loki Helm chart. + For general Loki usage questions, please use the [Grafana Community Slack](https://slack.grafana.com/) and join the `#loki` channel. This issue template is for feature requests specific to the Loki Helm chart. - type: input id: chart-name @@ -25,7 +25,7 @@ body: id: deployment-mode attributes: label: What's your Loki deployment mode? - description: Select the deployment mode this feature request relates to + description: Select the deployment mode this feature request relates to. options: - Single Binary (monolithic) - Simple Scalable (read, write, backend components) @@ -40,7 +40,7 @@ body: attributes: label: Is your feature request related to a problem ? description: Give a clear and concise description of what the problem is. - placeholder: ex. I'm always frustrated when [...] + placeholder: For example, I'm always frustrated when [...] validations: required: true @@ -56,7 +56,7 @@ body: id: alternatives attributes: label: Describe alternatives you've considered. - description: A clear and concise description of any alternative solutions or features you've considered. If nothing, please enter `NONE` + description: A clear and concise description of any alternative solutions or features you've considered. If nothing, please enter `NONE`. validations: required: true diff --git a/.github/ISSUE_TEMPLATE/helm_issue.md b/.github/ISSUE_TEMPLATE/helm_issue.md index 8a010690ff44a..9bc8913839c3e 100644 --- a/.github/ISSUE_TEMPLATE/helm_issue.md +++ b/.github/ISSUE_TEMPLATE/helm_issue.md @@ -8,13 +8,13 @@ body: - type: markdown attributes: value: | - Thanks for taking the time to fill out this bug report for the Loki Helm chart! Please be cautious with the sensitive information/logs while filing the issue. + Thanks for taking the time to fill out this bug report for the Loki Helm chart! Please be cautious with any sensitive information/logs while filing your issue. - For general Loki usage questions, please use the [Grafana Community Slack](https://slack.grafana.com/) and join the `#loki` channel. This issue tracker is for bugs specific to the Loki Helm chart. + For general Loki usage questions, please use the [Grafana Community Slack](https://slack.grafana.com/) and join the `#loki` channel. This issue template is for bugs specific to the Loki Helm chart. - type: textarea id: desc attributes: - label: Describe the bug a clear and concise description of what the bug is. + label: Describe the bug, providing a clear and concise description of what the bug is. validations: required: true @@ -22,15 +22,15 @@ body: id: helm-version attributes: label: What's your helm version? - description: Enter the output of `$ helm version` - placeholder: Copy paste the entire output of the above + description: Enter the output of `$ helm version`. + placeholder: Copy paste the entire output of the above. validations: required: true - type: input id: kubectl-version attributes: label: What's your kubectl version? - description: Enter the output of `$ kubectl version` + description: Enter the output of `$ kubectl version`. validations: required: true @@ -47,7 +47,7 @@ body: id: chart-version attributes: label: What's the chart version? - description: Enter the version of the chart that you encountered this bug. + description: Enter the version of the chart where you encountered this bug. validations: required: true @@ -55,7 +55,7 @@ body: id: deployment-mode attributes: label: What's your Loki deployment mode? - description: Select the deployment mode you're using + description: Select the deployment mode you're using. options: - Single Binary (monolithic) - Simple Scalable (read, write, backend components) @@ -120,8 +120,8 @@ body: description: | Include any relevant information such as: - Kubernetes cluster details (version, provider) - - Storage backend being used (filesystem, S3, GCS, etc.) + - Storage backend being used (filesystem, S3, GCS, Azure, etc.) - Log volume or specific error messages - - Related components (Grafana, Promtail, etc.) + - Related components (Grafana, Alloy, Promtail, etc.) validations: required: false \ No newline at end of file