Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/sources/setup/upgrade/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,24 @@ The next Loki release introduces a new configuration option (i.e. `-distibutor.m

Supported clients should check the configuration options for max send message size if applicable.

## Helm Chart Upgrades

### Helm Chart 6.34.0 - Zone-aware Ingester Breaking Change

{{< admonition type="warning" >}}
Helm chart version 6.34.0 introduces a **breaking change** that affects users with zone-aware ingester replication enabled.
{{< /admonition >}}

If you are using zone-aware ingesters (`ingester.zoneAwareReplication.enabled: true`), upgrading to Helm chart 6.34.0 requires manual StatefulSet deletion before the upgrade. This is due to a fix for the `serviceName` field in zone-aware ingester StatefulSets, which is an immutable field in Kubernetes.

**For detailed upgrade instructions, see**: [Helm Chart 6.x Upgrade Guide - Zone-aware Ingester Breaking Change](https://grafana.com/docs/loki/latest/setup/upgrade/upgrade-to-6x/#breaking-zone-aware-ingester-statefulset-servicename-fix-6340)

Key points:
- Only affects deployments with `ingester.zoneAwareReplication.enabled: true`
- Requires manual StatefulSet deletion with `--cascade=orphan`
- **No data loss** - PersistentVolumeClaims and data are preserved
- New StatefulSets will be created with correct service references

## 3.4.0

### Loki 3.4.0
Expand Down
36 changes: 36 additions & 0 deletions docs/sources/setup/upgrade/upgrade-to-6x/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,42 @@ resultsCache:

With these caches disabled, Loki will return to defaults which enables an in-memory results and chunks cache, so you will still get some caching.

#### BREAKING: Zone-aware ingester StatefulSet serviceName fix (6.34.0+)

**Affected users**: Only deployments using zone-aware ingester replication (`ingester.zoneAwareReplication.enabled: true`)

In Helm chart version 6.34.0, [PR #18558](https://github.com/grafana/loki/pull/18558) fixed the `serviceName` field in zone-aware ingester StatefulSets to correctly reference headless services. However, since `serviceName` is an immutable field in Kubernetes StatefulSets, upgrading to 6.34.0 requires manual intervention.

**Required action before upgrading to 6.34.0**:

1. **Check if you're affected**:
```bash
helm get values <release-name> | grep -A5 zoneAwareReplication
```
If `enabled: true` appears, you need to follow these steps.

2. **Delete the StatefulSets** (data will be preserved):
```bash
kubectl delete statefulset \
<release-name>-ingester-zone-a \
<release-name>-ingester-zone-b \
<release-name>-ingester-zone-c \
--cascade=orphan
```

3. **Proceed with the Helm upgrade**:
```bash
helm upgrade <release-name> grafana/loki --version 6.34.0
```

**What happens**:
- PersistentVolumeClaims and data are preserved
- New StatefulSets will be created with correct service references
- Pods will restart and reattach to existing storage

**Why this change was necessary**:
The previous configuration caused ingester scaling operations to fail because the rollout-operator couldn't find the correct headless services for the `/ingester/prepare-downscale` endpoint.

#### Distributed mode

This chart introduces the ability to run Loki in distributed, or [microservices mode](https://grafana.com/docs/loki/<LOKI_VERSION>/get-started/deployment-modes/#microservices-mode). Separate instructions on how to enable this as well as how to migrate from the existing community chart will be coming shortly.
5 changes: 2 additions & 3 deletions production/helm/loki/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ Entries should include a reference to the pull request that introduced the chang
## 6.34.0

- [CHANGE] Updated version of Grafana Loki to 3.5.3
- [CHANGE] **BREAKING** Fixed serviceName in zone-aware ingester StatefulSets to reference correct headless services [#18558](https://github.com/grafana/loki/pull/18558). **Upgrade Impact**: Users with zone-aware ingesters (`ingester.zoneAwareReplication.enabled: true`) must manually delete the ingester StatefulSets before upgrading: `kubectl delete statefulset <release-name>-ingester-zone-a <release-name>-ingester-zone-b <release-name>-ingester-zone-c --cascade=orphan`. PersistentVolumeClaims and data will be preserved. **See**: [Helm chart upgrade guide](https://grafana.com/docs/loki/latest/setup/upgrade/upgrade-to-6x/#breaking-zone-aware-ingester-statefulset-servicename-fix-6340)
- [FEATURE] Expose labels on volumeClaimTemplates [#13317](https://github.com/grafana/loki/pull/13317)
- [FEATURE] Allow setting custom labels for ingester statefulsets [#18536](https://github.com/grafana/loki/pull/18536)
- [FEATURE] Added support for chunk-cache-l2 [#17556](https://github.com/grafana/loki/pull/17556)
Expand All @@ -25,9 +26,7 @@ Entries should include a reference to the pull request that introduced the chang
- [ENHANCEMENT] Add configurable `startupProbe` to the loki-sc-rules sidecar container in the backend pods. [#18547](https://github.com/grafana/loki/pull/18547)
- [BUGFIX] Add validation for loki.storage.bucketNames [#13781](https://github.com/grafana/loki/pull/13781)
- [BUGFIX] Create missing RBAC for the rules sidecar when RBAC is namespaced [#16776](https://github.com/grafana/loki/pull/16776)
- [BUGFIX] Ensure the scc is actually allowing to use hostPath volumes when the rbac.sccAllowHostDirVolumePlugin is set to true [#17680](https://github.com/grafana/loki/pull/17680)
- [BUGFIX] Fix serviceName in zone-aware ingester statefulsets [#18558](https://github.com/grafana/loki/pull/18558)

- [BUGFIX] Ensure the scc is actually allowing to use hostPath volumes when the rbac.sccAllowHostDirVolumePlugin is set to true [#17680](https://github.com/grafana/loki/pull/17680)
## 6.33.0

- [FEATURE] Allow passing tenant password hash instead of password. [#17049](https://github.com/grafana/loki/pull/17049)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,11 @@ loki:
querier:
# Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
max_concurrent: 4

storage:
bucketNames:
chunks: chunks
ruler: ruler
admin: admin
#gateway:
# ingress:
# enabled: true
Expand Down
Loading