Skip to content

Document volume deletion for Grafana #1887

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 15, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 31 additions & 9 deletions docs/clusters/aws/uninstall.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ cortex cluster down

## Delete metadata and log groups

Since you may wish to have access to your data after spinning down your cluster, Cortex's bucket, log groups, and Prometheus volume are not automatically deleted when running `cortex cluster down`.
Since you may wish to have access to your data after spinning down your cluster, Cortex's bucket, log groups, and
Prometheus volume are not automatically deleted when running `cortex cluster down`.

To delete them:

Expand All @@ -25,24 +26,45 @@ aws s3 rb --force s3://<bucket>
aws logs describe-log-groups --log-group-name-prefix=<cluster_name> --query logGroups[*].[logGroupName] --output text | xargs -I {} aws logs delete-log-group --log-group-name {}
```

To delete the Prometheus volume, navigate to the [EC2 volumes page](https://console.aws.amazon.com/ec2/v2/home?#Volumes) in the AWS console (be sure to set the appropriate region), select the volume, and click "Actions" and then "Delete Volume". A Prometheus volume that Cortex created has a name that starts with `kubernetes-dynamic-pvc`, the `kubernetes.io/cluster/<cluster name>` tag is set to `owned`, and the `kubernetes.io/created-for/pvc/name` tag starts with `prometheus-`.
## Delete Volumes

If you've configured a custom domain for your APIs, you can remove the SSL Certificate and Hosted Zone for the domain by following these [instructions](networking/custom-domain.md#cleanup).
The volumes used by Cortex's Prometheus and Grafana instances are not deleted by default, as they might contain important
information. If these volumes are not required anymore, you can delete them in the AWS console.

To delete the volumes, navigate to the [EC2 volumes page](https://console.aws.amazon.com/ec2/v2/home?#Volumes)
in the AWS console (be sure to set the appropriate region), select the volumes, click "Actions" and then "Delete Volume"
. Both volumes for Prometheus and Grafana that Cortex created have a name that starts with `kubernetes-dynamic-pvc`,
the `kubernetes.io/cluster/<cluster name>` tag is set to `owned`, and the `kubernetes.io/created-for/pvc/name` tag start
with `prometheus-` and `grafana-` respectively.

## Delete Certificates

If you've configured a custom domain for your APIs, you can remove the SSL Certificate and Hosted Zone for the domain by
following these [instructions](networking/custom-domain.md#cleanup).

## Troubleshooting

On rare occasions, `cortex cluster down` may not be able to spin down your Cortex cluster. When this happens, follow these steps:
On rare occasions, `cortex cluster down` may not be able to spin down your Cortex cluster. When this happens, follow
these steps:

1. If you've manually created any AWS networking resources that are pointed to the cluster or its VPC (e.g. API Gateway VPC links, custom domains, etc), delete them from the AWS console.
1. If you've manually created any AWS networking resources that are pointed to the cluster or its VPC (e.g. API Gateway
VPC links, custom domains, etc), delete them from the AWS console.

1. Replace "<region>" and "<cluster_name>" in the following URL, and open it in your browser: `https://console.aws.amazon.com/cloudformation/home?region=<region>#/stacks?filteringText=eksctl-<cluster_name>-`
1. Replace "<region>" and "<cluster_name>" in the following URL, and open it in your
browser: `https://console.aws.amazon.com/cloudformation/home?region=<region>#/stacks?filteringText=eksctl-<cluster_name>-`

![image](https://user-images.githubusercontent.com/808475/97790394-963b4880-1b85-11eb-8e27-ba5a551606b3.png)
![image](https://user-images.githubusercontent.com/808475/97790394-963b4880-1b85-11eb-8e27-ba5a551606b3.png)

1. For each CloudFormation stack which contains the word "nodegroup", select the stack and click "Delete".

1. Select the final stack (the one that ends in "-cluster") and click "Delete".

If deleting the stack fails, navigate to the EC2 dashboard in the AWS console, delete the load balancers that are associated with the cluster, and try again (you can determine which load balancers are associated with the cluster by setting the correct region in the console and checking the `cortex.dev/cluster-name` tag on all load balancers). If the problem still persists, delete any other AWS resources that are blocking the stack deletion and try again.
If deleting the stack fails, navigate to the EC2 dashboard in the AWS console, delete the load balancers that are
associated with the cluster, and try again (you can determine which load balancers are associated with the cluster by
setting the correct region in the console and checking the `cortex.dev/cluster-name` tag on all load balancers). If
the problem still persists, delete any other AWS resources that are blocking the stack deletion and try again.

1. In rare cases, you may need to delete other AWS resources associated with your Cortex cluster. For each the following resources, go to the appropriate AWS Dashboard (in the region that your cluster was in), and confirm that there are no resources left behind by the cluster: CloudWatch Dashboard, SQS Queues, S3 Bucket, and CloudWatch LogGroups (the Cortex bucket and log groups are not deleted by `cluster down` in order to preserve your data).
1. In rare cases, you may need to delete other AWS resources associated with your Cortex cluster. For each the following
resources, go to the appropriate AWS Dashboard (in the region that your cluster was in), and confirm that there are
no resources left behind by the cluster: CloudWatch Dashboard, SQS Queues, S3 Bucket, and CloudWatch LogGroups (the
Cortex bucket and log groups are not deleted by `cluster down` in order to preserve your data).
12 changes: 9 additions & 3 deletions docs/clusters/gcp/uninstall.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,14 @@
cortex cluster-gcp down
```

The `cortex cluster-gcp down` command doesn't wait for the cluster to spin down. You can ensure that the cluster has spun down by checking the GKE console.
The `cortex cluster-gcp down` command doesn't wait for the cluster to spin down. You can ensure that the cluster has
spun down by checking the GKE console.

## Delete Prometheus Volume
## Delete Volumes

The volume used by Cortex's Prometheus instance is not deleted by default, as it might contain important information. If this volume is not required anymore, you can delete it in the GCP console. Navigate to the [Disks](https://console.cloud.google.com/compute/disks) page (be sure to set the appropriate project), select the volume, and click "Delete". A Prometheus volume that Cortex created has a name that starts with `gke-<cluster name>-`, and the `kubernetes.io/created-for/pvc/name` tag starts with `prometheus-`.
The volumes used by Cortex's Prometheus and Grafana instances are not deleted by default, as they might contain important
information. If these volumes are not required anymore, you can delete them in the GCP console. Navigate to
the [Disks](https://console.cloud.google.com/compute/disks) page (be sure to set the appropriate project), select the
volumes, and click "Delete". The Prometheus and Grafana volumes that Cortex created have a name that starts
with `gke-<cluster name>-`, and the `kubernetes.io/created-for/pvc/name` tag starts with `prometheus-` and `grafana-`
respectively.