-
Notifications
You must be signed in to change notification settings - Fork 229
Update galexie docs for helm chart #2058
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 3 commits
0bf502f
8fe0862
637ab7f
e67a2b2
b919971
4370648
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,35 +7,102 @@ sidebar_position: 10 | |
|
|
||
| ## Google Cloud Platform (GCP) for GCS | ||
|
|
||
| ### Google Cloud Platform (GCP) credentials | ||
|
|
||
| Create application default credentials by using your user account for your GCP project by following these steps: | ||
|
|
||
| 1. Download the [SDK](https://cloud.google.com/sdk/docs/install). | ||
| 2. Install and initialize the [gcloud CLI](https://cloud.google.com/sdk/docs/initializing). | ||
| 3. Create [application default credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc#google-idp) and it should automatically store in this location: `$HOME/.config/gcloud/application_default_credentials.json.` | ||
| 4. Verify that this file exists before moving on to the next step. | ||
|
|
||
| ### Google Cloud Storage (GCS) bucket | ||
|
|
||
| If you already have a GCS bucket with read and write permissions, you can skip this section. If not, follow these steps: | ||
| If you already have a GCS bucket ready for Galexie to push data, you can skip this section. If not, follow these steps: | ||
|
|
||
| 1. Visit the GCP Console's Storage section (https://console.cloud.google.com/storage) and create a new bucket. | ||
| 2. Choose a descriptive name for the bucket, such as `stellar-ledger-data`. Refer to [Google Cloud Storage Bucket Naming Guideline](https://cloud.google.com/storage/docs/buckets#naming) for bucket naming conventions. Note down the bucket name, you will need it later during the configuration process. | ||
|
|
||
| ## Amazon Web Services (AWS) for S3 | ||
| ### Google Cloud Platform (GCP) Authentication | ||
|
|
||
| #### Google Kubernetes Engine Cluster | ||
|
|
||
| When running Galexie inside of a GKE cluster follow the Google cloud documentation for [workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) to make sure Galexie has the correct bucket access | ||
|
|
||
| #### GCP VM | ||
|
|
||
| 1. [Create a Service Account](https://docs.cloud.google.com/iam/docs/service-accounts-create) | ||
| 2. Use that Service Account when creating the GCP VM | ||
| 3. Make sure the Service Account has the correct bucket access | ||
|
|
||
| #### Credentials (Not Recommended) | ||
|
|
||
| In order to use static credentials, find the authentication route that works best in the Galexie environment and follow the Google cloud documentation for [creating credentials](https://developers.google.com/workspace/guides/create-credentials) making sure the principal of the credentials has access to the correct bucket | ||
|
|
||
| #### IAM Role Permissions | ||
|
|
||
| ### Amazon Web Services (AWS) credentials | ||
| When using GCP IAM to authenticate Galexie to access a bucket, the following permissions are required: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This may benefit by describing the break out of bucket permissions for the two use cases of Galexie as a Consumer or Publisher. For consumer, the instance just reads from buckets and would only need reduced set of read permissions stated. And then for Publisher, it would be good to mention the extended write permissions needed.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. when does Galexie act as a consumer? |
||
|
|
||
| Create application default credentials by using your user account for your AWS project by following these steps: | ||
| - storage.buckets.get | ||
| - storage.buckets.list | ||
| - storage.multipartUploads.abort | ||
| - storage.multipartUploads.create | ||
| - storage.multipartUploads.list | ||
| - storage.multipartUploads.listParts | ||
| - storage.objects.create | ||
| - storage.objects.delete | ||
| - storage.objects.get | ||
| - storage.objects.list | ||
| - storage.objects.restore | ||
| - storage.objects.update | ||
|
|
||
| 1. Download and install the [SDK](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#getting-started-install-instructions). | ||
| 2. Create [authentication credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-authentication.html). They should automatically store in this location: `$HOME/.aws/credentials` | ||
| 3. Verify that this file exists before moving on to the next step. | ||
| ## Amazon Web Services (AWS) for S3 | ||
|
|
||
| ### Amazon Simple Storage Service (S3) bucket | ||
|
|
||
| If you already have an S3 bucket with read and write permissions, you can skip this section. If not, follow these steps: | ||
| If you already have an S3 bucket ready for Galexie to push data, you can skip this section. If not, follow these steps: | ||
|
|
||
| 1. Visit the AWS Console's Storage section (https://console.aws.amazon.com/s3/) and create a new bucket. | ||
| 2. Choose a descriptive name for the bucket, such as `stellar-ledger-data`. Refer to [S3 General purpose bucket naming rules](https://cloud.google.com/storage/docs/buckets#naming) for bucket naming conventions. Note down the bucket name, you will need it later during the configuration process. | ||
|
|
||
| ### Amazon Web Services (AWS) Authentication | ||
|
|
||
| #### EKS Cluster | ||
|
|
||
| When running Galexie inside of a EKS cluster follow either the AWS documentation for [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) or [pod identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) | ||
|
|
||
| #### AWS EC2 | ||
|
|
||
| 1. [Creat an IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions_create-policies.html) | ||
| 2. Use that role in an [instance profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html) | ||
| 3. Use that instance profile in the creation of the EC2 instance | ||
| 4. Make sure the instance profile has the correct bucket access | ||
|
|
||
| #### Credentials (Not Recommended) | ||
|
|
||
| In order to use static credentials, [create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started-workloads.html) for Galexie making sure the principal of the credentials has access to the correct bucket and generate security credentials. | ||
|
|
||
| #### IAM Role Permissions | ||
|
|
||
| When using AWS IAM to authenticate Galexie to access a bucket, use this example policy making sure to use the correct bucket destination: | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Sid": "AllowS3BucketOperations", | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "s3:ListBucket", | ||
| "s3:GetBucketLocation", | ||
| "s3:ListBucketMultipartUploads" | ||
| ], | ||
| "Resource": "arn:aws:s3:::my-galexie-bucket-example" | ||
| }, | ||
| { | ||
| "Sid": "AllowS3ObjectAccess", | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "s3:PutObject", | ||
| "s3:GetObject", | ||
| "s3:DeleteObject", | ||
| "s3:AbortMultipartUpload", | ||
| "s3:ListMultipartUploadParts" | ||
| ], | ||
| "Resource": ["arn:aws:s3:::my-galexie-bucket-example/*"] | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think need to mention k8s on this page since the page's purpose is mostly runtime environment agnostic and more focused on just config data to network and bucket storage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cool, what do you think above flipping the structure here and starting with the key config values and then at the bottom having a section just describing copying the config file locally?