fix: Make datacatalog gRPC server MaxRecvMsgSize configurable #6313

honnix · 2025-03-06T16:09:51Z

Tracking issue

Why are the changes needed?

It is possible that a task might upload an artifact larger than 4MB that is default value of MaxRecvMsgSize of gRPC server, and when that happens propeller would get a failed call. It feels reasonable to leave the choice to users.

What changes were proposed in this pull request?

Make datacatalog gRPC server MaxRecvMsgSize configurable

How was this patch tested?

Partially unit tested

Labels

Please add one or more of the following labels to categorize your PR:

added: For new features.
changed: For changes in existing functionality.
deprecated: For soon-to-be-removed features.
removed: For features being removed.
fixed: For any bug fixed.
security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

Docs link

Summary by Bito

This PR makes gRPC MaxReceiveMessageSize configurable by introducing 'grpcMaxRecvMsgSizeMBs' parameter in chart and configuration files. It enables the datacatalog gRPC server to handle larger artifact uploads without failure. Deployment templates and Docker manifests have been updated to ensure consistency across environments.

Unit tests added: False

Estimated effort to review (1-5, lower is better): 3

flyte-bot · 2025-03-06T16:10:16Z

Code Review Agent Run Status

Limitations and other issues: ❌ Failure - The AI Code Review Agent skipped reviewing this change because it is configured to exclude certain pull requests based on the source/target branch or the pull request status. You can change the settings here, or contact the agent instance creator at [email protected].

codecov · 2025-03-06T16:13:20Z

Codecov Report

Attention: Patch coverage is 10.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 58.49%. Comparing base (695f5b5) to head (c32e22c).
Report is 65 commits behind head on master.

Files with missing lines	Patch %	Lines
datacatalog/pkg/rpc/datacatalogservice/service.go	0.00%	9 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6313   +/-   ##
=======================================
  Coverage   58.48%   58.49%           
=======================================
  Files         937      937           
  Lines       71091    71094    +3     
=======================================
+ Hits        41580    41586    +6     
+ Misses      26359    26356    -3     
  Partials     3152     3152

Flag	Coverage Δ
unittests-datacatalog	`59.03% <10.00%> (-0.04%)`	⬇️
unittests-flyteadmin	`56.30% <ø> (+0.02%)`	⬆️
unittests-flytecopilot	`30.99% <ø> (ø)`
unittests-flytectl	`64.70% <ø> (ø)`
unittests-flyteidl	`76.12% <ø> (ø)`
unittests-flyteplugins	`61.00% <ø> (ø)`
unittests-flytepropeller	`54.80% <ø> (ø)`
unittests-flytestdlib	`64.02% <ø> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

datacatalog/pkg/config/config_flags.go

flyte-bot · 2025-03-06T17:19:07Z

Code Review Agent Run #8929e2

Actionable Suggestions - 1

datacatalog/pkg/rpc/datacatalogservice/service.go - 1
- Consider checking for positive value instead · Line 142-142

Review Details

Files reviewed - 5 · Commit Range: d3e776d..d3e776d
- datacatalog/datacatalog_config.yaml
- datacatalog/pkg/config/config.go
- datacatalog/pkg/config/config_flags.go
- datacatalog/pkg/config/config_flags_test.go
- datacatalog/pkg/rpc/datacatalogservice/service.go
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful

AI Code Review powered by

flyte-bot · 2025-03-06T17:23:07Z

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Feature Improvement - gRPC Configuration Enhancements	- `values-keycloak-idp-flyteclients-without-browser.yaml` - Added grpcMaxRecvMsgSizeMBs: 6 to update client configuration. - `values.yaml` - Inserted grpcMaxRecvMsgSizeMBs: 6 for updated core settings. - `values.yaml` - Included grpcMaxRecvMsgSizeMBs: 6 with appropriate indentation for flyte charts. - `datacatalog_config.yaml` - Introduced grpcMaxRecvMsgSizeMBs: 6 in the datacatalog configuration. - `config.go` - Defined a new field for grpcMaxRecvMsgSizeMBs and updated the default config block. - `config_flags.go` - Added a command-line flag to configure grpcMaxRecvMsgSizeMBs. - `config_flags_test.go` - Implemented new testing for the grpcMaxRecvMsgSizeMBs flag override. - `service.go` - Modified gRPC server initialization to conditionally include a max receive message size option.
Other Improvements - Deployment and Docker Manifest Updates	- `flyte_aws_scheduler_helm_generated.yaml` - Appended grpcMaxRecvMsgSizeMBs: 6 and updated the config checksum value. - `flyte_helm_controlplane_generated.yaml` - Added grpcMaxRecvMsgSizeMBs: 6 and modified the config checksum for consistency. - `flyte_helm_generated.yaml` - Inserted grpcMaxRecvMsgSizeMBs: 6 and refreshed the checksum to match deployment changes. - `flyte_helm_controlplane_generated.yaml` - Included grpcMaxRecvMsgSizeMBs: 6 and updated the config checksum. - `flyte_helm_generated.yaml` - Appended grpcMaxRecvMsgSizeMBs: 6 and revised the checksum value. - `flyte_helm_generated.yaml` - Added grpcMaxRecvMsgSizeMBs: 6 and updated the checksum accordingly. - `complete-agent.yaml` - Updated haSharedSecret to a new value for enhanced security. - `complete.yaml` - Revised haSharedSecret and checksum/secret values to align with current configs. - `dev.yaml` - Modified haSharedSecret and checksum/secret entries to reflect updated secrets.

datacatalog/pkg/rpc/datacatalogservice/service.go

flyte-bot · 2025-03-06T21:19:08Z

Code Review Agent Run #c55beb

Actionable Suggestions - 0

Review Details

Files reviewed - 5 · Commit Range: fd7b760..fd7b760
- datacatalog/datacatalog_config.yaml
- datacatalog/pkg/config/config.go
- datacatalog/pkg/config/config_flags.go
- datacatalog/pkg/config/config_flags_test.go
- datacatalog/pkg/rpc/datacatalogservice/service.go
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful

AI Code Review powered by

eapolinario

Can you also expose this in the flyte-core and flyte-binary charts?

honnix · 2025-03-11T22:06:55Z

Can you also expose this in the flyte-core and flyte-binary charts?

I did some changes in the last commit. Please let me know if I did something wrong or missed some. I'm not familiar with these things. Thank you.

flyte-bot · 2025-03-11T23:12:58Z

Code Review Agent Run #752826

Actionable Suggestions - 0

Review Details

Files reviewed - 3 · Commit Range: fd7b760..a58c9f9
- charts/flyte-core/values-keycloak-idp-flyteclients-without-browser.yaml
- charts/flyte-core/values.yaml
- charts/flyte/values.yaml
Files skipped - 2
- charts/flyte-core/README.md - Reason: Filter setting
- charts/flyte/README.md - Reason: Filter setting
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

Signed-off-by: Hongxin Liang <[email protected]>

flyte-bot · 2025-03-12T09:23:16Z

Code Review Agent Run #6335fa

Actionable Suggestions - 0

Review Details

Files reviewed - 17 · Commit Range: c32e22c..c32e22c
- charts/flyte-core/values-keycloak-idp-flyteclients-without-browser.yaml
- charts/flyte-core/values.yaml
- charts/flyte/values.yaml
- datacatalog/datacatalog_config.yaml
- datacatalog/pkg/config/config.go
- datacatalog/pkg/config/config_flags.go
- datacatalog/pkg/config/config_flags_test.go
- datacatalog/pkg/rpc/datacatalogservice/service.go
- deployment/eks/flyte_aws_scheduler_helm_generated.yaml
- deployment/eks/flyte_helm_controlplane_generated.yaml
- deployment/eks/flyte_helm_generated.yaml
- deployment/gcp/flyte_helm_controlplane_generated.yaml
- deployment/gcp/flyte_helm_generated.yaml
- deployment/sandbox/flyte_helm_generated.yaml
- docker/sandbox-bundled/manifests/complete-agent.yaml
- docker/sandbox-bundled/manifests/complete.yaml
- docker/sandbox-bundled/manifests/dev.yaml
Files skipped - 2
- charts/flyte-core/README.md - Reason: Filter setting
- charts/flyte/README.md - Reason: Filter setting
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

eapolinario · 2025-03-12T17:52:23Z

Thank you!

honnix added the fixed For any bug fixes label Mar 6, 2025

honnix force-pushed the honnix/datacatalog-max-rcv branch from 4ec063f to d3e776d Compare March 6, 2025 16:16

honnix commented Mar 6, 2025

View reviewed changes

datacatalog/pkg/config/config_flags.go Show resolved Hide resolved

honnix marked this pull request as ready for review March 6, 2025 16:28

honnix requested a review from hamersaw March 6, 2025 16:28

flyte-bot reviewed Mar 6, 2025

View reviewed changes

datacatalog/pkg/rpc/datacatalogservice/service.go Outdated Show resolved Hide resolved

honnix force-pushed the honnix/datacatalog-max-rcv branch from d3e776d to fd7b760 Compare March 6, 2025 20:17

honnix requested a review from eapolinario March 7, 2025 09:59

eapolinario reviewed Mar 11, 2025

View reviewed changes

honnix force-pushed the honnix/datacatalog-max-rcv branch 3 times, most recently from 5f49aea to 0b15ec8 Compare March 12, 2025 08:23

fix: Make datacatalog gRPC server MaxRecvMsgSize configurable

c32e22c

Signed-off-by: Hongxin Liang <[email protected]>

honnix force-pushed the honnix/datacatalog-max-rcv branch from 0b15ec8 to c32e22c Compare March 12, 2025 08:37

eapolinario approved these changes Mar 12, 2025

View reviewed changes

eapolinario merged commit 90eaf2e into flyteorg:master Mar 12, 2025
55 of 56 checks passed

honnix deleted the honnix/datacatalog-max-rcv branch March 12, 2025 18:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: Make datacatalog gRPC server MaxRecvMsgSize configurable #6313

fix: Make datacatalog gRPC server MaxRecvMsgSize configurable #6313

Uh oh!

honnix commented Mar 6, 2025 •

edited by flyte-bot

Loading

Uh oh!

flyte-bot commented Mar 6, 2025

Uh oh!

codecov bot commented Mar 6, 2025 •

edited

Loading

Uh oh!

Uh oh!

flyte-bot commented Mar 6, 2025 •

edited

Loading

Code Review Agent Run #8929e2

Uh oh!

flyte-bot commented Mar 6, 2025 •

edited

Loading

Changelist by Bito

Uh oh!

Uh oh!

flyte-bot commented Mar 6, 2025 •

edited

Loading

Code Review Agent Run #c55beb

Uh oh!

eapolinario left a comment

Uh oh!

honnix commented Mar 11, 2025

Uh oh!

flyte-bot commented Mar 11, 2025 •

edited

Loading

Code Review Agent Run #752826

Bito Usage Guide

Uh oh!

flyte-bot commented Mar 12, 2025 •

edited

Loading

Code Review Agent Run #6335fa

Bito Usage Guide

Uh oh!

eapolinario commented Mar 12, 2025

Uh oh!

Uh oh!

Uh oh!

fix: Make datacatalog gRPC server MaxRecvMsgSize configurable #6313

fix: Make datacatalog gRPC server MaxRecvMsgSize configurable #6313

Uh oh!

Conversation

honnix commented Mar 6, 2025 • edited by flyte-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Labels

Setup process

Screenshots

Check all the applicable boxes

Related PRs

Docs link

Summary by Bito

Uh oh!

flyte-bot commented Mar 6, 2025

Code Review Agent Run Status

Uh oh!

codecov bot commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

flyte-bot commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #8929e2

Uh oh!

flyte-bot commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelist by Bito

Uh oh!

Uh oh!

flyte-bot commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #c55beb

Uh oh!

eapolinario left a comment

Choose a reason for hiding this comment

Uh oh!

honnix commented Mar 11, 2025

Uh oh!

flyte-bot commented Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #752826

Bito Usage Guide

Uh oh!

flyte-bot commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #6335fa

Bito Usage Guide

Uh oh!

eapolinario commented Mar 12, 2025

Uh oh!

Uh oh!

Uh oh!

honnix commented Mar 6, 2025 •

edited by flyte-bot

Loading

codecov bot commented Mar 6, 2025 •

edited

Loading

flyte-bot commented Mar 6, 2025 •

edited

Loading

flyte-bot commented Mar 6, 2025 •

edited

Loading

flyte-bot commented Mar 6, 2025 •

edited

Loading

flyte-bot commented Mar 11, 2025 •

edited

Loading

flyte-bot commented Mar 12, 2025 •

edited

Loading