[GRPC] Adapt transport-grpc to use opensearch-protobufs 0.13.0

[karenyrx]

Description

This PR fixes all compilation errors in the transport-grpc module when upgrading opensearch-protobufs from 0.6.0 to 0.13.0.

Co-authored by:

• @sakrah - who authored the changes from 0.6.0 -> 0.7.0 (the previous PR Upgrade opensearch-protobufs dependency to 0.7.0 and update transport… #19003 on which this was stacked was closed)
• @lucy66hw - who authored the changes from 0.12.0 -> 0.13.0

Test Plan

Sample comparison of OpenSearch Request/Responses for HTTP and gRPC (before vs. after) -
Note: This is not comprehensive of all fields that had changes.

Sample Term Query

HTTP (JSON)

POST /test-index/_search
{
  "query": {
    "term": {
      "category": "books"
    }
  }
}

gRPC (Before - opensearch-protobufs 0.6.0)

{
  "index": ["test-index"],
  "request_body": {
    "query": {
      "term": {
        "field": "category",
        "value": {
          "string_value": "books"
        },
        "underscore_name": "my_term_query"
      }
    }
  }
}

gRPC (After - opensearch-protobufs 0.13.0)

{
  "index": ["test-index"],
  "request_body": {
    "query": {
      "term": {
        "field": "category",
        "value": {
          "string": "books"
        },
        "x_name": "my_term_query"
      }
    }
  }
}

Sample Terms Query

HTTP (JSON)

POST /test-index/_search
{
  "query": {
    "terms": {
      "category": ["books", "electronics", "clothing"]
    }
  }
}

gRPC (Before - opensearch-protobufs 0.6.0)

{
  "index": ["test-index"],
  "request_body": {
    "query": {
      "terms_query_field": {
        "terms_lookup_field_string_array_map": {
          "category": {
            "field_value_array": {
              "field_value_array": [
                {"string_value": "books"},
                {"string_value": "electronics"},
                {"string_value": "clothing"}
              ]
            }
          }
        },
        "underscore_name": "my_terms_query"
      }
    }
  }
}

gRPC (After - opensearch-protobufs 0.13.0)

{
  "index": ["test-index"],
  "request_body": {
    "query": {
      "terms": {
        "terms": {
          "category": {
            "field_value_array": {
              "field_value_array": [
                {"string": "books"},
                {"string": "electronics"},
                {"string": "clothing"}
              ]
            }
          }
        },
        "x_name": "my_terms_query"
      }
    }
  }
}

Sample Bulk Request

HTTP (JSON)

POST /_bulk
{"index": {"_index": "test-index", "_id": "1"}}
{"title": "New Document", "price": 29.99, "active": true}
{"create": {"_index": "test-index", "_id": "2"}}
{"title": "Created Document", "price": 19.99, "active": false}
{"update": {"_index": "test-index", "_id": "3"}}
{"doc": {"title": "Updated Document", "price": 39.99}}
{"delete": {"_index": "test-index", "_id": "4"}}

gRPC (Before - opensearch-protobufs 0.6.0)

{
  "request_body": [
    {
      "index": {
        "id": "1",
        "index": "test-index"
      },
      "doc": "eyJ0aXRsZSI6ICJOZXcgRG9jdW1lbnQiLCAicHJpY2UiOiAyOS45OSwgImFjdGl2ZSI6IHRydWV9"
    },
    {
      "create": {
        "id": "2",
        "index": "test-index"
      },
      "doc": "eyJ0aXRsZSI6ICJDcmVhdGVkIERvY3VtZW50IiwgInByaWNlIjogMTkuOTksICJhY3RpdmUiOiBmYWxzZX0="
    },
    {
      "update": {
        "id": "3",
        "index": "test-index"
      },
      "doc": "eyJkb2MiOiB7InRpdGxlIjogIlVwZGF0ZWQgRG9jdW1lbnQiLCAicHJpY2UiOiAzOS45OX19",
      "detect_noop": true,
      "doc_as_upsert": false
    },
    {
      "delete": {
        "id": "4",
        "index": "test-index"
      }
    }
  ]
}

gRPC (After - opensearch-protobufs 0.13.0)

{
  "request_body": [
    {
      "operation_container": {
        "index": {
          "x_id": "1",
          "x_index": "test-index"
        }
      },
      "object": "eyJ0aXRsZSI6ICJOZXcgRG9jdW1lbnQiLCAicHJpY2UiOiAyOS45OSwgImFjdGl2ZSI6IHRydWV9"
    },
    {
      "operation_container": {
        "create": {
          "x_id": "2",
          "x_index": "test-index"
        }
      },
      "object": "eyJ0aXRsZSI6ICJDcmVhdGVkIERvY3VtZW50IiwgInByaWNlIjogMTkuOTksICJhY3RpdmUiOiBmYWxzZX0="
    },
    {
      "operation_container": {
        "update": {
          "x_id": "3",
          "x_index": "test-index"
        }
      },
      "update_action": {
        "doc": "eyJkb2MiOiB7InRpdGxlIjogIlVwZGF0ZWQgRG9jdW1lbnQiLCAicHJpY2UiOiAzOS45OX19",
        "detect_noop": true,
        "doc_as_upsert": false
      }
    },
    {
      "operation_container": {
        "delete": {
          "x_id": "4",
          "x_index": "test-index"
        }
      }
    }
  ]
}

Sample Bulk Response

HTTP (JSON)

{
  "took": 15,
  "errors": false,
  "items": [
    {
      "index": {
        "_index": "test-index",
        "_id": "1",
        "_version": 1,
        "result": "created",
        "_shards": {"total": 2, "successful": 1, "failed": 0},
        "status": 201
      }
    },
    {
      "create": {
        "_index": "test-index",
        "_id": "2", 
        "_version": 1,
        "result": "created",
        "_shards": {"total": 2, "successful": 1, "failed": 0},
        "status": 201
      }
    },
    {
      "update": {
        "_index": "test-index",
        "_id": "3",
        "_version": 2,
        "result": "updated",
        "_shards": {"total": 2, "successful": 1, "failed": 0},
        "status": 200
      }
    },
    {
      "delete": {
        "_index": "test-index",
        "_id": "4",
        "_version": 2,
        "result": "deleted",
        "_shards": {"total": 2, "successful": 1, "failed": 0},
        "status": 200
      }
    }
  ]
}

gRPC (Before - opensearch-protobufs 0.6.0)

{
  "bulk_response_body": {
    "took": 15,
    "errors": false,
    "items": [
      {
        "index": {
          "type": "_doc",
          "id": {"string": "1"},
          "index": "test-index",
          "primary_term": 1,
          "result": "RESULT_CREATED",
          "seq_no": 0,
          "shards": {"total": 2, "successful": 1, "failed": 0},
          "version": 1,
          "status": 201
        }
      },
      {
        "create": {
          "type": "_doc", 
          "id": {"string": "2"},
          "index": "test-index",
          "primary_term": 1,
          "result": "RESULT_CREATED",
          "seq_no": 1,
          "shards": {"total": 2, "successful": 1, "failed": 0},
          "version": 1,
          "status": 201
        }
      },
      {
        "update": {
          "type": "_doc",
          "id": {"string": "3"},
          "index": "test-index", 
          "primary_term": 1,
          "result": "RESULT_UPDATED",
          "seq_no": 2,
          "shards": {"total": 2, "successful": 1, "failed": 0},
          "version": 2,
          "status": 200
        }
      },
      {
        "delete": {
          "type": "_doc",
          "id": {"string": "4"},
          "index": "test-index",
          "primary_term": 1,
          "result": "RESULT_DELETED", 
          "seq_no": 3,
          "shards": {"total": 2, "successful": 1, "failed": 0},
          "version": 2,
          "status": 200
        }
      }
    ]
  }
}

gRPC (After - opensearch-protobufs 0.13.0)

{
  "took": 15,
  "errors": false,
  "items": [
    {
      "index": {
        "x_type": "_doc",
        "x_id": {"string": "1"},
        "x_index": "test-index",
        "x_primary_term": 1,
        "result": "RESULT_CREATED",
        "x_seq_no": 0,
        "x_shards": {"total": 2, "successful": 1, "failed": 0},
        "x_version": 1,
        "status": 201
      }
    },
    {
      "create": {
        "x_type": "_doc",
        "x_id": {"string": "2"}, 
        "x_index": "test-index",
        "x_primary_term": 1,
        "result": "RESULT_CREATED",
        "x_seq_no": 1,
        "x_shards": {"total": 2, "successful": 1, "failed": 0},
        "x_version": 1,
        "status": 201
      }
    },
    {
      "update": {
        "x_type": "_doc",
        "x_id": {"string": "3"},
        "x_index": "test-index",
        "x_primary_term": 1,
        "result": "RESULT_UPDATED",
        "x_seq_no": 2,
        "x_shards": {"total": 2, "successful": 1, "failed": 0},
        "x_version": 2,
        "status": 200
      }
    },
    {
      "delete": {
        "x_type": "_doc",
        "x_id": {"string": "4"},
        "x_index": "test-index",
        "x_primary_term": 1,
        "result": "RESULT_DELETED",
        "x_seq_no": 3,
        "x_shards": {"total": 2, "successful": 1, "failed": 0},
        "x_version": 2,
        "status": 200
      }
    }
  ]
}

Sample Search Response

HTTP (JSON)

{
  "took": 3,
  "timed_out": false,
  "hits": {
    "total": {"value": 2, "relation": "eq"},
    "max_score": 1.0,
    "hits": [
      {
        "_index": "test-index",
        "_id": "1",
        "_score": 1.0,
        "_source": {"title": "Document 1", "category": "books"}
      },
      {
        "_index": "test-index", 
        "_id": "2",
        "_score": 1.0,
        "_source": {"title": "Document 2", "category": "electronics"}
      }
    ]
  }
}

gRPC (Before - opensearch-protobufs 0.6.0)

{
  "response_body": {
    "took": 3,
    "timed_out": false,
    "hits": {
      "total": {
        "total_hits": {
          "value": 2,
          "relation": "TOTAL_HITS_RELATION_EQ"
        }
      },
      "max_score": {"float_value": 1.0},
      "hits": [
        {
          "index": "test-index",
          "id": "1",
          "score": {"float_value": 1.0},
          "source": "eyJ0aXRsZSI6ICJEb2N1bWVudCAxIiwgImNhdGVnb3J5IjogImJvb2tzIn0="
        },
        {
          "index": "test-index",
          "id": "2", 
          "score": {"float_value": 1.0},
          "source": "eyJ0aXRsZSI6ICJEb2N1bWVudCAyIiwgImNhdGVnb3J5IjogImVsZWN0cm9uaWNzIn0="
        }
      ]
    }
  }
}

gRPC (After - opensearch-protobufs 0.13.0)

{
  "took": 3,
  "timed_out": false,
  "x_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "hits": {
    "total": {
      "total_hits": {
        "value": 2,
        "relation": "TOTAL_HITS_RELATION_EQ"
      }
    },
    "max_score": {"float": 1.0},
    "hits": [
      {
        "x_index": "test-index",
        "x_id": "1",
        "x_score": {"double": 1.0},
        "x_source": "eyJ0aXRsZSI6ICJEb2N1bWVudCAxIiwgImNhdGVnb3J5IjogImJvb2tzIn0="
      },
      {
        "x_index": "test-index",
        "x_id": "2",
        "x_score": {"double": 1.0}, 
        "x_source": "eyJ0aXRsZSI6ICJEb2N1bWVudCAyIiwgImNhdGVnb3J5IjogImVsZWN0cm9uaWNzIn0="
      }
    ]
  }
}

Related Issues

#19020

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 6a30ba5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 765acec: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for dfad397: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 7bb83ab: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for daa3789: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[karenyrx]

Will retry the gradle check flaky tests (which are unrelated to this PR) once #19003 is merged, since it will require a new merge commit regardless

> Task :server:internalClusterTest

Tests with failures:
 - org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness
 - org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testIndexWhileRecoveringReplica

5497 tests completed, 2 failed, 56 skipped

[github-actions]

✅ Gradle check result for 63a5820: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 79.45205% with 60 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.91%. Comparing base (53dc1ea) to head (ae21a00).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
...uest/search/query/TermsQueryBuilderProtoUtils.java	59.72%	22 Missing and 7 partials ⚠️
...pc/proto/response/common/FieldValueProtoUtils.java	82.50%	4 Missing and 3 partials ⚠️
...response/document/bulk/BulkResponseProtoUtils.java	57.14%	5 Missing and 1 partial ⚠️
...o/request/search/SearchAfterBuilderProtoUtils.java	66.66%	3 Missing and 2 partials ⚠️
.../request/search/SearchSourceBuilderProtoUtils.java	50.00%	2 Missing and 3 partials ⚠️
...est/document/bulk/BulkRequestParserProtoUtils.java	88.88%	0 Missing and 4 partials ⚠️
...o/request/common/FetchSourceContextProtoUtils.java	93.75%	0 Missing and 1 partial ⚠️
.../proto/request/search/SearchRequestProtoUtils.java	80.00%	0 Missing and 1 partial ⚠️
...onse/document/bulk/BulkItemResponseProtoUtils.java	75.00%	1 Missing ⚠️
...pc/proto/response/search/SearchHitsProtoUtils.java	87.50%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19007      +/-   ##
============================================
- Coverage     72.92%   72.91%   -0.01%     
- Complexity    69782    69787       +5     
============================================
  Files          5667     5667              
  Lines        320555   320592      +37     
  Branches      46348    46351       +3     
============================================
+ Hits         233762   233769       +7     
- Misses        67886    67973      +87     
+ Partials      18907    18850      -57     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

❌ Gradle check result for 89235c3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[finnegancarroll]

One general nit pick I have is the addition of "unspecified" or "not recognized" cases. It seems like most all of the fields with these cases are optional and we will need to check for existence anyways making these redundant.

[karenyrx]

One general nit pick I have is the addition of "unspecified" or "not recognized" cases. It seems like most all of the fields with these cases are optional and we will need to check for existence anyways making these redundant.

Do you mean for enums? Adding an UNSPECIFIED = 0; case for enums is to adhere to protobuf best practices: https://protobuf.dev/best-practices/dos-donts/#unspecified-enum

[github-actions]

✅ Gradle check result for 65ae49c: SUCCESS

[github-actions]

❌ Gradle check result for 00059c1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 78acf09: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 40de639: SUCCESS

[github-actions]

✅ Gradle check result for ae21a00: SUCCESS