Skip to content

Commit 0f9a483

Browse files
committed
Replace protoc with buf and add build script for generating code from the protos (#174)
## Problem The code generation using proto file was done using protoc and the process was manual. ## Solution Added a script to generate java classes using [buf](https://buf.build/docs/generate/overview/). ## Type of Change - [ ] Bug fix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] This change requires a documentation update - [ ] Infrastructure change (CI configs, etc) - [ ] Non-code change (docs, etc) - [ ] None of the above: (explain here) ## Test Plan Integration tests should run successfully.
1 parent 154dfd3 commit 0f9a483

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

58 files changed

+2919
-3143
lines changed

build.gradle

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ dependencies {
5252
implementation 'com.google.code.gson:gson:2.9.1'
5353
implementation 'io.gsonfire:gson-fire:1.8.5'
5454
implementation 'org.openapitools:jackson-databind-nullable:0.2.6'
55-
implementation 'com.google.protobuf:protobuf-java:3.25.2'
55+
implementation 'com.google.protobuf:protobuf-java:4.29.3'
5656
compileOnly "org.apache.tomcat:annotations-api:6.0.53" // necessary for Java 9+
5757

5858
testImplementation("com.squareup.okhttp3:mockwebserver:4.12.0")

codegen/apis

Submodule apis updated from 24c6261 to 63e97dc

codegen/buf.gen.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
version: v1
2+
plugins:
3+
- name: java
4+
out: gen/java
5+
- name: grpc-java
6+
out: gen/java

codegen/buf.lock

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# Generated by buf. DO NOT EDIT.
2+
version: v2
3+
deps:
4+
- name: buf.build/googleapis/googleapis
5+
commit: e93e34f48be043dab55be31b4b47f458
6+
digest: b5:cebe5dfac5f7d67c55296f37ad9d368dba8d9862777e69d5d99eb1d72dc95fa68cd6323b483ca42cf70e66060002c1bc36e1f5f754b217a5c771c108eb243dbf

codegen/buf.yaml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# For details on buf.yaml configuration, visit https://buf.build/docs/configuration/v2/buf-yaml
2+
version: v2
3+
lint:
4+
use:
5+
- STANDARD
6+
breaking:
7+
use:
8+
- FILE
9+
deps:
10+
- buf.build/googleapis/googleapis
11+
modules:
12+
- path: apis/_build/2025-01

codegen/build-grpc.sh

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/bin/bash
2+
3+
set -eux -o pipefail
4+
version=$1 # e.g. 2024-07
5+
6+
update_apis_repo() {
7+
echo "Updating apis repo"
8+
pushd codegen/apis
9+
git fetch
10+
git checkout main
11+
git pull
12+
just build
13+
popd
14+
}
15+
16+
update_buf_config() {
17+
pushd codegen
18+
# Update buf config to find correct proto version
19+
sed -i '' "s/[0-9][0-9][0-9][0-9]-[0-1][0-9]/${version}/g" buf.yaml
20+
21+
# Clean before building
22+
rm -rf gen
23+
24+
# Ensure path valid by running the buf build command
25+
buf build
26+
popd
27+
}
28+
29+
buf_generate() {
30+
pushd codegen
31+
# Generate the java code
32+
buf generate
33+
popd
34+
}
35+
36+
update_apis_repo
37+
update_buf_config
38+
buf_generate
39+
40+
dest="src/main/java/io/pinecone/proto/"
41+
42+
# Remove existing files in dest
43+
rm -rf "${dest}*.java"
44+
45+
# Copy the new generated files to dest directory
46+
cp codegen/gen/java/io/pinecone/proto/*.java ${dest}
47+
48+
# Cleanup the intermediate files that were generated
49+
rm -rf codegen/gen

src/integration/java/io/pinecone/helpers/TestUtilities.java

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,10 @@
11
package io.pinecone.helpers;
22

3-
import io.pinecone.clients.Index;
43
import io.pinecone.clients.Pinecone;
5-
import io.pinecone.proto.DescribeIndexStatsResponse;
6-
import io.pinecone.proto.NamespaceSummary;
74
import org.openapitools.db_control.client.model.*;
85
import org.slf4j.Logger;
96
import org.slf4j.LoggerFactory;
107

11-
import java.util.*;
12-
138
import static org.junit.jupiter.api.Assertions.assertEquals;
149
import static org.junit.jupiter.api.Assertions.fail;
1510

src/integration/java/io/pinecone/integration/controlPlane/pod/CollectionTest.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,9 @@
55
import io.pinecone.exceptions.PineconeException;
66
import io.pinecone.exceptions.PineconeValidationException;
77
import io.pinecone.helpers.RandomStringBuilder;
8-
import io.pinecone.proto.FetchResponse;
98
import io.pinecone.helpers.TestResourcesManager;
109

10+
import io.pinecone.proto.FetchResponse;
1111
import org.junit.jupiter.api.AfterAll;
1212
import org.junit.jupiter.api.BeforeAll;
1313
import org.junit.jupiter.api.Disabled;

src/main/java/io/pinecone/proto/VectorServiceOuterClass.java renamed to src/main/java/io/pinecone/proto/DbData202501.java

Lines changed: 173 additions & 142 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)