Skip to content

Commit f15f541

Browse files
rohanshah18rohanshah18
authored
Add Support for Dedicated Read Capacity (DRN) and Metadata Schema Configuration (#202)
## Problem Add Support for Dedicated Read Capacity (DRN) and Metadata Schema Configuration ## Solution This PR adds support for Dedicated Read Capacity (DRN) and Metadata Schema Configuration for serverless indexes. These features allow users to: 1. **Configure dedicated read nodes** for better performance and cost predictability 2. **Limit metadata indexing** to specific fields for improved performance 3. **Configure read capacity** on existing serverless indexes #### 1. Create Serverless Index with Read Capacity and Schema Added overloaded `createServerlessIndex` method that accepts: - `ReadCapacity` parameter for configuring OnDemand or Dedicated read capacity - `BackupModelSchema` parameter for configuring metadata schema ```java // Create index with Dedicated read capacity ScalingConfigManual manual = new ScalingConfigManual().shards(2).replicas(2); ReadCapacityDedicatedConfig dedicated = new ReadCapacityDedicatedConfig() .nodeType("t1") .scaling("Manual") .manual(manual); ReadCapacity readCapacity = new ReadCapacity( new ReadCapacityDedicatedSpec().mode("Dedicated").dedicated(dedicated)); IndexModel indexModel = pinecone.createServerlessIndex( indexName, "cosine", 1536, "aws", "us-west-2", "enabled", tags, readCapacity, null); // Create index with metadata schema Map<String, BackupModelSchemaFieldsValue> fields = new HashMap<>(); fields.put("genre", new BackupModelSchemaFieldsValue().filterable(true)); fields.put("year", new BackupModelSchemaFieldsValue().filterable(true)); BackupModelSchema schema = new BackupModelSchema().fields(fields); IndexModel indexModel = pinecone.createServerlessIndex( indexName, "cosine", 1536, "aws", "us-west-2", "enabled", tags, null, schema); ``` #### 2. Create Index for Model with Read Capacity and Schema Added overloaded `createIndexForModel` method that accepts: - `ReadCapacity` parameter - `BackupModelSchema` parameter #### 3. Configure Read Capacity on Existing Index Enhanced `configureServerlessIndex` method to accept flattened parameters for easier use: ```java // Switch to Dedicated read capacity IndexModel indexModel = pinecone.configureServerlessIndex( indexName, "enabled", tags, null, "Dedicated", "t1", 2, 2); // Switch to OnDemand read capacity IndexModel indexModel = pinecone.configureServerlessIndex( indexName, "enabled", tags, null, "OnDemand", null, null, null); ``` **Note:** Read capacity settings can only be updated once per hour per index. ### Documentation - Updated `README.md` with examples for: - Creating serverless indexes with dedicated read capacity - Creating serverless indexes with OnDemand read capacity - Creating serverless indexes with metadata schema - Configuring read capacity on existing indexes ## Type of Change - [ ] Bug fix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] This change requires a documentation update - [ ] Infrastructure change (CI configs, etc) - [ ] Non-code change (docs, etc) - [ ] None of the above: (explain here) ## Test Plan - Added comprehensive integration tests in `ReadCapacityAndSchemaTest.java`: - Create serverless index with OnDemand read capacity - Create serverless index with Dedicated read capacity - Create serverless index with metadata schema - Create serverless index with both read capacity and schema - Create index for model with read capacity and schema - Configure read capacity on existing index - Added helper method `waitUntilReadCapacityIsReady()` in `TestUtilities.java` to wait for read capacity status to be "Ready" before configuring (required by API). - Note: Tests for switching read capacity modes and scaling are omitted due to API rate limits (once per hour per index).
1 parent 5a5f743 commit f15f541

File tree

5 files changed

+719
-0
lines changed

5 files changed

+719
-0
lines changed

‎README.md‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,105 @@ tags.put("env", "test");
186186
IndexModel indexModel = pinecone.createServerlessIndex(indexName, similarityMetric, dimension, cloud, region, "enabled", tags);
187187
```
188188

189+
### Create a serverless index with dedicated read capacity
190+
191+
The following example creates a serverless index with dedicated read capacity nodes for better performance and cost predictability. For more information, see [Dedicated Read Nodes](https://docs.pinecone.io/guides/index-data/dedicated-read-nodes).
192+
193+
```java
194+
import io.pinecone.clients.Pinecone;
195+
import org.openapitools.db_control.client.model.IndexModel;
196+
import org.openapitools.db_control.client.model.ReadCapacity;
197+
import org.openapitools.db_control.client.model.ReadCapacityDedicatedSpec;
198+
import org.openapitools.db_control.client.model.ReadCapacityDedicatedConfig;
199+
import org.openapitools.db_control.client.model.ScalingConfigManual;
200+
import java.util.HashMap;
201+
...
202+
203+
Pinecone pinecone = new Pinecone.Builder("PINECONE_API_KEY").build();
204+
205+
String indexName = "example-index";
206+
String similarityMetric = "cosine";
207+
int dimension = 1538;
208+
String cloud = "aws";
209+
String region = "us-west-2";
210+
HashMap<String, String> tags = new HashMap<>();
211+
tags.put("env", "test");
212+
213+
// Configure dedicated read capacity with manual scaling
214+
ScalingConfigManual manual = new ScalingConfigManual().shards(2).replicas(2);
215+
ReadCapacityDedicatedConfig dedicated = new ReadCapacityDedicatedConfig()
216+
.nodeType("t1")
217+
.scaling("Manual")
218+
.manual(manual);
219+
ReadCapacity readCapacity = new ReadCapacity(
220+
new ReadCapacityDedicatedSpec().mode("Dedicated").dedicated(dedicated));
221+
222+
IndexModel indexModel = pinecone.createServerlessIndex(indexName, similarityMetric, dimension,
223+
cloud, region, "enabled", tags, readCapacity, null);
224+
```
225+
226+
### Create a serverless index with OnDemand read capacity
227+
228+
The following example explicitly creates a serverless index with OnDemand read capacity (the default mode). OnDemand provides pay-per-use pricing with automatic scaling.
229+
230+
```java
231+
import io.pinecone.clients.Pinecone;
232+
import org.openapitools.db_control.client.model.IndexModel;
233+
import org.openapitools.db_control.client.model.ReadCapacity;
234+
import org.openapitools.db_control.client.model.ReadCapacityOnDemandSpec;
235+
import java.util.HashMap;
236+
...
237+
238+
Pinecone pinecone = new Pinecone.Builder("PINECONE_API_KEY").build();
239+
240+
String indexName = "example-index";
241+
String similarityMetric = "cosine";
242+
int dimension = 1538;
243+
String cloud = "aws";
244+
String region = "us-west-2";
245+
HashMap<String, String> tags = new HashMap<>();
246+
tags.put("env", "test");
247+
248+
// Configure OnDemand read capacity (optional - this is the default)
249+
ReadCapacity readCapacity = new ReadCapacity(new ReadCapacityOnDemandSpec().mode("OnDemand"));
250+
251+
IndexModel indexModel = pinecone.createServerlessIndex(indexName, similarityMetric, dimension,
252+
cloud, region, "enabled", tags, readCapacity, null);
253+
```
254+
255+
### Create a serverless index with metadata schema
256+
257+
The following example creates a serverless index with metadata schema configuration to limit metadata indexing to specific fields for improved performance.
258+
259+
```java
260+
import io.pinecone.clients.Pinecone;
261+
import org.openapitools.db_control.client.model.IndexModel;
262+
import org.openapitools.db_control.client.model.BackupModelSchema;
263+
import org.openapitools.db_control.client.model.BackupModelSchemaFieldsValue;
264+
import java.util.HashMap;
265+
...
266+
267+
Pinecone pinecone = new Pinecone.Builder("PINECONE_API_KEY").build();
268+
269+
String indexName = "example-index";
270+
String similarityMetric = "cosine";
271+
int dimension = 1538;
272+
String cloud = "aws";
273+
String region = "us-west-2";
274+
HashMap<String, String> tags = new HashMap<>();
275+
tags.put("env", "test");
276+
277+
// Configure metadata schema to only index specific fields
278+
HashMap<String, BackupModelSchemaFieldsValue> fields = new HashMap<>();
279+
fields.put("genre", new BackupModelSchemaFieldsValue().filterable(true));
280+
fields.put("year", new BackupModelSchemaFieldsValue().filterable(true));
281+
fields.put("description", new BackupModelSchemaFieldsValue().filterable(true));
282+
BackupModelSchema schema = new BackupModelSchema().fields(fields);
283+
284+
IndexModel indexModel = pinecone.createServerlessIndex(indexName, similarityMetric, dimension,
285+
cloud, region, "enabled", tags, null, schema);
286+
```
287+
189288
### Create a sparse serverless index
190289

191290
The following is an example of creating a sparse serverless index in the `us-east-1` region of AWS. For more information on
@@ -341,6 +440,38 @@ tags.put("env", "test");
341440
pinecone.configureServerlessIndex(indexName, "enabled", tags);
342441
```
343442

443+
### Configure read capacity on an existing serverless index
444+
445+
The following example shows how to configure or change the read capacity mode of an existing serverless index. You can switch between OnDemand and Dedicated modes, or scale dedicated read nodes.
446+
447+
**Note:** Read capacity settings can only be updated once per hour per index.
448+
449+
```java
450+
import io.pinecone.clients.Pinecone;
451+
import org.openapitools.db_control.client.model.IndexModel;
452+
import java.util.HashMap;
453+
...
454+
455+
Pinecone pinecone = new Pinecone.Builder("PINECONE_API_KEY").build();
456+
457+
String indexName = "example-index";
458+
HashMap<String, String> tags = new HashMap<>();
459+
tags.put("env", "test");
460+
461+
// Switch to Dedicated read capacity with manual scaling
462+
// Parameters: indexName, deletionProtection, tags, embed, readCapacityMode, nodeType, shards, replicas
463+
IndexModel indexModel = pinecone.configureServerlessIndex(
464+
indexName, "enabled", tags, null, "Dedicated", "t1", 3, 2);
465+
466+
// Switch to OnDemand read capacity
467+
IndexModel onDemandIndex = pinecone.configureServerlessIndex(
468+
indexName, "enabled", tags, null, "OnDemand", null, null, null);
469+
470+
// Verify the configuration was applied
471+
IndexModel desc = pinecone.describeIndex(indexName);
472+
// Check desc.getSpec().getServerless().getReadCapacity()...
473+
```
474+
344475
## Describe index statistics
345476

346477
The following example returns statistics about an index.

‎src/integration/java/io/pinecone/helpers/TestUtilities.java‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,81 @@ public static IndexModel waitUntilIndexIsReady(Pinecone pineconeClient, String i
3838
return waitUntilIndexIsReady(pineconeClient, indexName, 200000);
3939
}
4040

41+
/**
42+
* Waits until the read capacity status is Ready for a serverless index.
43+
* This is needed before configuring read capacity, as the API requires read capacity to be Ready before updates.
44+
*
45+
* @param pineconeClient The Pinecone client instance
46+
* @param indexName The name of the index
47+
* @param totalMsToWait Maximum time to wait in milliseconds
48+
* @return The IndexModel with read capacity status Ready
49+
* @throws InterruptedException if the thread is interrupted
50+
*/
51+
public static IndexModel waitUntilReadCapacityIsReady(Pinecone pineconeClient, String indexName, Integer totalMsToWait) throws InterruptedException {
52+
IndexModel index = pineconeClient.describeIndex(indexName);
53+
int waitedTimeMs = 0;
54+
int intervalMs = 2000;
55+
56+
while (true) {
57+
// Check if index has serverless spec with read capacity
58+
if (index.getSpec() != null && index.getSpec().getIndexModelServerless() != null) {
59+
ServerlessSpecResponse serverless = index.getSpec().getIndexModelServerless().getServerless();
60+
if (serverless != null && serverless.getReadCapacity() != null) {
61+
ReadCapacityResponse readCapacityResponse = serverless.getReadCapacity();
62+
ReadCapacityStatus status = null;
63+
64+
// Get status from the appropriate response type
65+
try {
66+
ReadCapacityDedicatedSpecResponse dedicatedResponse = readCapacityResponse.getReadCapacityDedicatedSpecResponse();
67+
status = dedicatedResponse.getStatus();
68+
} catch (ClassCastException e) {
69+
try {
70+
ReadCapacityOnDemandSpecResponse onDemandResponse = readCapacityResponse.getReadCapacityOnDemandSpecResponse();
71+
status = onDemandResponse.getStatus();
72+
} catch (ClassCastException e2) {
73+
logger.warn("Unknown read capacity response type for index " + indexName);
74+
}
75+
}
76+
77+
if (status != null && "Ready".equals(status.getState())) {
78+
logger.info("Read capacity for index " + indexName + " is ready after " + waitedTimeMs + "ms");
79+
break;
80+
}
81+
} else {
82+
// If no read capacity is configured (OnDemand by default), consider it ready
83+
logger.info("Index " + indexName + " has no read capacity configured (defaults to OnDemand), considering ready");
84+
break;
85+
}
86+
} else {
87+
// Not a serverless index or spec not available yet
88+
logger.info("Index " + indexName + " spec not available yet, waiting...");
89+
}
90+
91+
if (waitedTimeMs >= totalMsToWait) {
92+
logger.info("WARNING: Read capacity for index " + indexName + " not ready after " + waitedTimeMs + "ms");
93+
break;
94+
}
95+
96+
Thread.sleep(intervalMs);
97+
waitedTimeMs += intervalMs;
98+
logger.info("Waited " + waitedTimeMs + "ms for read capacity of " + indexName + " to get ready");
99+
index = pineconeClient.describeIndex(indexName);
100+
}
101+
return index;
102+
}
103+
104+
/**
105+
* Waits until the read capacity status is Ready for a serverless index (default timeout: 200 seconds).
106+
*
107+
* @param pineconeClient The Pinecone client instance
108+
* @param indexName The name of the index
109+
* @return The IndexModel with read capacity status Ready
110+
* @throws InterruptedException if the thread is interrupted
111+
*/
112+
public static IndexModel waitUntilReadCapacityIsReady(Pinecone pineconeClient, String indexName) throws InterruptedException {
113+
return waitUntilReadCapacityIsReady(pineconeClient, indexName, 200000);
114+
}
115+
41116
public static CollectionModel createCollection(Pinecone pineconeClient, String collectionName, String indexName, boolean waitUntilReady) throws InterruptedException {
42117
CollectionModel collection = pineconeClient.createCollection(collectionName, indexName);
43118

‎src/integration/java/io/pinecone/integration/controlPlane/pod/ConfigureIndexTest.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
import io.pinecone.exceptions.PineconeForbiddenException;
66
import io.pinecone.exceptions.PineconeNotFoundException;
77
import io.pinecone.helpers.TestResourcesManager;
8+
import okhttp3.OkHttpClient;
89
import org.junit.jupiter.api.AfterEach;
910
import org.junit.jupiter.api.BeforeAll;
1011
import org.junit.jupiter.api.Test;
@@ -13,6 +14,8 @@
1314
import org.slf4j.Logger;
1415
import org.slf4j.LoggerFactory;
1516

17+
import java.util.concurrent.TimeUnit;
18+
1619
import static io.pinecone.helpers.AssertRetry.assertWithRetry;
1720
import static org.junit.jupiter.api.Assertions.*;
1821

@@ -22,6 +25,11 @@ public class ConfigureIndexTest {
2225
private static final Pinecone controlPlaneClient = new Pinecone
2326
.Builder(System.getenv("PINECONE_API_KEY"))
2427
.withSourceTag("pinecone_test")
28+
.withOkHttpClient(new OkHttpClient.Builder()
29+
.connectTimeout(10, TimeUnit.SECONDS)
30+
.readTimeout(60, TimeUnit.SECONDS)
31+
.writeTimeout(30, TimeUnit.SECONDS)
32+
.build())
2533
.build();
2634
private static String indexName;
2735

0 commit comments

Comments
 (0)