Datastore Entity Builder: Support unindexed properties

[simon-temple]

Unindexed properties are essential when storing large blob properties as they can exceed the maximum allowed length for indexed values. Using a local gcd I get:

INFO: Exception executing rpc. com.google.apphosting.client.serviceapp.RpcException: Indexed blob value has more than permitted 500 bytes.

I can't see how to select the indexing policy using the EntityBuilder. I guess we need the ability to set the indexed attribute on each Builder.set( String name, Value value, boolean indexed )

[simon-temple]

This looks like a change to all classes that extend Value:
BlobValue, StringValue, BooleanValue etc.
to expose the Value.indexed attribute

[aozarov]

You specify the index attribute on the Value

https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/ValueBuilder.java#L30

[simon-temple]

Thanks for the help with this. I switched to using a Value.Builder instead of relying on the Entity.Builder to build my Value(s).

Not totally intuitive but this is a young API and examples are minimal... and I'm a newbie!

[volgin]

excludeFromIndexes(true) on a ListValue results in

"Exclude from indexes cannot be set on a list value."

Is there a way to exclude a list of values from indexes?

[eddavisson]

The Cloud Datastore API currently requires that each individual value in the list sets excludeFromIndexes(true).

[volgin]

I just tried it - the property is still indexed in the Datastore.

[eddavisson]

I haven't been able to reproduce this in the underlying API. Can you provide details on how you're checking that the property is still indexed?

One thing to note is that calling excludeFromIndexes(true) will only affect the values in the updated entity, not values in entities that were previously inserted.

[volgin]

I created 20 new entities using this code:

IncompleteKey playlistKey = IncompleteKey.builder(PROJECT, "PL").build();
FullEntity.Builder<IncompleteKey> playlistBuilder = FullEntity.builder(playlistKey);
playlistBuilder.set("title", StringValue.builder(playlistName).excludeFromIndexes(true).build());

Builder recordingsBuilder = ListValue.builder();
for (Long recording : recordings) {
    recordingsBuilder.addValue(LongValue.builder(recording).excludeFromIndexes(true).build());
}
playlistBuilder.set("recordings", recordingsBuilder.build());
datastore.add(playlistBuilder.build());

All entities have "title" property as unindexed and "recordings" property as indexed according to the Datastore viewer in the console. BTW, it's impossible to uncheck "indexed" checkbox for List properties in the console.

[eddavisson]

Thanks for sharing your code. It looks like this is a bug in the Datastore Viewer. It's using the value of excludeFromIndexes on the overall list (which can only ever be false) instead of looking at the individual values in the list. I've contacted that team about clarifying this in the UI.