Indexing nested JSON object impacts FT.SEARCH returning the records.

[radhakrishna67]

I have issue with indexing nested object. When i index nested object, FT.SEARCH is not returning the records.

Redis OM Spring version: 0.8.3
Spring Boot Version: 3.1.1
Java Version: openjdk version "17.0.2" 2022-01-18

here are Class details and indexing,

@Data
@ToString
@Document("User")
@RequiredArgsConstructor(staticName = "of")
@AllArgsConstructor(access = AccessLevel.PROTECTED)
public class User {

    @NotEmpty(message ="User email field value should not be null or empty")
    @Id
    private String userEmail;
    @Indexed private List<Role> roles;
    @Indexed private Set<String> projects;
    private List<Document> documents;
}

@Data
@RequiredArgsConstructor(staticName = "of")
public class Role {
    @Indexed private String projectId;
    @Indexed private List<String> roles;
    @Indexed private LocalDateTime joinTime;
	
}

"FT.SEARCH" "UserIdx" "@userEmail:{ra****\\@gmail\\.com}"
1) "0"
This is because of indexing  roles

"JSON.GET" "User:ra****@gmail.com"  works totally fine.

FT.SEARCH works totally fine when removing roles Indexing.

[joedickeyredis]

The issue has been identified with the schema generated in the FT.CREATE command when indexing nested arrays. In this case, when trying to index the list of Roles within the list of ProjectRoles. Currently the FT.CREATE command is generated as follows:

"FT.CREATE" "UserIdx" "ON" "JSON" "PREFIX" "1" "User" "LANGUAGE" "english" "SCHEMA" "$.userUuid" "AS" "userUuid" "TAG" "SEPARATOR" "|" "$.userFirstName" "AS" "userFirstName" "TAG" "SEPARATOR" "|" "$.userSecondName" "AS" "userSecondName" "TAG" "SEPARATOR" "|" "$.creatorEmail" "AS" "creatorEmail" "TAG" "SEPARATOR" "|" "$.creatorUuid" "AS" "creatorUuid" "TAG" "SEPARATOR" "|" "$.projectRoles[0:].projectId" "AS" "projectRoles_projectId" "TAG" "SEPARATOR" "|" "$.projectRoles[0:].roles" "AS" "projectRoles_roles" "TAG" "SEPARATOR" "|" "$.projectRoles[0:].joinTime" "AS" "projectRoles_joinTime" "NUMERIC" "$.projects[*]" "AS" "projects" "TAG" "SEPARATOR" "|" "$.userEmail" "AS" "userEmail" "TAG" "SEPARATOR" "|"

The issue has been pinpointed to "$.projectRoles[0:].roles" which to fix should be "$.projectRoles[0:].roles[*]" .

[djptek]

JsonPath permits distinct notations for some equivalent paths, so:

"$.projectRoles[0:].roles[*]"

can also be written as

"$.projectRoles[0:].roles[0:]"

Which delivers the same result and appears more consistent

[djptek]

see: JSONPath syntax

[bsbodden]

@joedickeyredis @djptek @radhakrishna67 see the example https://github.com/bsbodden/hpe-demo to drive the discussion, in that example I partially recreated the Radha example:

public class User {

  @NotEmpty(message ="User email field value should not be null or empty")
  @Id
  private String userEmail;
  @Indexed
  private List<Role> roles;
  @Indexed private Set<String> projects;
}

public class Role {
  @Indexed
  private String projectId;
  @Indexed private List<String> roles;
  @Indexed private LocalDateTime joinTime;
}

This is using the latest 0.8.7-SNAPSHOT and it produces the following index:

"FT.CREATE" "com.example.hpedemo.UserIdx" "ON" "JSON" "PREFIX" "1" "User" "LANGUAGE" "english" 
  "SCHEMA" 
    "$.roles[0:].projectId" "AS" "roles_projectId" "TAG" "SEPARATOR" "|" 
    "$.roles[0:].roles" "AS" "roles_roles" "TAG" "SEPARATOR" "|" 
    "$.roles[0:].joinTime" "AS" "roles_joinTime" "NUMERIC" 
    "$.projects[*]" "AS" "projects" "TAG" "SEPARATOR" "|" 
    "$.userEmail" "AS" "userEmail" "TAG" "SEPARATOR" "|"

The demo creates two user instances:

127.0.0.1:6379> KEYS *
1) "User:user2@test.com"
2) "User:user1@test.com"
127.0.0.1:6379> JSON.GET "User:user2@test.com"
"{\"userEmail\":\"user2@test.com\",\"roles\":[{\"roles\":[\"role3\",\"role4\"]}],\"projects\":[\"project4\",\"project3\"]}"
127.0.0.1:6379> JSON.GET "User:user1@test.com"
"{\"userEmail\":\"user1@test.com\",\"roles\":[{\"roles\":[\"role1\",\"role2\"]}],\"projects\":[\"project2\",\"project1\"]}"
127.0.0.1:6379> 

I added a simple search by email using EntityStreams and it seems to work:

try (SearchStream<User> searchStream = es.of(User.class)) {
  List<User> users = searchStream.filter(User$.USER_EMAIL.eq("user1@test.com")).collect(Collectors.toList());
  users.forEach(u -> logger.info("Found user " + u.getUserEmail()));
}

It results in:

"FT.SEARCH" "com.example.hpedemo.UserIdx" "( @userEmail:{user1\\@test\\.com})" "LIMIT" "0" "10000" "DIALECT" "1"

Can all of you fork that repo and do whatever you need to make it fail? The initial report doesn't match what I'm seeing so let's get to the bottom of it and get this figured out. Thanks!