[BUG] COMPLEMENT does not work in Regexp queries

[kolchfa-aws]

Describe the bug

The COMPLEMENT value in RegexpFlag is not working. The code refers to deprecated complement in Lucene https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/query/RegexpFlag.java#L66. See related Lucene pull apache/lucene#13739.

Related component

No response

To Reproduce

When documenting regex syntax, I ran the following example to test the COMPLEMENT value of the flags parameter:

PUT /test/_doc/1
{
  "text": "abc"
}

PUT /test/_doc/2
{
  "text": "adc"
}

PUT /test/_doc/3
{
  "text": "acc"
}

GET /test/_search
{
  "query": {
    "regexp": {
      "text.keyword": {
        "value": "a~bc",
        "flags": "COMPLEMENT"
      }
    }
  }
}

The query returns no results.

Expected behavior

I'm expecting adc and acc to be returned.

Additional Details

OpenSearch 3.0

[sandeshkr419]

[Search Triage] Do we have any alternatives to achieve the same functionality?
I'm trying to understand if the deprecation was in favor of anything new functionality.

[kolchfa-aws]

As I understand, Lucene deprecated the COMPLEMENT operator in version 10 because its implementation required full determinization of regex patterns, which can be exponentially expensive. In Lucene 10, regex queries can execute using NFA, but COMPLEMENT is incompatible with NFA. Lucene is providing the DEPRECATED_COMPLEMENT in version 10, so the code was updated to use it, but, in my experience, this doesn't work as expected.

[msfroh]

I think the problem comes here:

OpenSearch/server/src/main/java/org/opensearch/index/query/RegexpQueryBuilder.java

Line 302 in aca2e9d

int sanitisedSyntaxFlag = syntaxFlagsValue & RegExp.ALL;

With the deprecation, DEPRECATED_COMPLEMENT is not included in RegExp.ALL. Since we & the flags, we drop DEPRECATED_COMPLEMENT. I think we need to change that line to:

 int sanitisedSyntaxFlag = syntaxFlagsValue & (RegExp.ALL | RegExp.DEPRECATED_COMPLEMENT);