Invalid NEST response built from a null ApiCall

[gwenzek]

NEST/Elasticsearch.Net version: 2.3.3

Elasticsearch version: 2.3.3

Description of the problem including expected versus actual behavior:

When doing a multi-term-vector request the TermVectorsResponse in the field Documents field have an ApiCall set to null.
This triggers the following debug message:
Invalid NEST response built from a null ApiCall which is highly exceptional, please open a bug if you see this. (Relevant source code)
So here I am :-)
This happens even if the response actually contains the TermVector I requested and seems totally fine. Also the response is marked as valid and contains the ApiCall object.
It seems like the serializer fails to transmit some info from the request to the documents.

Steps to reproduce:

Code used to request ES:

var ids = new []{ "0", "1" };
string tfField = "text";
var response = client.MultiTermVectors(new MultiTermVectorsRequest(indexName, MyDocument.TYPE)
{
    Positions = true,
    Documents = ids.Select(id => new MultiTermVectorOperation<MyDocument>(id)),
    Fields = (Field)tfField
});
var tokenized = response.Documents.Select(doc => doc
    .TermVectors[tfField].Terms
    .SelectMany(term_stats => term_stats.Value.Tokens
        .Select(stat => Tuple.Create(term_stats.Key, stat)))
    .OrderBy(token => token.Item2.Position)
);

Json send to ES:

{
  "docs": [
    {
      "_index": "my_index",
      "_type": "my_document",
      "_id": "0"
    },
    {
      "_index": "my_index",
      "_type": "my_document",
      "_id": "1"
    }
  ]
}

Json from ES:

{
  "docs" : [ {
    "_index" : "my_index",
    "_type" : "my_document",
    "_id" : "0",
    "_version" : 2,
    "found" : true,
    "took" : 1,
    "term_vectors" : {
      "text" : {
        "field_statistics" : {
          "sum_doc_freq" : 249530,
          "doc_count" : 1231,
          "sum_ttf" : -1
        },
        "terms" : {
          "Die" : {
            "term_freq" : 1,
            "tokens" : [ {
              "position" : 0,
              "start_offset" : 0,
              "end_offset" : 3
            } ]
          },
          "Donaudampfschiff" : {
            "term_freq" : 1,
            "tokens" : [ {
              "position" : 6,
              "start_offset" : 51,
              "end_offset" : 67
            } ]
          },
          "Jahresfeier" : {
            "term_freq" : 1,
            "tokens" : [ {
              "position" : 1,
              "start_offset" : 4,
              "end_offset" : 15
            } ]
          },
          "Rechtsanwaltskanzleien" : {
            "term_freq" : 1,
            "tokens" : [ {
              "position" : 3,
              "start_offset" : 20,
              "end_offset" : 42
            } ]
          }
         // more terms ...
        }
      }
    }
  }, {
    "_index" : "my_index",
    "_type" : "my_document",
    "_id" : "1",
    "_version" : 2,
    "found" : true,
    "took" : 1,
    "term_vectors" : {
      "exact" : {
        "field_statistics" : {
          "sum_doc_freq" : 249530,
          "doc_count" : 1231,
          "sum_ttf" : -1
        },
        "terms" : {
          "Café" : {
            "term_freq" : 1,
            "tokens" : [ {
              "position" : 6,
              "start_offset" : 27,
              "end_offset" : 31
            } ]
          },
          "Ein" : {
            "term_freq" : 1,
            "tokens" : [ {
              "position" : 0,
              "start_offset" : 0,
              "end_offset" : 3
            } ]
          }
         // more terms ...
        }
      }
    }
  }]
}

[gmarz]

Hi @gwenzek can you paste your ConnectionSettings here as well?

[gmarz]

@gwenzek any update on this?

[gwenzek]

@gmarz Yes, here you go:

var settings = new ConnectionSettings(new Uri(c.LocalUrl))
    .DefaultIndex(Index.Name)
    .DisableDirectStreaming()
    .OnRequestCompleted(details =>
    {
        Trace.WriteLine("### ES REQUEST ###");
        if (details.RequestBodyInBytes != null) Trace.WriteLine(Encoding.UTF8.GetString(details.RequestBodyInBytes));
        Trace.WriteLine("### ES RESPONSE ###");
        if (details.ResponseBodyInBytes != null) Trace.WriteLine(Encoding.UTF8.GetString(details.ResponseBodyInBytes));
    })
    .PrettyJson()
    ;

[russcam]

@gwenzek Unfortunately this is due to MultiTermVectorsResponse containing an IEnumerable<TermVectorsResponse>, where TermVectorsResponse is a response type that has been repurposed.

There is a note to not use TermVectorsResponse in the next version of the client. In 2.x however, I think we should handle the confusion by writing a custom serializer that sets IsValid, etc. on each TermVectorsResponse based on the values for MultiTermVectorsResponse.

[russcam]

I've added #2429 to set CallDetails on each TermVectorsResponse and #2428 to use a specific type in 5.x that does not inherit from IResponse