Quickwit merge pipelines begins to fail during indexing

[ChillFish8]

While indexing the GitHub Archive dataset of about 1 month of data, I noticed Quickwit seems to start intermittently failing to run merge pipelines while indexing and has some unusual affects:

• A merge will occur and attempt merge splits A, B, C this operation succeeds.
• The system will mark the splits for deletion
• Sometime later a second merge operation gets scheduled to attempt to merge split C again, this operation fails.
• Over time this happens more often.

Logs:
logs.txt

Reproduction:
The issue seems to be somewhat intermittent, but occurs on a recent update, the easiest way to try reproduce would be via docker:

• Download the 2015 GH archive dataset for the month of January.
• Run the chillfish8/quickwit-main:latest docker image (it's just a docker build of what ever version of main is I'm testing.
• Feed the data into quickwit

You may need to increase the concurrent connections pushing data into quickwit in order to create this instability, I first saw this issue when running 4+ simultaneous connections.

Index File:

version: "0.4"
index_id: gharchive

doc_mapping:
  store_source: true
  field_mappings:
    - name: id
      type: text
      tokenizer: raw
    - name: type
      type: text
      tokenizer: raw
      fast: true
    - name: actor
      type: object
      field_mappings:
        - name: id
          type: u64
          fast: true
        - name: login
          type: text
          fast: true
          tokenizer: raw
        - name: gravatar_id
          type: text
          tokenizer: raw
        - name: url
          type: text
          tokenizer: raw
        - name: avatar_url
          type: text
          tokenizer: raw
    - name: repo
      type: object
      field_mappings:
        - name: id
          type: u64
          fast: true
        - name: name
          type: text
          fast: true
        - name: url
          type: text
          tokenizer: raw
    - name: payload
      type: json
      indexed: true
      tokenizer: default
      expand_dots: false
      record: position
    - name: created_at
      type: datetime
      precision: seconds
      fast: true
  timestamp_field: created_at

index_uri: s3://gharchive

Additional quickwit configuration:

ingest_api:
  max_queue_memory_usage: 8GB
  max_queue_disk_usage: 16GB

[ChillFish8]

The build is likely on the commit ceede196bae2e03d31cae94cb62c02cd6ef455bc but I cannot be 100%

[fulmicoton]

01GVP3YMDEF8XQ93CV7BZR9V10 is being marked for deletion and then being merge

It was deleted due to a call to delete-index
2023-03-16 22:20:33 2023-03-16T22:20:33.857Z INFO quickwit_serve::index_api::rest_handler: delete-index index_id=gharchive dry_run=false

Somehow we don't kill recreate the merge pipeline.
(Most probably, we also don't update the state of the MergePlanner on a clear).
Quickwit does its business as usual and only spots the discrepancy on publish, when it notices one of the splits being merged does not exist.

We need to make sure we restart the merge pipeline when there is delete index (and a recreate follows right away), and when there is a clear index.

[fulmicoton]

Once "polluted" the merge pipeline will keep creating errors and does not repair itself

[trinity-1686a]

is this impacted by #3212 (fixed, or otherwise made easier to fix)?

[fulmicoton]

Fixed by #3212