Incorrect dissection behavior after consecutive delimiters

[nielsbauman]

When we dissect a field, we're behaving incorrectly when the following conditions apply:

• There is a delimiter in the pattern that is repeated multiple consecutive times - i.e. %{}|%{}|.
• After that last delimiter, there are one or more characters that are not part of the extracted fields - i.e. foo=.
• The value directly before the field that breaks the delimiter repetition is empty - i.e. baz|| or ||

If we use the pattern %{}|%{}|foo=%{field} to dissect the input string ||foo=bar, the field field is extracted as foo=bar whereas the expected value would be bar.

Possible workarounds are either removing the foo= part from the dissect pattern and removing it in an additional processor, or leaving the dissect pattern as-is and removing foo= in an additional processor (that is conditional or ignores failures).

Reproduction example

PUT _ingest/pipeline/my-pipeline
{
  "processors":
  [
    {
      "dissect":
      {
        "field": "message",
        "pattern": "%{}|%{}|foo=%{field}"
      }
    }
  ]
}


POST _ingest/pipeline/my-pipeline/_simulate
{
    "docs": [
        {
            "_source": {
                "message": "||foo=bar"
            }
        }
    ]
}

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

My guess is that is caused somewhere around where we handle consecutive delimiters:

elasticsearch/libs/dissect/src/main/java/org/elasticsearch/dissect/DissectParser.java

Line 229 in 8dbb25d

// look for consecutive delimiters (e.g. a,,,,d,e)