[BUG] Slow convert processor in opensearch 2.11.0

[shikeli]

Describe the bug

We are migrating to opensearch 2.11.0 from elasticsearch 6.8.1. We faced ingestion throughput 70% drop. After investigation, we found the convert processor is very slow compared to elasticsearch 6.8.1. We removed the processor as work around. I know most of team doesn't run processor out of indexing. We are doing this first then we do indexing.

Related component

Indexing:Performance

To Reproduce

Write one plugin to call the processor.

Expected behavior

We expect the same performance or better.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[andrross]

Thanks @shikeli. Can you share more detail? i.e. what kind of documents are you sending, what does your ingest pipeline configuration look like, any profiling information you captured, etc. If you have a basic reproduction you can share that would be very helpful too.

[shikeli]

We have one pipeline added that is using convert processor

processors:
  - convert: { field: 'foo', target_field: 'bar', type: 'long', ignore_failure: true }

In our ingestion plugin, we have one class PipelineExecutor, this will execute all pipeline for each document before we do bulk indexing. Below is my profiling data.