You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are migrating to opensearch 2.11.0 from elasticsearch 6.8.1. We faced ingestion throughput 70% drop. After investigation, we found the convert processor is very slow compared to elasticsearch 6.8.1. We removed the processor as work around. I know most of team doesn't run processor out of indexing. We are doing this first then we do indexing.
Related component
Indexing:Performance
To Reproduce
Write one plugin to call the processor.
Expected behavior
We expect the same performance or better.
Additional Details
Plugins
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):
OS: [e.g. iOS]
Version [e.g. 22]
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered:
Thanks @shikeli. Can you share more detail? i.e. what kind of documents are you sending, what does your ingest pipeline configuration look like, any profiling information you captured, etc. If you have a basic reproduction you can share that would be very helpful too.
In our ingestion plugin, we have one class PipelineExecutor, this will execute all pipeline for each document before we do bulk indexing. Below is my profiling data.
Describe the bug
We are migrating to opensearch 2.11.0 from elasticsearch 6.8.1. We faced ingestion throughput 70% drop. After investigation, we found the convert processor is very slow compared to elasticsearch 6.8.1. We removed the processor as work around. I know most of team doesn't run processor out of indexing. We are doing this first then we do indexing.
Related component
Indexing:Performance
To Reproduce
Write one plugin to call the processor.
Expected behavior
We expect the same performance or better.
Additional Details
Plugins
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: