You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It should be possible to run Sinks in a parallel fashion while still requiring them to be completed before exiting the pipeline.
Current implementation makes a distinction between serial synchronous and parallel asynchronous ; instead we most often want parallel synchronous and parallel asynchronous.
This can be straightforward to implement using the Future API.
It should be noted however that this requires additional attention to the ServiceExecutor: making most/all Sink parallel requires appropriate sizing, and possibly thread pooling between pipelines. We can already make such adjustments via the setSinkExecutor and setSinkExecutorProvider methods on the pipeline builders, but it may need to be better integrated.
The text was updated successfully, but these errors were encountered:
It should be possible to run
Sink
s in a parallel fashion while still requiring them to be completed before exiting the pipeline.Current implementation makes a distinction between serial synchronous and parallel asynchronous ; instead we most often want parallel synchronous and parallel asynchronous.
This can be straightforward to implement using the
Future
API.It should be noted however that this requires additional attention to the
ServiceExecutor
: making most/allSink
parallel requires appropriate sizing, and possibly thread pooling between pipelines. We can already make such adjustments via thesetSinkExecutor
andsetSinkExecutorProvider
methods on the pipeline builders, but it may need to be better integrated.The text was updated successfully, but these errors were encountered: