You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Consider the API / dependency implications for leveraging virtual threads (final release in Java 21) as a possible implementation for step and sink parallelism.
An important note: keep in mind that pipelines are currently expected to be ran concurrently (eg. piggy-backing on a Kafka consumer, or in the context of an HTTP server), so having them rely on a physical-thread backed internal parallelism mechanism is probably not the best possible implementation.
It is a possibility that the kind of parallelism we want within our pipelines most of the time is what virtual threads offer.
Possible implications to consider:
we don't want to make the library require Java 21, so this may need to be an add-on
yet, if we want to make this a default, or easily attainable setting, that might be a hurdle to overcome
afaik virtual threads should be almost identical to physical threads from an API PoV (meaning we should be able to use facilities like ServiceExecutor and parallel Stream APIs the exact same way we currently do, or plan to do in other issues)
The text was updated successfully, but these errors were encountered:
Consider the API / dependency implications for leveraging virtual threads (final release in Java 21) as a possible implementation for step and sink parallelism.
Tied to #17 and #29.
Previous comments:
An important note: keep in mind that pipelines are currently expected to be ran concurrently (eg. piggy-backing on a Kafka consumer, or in the context of an HTTP server), so having them rely on a physical-thread backed internal parallelism mechanism is probably not the best possible implementation.
It is a possibility that the kind of parallelism we want within our pipelines most of the time is what virtual threads offer.
Possible implications to consider:
ServiceExecutor
and parallelStream
APIs the exact same way we currently do, or plan to do in other issues)The text was updated successfully, but these errors were encountered: