Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Concurrency: virtual threads for parallelism #30

Open
eledhwen opened this issue Mar 6, 2024 · 0 comments
Open

Concurrency: virtual threads for parallelism #30

eledhwen opened this issue Mar 6, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@eledhwen
Copy link
Collaborator

eledhwen commented Mar 6, 2024

Consider the API / dependency implications for leveraging virtual threads (final release in Java 21) as a possible implementation for step and sink parallelism.

Tied to #17 and #29.


Previous comments:


An important note: keep in mind that pipelines are currently expected to be ran concurrently (eg. piggy-backing on a Kafka consumer, or in the context of an HTTP server), so having them rely on a physical-thread backed internal parallelism mechanism is probably not the best possible implementation.

It is a possibility that the kind of parallelism we want within our pipelines most of the time is what virtual threads offer.


Possible implications to consider:

  • we don't want to make the library require Java 21, so this may need to be an add-on
  • yet, if we want to make this a default, or easily attainable setting, that might be a hurdle to overcome
  • afaik virtual threads should be almost identical to physical threads from an API PoV (meaning we should be able to use facilities like ServiceExecutor and parallel Stream APIs the exact same way we currently do, or plan to do in other issues)
@eledhwen eledhwen added new feature enhancement New feature or request and removed new feature labels Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant