Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate direct path #31902

Open
wants to merge 26 commits into
base: master
Choose a base branch
from
Open

Integrate direct path #31902

wants to merge 26 commits into from
+2,233 −1,217

Commits on Oct 10, 2024

  1. add getdata clients and shareable logic for GetData w/ throttling and… 

    … metric tracking.
    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    15675d0 View commit details
    Browse the repository at this point in the history

  2. add HeartbeatSender interface

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    8c1ca87 View commit details
    Browse the repository at this point in the history

  3. address PR comments

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    9f42751 View commit details
    Browse the repository at this point in the history

  4. move side input state API into Work instance w/ keyed state fetching.… 

    … Future proofs this logic for direct path.
    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    e623fc7 View commit details
    Browse the repository at this point in the history

  5. don't let FanOutWorkRefreshClient exceptions crash the user worker

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    0735a0c View commit details
    Browse the repository at this point in the history

  6. address PR comments

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    998c07a View commit details
    Browse the repository at this point in the history

  7. address PR comments

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    659d5aa View commit details
    Browse the repository at this point in the history

  8. add WorkProvider interface and have StreamingEngine and Appliance impls

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    554b73e View commit details
    Browse the repository at this point in the history

  9. change StreamingEngineClient to FanOutStreamingEngineWorkProvider and… 

    … have it implement WorkProvider interface. Move class to windmill/work/provider directory, update visibility for dependent classes and move tests, add GetWorkBudgetOwnerInterface
    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    a7a9efa View commit details
    Browse the repository at this point in the history

  10. add WorkProvider interface and have StreamingEngine and Appliance impls

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    52c1fec View commit details
    Browse the repository at this point in the history

  11. change StreamingEngineClient to FanOutStreamingEngineWorkProvider and… 

    … have it implement WorkProvider interface. Move class to windmill/work/provider directory, update visibility for dependent classes and move tests, add GetWorkBudgetOwnerInterface
    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    83655fa View commit details
    Browse the repository at this point in the history

  12. add WorkProvider interface and have StreamingEngine and Appliance impls

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    e27fde2 View commit details
    Browse the repository at this point in the history

  13. change StreamingEngineClient to FanOutStreamingEngineWorkProvider and… 

    … have it implement WorkProvider interface. Move class to windmill/work/provider directory, update visibility for dependent classes and move tests, add GetWorkBudgetOwnerInterface
    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    9bf73c4 View commit details
    Browse the repository at this point in the history

  14. add stream management and correct stream shutdown mechanisms for dire… 

    …ct path
    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    9beeec9 View commit details
    Browse the repository at this point in the history

  15. integrate direct path components

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    0d60a18 View commit details
    Browse the repository at this point in the history

  16. address PR comments

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    361ec3c View commit details
    Browse the repository at this point in the history

  17. simplify budgeting logic for GrpcDirectGetWorkStream

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    fd6549e View commit details
    Browse the repository at this point in the history

  18. fix stuckness on range movements

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    36015f8 View commit details
    Browse the repository at this point in the history

  19. address PR comments, fix broken tests

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    14025db View commit details
    Browse the repository at this point in the history

  20. address PR comments

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    2144cd9 View commit details
    Browse the repository at this point in the history

  21. remove thread polling for new metadata; propogate metadata version to… 

    … WindmillEndpoints and don't process any version that is older than the current version in FanOutStreamingEngineWorkerHarness
    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    92414f7 View commit details
    Browse the repository at this point in the history

  22. deflake FanOutStreamingEngineWorkerHarnessTest.testOnNewWorkerMetadat… 

    …a_redistributesBudget()
    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    b09ba8f View commit details
    Browse the repository at this point in the history

  23. address PR comments

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    3db3b7f View commit details
    Browse the repository at this point in the history

  24. address PR comments

    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    06836da View commit details
    Browse the repository at this point in the history

  25. remove MAX_WAIT_SECONDS and just use the deadline passed in, its alre… 

    …ady 600 seconds anyway. Move DEFAULT_STREAM_RPC_DEADLINE_SECONDS to where it is being used and remove references in tests
    @m-trieu
    m-trieu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    6df1adf View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. address PR comments, remove unused IPv6 WindmillServiceAddress, renam… 

    …e StreamingEngineConnectionsState to StreamingEngineBackends
    @m-trieu
    m-trieu committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    4f5b381 View commit details
    Browse the repository at this point in the history