Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Only send needed data to task runner (no-changelog) #11487

Open
wants to merge 12 commits into
base: master
Choose a base branch
from

Commits on Oct 31, 2024

  1. feat: Only send needed data to task runner

    When executing JS code in the Code Node, the task runner has currently
    been fetching the entire workflow execution context data. This can
    be a lot of data, and can cause OOMs on large workflows. Most often
    Code Node is used in such a way that it only uses the input data and
    maaaaybe some other node's data. Hence sending all the data is an
    overkill.
    
    This PR changes the behaviour to send only the needed data. This is
    implemented by running a static analysis of the code and identifying
    which built-in variables are accessed within the code. Based on that
    analysis we only send the needed data.
    
    There can be corner cases where we can't statically analyse for
    example which nodes' data is needed. This can happen for example
    when a variable is used as parameter to $() function. In these
    cases we send all the data.
    
    Based on some naive measurements, the message size is reduced
    something between 30-70%.
    
    Next steps:
    After this change there is room for optimization. For example,
    there is still duplication in the message send to the task runner.
    The input data of the Code Node is located in multiple parts of
    the message, which all end up being separate objects when
    deserialized in the task runner.
    tomi committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    6eba219 View commit details
    Browse the repository at this point in the history
  2. Fix parameter

    tomi committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    39422d9 View commit details
    Browse the repository at this point in the history
  3. Fix test

    tomi committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    c1c717c View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2024

  1. Configuration menu
    Copy the full SHA
    3c797ce View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    13d2eae View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d75cbd8 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    7456880 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    d69c339 View commit details
    Browse the repository at this point in the history
  6. Add documentation

    tomi committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    a11f71e View commit details
    Browse the repository at this point in the history
  7. Fix test name

    tomi committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    213c8f1 View commit details
    Browse the repository at this point in the history
  8. Mark methods as private

    tomi committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    19c17b9 View commit details
    Browse the repository at this point in the history
  9. Fix test

    tomi committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    c11b0b4 View commit details
    Browse the repository at this point in the history