-
Notifications
You must be signed in to change notification settings - Fork 7.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Only send needed data to task runner (no-changelog) #11487
Open
tomi
wants to merge
12
commits into
master
Choose a base branch
from
pay-2174-send-only-needed-data-to-the-runner
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,511
−252
Commits on Oct 31, 2024
-
feat: Only send needed data to task runner
When executing JS code in the Code Node, the task runner has currently been fetching the entire workflow execution context data. This can be a lot of data, and can cause OOMs on large workflows. Most often Code Node is used in such a way that it only uses the input data and maaaaybe some other node's data. Hence sending all the data is an overkill. This PR changes the behaviour to send only the needed data. This is implemented by running a static analysis of the code and identifying which built-in variables are accessed within the code. Based on that analysis we only send the needed data. There can be corner cases where we can't statically analyse for example which nodes' data is needed. This can happen for example when a variable is used as parameter to $() function. In these cases we send all the data. Based on some naive measurements, the message size is reduced something between 30-70%. Next steps: After this change there is room for optimization. For example, there is still duplication in the message send to the task runner. The input data of the Code Node is located in multiple parts of the message, which all end up being separate objects when deserialized in the task runner.
Configuration menu - View commit details
-
Copy full SHA for 6eba219 - Browse repository at this point
Copy the full SHA 6eba219View commit details -
Configuration menu - View commit details
-
Copy full SHA for 39422d9 - Browse repository at this point
Copy the full SHA 39422d9View commit details -
Configuration menu - View commit details
-
Copy full SHA for c1c717c - Browse repository at this point
Copy the full SHA c1c717cView commit details
Commits on Nov 1, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 3c797ce - Browse repository at this point
Copy the full SHA 3c797ceView commit details -
Configuration menu - View commit details
-
Copy full SHA for 13d2eae - Browse repository at this point
Copy the full SHA 13d2eaeView commit details -
Configuration menu - View commit details
-
Copy full SHA for d75cbd8 - Browse repository at this point
Copy the full SHA d75cbd8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7456880 - Browse repository at this point
Copy the full SHA 7456880View commit details -
Configuration menu - View commit details
-
Copy full SHA for d69c339 - Browse repository at this point
Copy the full SHA d69c339View commit details -
Configuration menu - View commit details
-
Copy full SHA for a11f71e - Browse repository at this point
Copy the full SHA a11f71eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 213c8f1 - Browse repository at this point
Copy the full SHA 213c8f1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 19c17b9 - Browse repository at this point
Copy the full SHA 19c17b9View commit details -
Configuration menu - View commit details
-
Copy full SHA for c11b0b4 - Browse repository at this point
Copy the full SHA c11b0b4View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.