dws: fix Postrun workflow error hang #169
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: if a workflow goes to status: Error, coral2_dws will raise an exception. For most states the exception will transition the job to CLEANUP, which eventually moves the workflow to Teardown. However, if the job is already in CLEANUP and is being held in the dws-epilog epilog when the exception is thrown, the exception has no effect and the epilog will never be finished because the workflow does not progress. If a workflow moves to status: Error in PostRun or DataOut states, move it directly to Teardown.
Problem: there are no tests for copy_in and copy_out directives. The tests will have to be simple because no global lustre file system will exist for doing the actual copying. Add some basic tests.
jameshcorbett
force-pushed
the
postrun-workflow-error-hang
branch
from
453f60e
to
2d7ebb8
Compare
|
Rebased after #168 was merged and dropped the draft label, no code changes. Thanks for the review! Setting MWP. |
jameshcorbett
added a commit
to jameshcorbett/flux-coral2
that referenced
this pull request
Jul 17, 2024
Problem: as in flux-framework#169, for most states, raising an exception should be enough to trigger other logic that eventually moves the workflow to Teardown. However, if the workflow is in PostRun or DataOut, the exception won't affect the dws-epilog action holding the job, so the workflow should be moved to Teardown immediately. Move workflows that are stuck in TransientCondition in DataOut or PostRun to Teardown immediately.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem: if a workflow goes to status: Error, coral2_dws will raise
an exception. For most states the exception will transition the job
to CLEANUP, which eventually moves the workflow to Teardown.
However, if the job is already in CLEANUP and is being held in the
dws-epilog epilog when the exception is thrown, the exception has
no effect and the epilog will never be finished because the workflow
does not progress.
If a workflow moves to status: Error in PostRun or DataOut states,
move it directly to Teardown.