-
Notifications
You must be signed in to change notification settings - Fork 201
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[cuebot/rqd] Add feature to run frames on a containerized environment using docker #1549
Open
DiegoTavares
wants to merge
36
commits into
AcademySoftwareFoundation:master
Choose a base branch
from
DiegoTavares:rqd_multiple_os
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
[cuebot/rqd] Add feature to run frames on a containerized environment using docker #1549
DiegoTavares
wants to merge
36
commits into
AcademySoftwareFoundation:master
from
DiegoTavares:rqd_multiple_os
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When RUN_ON_DOCKER is set on rqd.conf, each frame will be launched as a docker container using the base image configured as DOCKER_IMAGE.
When RUN_ON_DOCKER is set on rqd.conf, each frame will be launched as a docker container using the base image configured as DOCKER_IMAGE.
Signed-off-by: Diego Tavares <[email protected]>
Logging was added on the wrong scope, which led to a "Frame not found in cache" when a frame was actually found.
New spec is required to allow passing the layer's expected OS.
When rqd is running on docker mode, it can report multiple supported OSs. On rqd.conf, multiple images can be provided under [docker.images] and each image refers to a supported OS.
Signed-off-by: Diego Tavares <[email protected]>
Previously it was safe to use the host's OS when querying for procs, now the job OS needs to be used as a host can have multiple OSs.
To be able to run as the frame's owner, the entrypoint needs to ensure the user exists before running the frame's cmd.
Not having nimby installed is an expected event, not an exception.
…le (AcademySoftwareFoundation#1542) - Updated `viewComments` method in `MenuActions.py` to wrap single Job objects in a list. - This prevents `TypeError` when attempting to iterate over a non-iterable Job object.
…on#1543) - Add `rocky9` log root to `render_logs.root` in `cuegui.yaml`
… directly (AcademySoftwareFoundation#1547) **Summarize your change.** Have changed most tests to use `-m unittest discover` instead og `setup.py test` The old `setup.py test` doesn't work in newer versions of python since it has been deprecated
unittest was not reporting test failures and interruptions as expected, which caused us to be running with failed unit tests for a long time. This commit replaces unittest with pytest for rqd and fixes some of the relevant unit tests.
…oundation#1554) Deleting an item from the dict being iterated over on sanitizeFrames caused the error: "Dictionary changed size during iteration".
…to3 (AcademySoftwareFoundation#1557) **Link the Issue(s) this Pull Request is related to.** This is to fix AcademySoftwareFoundation#1555 **Summarize your change.** Replaces 2to3 with a simple script that adds "from ." in front of pb2 imports. This is done to support newer versions of python where 2to3 has been removed.
Since AcademySoftwareFoundation#1308 rqd stopped supporting stats files containing whitespaces and parenthesis.
When RUN_ON_DOCKER is set on rqd.conf, each frame will be launched as a docker container using the base image configured as DOCKER_IMAGE.
Signed-off-by: Diego Tavares <[email protected]>
Signed-off-by: Diego Tavares <[email protected]>
This change has been rebased from #1560 to allow running unit tests on rqd. |
Signed-off-by: Diego Tavares <[email protected]>
DiegoTavares
force-pushed
the
rqd_multiple_os
branch
from
October 29, 2024 15:08
d4904cb
to
674f494
Compare
DiegoTavares
force-pushed
the
rqd_multiple_os
branch
from
October 29, 2024 15:15
674f494
to
88b8892
Compare
When creating an user on a container, add a randomly generated password for security
DiegoTavares
changed the title
[EXPERIMENTAL] Add feature to run frames on a containerized environment using docker
[cuebot/rqd] Add feature to run frames on a containerized environment using docker
Oct 29, 2024
DiegoTavares
requested review from
bcipriano and
ramonfigueiredo
as code owners
October 29, 2024 23:03
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Running OpenCue In a multi operational system environment requires segregating the farm, which means hosts have to be assigned to one OS and cannot be shared between shows that have different OS requirements. This can be a challenge when sharing resources between shows is necessary.
Proposed solution
A new execution mode on rqd
runDocker
to live alongsiderunLinux
andrunWindows
. This mode will launch the frame command on a docker container based on the frame expected OS. With this, rqd is now able to run jobs from different OSs on the same host.But to make this possible, a rqd host needs to advertise itself not with its own OS code (defined by
SP_OS
on rqd.conf), but with all the OSs of images it is capable of executing.Configuration changes
The following sections were added to rqd.conf:
In this case, the rqd host would advertise itself with
OS=centos7,rocky9
, and the dispatch logic has been changed accordingly to account for dispatching frames to nodes that support multiple OSs.