Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up docker containers #55

Merged
merged 4 commits into from
Jan 11, 2024
Merged

Clean up docker containers #55

merged 4 commits into from
Jan 11, 2024

Conversation

nweires
Copy link
Collaborator

@nweires nweires commented Jan 10, 2024

Clean up our docker containers at the end of each task.

This should be handled by GCP Batch, but since it's not, do it ourselves.

Tested:

  • Before this change, containers would build up as multiple tasks ran on the same VM. For example:
$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          1         1         4.53GB    0B (0%)
Containers      9         4         1.205GB   533.3MB (44%)
Local Volumes   9         9         977.4MB   0B (0%)
Build Cache     0         0         0B        0B
  • With this change, they are cleaned up along the way:
$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          1         1         4.531GB   0B (0%)
Containers      3         3         99.8MB    0B (0%)
Local Volumes   3         3         409.4MB   0B (0

Copy link

github-actions bot commented Jan 10, 2024

File Coverage
All files 86%
base.py 91%
exc.py 57%
hpc.py 78%
local.py 70%
postprocessing.py 84%
utils.py 91%
cloud/docker_base.py 78%
sampler/base.py 79%
sampler/downselect.py 33%
sampler/precomputed.py 93%
sampler/residential_quota.py 61%
test/shared_testing_stuff.py 85%
test/test_docker.py 33%
test/test_local.py 97%
test/test_validation.py 97%
workflow_generator/base.py 90%
workflow_generator/commercial.py 53%
workflow_generator/residential_hpxml.py 86%

Minimum allowed coverage is 33%

Generated by 🐒 cobertura-action against 856b75c

@@ -577,7 +581,7 @@ def start_batch_job(self, batch_info):
# Give three minutes per simulation, plus ten minutes for job overhead
task_duration_secs = 60 * (10 + batch_info.n_sims_per_job * 3)
task = batch_v1.TaskSpec(
runnables=[runnable],
runnables=[runnable, prune],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that there are two runnables, can we rename runnable to something like bsb_runnable?

And maybe a better name than prune might be cleanup_runnable (hide the implementation detail) or docker_prune_runnable (qualify what's being pruned). I lean towards the former, fwiw.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - this was a quick version to start testing with, but I'll clean it up before merging.

@nweires nweires requested a review from lathanh January 10, 2024 21:33
buildstockbatch/gcp/gcp.py Outdated Show resolved Hide resolved
@nweires nweires merged commit 03b7e88 into gcp Jan 11, 2024
6 checks passed
@nweires nweires deleted the natalie/disk_space branch January 12, 2024 15:16
@nweires nweires mentioned this pull request Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants