Skip to content

Commit

Permalink
Improve restarting of workers to prevent issues with batch request
Browse files Browse the repository at this point in the history
  • Loading branch information
aequitas committed Oct 18, 2024
1 parent b775fb2 commit 7b8aaaa
Showing 1 changed file with 8 additions and 2 deletions.
10 changes: 8 additions & 2 deletions docker/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,8 @@ services:
# time after which a SIGKILL is sent to celery after a SIGTERM (warm shutdown), default 10s
# insufficient short grace period causes issues on batch when tasks are killed during the hourly worker restart
stop_grace_period: 10m
# SIGTERM is default, but make it explicit
stop_signal: SIGTERM

depends_on:
db-migrate:
Expand Down Expand Up @@ -1134,8 +1136,12 @@ configs:
content: |
#!/bin/sh
set -e
# find nassl worker and restart the container(s)
docker restart $(docker ps --filter label=com.docker.compose.service=worker-nassl --quiet)
# stop and start worker one at a time to ensure (batch) tasks are still being picked up
# workers are sent a TERM signal with which a 10 minute grace period before QUIT is sent
for worker in docker ps --filter label=com.docker.compose.service=worker-nassl --quiet; do
docker stop "$worker"
docker start "$worker"
dones
docker_image_prune:
content: |
Expand Down

0 comments on commit 7b8aaaa

Please sign in to comment.