stopcronjobs() hangs the server (on macOS)

[doorisajar]

I've been trying to use the cron management functionality, but every time I try to stop cron jobs in order to replace them with new ones, my server hangs. A small example:

using Oxygen, HTTP


function run_job(req)
    @info "run_job request body: $(req.body)"
    @info "Done"
    return 1
end


@post "/start" function (req::HTTP.Request)
    @info "/start POST endpoint hit; running job"
    stopcronjobs()

    @cron "*/10" function ()
        run_job(req)
    end

    startcronjobs()
    out = run_job(req)
    return out
end


@post "/stop" function (req::HTTP.Request)
    @info "/stop POST endpoint hit"
    terminate()
end


@info "Starting server"
serve(; host="0.0.0.0", port=5000)

@info "Server stopped"

After starting the server, everything runs as expected until stopcronjobs() is called (either directly or inside terminate()) while there are already registered cron jobs.

For example, I can run the server, hit the /stop endpoint, and see it shut down gracefully. Or I can run the server, hit the /start endpoint, and see the cron job start running every 10 seconds. But if I hit /start a second time, or hit /stop once the cron job is registered, the server hangs.

I've tested this directly on macOS 14.1.1 with Julia 1.9.3 and Oyxgen.jl 1.2.0, as well as indirectly in a Ubuntu 20.04 deployment environment that I don't have direct control over. I could do more direct testing on Linux, but I'm far from an expert.

[ndortega]

Hi @doorisajar,

Thanks for bringing up this issue. I never exported the resetcronstate() function from the cron module. From a first glance this probably happening when Julia tries to stop a task that has already been terminated due to the old references still in the internal vector.

[ndortega]

Hi @doorisajar,

thanks for the example script, this was very helpful in debugging this issue. Try installing a build from this branch which should address this issue: https://github.com/ndortega/Oxygen.jl/tree/bugfix/cron-job-restarting-issue

This issue stemmed from the way I was terminating running tasks which caused the server to close which looked like it was "hanging". With these changes, I'm using an internal global reference which is checked every second within each task to see if each task should continue or stop.

It's worth mentioning that stopping cron jobs now completely clears any internal references to stored jobs and requires re-registering to run them again (like in your example).

[doorisajar]

This does fix the issue, thank you!

[ndortega]

Hi @doorisajar,

Sorry to mess with a working solution, but I've made a couple of additional changes around this.

1. stopcronjobs() just stops all running jobs (doesn't clear jobs). This enables us to restart these cron jobs
2. added a new clearcronjobs() function which will wipe out any cron job definitions (will not impact running jobs)
3. It's no longer possible to add duplicate cron jobs (same name, function and expression)

Checkout my demo script here to see an example of this in action:
https://github.com/ndortega/Oxygen.jl/blob/bugfix/cron-job-restarting-issue/demo/cronmanagementdemo.jl

Let me know what you think!

[doorisajar]

^^ This update works perfectly. I have a good use for clearcronjobs, so I think it's a worthwhile addition. The separation between stop and clear makes sense.

Thanks!