Schedule repeat tasks as a cron job

[the-noble-argon]

I see that Oxygen.jl supports tasks that happen at an interval, but is there any way to supply a cron expression to determine the offset timing of the repeat task? I guess I could make a repeat task that fires every minute to check if a task is due in the next interval and wait for that time, I'm just wondering if there is an easier way to do this than building this myself.

[ndortega]

Hi @the-noble-argon ,

Unfortunately, that exact functionality is not available at the moment but it's definitely a great idea. I don't think there's any CRON.jl package to do this for me, so I'll have to make it myself. Personally, I'd want something similar to the cron expressions you can use in spring boot. What do you think?

[the-noble-argon]

That would be amazing. A lot of what I do in Azure Function Apps are timed tasks like this (and due to the .exe issues, it's really hard to use Julia inside a function app), but docker images running REST APIs on app services run great in Azure. Having this ability would greatly expand the usability of Julia in Azure (the only other problem with Julia in Azure is authentication without using PyCall).

[ndortega]

Hey @the-noble-argon,

I have some good news, I have an early working example of cron expressions in my feature branch. At the moment, you can either set the cron expressions from within the router() function or you can use the 'cron' macro to register regular functions.

Currently, I'm using the modulo operator to determine when a function needs to get called. So for example, if we register a function to run every 3 seconds it will only get called on seconds 3,6,9...etc. (There's probably a better way to do this)

Quick notes

• the cron expressions in oxygen are based on this: https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/scheduling/support/CronExpression.html
• You don't need trailing "*" characters to your expressions (you can leave them out). They will get defaulted to "*" expressions if they are not set by you

using Oxygen

value = 0

# Here's utility macro to call functions directly
@cron "*/5" function()
    println("every 5 seconds")
end

# You can also just use the 'cron' keyword that's apart of the router() function
@get router("/increment", cron="*/3") function(req)
    global value += 1
    println("every 3 seconds: $value")
    return value
end

serve()

You can run the following command in the package repl to try it out yourself

pkg> add https://github.com/ndortega/Oxygen.jl.git#feature/cron-expressions

[the-noble-argon]

Nice. How are you implementing incremental background tasks currently? Are you making a bunch of async tasks and having them sleep for that interval?

The trick for cron expressions is to

1. Break down compound cron expressions into N simple ones (for example, one that runs on 8:00, 9:00, and 10:00 would be three different expressions)
2. Take the current time
3. Create new time(s) by substituting the current time with the desired cron expression values
4. Subtract now from the cron substitution(s) to get the wait time(s)
5. If a wait time is negative (which it usually is), add the cron expression interval
6. Sleep the task for the smallest wait time

[ndortega]

Here's the current approach:

• Start up a single async task which is a simple spin loop that only executes on every whole second
• At each whole second, we take the current time and see if it matches any registered cron expression. If so, the corresponding function handler is called asynchronously.

Here's what happens during the "matching" logic:

• Break the cron expression into sub-expressions ( 6 parts )
• Pass each sub-expression to a function to parse and evaluate individually against the current time.
• In order for a cron expression to "match", all sub-expressions need to evaluate to true.

---

Pros:

• This approach is pretty straightforward and we'll never "miss" a cron expression

Cons

• Definitely is eating up more CPU by running and checking all expressions each second.

---

Next Steps

• I'm thinking of adapting your approach by spinning up a separate task for each registered cron job and trying to calculate the next valid value and sleeping until then.

[the-noble-argon]

Right. Your current method is definitely easier to troubleshoot. If you're checking every whole second it's probably not THAT much overhead unless you have a lot of cron tasks. If you see a lot of overhead it might actually be caused something like type instability caused by global variables; I see you have a lot of global variables which kills performance. If you know what type the global variable is, you can use const Ref instead like this

const MY_GLOBAL = Ref{Float64}()
MY_GLOBAL[] = 2.3

function add_to_my_global(x)
    return x + MY_GLOBAL[]
end

function change_my_global(x)
    return MY_GLOBAL[] = x
end

You don't even have to do this if your const is mutable. For example, with your

global repeattasks = []

It's probably much more performant if you use (note caps are only for readability conventions)

const REPEATTASKS = Task[]

If you want to reset it and empty everything, use

empty!(REPEATTASKS)

I see you're already using push!() on it. If you want to delete things, use deleteat!()

If this doesn't solve your performance problems, that calculation I told you about would only have to run on startup for the initial offset because once you do that calculation, you can
(1) Store the time remaining for each task
(2) Find the time time remaining until the next task
(3) Subtract that wait time from every task
(4) Sleep the scheduler for the allotted time
(5) When executing any cron task, redo the time remaining calculation for that task only

If you do your multiple task sleep approach, you may have to redo these calculations anyway just to prevent compound roundoff error.

[the-noble-argon]

Shoot, I just realized that calculating the time until next execution is difficult if the cron expression has intermediate wildcards like
"0 0 9-17 * * MON-FRI". My previous methods for doing this didn't account for that. Checking once every second might just be the easier way to go for now. You will likely get good enough performance if you change your global x = ... to const x = Ref(...), and if you make sure your repeattasks array is typed when you initialize it repeattasks = [] will give you an array of type Any which is really slow; use const repeattasks = MyTaskType[].

Julia 1.8 now allows for type annotation for global variables, but the "const Ref" trick is more backwards-compatible (especially for LTS) and is widely accepted as the solution.

[ndortega]

It actually shouldn't be too bad since I already have the code to match any sub expression. I can "calculate" the next execution time by brute forcing it within 59 iterations.

Strategy

• Foor Loop (1-59)
  • Add 1 second to the current date until the seconds expression matches
  • Add 1 minute to the current date until the minutes expression matches
  • Add 1 hour to the current date until the hour expression matches
  • Add 1 day to the current date until the day of month & day of week expression matches
  • Add 1 month to the current date until the month expression matches
  • Return from the function when all expressions match

I'll have to implement this and play around and see how the CPU utilization compares to the original to see if this extra work is worth it. If I run into any weird bugs, I'll just stick to the current approach which is more straightforward.

[ndortega]

Hi @the-noble-argon,

I just released these changes in v1.1.6, give it a shot and let me know what you think!