[SDESK-7371] Support eager mode in async celery

[eos87]

Purpose

The purpose of this PR is to modify the HybridAppContextTask class to support different configurations for Celery task execution, specifically ensuring compatibility with both eager and non-eager execution modes. The changes also aim to make apply_async() and delay() methods asynchronous to allow awaiting the scheduling of a task or either returning the result of the task directly if eager mode is on.

What has changed

• Added config-based handling for Celery execution:
  • Eager Mode (CELERY_TASK_ALWAYS_EAGER is True): Tasks are executed synchronously and awaited.
  • Non-Eager Mode (CELERY_TASK_ALWAYS_EAGER is False): If no loop is running, a new one is created; otherwise, the task is scheduled asynchronously.
• Made apply_async() and delay() await task results in eager mode (no need to override delay since it is essentially an alias method of apply_async)
• Added background_tasks to manage references to tasks created with asyncio.create_task().
• Added _is_always_eager() utility function to determine eager mode.

How to test

Here are two demo tasks that you can call either in eager and non-eager mode and observe the result and behaviour

import asyncio

@celery.task
async def dump_task():
    print("start dump_task")
    await asyncio.sleep(5)
    print("finish dump_task after sleeping 5 seconds")
    return "result dump task"

@celery.task
async def dump_task_2():
    print("start dump_task_2")
    print("finish dump_task 2")
    return "result dump task 2"

then run them like

result = await dump_task.delay()
print(result)

result = await dump_task_2.delay()
print(result)

print("COMPLETED")

with eager mode ON, you should have something like this (notice that they run sync)

start dump_task
finish dump_task after sleeping 5 seconds
result dump task
start dump_task_2
finish dump_task 2
result dump task 2
COMPLETED

with eager mode OFF, should be like this below. Notice that dump_task_2 is executed while dump_task sleeps for 5 seconds. Then finally dump_task finishes.

start dump_task
81578173-5564-46af-ba53-292da18795ed
91fb2f57-ccb7-4ff8-b571-0bb5382e1c2f
COMPLETED

start dump_task_2
finish dump_task 2
finish dump_task after sleeping 5 seconds

Thanks for checking!

Resolves: SDESK-7371

[MarkLark86]

Looks good @eos87, nice one 👍