Job Queues

Rodan Job Queues

Rodan assigns jobs to different queues in Celery depending on the kind of job it is. At the time of writing the queues are:

• None or celery: this is only for core jobs and runs on the celery container;
• Python2: this is for all Python2 jobs and runs on the py2-celery container;
• Python3: this is for all Python3 jobs and runs on the py3-celery container; and
• GPU: this is for all Python3 jobs that need a GPU and runs on the gpu-celery container.

A warning concerning dependencies

Rodan, which exists in the core celery container, must be able to load each job's wrapper to register it in the database. As a consequence of this, all jobs must be able to be loaded in the core celery container, which contains only Python 2. Even if a job only runs correctly in Python 3, Python 2 versions of its dependencies must be installed in the core container so that Rodan can load the job. There is no requirement that these Python 2 versions actually work (could be a totally empty module, if not needed by any other job); they just need to be imported without error.

Specifying a Queue for a Job

There are two places a job queue must be specified for a job: in Rodan's settings.py and the settings parameter of the job itself.

Updating settings.py

In the file /rodan/code/rodan/settings.py there are three arrays representing queues for rodan jobs. These are RODAN_PYTHON2_JOBS, RODAN_PYTHON3_JOBS, and RODAN_GPU_JOBS for the Python2, Python3, and GPU queues respectively. These contain the location of the module for your job. So if your job some-job is in the jobs folder, it would be represented as rodan.jobs.some-job.

Each job is commented out in settings.py by default. It should be prefixed with the appropriate comment for that queue. The prefixes for the Python2, Python3, and GPU queues are, respectively, #py2 , #py3 , and #gpu . Note that each contains a space at the end. This would make the full line for your job if it was in the Python3 queue #py3 "rodan.jobs.some-job",.

Specifying Queue in the Job

Each Rodan job can have a settings attribute among the others as specified on the Rodan wiki. This settings attribute should be a dictionary containing a key job_queue whose value is the desired queue.

A queue should be specified. If it is not it will default to the Python2 queue.

The settings dictionary should look something like this:

settings = {
    ...
    "job_queue": "Python3",
    ...
}