paddles: Adding a queueing mechanism to Paddles

[amathuria]

This change adds support for a queuing mechanism in Paddles. The idea is to eventually use only the Paddles backend for teuthology and remove the need for the Beanstalk queue

The following are the changes made:

1. Addition of new fields to the Jobs table in order to store the entire teuthology config.
2. Addition of a priority field to mimic a priority queue (similar to Beanstalk)
3. A new Queue table to keep track of the queues and their statuses on various machine types
4. Generation of a unique Job ID in Paddles
5. Rest APIs for retrieving a job from the queue, creating a new queue, updating status of a queue and retrieving stats for a particular queue

Signed-off-by: Aishwarya Mathuria [email protected]

[jdurgin]

This is looking good! IIRC you had some alembic changes to the fields as well - looks like they weren't added in this commit.

It'd be great to add some unit tests for the new/updated models and controllers, particularly ones with logic - e.g. for updating job priority it looks like you had to modify the model to work without targets, and only an update of priority

[kshtsk]

I guess we need to deploy a test instance for this.

[kshtsk]

travis tests are failing, btw

[amathuria]

travis tests are failing, btw

@kshtsk I can see a connection refused error in the test logs here: https://travis-ci.org/github/ceph/paddles/builds/771656065#L711. Any idea how I can fix this?

[jdurgin]

could you add a unit test for updating job priority? the new queue tests look good.

For the travis failure, do the same commands work for you locally? i.e. the commands starting here https://travis-ci.com/github/ceph/paddles/builds/227874146#L196

Though travis doesn't show the paddles log, it looks like paddles is crashing quickly after startup

[kshtsk]

travis tests are failing, btw

@kshtsk I can see a connection refused error in the test logs here: https://travis-ci.org/github/ceph/paddles/builds/771656065#L711. Any idea how I can fix this?

This means that server is not started in fact, you must try reproduce it on your local machine.

[kshtsk]

E           TypeError: Invalid argument(s) 'listeners' sent to create_engine(), using configuration SQLiteDialect_pysqlite/NullPool/Engine.  Please check that the keyword arguments are appropriate for this combination of components.

[kshtsk]

paddles/models/__init__.py:124:1: F401 '.queue.Queue' imported but unused
paddles/models/queue.py:1:1: F401 'sqlalchemy.exc' imported but unused

[amathuria]

travis tests are failing, btw

@kshtsk I can see a connection refused error in the test logs here: https://travis-ci.org/github/ceph/paddles/builds/771656065#L711. Any idea how I can fix this?

This means that server is not started in fact, you must try reproduce it on your local machine.

Sure I will do that.

[kshtsk]

> docker run --name=xxxx -d -p 80:8080 -e PADDLES_SERVER_HOST=0.0.0.0 -e PADDLES_SQLALCHEMY_URL=sqlite:///dev.db ceph-paddles-test
9215e5fe3b5bb24e433b6ac2586df37523fcae114497b2495fccfb1c20615949
> docker logs xxxx
==> LOADING ENVIRONMENT
Traceback (most recent call last):
  File "/usr/local/bin/pecan", line 11, in <module>
    sys.exit(CommandRunner.handle_command_line())
  File "/usr/local/lib/python3.6/dist-packages/pecan/commands/base.py", line 96, in handle_command_line
    runner.run(sys.argv[1:])
  File "/usr/local/lib/python3.6/dist-packages/pecan/commands/base.py", line 91, in run
    self.commands[ns.command_name]().run(ns)
  File "/usr/local/lib/python3.6/dist-packages/paddles/commands/populate.py", line 20, in run
    self.load_app()
  File "/usr/local/lib/python3.6/dist-packages/pecan/commands/base.py", line 162, in load_app
    return load_app(self.args.config_file)
  File "/usr/local/lib/python3.6/dist-packages/pecan/core.py", line 218, in load_app
    app = module.app.setup_app(_runtime_conf, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/paddles/app.py", line 7, in setup_app
    models.init_model()
  File "/usr/local/lib/python3.6/dist-packages/paddles/models/__init__.py", line 74, in init_model
    conf.sqlalchemy.engine = _engine_from_config(conf.sqlalchemy)
  File "/usr/local/lib/python3.6/dist-packages/paddles/models/__init__.py", line 86, in _engine_from_config
    return create_engine(url, **configuration)
  File "<string>", line 2, in create_engine
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/deprecations.py", line 298, in warned
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/create.py", line 645, in create_engine
    engineclass.__name__,
TypeError: Invalid argument(s) 'listeners' sent to create_engine(), using configuration SQLiteDialect_pysqlite/NullPool/Engine.  Please check that the keyword arguments are appropriate for this combination of components.

[kshtsk]

@amathuria usually you must make the PR run without unittests failures first ;-)

[amathuria]

@amathuria usually you must make the PR run without unittests failures first ;-)

Thank you for the logs. I was testing my changes on a mira setup with postgres and tested the sqlalchemy library upgrade there (which is causing the issue here in the docker test) and not with sqlite so I missed this. I'm working on the fix.

[jdurgin]

replace beanstalk queue

[zmc]

There's a lot of good work here! I don't think this is too far from being ready.

I've left a lot of comments, and tried to be clear, but if you have any questions, or if I just got something wrong, let me know.