Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

403/Forbidden and "Could not open job state file" #571

Open
marco-brandizi opened this issue Nov 4, 2020 · 6 comments
Open

403/Forbidden and "Could not open job state file" #571

marco-brandizi opened this issue Nov 4, 2020 · 6 comments

Comments

@marco-brandizi
Copy link

I'm trying to set up a local Galaxy instance, using this Docker project.

I've tried this command:

# docker-run.sh
cd `dirname "$0"`
docker run -it -p 8080:80 -v `pwd`/galaxy-store/:/export/ --name galaxy bgruening/galaxy-stable

Initially, it says: "Checking /export..." for about 30 mins, then it starts yielding several messages, including errors like:

FATAL:  the database system is starting up # repeated multiple times
...
==> /home/galaxy/logs/slurmctld.log <==
[2020-11-04T14:14:55.442] error: Could not open job state file /tmp/slurm/job_state: No such file or directory
[2020-11-04T14:14:55.442] error: NOTE: Trying backup state save file. Jobs may be lost!
[2020-11-04T14:14:55.442] No job state file (/tmp/slurm/job_state.old) found

I get the messages about job_state after I tried http://localhost:8080 with the host browser. The latter returns "403 Forbidden".

Any ideas?
@benjaminbrumbaugh benjaminbrumbaugh mentioned this issue Jul 30, 2021
Open
@shenweiyan
Copy link

I got the same error .

@bgruening
Copy link
Owner

@shenweiyan are you also using the export? What happens if you do not use the export?

@shenweiyan
Copy link

@shenweiyan are you also using the export? What happens if you do not use the export?

I got something wrong as follow:

$ docker run -it -p 9080:80 -p 9081:21 -p 9082:22 bgruening/galaxy-stable
Enable Galaxy reports authentification
Checking /export...
Disable Galaxy Interactive Environments. Start with --privileged to enable IE's.
Starting postgres
postgresql: started
Checking if database is up and running
Traceback (most recent call last):
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/check_database.py", line 25, in <module>
    query.count()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3749, in count
    return self.from_self(col).scalar()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3469, in scalar
    ret = self.one()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3436, in one
    ret = self.one_or_none()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3405, in one_or_none
    ret = list(self)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
    return self._execute_and_instances(context)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
    return meth(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
    distilled_params,
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
    e, statement, parameters, cursor, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

.....

==> /home/galaxy/logs/uwsgi.log <==
Traceback (most recent call last):
  File "lib/galaxy/webapps/galaxy/buildapp.py", line 48, in app_factory
    app = galaxy.app.UniverseApplication(global_conf=global_conf, **kwargs)
  File "lib/galaxy/app.py", line 97, in __init__
    self._configure_models(check_migrate_databases=self.config.check_migrate_databases, check_migrate_tools=check_migrate_tools, config_file=config_file)
  File "lib/galaxy/config/__init__.py", line 1226, in _configure_models
    create_or_verify_database(db_url, config_file, self.config.database_engine_options, app=self, map_install_models=combined_install_database)
  File "lib/galaxy/model/migrate/check.py", line 140, in create_or_verify_database
    raise Exception("{}. {}{}".format(expect_msg, instructions, backup_msg))
Exception: Your database has version '1' but this code expects version '170'. Please backup your database and then migrate the database schema by running 'sh manage_db.sh upgrade'.

....

@bgruening
Copy link
Owner

Can you try docker run -it -p 9080:80 -p 9081:21 -p 9082:22 quay.io/bgruening/galaxy

@shenweiyan
Copy link

shenweiyan commented Oct 15, 2021 •
edited
Loading

On my CentOS 7, the error is still the same. But I can see the galaxy web page on http://localhost:9080.

$ docker run -it -p 9080:80 -p 9081:21 -p 9082:22 quay.io/bgruening/galaxy
Enable Galaxy reports authentification
Checking /export...
Disable Galaxy Interactive Environments. Start with --privileged to enable IE's.
Starting postgres
postgresql: started
Checking if database is up and running
Traceback (most recent call last):
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/check_database.py", line 25, in <module>
    query.count()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3749, in count
    return self.from_self(col).scalar()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3469, in scalar
    ret = self.one()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3436, in one
    ret = self.one_or_none()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3405, in one_or_none
    ret = list(self)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
    return self._execute_and_instances(context)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
    return meth(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
    distilled_params,
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
    e, statement, parameters, cursor, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

[SQL: SELECT count(*) AS count_1
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token
FROM galaxy_user
WHERE galaxy_user.email = %(email_1)s) AS anon_1]
[parameters: {'email_1': '[email protected]'}]
(Background on this error at: http://sqlalche.me/e/f405)
Waiting for database
Traceback (most recent call last):
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/check_database.py", line 25, in <module>
    query.count()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3749, in count
    return self.from_self(col).scalar()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3469, in scalar
    ret = self.one()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3436, in one
    ret = self.one_or_none()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3405, in one_or_none
    ret = list(self)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
    return self._execute_and_instances(context)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
    return meth(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
    distilled_params,
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
    e, statement, parameters, cursor, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

[SQL: SELECT count(*) AS count_1
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token
FROM galaxy_user
WHERE galaxy_user.email = %(email_1)s) AS anon_1]
[parameters: {'email_1': '[email protected]'}]
(Background on this error at: http://sqlalche.me/e/f405)
Waiting for database
Traceback (most recent call last):
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/check_database.py", line 25, in <module>
    query.count()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3749, in count
    return self.from_self(col).scalar()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3469, in scalar
    ret = self.one()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3436, in one
    ret = self.one_or_none()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3405, in one_or_none
    ret = list(self)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
    return self._execute_and_instances(context)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
    return meth(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
    distilled_params,
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
    e, statement, parameters, cursor, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

[SQL: SELECT count(*) AS count_1
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token
FROM galaxy_user
WHERE galaxy_user.email = %(email_1)s) AS anon_1]
[parameters: {'email_1': '[email protected]'}]
(Background on this error at: http://sqlalche.me/e/f405)
Waiting for database
Traceback (most recent call last):
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/check_database.py", line 25, in <module>
    query.count()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3749, in count
    return self.from_self(col).scalar()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3469, in scalar
    ret = self.one()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3436, in one
    ret = self.one_or_none()
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3405, in one_or_none
    ret = list(self)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
    return self._execute_and_instances(context)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
    return meth(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
    distilled_params,
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
    e, statement, parameters, cursor, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/galaxy_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "galaxy_user" does not exist
LINE 3: FROM galaxy_user
             ^

[SQL: SELECT count(*) AS count_1
FROM (SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.last_password_change AS galaxy_user_last_password_change, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token
FROM galaxy_user
WHERE galaxy_user.email = %(email_1)s) AS anon_1]
[parameters: {'email_1': '[email protected]'}]
(Background on this error at: http://sqlalche.me/e/f405)
Waiting for database
Database connected
Starting cron
cron: started
Starting ProFTP
proftpd: started
Starting Galaxy reports webapp
reports: started
Starting nodejs
galaxy:galaxy_nodejs_proxy: started
Starting condor
condor: started
Starting slurmctld
Starting slurmd
Creating admin user admin with key fakekey and password password if not existing
==> /home/galaxy/logs/handler0.log <==
galaxy.jobs.handler INFO 2021-10-15 09:07:50,893 job handler queue started
galaxy.jobs.handler INFO 2021-10-15 09:07:50,910 job handler stop queue started
galaxy.workflow.scheduling_manager INFO 2021-10-15 09:07:50,979 No workflow schedulers plugin config file defined, using default scheduler.
galaxy.workflow.scheduling_manager DEBUG 2021-10-15 09:07:50,979 Starting workflow schedulers
galaxy.queue_worker INFO 2021-10-15 09:07:51,524 Binding and starting galaxy control worker for handler0
galaxy.queue_worker INFO 2021-10-15 09:07:51,798 Queuing async task rebuild_toolbox_search_index for handler0.
galaxy.app INFO 2021-10-15 09:07:54,088 Galaxy app startup finished (96604.370 ms)
galaxy.web_stack INFO 2021-10-15 09:07:54,088 Galaxy server instance 'handler0' is running
galaxy.queue_worker INFO 2021-10-15 09:07:55,100 Instance 'handler0' received 'rebuild_toolbox_search_index' task, executing now.
galaxy.queue_worker DEBUG 2021-10-15 09:07:55,100 App is not a webapp, not building a search index

==> /home/galaxy/logs/handler1.log <==
galaxy.workflow.scheduling_manager INFO 2021-10-15 09:07:46,616 No workflow schedulers plugin config file defined, using default scheduler.
galaxy.workflow.scheduling_manager DEBUG 2021-10-15 09:07:46,616 Starting workflow schedulers
galaxy.queue_worker INFO 2021-10-15 09:07:47,097 Binding and starting galaxy control worker for handler1
galaxy.queue_worker INFO 2021-10-15 09:07:47,217 Queuing async task rebuild_toolbox_search_index for handler1.
galaxy.model.database_heartbeat DEBUG 2021-10-15 09:07:48,212 handler1 is config watcher
galaxy.util.watcher DEBUG 2021-10-15 09:07:48,896 Watching for changes in directory (recursively): /galaxy-central/tool-data
galaxy.app INFO 2021-10-15 09:07:49,682 Galaxy app startup finished (92769.326 ms)
galaxy.web_stack INFO 2021-10-15 09:07:49,683 Galaxy server instance 'handler1' is running
galaxy.queue_worker INFO 2021-10-15 09:07:50,611 Instance 'handler1' received 'rebuild_toolbox_search_index' task, executing now.
galaxy.queue_worker DEBUG 2021-10-15 09:07:50,611 App is not a webapp, not building a search index

==> /home/galaxy/logs/reports.log <==
galaxy.web.framework.base DEBUG 2021-10-15 09:07:50,801 Enabling 'jobs' controller, class: Jobs
galaxy.web.framework.base DEBUG 2021-10-15 09:07:50,806 Enabling 'users' controller, class: Users
galaxy.web.framework.base DEBUG 2021-10-15 09:07:50,827 Enabling 'system' controller, class: System
galaxy.webapps.util DEBUG 2021-10-15 09:07:50,879 Enabling 'paste.httpexceptions' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:07:51,885 Enabling 'RecursiveMiddleware' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:07:52,079 Enabling 'ErrorMiddleware' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:07:52,087 Enabling 'TransLogger' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:07:52,092 Enabling 'XForwardedHostMiddleware' middleware
Starting server in PID 616.
serving on http://127.0.0.1:9001

==> /home/galaxy/logs/slurmctld.log <==
[2021-10-15T09:06:26.382] No trigger state file (/tmp/slurm/trigger_state.old) to recover
[2021-10-15T09:06:26.382] _preserve_plugins: backup_controller not specified
[2021-10-15T09:06:26.382] Reinitializing job accounting state
[2021-10-15T09:06:26.382] cons_res: select_p_reconfigure
[2021-10-15T09:06:26.382] cons_res: select_p_node_init
[2021-10-15T09:06:26.382] cons_res: preparing for 1 partitions
[2021-10-15T09:06:26.383] Running as primary controller
[2021-10-15T09:06:26.414] No parameter for mcs plugin, default values set
[2021-10-15T09:06:26.414] mcs: MCSParameters = (null). ondemand set.
[2021-10-15T09:06:29.806] SchedulerParameters=default_queue_depth=100,max_rpc_cnt=0,max_sched_time=2,partition_job_depth=0,sched_max_job_start=0,sched_min_interval=2

==> /home/galaxy/logs/slurmd.log <==
[2021-10-15T09:06:28.014] error: Domain socket directory /var/spool/slurmd: No such file or directory
[2021-10-15T09:06:28.182] Message aggregation disabled
[2021-10-15T09:06:28.186] CPU frequency setting not configured for this node
[2021-10-15T09:06:28.325] slurmd version 17.11.2 started
[2021-10-15T09:06:28.386] slurmd started on Fri, 15 Oct 2021 09:06:28 +0000
[2021-10-15T09:06:28.398] CPUs=4 Boards=1 Sockets=4 Cores=1 Threads=1 Memory=15871 TmpDisk=1007799 Uptime=4720391 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)

==> /home/galaxy/logs/uwsgi.log <==
galaxy.web.framework.base DEBUG 2021-10-15 09:07:42,413 [p:499,w:0,m:0] [MainThread] Enabling 'error' controller, class: Error
galaxy.web.framework.base DEBUG 2021-10-15 09:07:42,716 [p:499,w:0,m:0] [MainThread] Enabling 'visualization' controller, class: VisualizationController
galaxy.web.framework.base DEBUG 2021-10-15 09:07:43,505 [p:499,w:0,m:0] [MainThread] Enabling 'history' controller, class: HistoryController
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,799 [p:499,w:0,m:0] [MainThread] Enabling 'workflow' controller, class: WorkflowController
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,891 [p:499,w:0,m:0] [MainThread] Enabling 'authnz' controller, class: JSAppLauncher
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,892 [p:499,w:0,m:0] [MainThread] Enabling 'authnz' controller, class: OIDC
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,907 [p:499,w:0,m:0] [MainThread] Enabling 'root' controller, class: RootController
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,908 [p:499,w:0,m:0] [MainThread] Enabling 'library' controller, class: Library
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,909 [p:499,w:0,m:0] [MainThread] Enabling 'admin' controller, class: AdminGalaxy
galaxy.web.framework.base DEBUG 2021-10-15 09:07:45,916 [p:499,w:0,m:0] [MainThread] Enabling 'forms' controller, class: Forms
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,199 [p:499,w:0,m:0] [MainThread] Enabling 'page' controller, class: PageController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,208 [p:499,w:0,m:0] [MainThread] Enabling 'tag' controller, class: TagsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,216 [p:499,w:0,m:0] [MainThread] Enabling 'user' controller, class: User
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,227 [p:499,w:0,m:0] [MainThread] Enabling 'roles' API controller, class: RoleAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,280 [p:499,w:0,m:0] [MainThread] Enabling 'tags' API controller, class: TagsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,288 [p:499,w:0,m:0] [MainThread] Enabling 'tool_shed_repositories' API controller, class: ToolShedRepositoriesController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,291 [p:499,w:0,m:0] [MainThread] Enabling 'trs_consumer' API controller, class: TrsConsumeAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,297 [p:499,w:0,m:0] [MainThread] Enabling 'groups' API controller, class: GroupAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,300 [p:499,w:0,m:0] [MainThread] Enabling 'webhooks' API controller, class: WebhooksController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,304 [p:499,w:0,m:0] [MainThread] Enabling 'dataset_collections' API controller, class: DatasetCollectionsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,315 [p:499,w:0,m:0] [MainThread] Enabling 'remote_files' API controller, class: RemoteFilesAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,320 [p:499,w:0,m:0] [MainThread] Enabling 'folders' API controller, class: FoldersController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,322 [p:499,w:0,m:0] [MainThread] Enabling 'tool_entry_points' API controller, class: ToolEntryPointsAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,326 [p:499,w:0,m:0] [MainThread] Enabling 'authenticate' API controller, class: AuthenticationController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,903 [p:499,w:0,m:0] [MainThread] Enabling 'search' API controller, class: SearchController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,908 [p:499,w:0,m:0] [MainThread] Enabling 'folder_contents' API controller, class: FolderContentsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,911 [p:499,w:0,m:0] [MainThread] Enabling 'history_content_provenance' API controller, class: HDAProvenanceController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,912 [p:499,w:0,m:0] [MainThread] Enabling 'ldda_provenance' API controller, class: LDDAProvenanceController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,914 [p:499,w:0,m:0] [MainThread] Enabling 'page_revisions' API controller, class: PageRevisionsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,987 [p:499,w:0,m:0] [MainThread] Enabling 'tools' API controller, class: ToolsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,989 [p:499,w:0,m:0] [MainThread] Enabling 'metrics' API controller, class: MetricsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,993 [p:499,w:0,m:0] [MainThread] Enabling 'job_files' API controller, class: JobFilesAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:09,999 [p:499,w:0,m:0] [MainThread] Enabling 'quotas' API controller, class: QuotaAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,001 [p:499,w:0,m:0] [MainThread] Enabling 'display_applications' API controller, class: DisplayApplicationsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,017 [p:499,w:0,m:0] [MainThread] Enabling 'cloudauthz' API controller, class: CloudAuthzController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,020 [p:499,w:0,m:0] [MainThread] Enabling 'configuration' API controller, class: ConfigurationController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,103 [p:499,w:0,m:0] [MainThread] Enabling 'cloud' API controller, class: CloudController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,105 [p:499,w:0,m:0] [MainThread] Enabling 'history_content_tags' API controller, class: HistoryContentTagsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,106 [p:499,w:0,m:0] [MainThread] Enabling 'history_tags' API controller, class: HistoryTagsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,106 [p:499,w:0,m:0] [MainThread] Enabling 'workflow_tags' API controller, class: WorkflowTagsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,116 [p:499,w:0,m:0] [MainThread] Enabling 'history_contents' API controller, class: HistoryContentsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,119 [p:499,w:0,m:0] [MainThread] Enabling 'datatypes' API controller, class: DatatypesController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,122 [p:499,w:0,m:0] [MainThread] Enabling 'visualizations' API controller, class: VisualizationsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,124 [p:499,w:0,m:0] [MainThread] Enabling 'genomes' API controller, class: GenomesController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,198 [p:499,w:0,m:0] [MainThread] Enabling 'job_ports' API controller, class: JobPortsAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,206 [p:499,w:0,m:0] [MainThread] Enabling 'libraries' API controller, class: LibrariesController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,208 [p:499,w:0,m:0] [MainThread] Enabling 'group_users' API controller, class: GroupUsersAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,211 [p:499,w:0,m:0] [MainThread] Enabling 'tours' API controller, class: ToursController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,218 [p:499,w:0,m:0] [MainThread] Enabling 'histories' API controller, class: HistoriesController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,284 [p:499,w:0,m:0] [MainThread] Enabling 'workflows' API controller, class: WorkflowsAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,299 [p:499,w:0,m:0] [MainThread] Enabling 'library_datasets' API controller, class: LibraryDatasetsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,302 [p:499,w:0,m:0] [MainThread] Enabling 'history_annotations' API controller, class: HistoryAnnotationsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,302 [p:499,w:0,m:0] [MainThread] Enabling 'history_content_annotations' API controller, class: HistoryContentAnnotationsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,303 [p:499,w:0,m:0] [MainThread] Enabling 'workflow_annotations' API controller, class: WorkflowAnnotationsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,309 [p:499,w:0,m:0] [MainThread] Enabling 'jobs' API controller, class: JobController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,311 [p:499,w:0,m:0] [MainThread] Enabling 'group_roles' API controller, class: GroupRolesAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,319 [p:499,w:0,m:0] [MainThread] Enabling 'forms' API controller, class: FormDefinitionAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,320 [p:499,w:0,m:0] [MainThread] Enabling 'trs_search' API controller, class: TrsSearchAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,387 [p:499,w:0,m:0] [MainThread] Enabling 'users' API controller, class: UserAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,390 [p:499,w:0,m:0] [MainThread] Enabling 'history_dataset_extended_metadata' API controller, class: HistoryDatasetExtendMetadataController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,390 [p:499,w:0,m:0] [MainThread] Enabling 'library_dataset_extended_metadata' API controller, class: LibraryDatasetExtendMetadataController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,391 [p:499,w:0,m:0] [MainThread] Enabling 'container_resolution' API controller, class: ContainerResolutionAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,394 [p:499,w:0,m:0] [MainThread] Enabling 'uploads' API controller, class: UploadsAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,396 [p:499,w:0,m:0] [MainThread] Enabling 'pages' API controller, class: PagesController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,397 [p:499,w:0,m:0] [MainThread] Enabling 'toolshed' API controller, class: ToolShedController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,400 [p:499,w:0,m:0] [MainThread] Enabling 'tool_dependencies' API controller, class: ToolDependenciesAPIController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,403 [p:499,w:0,m:0] [MainThread] Enabling 'tool_data' API controller, class: ToolData
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,404 [p:499,w:0,m:0] [MainThread] Enabling 'plugins' API controller, class: PluginsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,408 [p:499,w:0,m:0] [MainThread] Enabling 'library_contents' API controller, class: LibraryContentsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,413 [p:499,w:0,m:0] [MainThread] Enabling 'datasets' API controller, class: DatasetsController
galaxy.web.framework.base DEBUG 2021-10-15 09:08:10,415 [p:499,w:0,m:0] [MainThread] Enabling 'dynamic_tools' API controller, class: DynamicToolsController
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,654 [p:499,w:0,m:0] [MainThread] Enabling 'paste.httpexceptions' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,690 [p:499,w:0,m:0] [MainThread] Enabling 'RecursiveMiddleware' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,690 [p:499,w:0,m:0] [MainThread] Enabling 'ErrorMiddleware' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,696 [p:499,w:0,m:0] [MainThread] Enabling 'TransLogger' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,702 [p:499,w:0,m:0] [MainThread] Enabling 'XForwardedHostMiddleware' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,702 [p:499,w:0,m:0] [MainThread] Enabling 'RequestIDMiddleware' middleware
galaxy.webapps.util DEBUG 2021-10-15 09:08:11,702 [p:499,w:0,m:0] [MainThread] Enabling 'BatchMiddleware' middleware
galaxy.webapps.galaxy.buildapp DEBUG 2021-10-15 09:08:11,709 [p:499,w:0,m:0] [MainThread] Prior to webapp return, Galaxy thread <_MainThread(MainThread, started 140052499973952)> is alive.
galaxy.webapps.galaxy.buildapp DEBUG 2021-10-15 09:08:11,710 [p:499,w:0,m:0] [MainThread] Prior to webapp return, Galaxy thread <SqliteMultithread(Thread-4, started daemon 140052303128320)> is alive.
Fri Oct 15 09:08:11 2021 - WSGI app 0 (mountpoint='') ready in 202 seconds on interpreter 0x55b1989f3dc0 pid: 499 (default app)
Fri Oct 15 09:08:11 2021 - *** uWSGI is running in multiple interpreter mode ***
Fri Oct 15 09:08:11 2021 - spawned uWSGI master process (pid: 499)
Fri Oct 15 09:08:11 2021 - spawned uWSGI worker 1 (pid: 853, cores: 4)
Fri Oct 15 09:08:11 2021 - spawned 2 offload threads for uWSGI worker 1
Fri Oct 15 09:08:11 2021 - spawned uWSGI worker 2 (pid: 856, cores: 4)
Fri Oct 15 09:08:11 2021 - spawned 2 offload threads for uWSGI worker 2
galaxy.web_stack DEBUG 2021-10-15 09:08:11,820 [p:853,w:1,m:0] [MainThread] Calling postfork function: <bound method ApplicationStack.set_postfork_server_name of <galaxy.web_stack.UWSGIApplicationStack object at 0x7f605e9ed358>>
Fri Oct 15 09:08:11 2021 - *** Stats server enabled on 127.0.0.1:9191 fd: 17 ***
Fri Oct 15 09:08:12 2021 - spawned uWSGI http 1 (pid: 859)
Fri Oct 15 09:08:12 2021 - running "unix_signal:2 gracefully_kill_them_all" (master-start)...
Fri Oct 15 09:08:12 2021 - running "unix_signal:15 gracefully_kill_them_all" (master-start)...
galaxy.web_stack DEBUG 2021-10-15 09:08:11,961 [p:856,w:2,m:0] [MainThread] Calling postfork function: <bound method ApplicationStack.set_postfork_server_name of <galaxy.web_stack.UWSGIApplicationStack object at 0x7f605e9ed358>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,320 [p:856,w:2,m:0] [MainThread] server_name set to: main.web.2
galaxy.web_stack DEBUG 2021-10-15 09:08:12,320 [p:856,w:2,m:0] [MainThread] Calling postfork function: <bound method ToolDocumentCache.reopen_ro of <galaxy.tools.cache.ToolDocumentCache object at 0x7f605d7c6588>>
sqlitedict INFO 2021-10-15 09:08:12,322 [p:856,w:2,m:0] [MainThread] opening Sqlite table 'unnamed' in '/galaxy-central/database/tool_cache/cache.sqlite'
galaxy.web_stack DEBUG 2021-10-15 09:08:12,391 [p:853,w:1,m:0] [MainThread] server_name set to: main.web.1
galaxy.web_stack DEBUG 2021-10-15 09:08:12,393 [p:853,w:1,m:0] [MainThread] Calling postfork function: <bound method ToolDocumentCache.reopen_ro of <galaxy.tools.cache.ToolDocumentCache object at 0x7f605d7c6588>>
sqlitedict INFO 2021-10-15 09:08:12,414 [p:853,w:1,m:0] [MainThread] opening Sqlite table 'unnamed' in '/galaxy-central/database/tool_cache/cache.sqlite'
galaxy.web_stack DEBUG 2021-10-15 09:08:12,626 [p:856,w:2,m:0] [MainThread] Calling postfork function: <bound method JobManager.start of <galaxy.jobs.manager.JobManager object at 0x7f605cc84d68>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,680 [p:856,w:2,m:0] [MainThread] Calling postfork function: <bound method DatabaseHeartbeat.start of <galaxy.model.database_heartbeat.DatabaseHeartbeat object at 0x7f605cc78898>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,802 [p:856,w:2,m:0] [MainThread] Calling postfork function: <bound method UWSGIApplicationStack.start of <galaxy.web_stack.UWSGIApplicationStack object at 0x7f605e9ed358>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,804 [p:856,w:2,m:0] [MainThread] Calling postfork function: <bound method GalaxyQueueWorker.bind_and_start of <GalaxyQueueWorker(Thread-1, initial daemon)>>
galaxy.queue_worker INFO 2021-10-15 09:08:12,807 [p:856,w:2,m:0] [MainThread] Binding and starting galaxy control worker for main.web.2
galaxy.web_stack DEBUG 2021-10-15 09:08:12,907 [p:853,w:1,m:0] [MainThread] Calling postfork function: <bound method JobManager.start of <galaxy.jobs.manager.JobManager object at 0x7f605cc84d68>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,910 [p:853,w:1,m:0] [MainThread] Calling postfork function: <bound method DatabaseHeartbeat.start of <galaxy.model.database_heartbeat.DatabaseHeartbeat object at 0x7f605cc78898>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,985 [p:853,w:1,m:0] [MainThread] Calling postfork function: <bound method UWSGIApplicationStack.start of <galaxy.web_stack.UWSGIApplicationStack object at 0x7f605e9ed358>>
galaxy.web_stack DEBUG 2021-10-15 09:08:12,987 [p:853,w:1,m:0] [MainThread] Calling postfork function: <bound method GalaxyQueueWorker.bind_and_start of <GalaxyQueueWorker(Thread-1, initial daemon)>>
galaxy.queue_worker INFO 2021-10-15 09:08:12,990 [p:853,w:1,m:0] [MainThread] Binding and starting galaxy control worker for main.web.1
galaxy.web_stack DEBUG 2021-10-15 09:08:13,001 [p:856,w:2,m:0] [MainThread] Calling postfork function: <function UniverseApplication.__init__.<locals>.<lambda> at 0x7f605ccdad90>
galaxy.queue_worker INFO 2021-10-15 09:08:13,006 [p:856,w:2,m:0] [MainThread] Queuing async task rebuild_toolbox_search_index for main.web.2.
galaxy.web_stack DEBUG 2021-10-15 09:08:13,318 [p:853,w:1,m:0] [MainThread] Calling postfork function: <function UniverseApplication.__init__.<locals>.<lambda> at 0x7f605ccdad90>
galaxy.queue_worker INFO 2021-10-15 09:08:13,325 [p:853,w:1,m:0] [MainThread] Queuing async task rebuild_toolbox_search_index for main.web.1.
galaxy.model.database_heartbeat DEBUG 2021-10-15 09:08:15,214 [p:856,w:2,m:0] [database_heartbeart_main.web.2.thread] main.web.2 is config watcher
galaxy.web_stack DEBUG 2021-10-15 09:08:15,615 [p:856,w:2,m:0] [MainThread] Calling postfork function: <built-in function register>
galaxy.web_stack DEBUG 2021-10-15 09:08:15,620 [p:856,w:2,m:0] [MainThread] Calling postfork function: <function postfork_setup at 0x7f605e9ebae8>
galaxy.web_stack INFO 2021-10-15 09:08:15,622 [p:856,w:2,m:0] [MainThread] Galaxy server instance 'main.web.2' is running
galaxy.queue_worker INFO 2021-10-15 09:08:15,723 [p:856,w:2,m:0] [Thread-1] Instance 'main.web.2' received 'rebuild_toolbox_search_index' task, executing now.
galaxy.tools.search DEBUG 2021-10-15 09:08:15,726 [p:856,w:2,m:0] [Thread-1] Starting to build toolbox index.
galaxy.web_stack DEBUG 2021-10-15 09:08:15,831 [p:853,w:1,m:0] [MainThread] Calling postfork function: <built-in function register>
galaxy.web_stack DEBUG 2021-10-15 09:08:15,834 [p:853,w:1,m:0] [MainThread] Calling postfork function: <function postfork_setup at 0x7f605e9ebae8>
galaxy.web_stack INFO 2021-10-15 09:08:15,883 [p:853,w:1,m:0] [MainThread] Galaxy server instance 'main.web.1' is running
Starting server in PID 499.
serving on http://127.0.0.1:8080
serving on uwsgi://127.0.0.1:4001
galaxy.util.watcher DEBUG 2021-10-15 09:08:15,907 [p:856,w:2,m:0] [database_heartbeart_main.web.2.thread] Watching for changes in directory (recursively): /galaxy-central/tool-data
galaxy.queue_worker INFO 2021-10-15 09:08:16,623 [p:853,w:1,m:0] [Thread-1] Instance 'main.web.1' received 'rebuild_toolbox_search_index' task, executing now.
galaxy.tools.search DEBUG 2021-10-15 09:08:16,639 [p:853,w:1,m:0] [Thread-1] Starting to build toolbox index.
galaxy.tools.search DEBUG 2021-10-15 09:08:17,314 [p:853,w:1,m:0] [Thread-1] Toolbox index finished (675.020 ms)
galaxy.tools.search DEBUG 2021-10-15 09:08:19,086 [p:856,w:2,m:0] [Thread-1] Toolbox index finished (3361.779 ms)

==> /home/galaxy/logs/handler1.log <==
galaxy.model.database_heartbeat DEBUG 2021-10-15 09:08:50,705 handler1 is not config watcher

==> /home/galaxy/logs/slurmctld.log <==
[2021-10-15T09:11:26.709] error: Could not open job state file /tmp/slurm/job_state: No such file or directory
[2021-10-15T09:11:26.710] error: NOTE: Trying backup state save file. Jobs may be lost!
[2021-10-15T09:11:26.710] No job state file (/tmp/slurm/job_state.old) found

@hexylena
Copy link
Contributor

hexylena commented Nov 9, 2021

I see this in the logs

migrate.versioning.repository DEBUG 2021-11-09 12:42:13,037 [p:1430,w:0,m:0] [MainThread] Config: OrderedDict([('db_settings', OrderedDict([('repository_id', 'Galaxy'), ('version_table', 'migrate_version'), ('required_dbs', '[]')]))])
Traceback (most recent call last):
  File "lib/galaxy/webapps/galaxy/buildapp.py", line 48, in app_factory
    app = galaxy.app.UniverseApplication(global_conf=global_conf, **kwargs)
  File "lib/galaxy/app.py", line 97, in __init__
    self._configure_models(check_migrate_databases=self.config.check_migrate_databases, check_migrate_tools=check_migrate_tools, config_file=config_file)
  File "lib/galaxy/config/__init__.py", line 1226, in _configure_models
    create_or_verify_database(db_url, config_file, self.config.database_engine_options, app=self, map_install_models=combined_install_database)
  File "lib/galaxy/model/migrate/check.py", line 140, in create_or_verify_database
    raise Exception("{}. {}{}".format(expect_msg, instructions, backup_msg))
Exception: Your database has version '1' but this code expects version '170'. Please backup your database and then migrate the database schema by running 'sh manage_db.sh upgrade'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hexylena @bgruening @marco-brandizi @shenweiyan