Documentation compilation is broken

[isms]

The new Docker services introduced by #55 to compile documentation are not currently working.

Expected Behavior

We expect on docker-compose -f local.yml up --build compile_docs for the documentation to build properly.

Current Behavior

We currently get numerous stack traces:

Attaching to concepttoclinic_compile_docs_1
Attaching to concepttoclinic_compile_docs_1
compile_docs_1   make: Entering directory '/app/docs'
compile_docs_1   sphinx-apidoc -f -o apidoc_interface/ ..//interface/
compile_docs_1   Creating file apidoc_interface/manage.rst.
compile_docs_1   Creating file apidoc_interface/backend.rst.
compile_docs_1   Creating file apidoc_interface/backend.api.rst.
compile_docs_1   Creating file apidoc_interface/backend.cases.rst.
compile_docs_1   Creating file apidoc_interface/backend.cases.migrations.rst.
compile_docs_1   Creating file apidoc_interface/backend.images.rst.
compile_docs_1   Creating file apidoc_interface/backend.images.migrations.rst.
compile_docs_1   Creating file apidoc_interface/backend.static.rst.
compile_docs_1   Creating file apidoc_interface/config.rst.
compile_docs_1   Creating file apidoc_interface/config.settings.rst.
compile_docs_1   Creating file apidoc_interface/modules.rst.
compile_docs_1   sphinx-apidoc -f -o apidoc_prediction/ ..//prediction/
compile_docs_1   Creating file apidoc_prediction/config.rst.
compile_docs_1   Creating file apidoc_prediction/src.rst.
compile_docs_1   Creating file apidoc_prediction/src.algorithms.rst.
compile_docs_1   Creating file apidoc_prediction/src.algorithms.classify.rst.
compile_docs_1   Creating file apidoc_prediction/src.algorithms.identify.rst.
compile_docs_1   Creating file apidoc_prediction/src.algorithms.segment.rst.
compile_docs_1   Creating file apidoc_prediction/src.preprocess.rst.
compile_docs_1   Creating file apidoc_prediction/src.tests.rst.
compile_docs_1   Creating file apidoc_prediction/modules.rst.
compile_docs_1   
compile_docs_1   apidoc build finished. The api source files are in /
compile_docs_1   Running Sphinx v1.5.1
compile_docs_1   loading pickled environment... done
compile_docs_1   building [mo]: targets for 0 po files that are out of date
compile_docs_1   building [html]: targets for 0 source files that are out of date
compile_docs_1   updating environment: 0 added, 8 changed, 0 removed
compile_docs_1   reading sources... [ 12%] apidoc_interface/backend.api
compile_docs_1   reading sources... [ 25%] apidoc_interface/backend.cases
compile_docs_1   reading sources... [ 37%] apidoc_interface/backend.cases.migrations
compile_docs_1   reading sources... [ 50%] apidoc_interface/backend.images
compile_docs_1   reading sources... [ 62%] apidoc_interface/backend.images.migrations
compile_docs_1   reading sources... [ 75%] apidoc_interface/backend.static
compile_docs_1   reading sources... [ 87%] apidoc_interface/config
compile_docs_1   reading sources... [100%] apidoc_interface/config.settings
compile_docs_1   
compile_docs_1   /app/docs/apidoc_interface/backend.api.rst:7: WARNING: toctree contains reference to nonexisting document 'apidoc_interface/backend.api.migrations'
compile_docs_1   /app/docs/apidoc_interface/backend.api.rst:17: WARNING: autodoc: failed to import module 'backend.api.apps'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/api/apps.py", line 1, in <module>
compile_docs_1       from django.apps import AppConfig
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.api.rst:25: WARNING: autodoc: failed to import module 'backend.api.serializers'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/api/serializers.py", line 1, in <module>
compile_docs_1       from rest_framework import serializers
compile_docs_1   ModuleNotFoundError: No module named 'rest_framework'
compile_docs_1   /app/docs/apidoc_interface/backend.api.rst:33: WARNING: autodoc: failed to import module 'backend.api.tests'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/api/tests.py", line 1, in <module>
compile_docs_1       from django.test import (
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.api.rst:41: WARNING: autodoc: failed to import module 'backend.api.urls'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/api/urls.py", line 1, in <module>
compile_docs_1       from django.conf.urls import (
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.api.rst:49: WARNING: autodoc: failed to import module 'backend.api.views'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/api/views.py", line 1, in <module>
compile_docs_1       from rest_framework import viewsets
compile_docs_1   ModuleNotFoundError: No module named 'rest_framework'
compile_docs_1   /app/docs/apidoc_interface/backend.cases.rst:17: WARNING: autodoc: failed to import module 'backend.cases.apps'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/cases/apps.py", line 1, in <module>
compile_docs_1       from django.apps import AppConfig
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.cases.rst:25: WARNING: autodoc: failed to import module 'backend.cases.factories'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/cases/factories.py", line 1, in <module>
compile_docs_1       import factory
compile_docs_1   ModuleNotFoundError: No module named 'factory'
compile_docs_1   /app/docs/apidoc_interface/backend.cases.rst:33: WARNING: autodoc: failed to import module 'backend.cases.models'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/cases/models.py", line 1, in <module>
compile_docs_1       from django.core.validators import (
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.cases.rst:41: WARNING: autodoc: failed to import module 'backend.cases.tests'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/cases/tests.py", line 1, in <module>
compile_docs_1       from django.test import TestCase
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.cases.migrations.rst:10: WARNING: autodoc: failed to import module 'backend.cases.migrations.0001_initial'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/cases/migrations/0001_initial.py", line 5, in <module>
compile_docs_1       import django.core.validators
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.images.rst:17: WARNING: autodoc: failed to import module 'backend.images.apps'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/images/apps.py", line 1, in <module>
compile_docs_1       from django.apps import AppConfig
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.images.rst:25: WARNING: autodoc: failed to import module 'backend.images.factories'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/images/factories.py", line 1, in <module>
compile_docs_1       import factory
compile_docs_1   ModuleNotFoundError: No module named 'factory'
compile_docs_1   /app/docs/apidoc_interface/backend.images.rst:33: WARNING: autodoc: failed to import module 'backend.images.models'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/images/models.py", line 1, in <module>
compile_docs_1       from django.db import models
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.images.rst:41: WARNING: autodoc: failed to import module 'backend.images.tests'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/images/tests.py", line 1, in <module>
compile_docs_1       from django.test import TestCase
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.images.migrations.rst:10: WARNING: autodoc: failed to import module 'backend.images.migrations.0001_initial'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/images/migrations/0001_initial.py", line 5, in <module>
compile_docs_1       from django.db import migrations, models
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.images.migrations.rst:18: WARNING: autodoc: failed to import module 'backend.images.migrations.0002_imageseries_patient_id'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/images/migrations/0002_imageseries_patient_id.py", line 5, in <module>
compile_docs_1       from django.db import migrations, models
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.static.rst:10: WARNING: autodoc: failed to import module 'backend.static.apps'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/static/apps.py", line 1, in <module>
compile_docs_1       from django.apps import AppConfig
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.static.rst:18: WARNING: autodoc: failed to import module 'backend.static.context_processors'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/static/context_processors.py", line 1, in <module>
compile_docs_1       from django.conf import settings
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.static.rst:26: WARNING: autodoc: failed to import module 'backend.static.tests'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/static/tests.py", line 1, in <module>
compile_docs_1       from django.test import TestCase
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.static.rst:34: WARNING: autodoc: failed to import module 'backend.static.urls'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/static/urls.py", line 1, in <module>
compile_docs_1       from django.conf.urls import url
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/backend.static.rst:42: WARNING: autodoc: failed to import module 'backend.static.views'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1     File "/app/interface/backend/static/views.py", line 1, in <module>
compile_docs_1       from django.shortcuts import render
compile_docs_1   ModuleNotFoundError: No module named 'django'
compile_docs_1   /app/docs/apidoc_interface/config.rst:17: WARNING: autodoc: failed to import module 'config.urls'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1   ModuleNotFoundError: No module named 'config.urls'; 'config' is not a package
compile_docs_1   /app/docs/apidoc_interface/config.rst:25: WARNING: autodoc: failed to import module 'config.wsgi'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1   ModuleNotFoundError: No module named 'config.wsgi'; 'config' is not a package
compile_docs_1   /app/docs/apidoc_interface/config.settings.rst:10: WARNING: autodoc: failed to import module 'config.settings.base'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1   ModuleNotFoundError: No module named 'config.settings'; 'config' is not a package
compile_docs_1   /app/docs/apidoc_interface/config.settings.rst:18: WARNING: autodoc: failed to import module 'config.settings.local'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1   ModuleNotFoundError: No module named 'config.settings'; 'config' is not a package
compile_docs_1   /app/docs/apidoc_interface/config.settings.rst:26: WARNING: autodoc: failed to import module 'config.settings.production'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1   ModuleNotFoundError: No module named 'config.settings'; 'config' is not a package
compile_docs_1   /app/docs/apidoc_interface/config.settings.rst:34: WARNING: autodoc: failed to import module 'config.settings.test'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1   ModuleNotFoundError: No module named 'config.settings'; 'config' is not a package
compile_docs_1   /app/docs/apidoc_interface/config.settings.rst:43: WARNING: autodoc: failed to import module 'config.settings'; the following exception was raised:
compile_docs_1   Traceback (most recent call last):
compile_docs_1     File "/usr/local/lib/python3.6/site-packages/sphinx/ext/autodoc.py", line 547, in import_object
compile_docs_1       __import__(self.modname)
compile_docs_1   ModuleNotFoundError: No module named 'config.settings'; 'config' is not a package
compile_docs_1   looking for now-outdated files... none found
compile_docs_1   pickling environment... done
compile_docs_1   checking consistency... /app/docs/configure git-lfs.md:: WARNING: document isn't included in any toctree
compile_docs_1   /app/docs/templates/algorithm_doc_template.md:: WARNING: document isn't included in any toctree
compile_docs_1   done
compile_docs_1   preparing documents... done
compile_docs_1   writing output... [  9%] apidoc_interface/backend
compile_docs_1   writing output... [ 18%] apidoc_interface/backend.api
compile_docs_1   writing output... [ 27%] apidoc_interface/backend.cases
compile_docs_1   writing output... [ 36%] apidoc_interface/backend.cases.migrations
compile_docs_1   writing output... [ 45%] apidoc_interface/backend.images
compile_docs_1   writing output... [ 54%] apidoc_interface/backend.images.migrations
compile_docs_1   writing output... [ 63%] apidoc_interface/backend.static
compile_docs_1   writing output... [ 72%] apidoc_interface/config
compile_docs_1   writing output... [ 81%] apidoc_interface/config.settings
compile_docs_1   writing output... [ 90%] apidoc_interface/modules
compile_docs_1   writing output... [100%] index
compile_docs_1   
compile_docs_1   generating indices... genindex py-modindex
compile_docs_1   writing additional pages... search
compile_docs_1   copying static files... done
compile_docs_1   copying extra files... done
compile_docs_1   dumping search index in English (code: en) ... done
compile_docs_1   dumping object inventory... done
compile_docs_1   build succeeded, 31 warnings.
compile_docs_1   
compile_docs_1   Build finished. The HTML pages are in _build/html.
compile_docs_1   Sphinx build finished. The build files are in _build/
compile_docs_1   make: Leaving directory '/app/docs'
concepttoclinic_compile_docs_1 exited with code 0

Possible Solution

There seem to be two problems:

1. Packages are not being installed, e.g. Django. See for example:

   compile_docs_1       from django.test import TestCase
   compile_docs_1   ModuleNotFoundError: No module named 'django'
   

2. The presence or absence of __init__.py are cause module errors.

Steps to Reproduce

1. Make sure you are up to date with master branch.
2. Rebuild and run the compile_docs container with docker-compose -f local.yml up --build compile_docs

[isms]

@musale Any chance you can look into this?

[musale]

Hi @isms
I will definitely have a look. Off the bat I can see that building the container using the prediction app requirements is a reason why there are missing packages. Let me try and reproduce on my end then I'll patch this up.

[ask7]

The problem is that the docs rely on Django, but it is not present in the prediction/requirements.
Using
dockerfile: ./compose/interface/Dockerfile-dev
instead works. Maybe the docs should have their own requirements file?

Also, may I ask why the "restart: always" was added (I'm new to Docker)? With the above error, it caused the build to run in an endless loop for me, because it tried to restart on every failure.

[musale]

Hello @ask7 I tried using the interface app Dockerfile and I came across some unsuccessful scenarios:

When I run:

$ docker-compose -f local.yml up --build compile_docs

Everything builds but the container won't finish building because the interface app Dockerfile starts the Django app and sleeps until a connection to Postgres is established.

compile_docs_1   | Postgres is unavailable - sleeping
compile_docs_1   | Postgres is unavailable - sleeping

When I run:

$ docker-compose -f local.yml up

Everything builds but I get

compile_docs_1   |   File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 39, in _setup
compile_docs_1   |     % (desc, ENVIRONMENT_VARIABLE))
compile_docs_1   | django.core.exceptions.ImproperlyConfigured: Requested settings, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
compile_docs_1   | The full traceback has been saved in /tmp/sphinx-err-0j6r37rq.log, if you want to report the issue to the developers.
compile_docs_1   | Please also report this if it was a user error, so that a better error message can be provided next time.
compile_docs_1   | A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
compile_docs_1   | make: *** [html] Error 1
compile_docs_1   | Makefile:29: recipe for target 'html' failed

current state

I am separating the documentation docker configs and it's requirements from interface app but so far, I am still getting the above error. Do you have any idea on how to set the DJANGO_SETTINGS_MODULE during docs compilation? I tried:

import os
os.envrion['DJANGO_SETTINGS_MODULE'] = 'interface.config.settings...'

But that doesn't seem to recognize the interface app. I will update when I make progress on this.

restart: always

Since the documentation is done in two separate containers i.e. compile_docs which compiles the files into "serveable" html files and documentation which renders the files using a simple python server. When you edit a document, it can be synced into the compile_docs container through volumes but it won't be compiled into a ready html file unless you manually restart the container.

[ask7]

@musale Thanks for explaining restart: always!

I think it's a good idea to separate the two requirements and configs. Did you also set

environment: - DJANGO_SETTINGS_MODULE=config.settings.local

? I'm not sure you can simply use the settings files of interface, as they reference local directories like backend. I tried setting up the docs/config/settings/... locally, but ATM it doesn't recognize config for some reason...

[isms]

As of fc898d4 the requirements have been untangled, but somebody needs to take a look at the actual building part and decide where the logic should live.