Generalize and make this test framework pluggable

[tlvu]

This repo basically allow to run jupyter notebooks from any external git repos and ensure expected output and exit code.

Current limitations:

1. Complex to add new repos containing notebooks.

   Have to edit several files (default_build_params, Jenkinsfile, downloadrepos, testall).

   Ideally all these files would be driven by only 1 config file.

2. No exclude list for notebooks. Can only enable/disable notebooks for a whole repo.

   Per repo toogle is in Jenkinsfile, forcing a commit if we want to enable/disable a repo later.

   Ideally the toogle would be configurable and not hardcoded in the code.

3. No pluggable mechanism to add new repos without having to touch the code.

   This means all the organizations deploying this test framework are running the exact same tests, without being able to customize their own list of notebooks to run.

   Ideally the list of repos and which notebooks to run by default would be provided by an externally (not committed here) config file which would allow different organizations to customize their own run.

4. No support to test private notebooks repos.

Related issue: #49

Once we have a new generic and pluggable framework, we would start a new "Jupyter-notebooks-e2e-tests" repo with this framework.

Edit:

• The matching Jenkins config https://github.com/Ouranosinc/jenkins-config fully pre-configured with jobs from this repo.
  • Have ability to plugin additional jobs but unable to change the list of repos or notebooks.
• Add "No support to test private notebooks repos."

[dbyrns]

I think that what we proposed here : bird-house/pavics-jupyter-base#2 (comment) will help to solve the current limitations :

1. Because the notebooks would be in images, the only concern of e2e would be to know on which image it should run the test. Adding repos is then only a matter of updating/switching images.
2. Depends on the goal here. If we want to cherry pick the notebooks, it does not help. But if we want to test for a specific topic only, the solution would be to have a test image for such topic and select it for the test.
3. Available images is already a per organization config, so the test would also apply to their images.
4. Private notebooks repos are not a big deal if the image is build by someone having access to them. But then the image must be private too? Again I'm not sure of the goal here, maybe for tests only? e2e could be provided with credentials to access to those images (weaver already have this functionnality to access private process image).

Still not a trivial task, but I think it provides a good fundation to solve this issue.

[tlvu]

I think that what we proposed here : bird-house/pavics-jupyter-base#2 (comment) will help to solve the current limitations :

(...)

Still not a trivial task, but I think it provides a good fundation to solve this issue.

Indeed a foundation to build upon.

I believe the notebooks are not shipped together with the image but the image have the scripts to download them so this script can be called many times to keep the notebooks up-to-date without having to rebuild the full image just to get updated notebooks.

So adding new notebooks is deferred to the image builder/owner, basically solve point 3 of adding new notebooks repos.

For the rest, like you said "still a not trivial task".

[dbyrns]

I believe the notebooks are not shipped together with the image but the image have the scripts to download them so this script can be called many times to keep the notebooks up-to-date without having to rebuild the full image just to get updated notebooks.

You're right, technically the image does not contain the notebooks, but from the e2e it does not matter as the image would be responsible to provide them.

[tlvu]

You're right, technically the image does not contain the notebooks, but from the e2e it does not matter as the image would be responsible to provide them.

Sorry, I should have referenced the proper reply. It's meant to clarify "But then the image must be private too?" So if the private notebooks are not part of the image, then the image do not have to be private. Also imply the download script need to be able to handle private repos to get those private notebooks.

[dbyrns]

Hum ok, that's a good point. For these cases images would required credentials and the complexity increases.