Skip to content
This repository has been archived by the owner on Jun 26, 2018. It is now read-only.

How to create a workspace

Jump to bottom
Marc Streit edited this page Oct 26, 2017 · 1 revision

Table of Contents

Goals

This tutorial shows how to create a workspace in which plugins and extensions for the datavisyn Target Discovery Platform (TDP) can be developed. The workspace enables developers to compile the current codebase into a running application.

Prerequisites

This tutorial requires to complete the following steps:

  • Install Docker and Docker Compose (If running a Windows version < 10 Pro + HyperV: install the Docker Toolbox instead. See Using Docker Toolbox (for Windows version < 10 Pro + HyperV for more information
  • Install Node.js version >= 6
  • Install Yeoman + Generator-phovea (i.e., sudo npm install -g yo github:phovea/generator-phovea)

Summary

Install software from Prerequisites section

sudo npm install -g yo https://github.com/phovea/generator-phovea

Docker Toolbox (if needed)

When using the Docker Toolbox install the workspace under C:\Users\... because otherwise the VM cannot access the host’s folder structure. Furthermore the host’s port 127.0.0.1:9000 must manually be forwarded to the guest’s port 9000. See Using Docker Toolbox (for Windows version < 10 Pro + HyperV).

Docker Toolbox + Proxy

If the computer on which the workspace should be created is behind a proxy server and Docker Toolbox is used, store them inside the Docker Toolbox start script under C:\Program Files\Docker Toolbox\start.sh:

export HTTP_PROXY="http://hostname:port/"
export HTTPS_PROXY="http://hostname:port/"

docker-machine restart default

Workspace creation

yo phovea:setup-workspace example_product

If a proxy server needs to be configured, change the respective proxy settings inside the docker-compose-patch.yml file:

...
environment:
http_proxy=yourproxylocation
https_proxy=yourproxylocation
...

After configuring the proxy server the workspace needs to be updated, such that the proxy settings are copied into the docker-compose.yml file:

yo phovea:update

Then restart the build process:

docker-compose build

Workspace update

./forEach git pull

yo phovea:update

Tutorial

Using Docker Toolbox (for Windows version lower than 10 Pro + HyperV)

Versions of Windows < 10 Pro + HyperV support, require Docker Toolbox to be installed. In this setup Docker will run inside a virtual machine using VirtualBox, which will be installed by Docker Toolbox. By default, only Windows C:\Users\... directory is shared with the virtual machine, therefore has the workspace to be created somewhere inside the Users directory, otherwise the installation will fail. Another important step is to manually forward the host’s port 9000 (which is the port the API Docker container listens to) to the guest’s port 9000. This is done with the following steps:

  • Open the VirtualBox
  • Settings
  • Network
  • In Adapter 1 (selected by default) click Advanced
  • Port Forwarding
  • Add a new rule with the following settings:
    • Protocol: TCP
    • Host IP: 127.0.0.1
    • Host Port: 9000
    • Guest IP: leave empty
    • Guest Port: 9000
    • Confirm

The virtual machine is started automatically when the Docker Toolbox shell is opened. The virtual machine is by default called default. The virtual machine can be stopped again with the following command (with the name of the VM as the last parameter):

docker-machine stop default

To restart the VM execute:

docker-machine restart default

When using the Docker Toolbox the directory to place the workspace in must be in C:\Users\... and the port 9000 needs to be forwarded manually.

Running Docker behind a Proxy Server

When installing the workspace from a company network with a proxy server the proxy settings for Docker need to be stored in two places. First, Docker Toolbox (if used) needs to know the proxy settings such that the VM can access the Internet and download the required packages. To add them, open C:\Program Files\Docker Toolbox\start.sh. This file needs to be changed to contain the following two lines:

export HTTP_PROXY=yourproxylocation
export HTTPS_PROXY=yourproxylocation

After adding the proxy settings to the start script, the VM needs to be restarted.

The second location to store the proxy settings is the docker-compose-patch.yml file, which is created after setting up the workspace (see Setting up the workspace first) in the root folder. Despite other configuration it contains proxy settings for the API and the server service, which look like in the following listing:

...
environment:
http_proxy=yourproxylocation
https_proxy=yourproxylocation
...

So both occurrences of the environment proxy settings need to be changed. Afterwards the workspace needs to be updated by using the generator’s update command:

yo phovea:update

If the containers are already running in background mode (with the -d option) restart them by running:

docker-compose restart

Or, if the containers are running in foreground mode restart by hitting CTRL-C and execute:

docker-compose up

Setting up the workspace

To set up a workspace open a terminal window and navigate to the folder in which a new workspace should be created. The following command shows how Phovea’s Yeoman generator can be used to execute all needed steps to create the new workspace.

yo phovea:setup-workspace Caleydo/example_product

Attention!

Replace the product definition (“Caleydo/example_product”) with a real one.

The first part of the command (yo) is the Yeoman command-line tool. The second part tells Yeoman that it should use the “phovea” generator and execute “setup-workspace” sub-generator. The third part is the product definition from which the workspace should be created. A prompt will appear asking if the dependencies should be cloned via SSH or HTTPS. Cloning via HTTPS should work everywhere, even behind a proxy, but it requires to authenticate via username and password to access private git repositories / plugins. When cloning via SSH authentication is handled automatically with private and public keys, however when being behind a proxy this method of cloning may not work. See also the documentation on GitHub.

This command will create a new folder, which is named after the product without the _product suffix (in this case example). Inside this directory all needed plugins will be cloned. Cloning private repositories requires having the correct access rights and authentication either with SSH or username and password. After cloning all needed repositories their dependencies are collected and merged into the corresponding file inside the root directory of the workspace (for more information see the Folder structure section). The generator then executes the command npm install to install the node modules. Finally the docker containers will be downloaded and installed. The corresponding command to achieve this is docker-compose build. Building the containers can fail when being behind a proxy server because Docker does not know anything about the proxy settings. If this is the case, please refer to Running Docker behind a Proxy Server. When the build fails it can be restarted using the following command inside the workspace:

docker-compose build

The workspace can now be opened using PyCharm.

Folder structure

┣━ Root folder (e.g. example)
┃    ┣━ _backup/
┃    ┣━ _data/
┃    ┣━ ordino/
┃    ┣━ phovea_server/
┃    ┣━ phovea_clue/
┃    ┣━ …
┃    ┣━ phovea.js
┃    ┣━ phovea_registry.js
┃    ┣━ package.json
┃    ┣━ requirements.txt
┃    ┣━ docker-compose.yml
┃    ┣━ docker-compose-patch.yml
┃    ┣━ forEach.sh
┃    ┣━ forEach.cmd
┃    ┣━ docker-backup.sh
┃    ┣━ docker-backup.cmd
┃    ┣━ …

The folders inside the workspace root are plugins. An exception to this are folders starting with an underscore (_), which are not included in the workspace by default. These folders are not plugins but the place to put in database dumps (_backup) to be restored or other files, which should be served by the web server (_data). The created workspace contains some plugins by default, which are needed for the specified product. Many of the files (e.g., package.json, requirements.txt, docker-compose.yml, ...) are automatically generated. For example, each plugin maintains its own package.json file, but when installing the node modules for the workspace a package.json file, which combines the dependencies of all plugins, is needed. After changing one of these files the workspace needs to be updated (see Updating the workspace) such that the global workspace files include the latest changes. This step is done by default when creating the workspace.

Attention!

The folders beginning with an underscore (_) are not part of the default workspace.

Restoring a database dump

The workspace contains a helper script for both Windows and Linux to restore a database dump. Place the dump under _backup and execute the following command:

./docker-backup restore

Updating the workspace

A workspace’s auto-generated files are created using the corresponding files inside each plugin. So for example the package.json file is created by taking the package.json file of each plugin and combining them. It is important to update the workspace whenever one of these files (e.g. package.json, requirements.txt, ...) changes inside a plugin, because otherwise the changes will not be available in the compiled application.

The workspace contains the forEach.sh shell script, which is used to update all plugins to the latest version. It will run through all plugins and execute git pull on the current branch.

./forEach git pull

To update the workspace’s auto-generated files, the Phovea’s Yeoman generator can be used again:

yo phovea:update

This command will check which files have changed and ask if they should be replaced.

Running the application

To compile and run the application inside a workspace, the Docker containers need to be started first because they contain the backend. The frontend application can be started using the NPM start script.

docker-compose up -d
npm start

The first command will start docker in the background (-d flag) and the current terminal window is free to execute the NPM start script. The logs from the docker containers can be printed by executing the following command: docker-compose logs

Alternatively, docker can also be started in the foreground, where it shows the logs instantly. In this case another terminal window has to be used to start the application via the NPM start script.

docker-compose up

Usually, the application runs on localhost:8080, but in case port 8080 is already occupied the server will use the next open port (e.g., 8081). The information on which port the application is running can be seen at the beginning of the output of the start script.

Shutting the application down

First shut down the NPM start script by clicking on the terminal window that runs the script and press CTRL+C. If docker-compose is running in the foreground, it can be stopped using the same steps, otherwise it can be stopped using:

docker-compose stop

If the Docker Toolbox is used, the virtual machine needs to be stopped by executing:

docker-machine stop default

Conclusion

Lessons learned:

  • How to configure proxy settings
  • How to use the Docker Toolbox when using an Operating System Docker does not run natively on
  • How to use Phovea’s Yeoman generator to create a workspace from a product definition
  • What the most important components of the folder structure are
  • How to update the workspace whenever auto-generated files (e.g., package.json, requirements.txt, …) change
  • How to run the application

Further reading

Clone this wiki locally