The repository for the "Online Membership System" (OMS), an open-source intranet project of the student/youth association AEGEE-Europe.
It makes use of docker, and docker-compose.
A short perspective: the educational value of this project. Head to roadmap.sh and see all you could learn thanks to this project (almost everything, short of blockchain):
- software architecture
- devops
- backend
- cybersecurity
- frontend
- QA
- ... and more ;)
Install
- First Git (you might already have Git installed through other sources if you worked with git/GitHub before)
- then Virtualbox,
- and finally Vagrant.
Even if you have a linux box, this is very recommended. If you decide to not do it, sigh... but don't come to cry to us.
If you decide you know better than us, install docker and docker-compose on your Windows/Linux/Mac machine, instead of Virtualbox and Vagrant. (Make sure you install the correct versions, they can be found in the provisioning scripts -- also, Mac and linux have different versions of grep so again, your problem ;-) )
Note: if you use Vagrant, Docker will be already automatically on the virtual machine.
Memory requirements for the VM bootstrapped with Vagrant: 2GB (i.e. you need a machine with at least 3GB physical RAM)
Explanation of the installation are here. Explanation of why we're doing it this way is at the bottom.
A note on terminology:
- The computer you run Virtualbox on is the HOST.
- The created VM which runs Docker is the GUEST.
NOTE on URL MAPPING: to be able to use advanced features, the hosts file has to be edited. The procedure is handled by a script (both for Linux and Windows machine) provided in the repo. Details are explained below.
For manual edits, see Advanced URL mapping and troubleshoot.
On the HOST
git clone --recursive https://github.com/AEGEE/MyAEGEE.git
cd MyAEGEE
./start.sh
URL MAPPING for Linux: You don't have to do anything, the mapping is handled by start.sh.
You will have to wait for up to 20'. A message appears when the bootstrap completes, and you can check if it works in the ways described in the Usage section. Note that sometimes there WILL be red output, but it is not necessarily indicator of an error (unless it's the VERY last message, and it begins with the word "ERROR")
See below for explanation of start.sh
Foreword: if you want to learn WSL (windows subsystem for linux) and help us improve the next steps, you're most welcome!
On the HOST
git clone --recursive https://github.com/AEGEE/MyAEGEE.git
cd MyAEGEE
URL MAPPING for Windows:
As a helper in the windows case, you have the script "run_as_win_administrator.bat" (not very advanced). When you open it, it will tell you what to do, which is written below. To open it:
- To open it you have to right-click it and click "run as administrator".
- It will open the file you need to edit in notepad, AND open a different terminal window that will tell you the line to copy (which you can find below).
- Paste the content at the last line of the file
192.168.168.168 appserver.test my.appserver.test traefik.appserver.test portainer.appserver.test pgadmin.appserver.test - Delete the file called
Vagrantfileand renameVagrantfile.windowsintoVagrantfile - Save, and exit.
For any troubleshoot, see Advanced URL mapping and troubleshoot.
Once set up the mapping, you can continue the installation:
vagrant up
You will have to wait for up to 25 minutes. A message appears when the bootstrap completes, and you can check if it works in the ways described in the Usage section.
MANUAL EDIT
If the script above did not work, you can also manually edit the /etc/hosts file on the HOST machine (on Windows: C:\Windows\system32\drivers\etc\hosts) to add the entry:
Vagrant case: 192.168.168.168 appserver.test my.appserver.test traefik.appserver.test portainer.appserver.test pgadmin.appserver.test
Pure docker case: 127.0.0.1 appserver.test my.appserver.test traefik.appserver.test portainer.appserver.test pgadmin.appserver.test
Windows write-permission issue For security reason, Windows could have rescrited writing permession. A workaround (original source) is to copy the hosts file to a different location:
- Go to
C:\Windows\system32\drivers\etc\hostsand locatehostsfile. - Copy it to your Desktop, or any other folder that you can easily access.
- Open the
hostsfile on your Desktop with Notepad or any other text editor. - Make the necessary changes (see above) and move the hosts file back to
C:\Windows\system32\drivers\etc\hostsdirectory.
Everything related to the behaviour of the app is defined in the top-most .env file. Most important parameters are:
ENABLED_SERVICES: telling which parts of the system are enabled
MYAEGEE_ENV: telling in which mode the system is run
<servicename>_SUBDOMAIN: telling how to access a specific service
See below for more info.
After launching the system, you have two ways to check everything is working:
- on the HOST
# on the _HOST_ you run the following
vagrant ssh
# ...which connects you to the _GUEST_, where docker is.
Let's run the commands and see that they should yield an output like
username@computername:~/Documents/aegee/MyAEGEE$ vagrant ssh
Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-111-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information disabled due to load higher than 1.0
* Kubernetes 1.19 is out! Get it in one command with:
sudo snap install microk8s --channel=1.19 --classic
https://microk8s.io/ has docs and details.
97 packages can be updated.
70 updates are security updates.
This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento
Last login: Fri Sep 18 14:13:43 2020 from 10.0.2.2
appserver[/vagrant]$ [stable !?]
So now you can run docker ps:
appserver[/vagrant]$ [stable !?] docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cbe689ea9ee2 aegee/discounts:dev "docker-entrypoint.s…" 5 hours ago Up 5 hours (healthy) 8084/tcp myaegee_discounts_1
ed1c20a4e65a aegee/statutory:dev "docker-entrypoint.s…" 5 hours ago Up 5 hours (healthy) 8084/tcp myaegee_statutory_1
f60476fa4a6c aegee/events:dev "docker-entrypoint.s…" 5 hours ago Up 5 hours (healthy) 8084/tcp myaegee_events_1
06add303acd0 aegee/core:dev "docker-entrypoint.s…" 5 hours ago Up 5 hours (healthy) 8084/tcp myaegee_core_1
72a7cb2d4100 postgres:10 "docker-entrypoint.s…" 5 hours ago Up 5 hours 5432/tcp myaegee_postgres-statutory_1
316ced849d6f postgres:10 "docker-entrypoint.s…" 5 hours ago Up 5 hours 5432/tcp myaegee_postgres-core_1
f3110b11d9c6 postgres:10 "docker-entrypoint.s…" 5 hours ago Up 5 hours 5432/tcp myaegee_postgres-discounts_1
9985f6cd3042 postgres:10 "docker-entrypoint.s…" 5 hours ago Up 5 hours 5432/tcp myaegee_postgres-events_1
623356e7ef0d dpage/pgadmin4:4.23 "/entrypoint.sh" 6 hours ago Up 6 hours 80/tcp, 443/tcp, 5050/tcp myaegee_pgadmin_1
0728da470558 erikdubbelboer/phpredisadmin:v1.11.4 "tini -- php -S 0.0.…" 6 hours ago Up 6 hours 80/tcp myaegee_redisadmin_1
8263f843df15 portainer/portainer:1.22.1 "/portainer -H unix:…" 6 hours ago Up 6 hours 9000/tcp myaegee_portainer_1
d056ce2c92e3 swaggerapi/swagger-ui:v3.28.0 "/docker-entrypoint.…" 6 hours ago Up 6 hours 80/tcp, 8080/tcp myaegee_swagger_1
4ee964dede44 traefik:v1.7.4-alpine "/entrypoint.sh --we…" 6 hours ago Up 6 hours (healthy) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp myaegee_traefik_1
30963e2faf63 aegee/portal:dev "docker-php-entrypoi…" 6 hours ago Up 6 hours 80/tcp myaegee_portal_1
49485e19dc87 aegee/frontend:dev "nginx" 6 hours ago Up 6 hours 80/tcp, 8083/tcp myaegee_frontend_1
e81bcc38bf76 aegee/nginx-static:latest "nginx -g 'daemon of…" 6 hours ago Up 6 hours (healthy) 80/tcp myaegee_statutory-static_1
dc2ea7b33c97 aegee/nginx-static:latest "nginx -g 'daemon of…" 6 hours ago Up 6 hours (healthy) 80/tcp myaegee_events-static_1
what do we see here? That we can connect to vagrant (we are inside the virtual machine with vagrant ssh), and that a bunch of containers are up (we ask the docker containers with docker ps). We'll see later that we should use some commands (make start) on the GUEST, so knowing how to access the virtual machine is important.
HOORAY! YOUR SYSTEM IS UP!
For any issue, see Troubleshooting
- you can navigate to it in your HOST web browser.
For accessing it, the three most important URLs (NB there is no https for development):
| Service | URL | Description |
|---|---|---|
| The app (MyAEGEE) | http://my.appserver.test | What you're here for |
| Traefik | http://traefik.appserver.test | Quick test to see if everything works well |
| Portainer | http://portainer.appserver.test | Visual docker manager |
Do you connect to any of these three URLs without troubles? HOORAY! YOUR SYSTEM IS UP!
See right below for the URL of extra services.
See at the bottom for the default credentials of MyAEGEE's fresh install.
read "subdomain.appserver.test"; e.g. you put in your HOST browser http://traefik.appserver.test
| Subdomain | What | Container |
|---|---|---|
| my | MyAEGEE | frontend |
| portainer | Easier container mgmt (development only) (under login) | portainer |
| traefik | Traefik's dashboard (under login) | traefik |
| pgadmin | Administration of databases (development only) (under login) | pgadmin |
| www | Website | wordpress |
| wiki | AEGEE's Wiki, the backbone of knowledge | mediawiki |
You can customise these subdomains by editing the .env file as mentioned above, and relaunching the script (see below about Makefile).
FIXME: For more detailed usage guides see this usage tips page. For container-specific usage guides see the container's repository.
(At the end of this section there will be links to detailed explanations, don't panic!)
There is a file called Makefile that gives some easy shortcut to do stuff. This must be launched on the GUEST.
On first run of vagrant, the bootstrap target will be invoked (you don't need to do it). If you are stubborn and decide to not use Vagrant, you still don't have to invoke it (it is invoked by start.sh)
The general flow is that once you edit the .env file, make start should be run (on the GUEST) to update the running configuration.
You can invoke the easy scripting in the following way (this shell command must be run in the same folder of the Makefile):
On the GUEST
| Command | What |
|---|---|
| make bootstrap | (init, build, start) in this order. (Run only the first time by vagrant/start.sh) |
| make init | Initialise the system (most likely you don't need to launch this) |
| make build | Build the containers registered in the .env file |
| make start | Run the containers registered in the .env file |
| make monitor | If you didn't enable kibana, then you may want to have a look at the logs through this |
| make live-refresh | Updates the containers to the new version (if any) and restarts them |
| make stop/restart/hard-restart | Just don't use them on the server, EVER |
| make bump | Only for development: updates the submodules |
Guest? Host? wtf? read the under the hood section, and the difference between start.sh and makefile.
For now, if one wants to follow some specific logs, they have to invoke helper.sh manually e.g.
./helper.sh --monitor container1 container2...containerN
Likewise, for now if one wants to execute a command on a container they have to invoke helper.sh manually e.g.
./helper.sh --execute containername command
It's your time to shine! Your system is set up, now it's up to you to create a new service, to improve an existing one, or to play with integrations leveraging existing services (e.g. wordpress, magento...). See the moving parts section that explain how to configure the system after you add something new.
For better development experience: the files in your HOST folder MyAEGEE are mounted inside the GUEST (location /vagrant), which means that if you create a file in the GUEST /vagrant folder you will find it in your HOST <your cloned dir>/MyAEGEE, and vice versa if you edit a file in your repository it is reflected in the VM. This way you can use the IDE you use normally on the system you use normally, without the need of getting accustomed to some new tool!
Make sure however to use an extension in your IDE called editorconfig! It will avoid pains especially if your HOST is a windows machine.
- change subdomain name, instead of accessing the app at
my.appserver.testmake itimthebest.appserver.test - change the top left logo of AEGEE-Europe to some other (small enough) logo
- change background colour, instead of a white one make it green (and I don't mean by going to "inspect element" and changing it temporarily!)
- change content and/or style of the footer
- change order/remove one of the microservices from the menubar on the left
- In resources > resources, add another box linking to your antenna's website
- Add an user to an antenna/circle (this is less programming-wise, more getting to know the platform)
- Change the email address of the first 5 users generated: instead of @example.com, make them @example.org
- Core-specific: change how many results you can visualise at a time
- Events-specific: in the Board view add an emoji to statuses "Accepted"/"Pending"/"Rejected"
- Statutory-specific: ?? (propose your own!)
- SU-specific: ?? (propose your own!)
You can read more about contributing on our confluence.
We use JIRA as our preferred issue tracker.
Apache License 2.0, see LICENSE.txt for more information.
Virtualbox is a utility that lets people creating virtual machines (VM) on your computer. The created VM is the GUEST. The computer you run Virtualbox on is the HOST
Vagrant is used as a tool to define VMs characteristics, that will be then run through Virtualbox - in other words, it is used so we can write a manifesto that defines the characteristics of a VM, and the VM generated has always the same characteristics. It is useful in this case to model the development VM just as if it was the server on which we will run the application.
Make is a tool that, among other things, chains commands together. So, for instance, you write in the Makefile that a runs a specific long command, b a different long command, and you can call the commands with make a or make b. You can also write a command c which is a chain of a followed by b. We use it to set a 'flow' of operations that should be followed (e.g. as explained, make bootstrap chains 3 operations, and one such operation is used very often i.e. build and/or start). This is used on the GUEST.
start.sh runs either vagrant up or make bootstrap (according to how you want to run your system in local) so one has to literally only launch one command and it's set to be working, after the startup time of around 10-20 minutes (according to internet connection speed). This is used on the HOST.
start.sh
On the HOST, i.e. the machine that runs the virtual machine, you use start.sh which can either:
- Start the vm
- Use
./start.shfor normal development cycle: app runs in development mode - Use
./start.sh --fastfor sysops/integration development cycle: app runs in production mode so you can concentrate on developing integration to the app, not the app itself
- Use
- Reset the settings to recreate the virtual machine (
./start.sh --reset). This is in case you experimented so hard that you made something exploooode. Doing so, you will lose the users and other content you created on your local instance of MyAEGEE, but this will not remove your source code.
If you are a know-it-all who doesn't want to use Vagrant, use ./start.sh --no-vagrant (but again, if you're in trouble you will only get superficial support from our side)
Makefile
On the GUEST, i.e. the virtual machine that runs docker, you use make which uses the Makefile (explained above).
For prerequisites and installation of individual containers, see their docker(/-compose) files, located in the (service)/docker folder in their respective repository.
For more detailed info, we hoped to have a better knowledge base here, it's not great right now but it's a something ¯\_(ツ)_/¯
The file contains variables where e.g. you define the base url (aegee.test) and where will various app be reachable (e.g. my. for my.aegee.test to reach the frontend).
List of defined variables:
- base url
- subdomain urls
- activated services
- runtime environment
- some default passwords
- SMTP user/pass/server for mailer
- Sendgrid user/pass
- Superadmin credentials (for other services to read; it does not set them in the system)
- folder locations
so for instance...
Example 1: you would use this file if you had a problem with 1 microservice and wanted to remove it from the setup. Note: the removal of the ms would not stop a container if there was one running already, so make sure you cleanup (not mandatory, just avoids headaches in case of troubleshooting and "oh, I forgot about this). How would you do that? Example: discounts
- Edit .env file
- Remove the service from the array
ENABLED_SERVICES docker stop myaegee_discounts_1 && docker rm myaegee_discounts_1(on the GUEST)
Example 2: you would use this file if you wrote a new microservice and wanted to add it from the setup.
How would you do that? Example: your service is into a folder called boombastic (at the same level of the folders core,events etc)
- Edit .env file
- Add the service at the end of the array
ENABLED_SERVICES=<whatever is here already>:boombastic. Remember: all the services are separated by colon (:) make start(on the GUEST)
In the docker-compose files there are the definitions of where an app should be reached.
Docker-compose will use the variables defined above, and put them under the labels section of a container (if a container needs it). The labels section is parsed by traefik to route all the HTTP calls to the correct containers. In other words, this is where the values contained in the .env file are used to specify that the app replies on 'my.appserver.test' instead of e.g. 'magic.appserver.test'.
This file which is located on the HOST machine is used to add new subdomains for the services: e.g. add pgadmin to be able to go to pgadmin.appserver.test and use a visual tool for the databases. The important part is that this name of subdomain matches the one defined in the file .env
-
Remember to check what is running on your GUEST via
docker ps(or portainer). Enter your GUEST withvagrant ssh(from your HOST). -
By running the system in development mode, you may get "NGINX error 403 (forbidden)" when you vist
my.appserver.testthe first time. To fix it, follow the steps afterfrontend helperin the file orchestrate_docker.sh -
If you are stuck and something doesn't seem to work, make sure you don't have a mismatch between
/etc/hosts, the URL you type in the browser, and the address that the system expects. See filecurrent-config.ymlfor that: it is a file which is generated every timemake startis launched. It contains the description of the desired state of the app. NB:/etc/hostsis the only file mentioned in this README that only exists in the HOST and is not mounted (as described above). -
You can add dev-tools to the array of services and troubleshoot docker using portainer (mentioned above). Make sure that you have added also the URL to the hostfile (
/etc/hosts) and that it matches the variable defined in.env -
As written in the 'moving parts' section: if you remove services from array
ENABLED_SERVICES, they are not stopped. This is not a problem in general, just don't be surprised if when you start (having the configurationENABLED_SERVICES=core:frontend:events) and stop later (with the configurationENABLED_SERVICES=events) docker writes a messageWARNING: found orphan containers. -
make startcalls a scripthelper.shthat will look into all folders described inENABLED_SERVICESfor adocker-compose.ymlfile. Then it creates in-memory a giant docker-compose configuration which is what will be run. This configuration is also outputted to a file (current-config.yml) for ease of troubleshoot. -
Running the system in
developmentmode means that everydocker-compose.ymlhas extra-settings in adocker-compose.dev.ymlfile (in the same folder). Some modules are not thought for development mode, if you find an error about a missingdocker-compose.dev.ymljust create an empty one where the system wants it.
?? @serge1peshcoff
From the core readme: there are test users. All users have 5ecr3t5ecr3t as password.
[email protected]- admin[email protected]- board member of antenna[email protected]- regular member of antenna[email protected]- member who is not confirmed[email protected]- member who requested a password reset[email protected]- a suspended member
You can use 5ecr3t for a password reset token (for a member with email [email protected]) and 5ecr3t for a mail confirmation (for a member with email [email protected]).
NOTICE if you use the --fast mode of start.sh, then core does NOT provision in production mode, so you will be fast in bootstrapping but without users to play with. This should not be an issue for you if you are interested in only the infrastructure work; if you however need a user, you can register yourself.
A couple of options
- As mentioned above, you can enable dev-tools for pgadmin. From there you can delete the db
- Notice unfortunately you have to configure pgadmin when you first login: specify the host, username, password.
- Username and password of the database is NOT the same username and password of pgadmin. Find everything in the files mentioned in 'moving parts', or
current-config.ymlfor ease. - You put the name of the service as hostname (docker internally resolves stuff with its internal DNS). In other words, every container is reachable at the host named like its service (e.g.
frontend,core,postgres-core, etc). The 'service' is named in thedocker-compose.ymlunder the keyservices:. Again: the title of the container is also the hostname of the container
- Username and password of the database is NOT the same username and password of pgadmin. Find everything in the files mentioned in 'moving parts', or
- Use portainer to delete the service and db, then
make startto fix everything.
- The service must be deleted/restarted because it runs the migrations and therefore fills the DB with the important data. If you delete only the db, the service will expect the db to be filled with data, causing errors