Merge pull request #76 from geoadmin/feature-improve_setup

Improve Dev Setup
swisstopo · Apr 12, 2024 · 3c4db5f · 3c4db5f
2 parents 607eb3f + 9cc7ed2
commit 3c4db5f
Show file tree

Hide file tree

Showing 7 changed files with 182 additions and 57 deletions.
diff --git a/README.md b/README.md
@@ -1,52 +1,131 @@
 # SwissGeol Asset
 
-## Setting up the development environment
-
+## Development
 The following components must be installed on the development computer:
 
 ✔️ Git  
 ✔️ Docker  
 ✔️ Node.js 20 LTS
 
-The following steps must be carried out once:
+### Setting Up the Development Environment
+Follow these steps to set up the development environment on your local machine:
+* [1. Configure Local Systems](#1-Configure-Local-Systems)
+* [2. Configure the Asset Server](#2-Configure-the-Asset-Server)
+* [3. Install Dependencies](#3-Install-Dependencies)
+* [4. Build Local Systems](#4-Build-Local-Systems)
+* [5. Initialize MinIO](#5-Initialize-MinIO)
+
+#### 1. Configure Local Systems
+Configure `development/.env` according to the [development services configuration](#Development-Services-Configuration).
+
+#### 2. Configure the Asset Server
+Create an empty copy of the [web server configuration](#Asset-Server-Configuration) as [`apps/server-asset-sg/.env.local`](apps/server-asset-sg/.env.local).
+Configure the following variables:
+* Set `AUTH_URL=http://localhost:8866`.
+* Set `FRONTEND_URL=http://localhost:4200`.
+* Set `DATABASE_URL=postgres://asset-swissgeol:asset-swissgeol@localhost:5432/postgres?schema=public`.
+* Set `GOTRUE_JWT_SECRET` to the same value as in [`development/.env`](development/.env).
+* Leave `OCR_URL` empty.
+* Leave `OCR_CALLBACK_URL` empty.
+
+#### 3. Install Dependencies
+Install node modules:
+```bash
+npm run install
+```
+
+Decorate the Angular CLI with the Nx CLI:
+```bash
+npm run postinstall
+```
+
+#### 4. Build Local Services
+Generate prisma-client for database-access:
+```bash
+cd apps/server-asset-sg/
+ng gen-prisma-client
+```
 
-1. Clone git repository whit `> git clone https://github.com/geoadmin/web-asset-swissgeol.git`.
-2. Install all necessary modules with `web-asset-swissgeol> npm install`.
-3. Decorate the Angular CLI with the Nx CLI `web-asset-swissgeol> npm run postinstall`.
-4. Create prisma-client for database-access with `web-asset-swissgeol\apps\server-asset-sg> ng gen-prisma-client`.
-5. Set environment variables for dockers in `web-asset-swissgeol\development\.env`
-6. Set environment variables for assets-server in `web-asset-swissgeol\apps\server-asset-sg\.env.local`
-7. Create local postgis-gotrue docker-image `development\images\db> docker build -t postgis-gotrue .`
-8. Creating elastic-search index. Execute in Kibana (`http://localhost:5601/app/dev_tools#/console`) the commands fro file `web-asset-swissgeol\development\init\elasticsearch\index`. --> docker must be started (see below)
+Build postgis-gotrue docker image:
+```bash
+cd development/images/db
+docker build -t postgis-gotrue .
+```
 
-### Starting the development environment
+#### 5. Initialize MinIO
+* [Start the development services](#Starting-the-Development-Environment).
+* Open http://localhost:9001
+* Sign in using the `STORAGE_USER` and `STORAGE_PASSWORD` of your development environment.
+* Navigate to [Buckets](http://localhost:9001/buckets) and create a new bucket with the name `asset-sg`.
+* Navigate to [the new bucket's browser](http://localhost:9001/browser/asset-sg) and create an empty folder with the name `asset-sg`.
+* Navigate to [Configuration](http://localhost:9001/settings/configurations/region) and change the server region to `local`.
+* Navigate to [Access Keys](http://localhost:9001/access-keys) and create a new access key.
+* Open your Asset Server Configuration at [`apps/server-asset-sg/.env.local`](apps/server-asset-sg/.env.local) and make the following changes:
+  * `S3_REGION=local`
+  * `S3_ENDPOINT=http://localhost:9000`
+  * `S3_BUCKET_NAME=asset-sg`
+  * `S3_ASSET_FOLDER=asset-sg`
+  * `S3_ACCESS_KEY_ID` as your newly generated access key.
+  * `S3_SECRET_ACCESS_KEY` as your newly generated access key's secret.
 
-1. Start docker with `web-asset-swissgeol\development>docker-compose up`.
-2. Start asset-swissgeol app with `web-asset-swissgeol>npm run start`.
+### Starting the Development Environment
+Start development services:
+```bash
+cd development
+docker compose up
+```
+Start the application:
+```bash
+npm run start
+```
 
-**The following services/applications are then available**
+### Local Services and Applications
+| 🔖App/Service            | 🔗Link                                           | 🧞User                   | 🔐Password               |
+|:-------------------------|:-------------------------------------------------|:-------------------------|:-------------------------|
+| Assets (client)          | [localhost:4200](http://localhost:4200/)         | `[email protected]` | `swissgeol_assets`       |
+| Assets REST API (server) | [localhost:3333/api/](http://localhost:3333/api) | n/a                      | n/a                      |
+| postgresSQL (docker)     | localhost:5432                                   | .env `$DB_USER`          | .env `$DB_PASSWORD`      |
+| Elasticsearch (docker)   | [localhost:9200](http://localhost:9200)          | n/a                      | n/a                      |
+| Kibana (docker)          | [localhost:5601](http://localhost:5601)          | n/a                      | n/a                      |
+| pgAdmin (docker)         | [localhost:5051](http://localhost:5051/)         | .env `$PGADMIN_EMAIL`    | .env `$PGADMIN_PASSWORD` |
+| MinIO (docker)           | [localhost:9001](http://localhost:9001/)         | .env `$STORAGE_USER`     | .env `$STORAGE_PASSWORD` |
+| smtp4dev (docker)        | [localhost:5000](http://localhost:5000/)         | n/a                      | n/a                      |
+| oidc-server (docker)     | [localhost:4011](http://localhost:4011/)         | n/a                      | n/a                      |
 
-| 🔖App/Service            | 🔗Link                                           | 🧞User                  | 🔐Password             |
-| :-------------------------| :------------------------------------------------| :-----------------------| :-----------------------|
-| Assets (client)           | [localhost:4200](http://localhost:4200/)         | `[email protected]`| `swissgeol_assets`    |
-| Assets REST API (server)  | [localhost:3333/api/](http://localhost:3333/api) | n/a                     | n/a                     |
-| postgresSQL (docker)      | localhost:5432                                   | .env `$DB_USER`         |.env `$DB_PASSWORD`      |
-| Elasticsearch (docker)    | [localhost:9200](http://localhost:9200)          | n/a                     | n/a                     |
-| Kibana (docker)           | [localhost:5601](http://localhost:5601)          | n/a                     | n/a                     |
-| pgAdmin (docker)          | [localhost:5051](http://localhost:5051/)         | .env `$PGADMIN_EMAIL`   |.env `$PGADMIN_PASSWORD` |
-| MinIO (docker)            | [localhost:9001](http://localhost:9001/)         | .env `$STORAGE_USER`    |.env `$STORAGE_PASSWORD` |
-| smtp4dev (docker)         | [localhost:5000](http://localhost:5000/)         | n/a                     | n/a                     |
-| oidc-server (docker)      | [localhost:4011](http://localhost:4011/)         | n/a                     | n/a                     |
+# Configuration
+## Asset Server Configuration
+The file `apps/server-asset-sg/.env.local` configures secrets for the SwissGeol Asset server.
+An empty template for the file can be found in [`apps/server-asset-sg/.env.template`](apps/server-asset-sg/.env.template).
 
-### Creating elastic-search index
+| Variable             | Example                                                                                    | Description                                                |
+|----------------------|--------------------------------------------------------------------------------------------|------------------------------------------------------------|
+| AUTH_URL             | http://my.gotrue.example:8866                                                              | URL of the GoTrue auth service.                            |
+| FRONTEND_URL         | http://assets.geo.admin.ch                                                                 | Public URL of the SwissGeol Asset web client.              |
+| S3_REGION            | euw-3                                                                                      | Region of the S3 instance.                                 |
+| S3_ENDPOINT          | http://compute-1.amazonaws.com                                                             | URL to the S3 instance.                                    |
+| S3_ACCESS_KEY_ID     | AP6wpeXraSc0IH4d42IN                                                                       | Access Key for the S3 instance.                            |
+| S3_SECRET_ACCESS_KEY | fSx5Bfib0OeAyG1mwtslKA04Qj6oPStLcpnkACmF                                                   | Secret Key for the S3 instance.                            |
+| S3_BUCKET_NAME       | asset-sg                                                                                   | S3 bucket name.                                            |
+| S3_ASSET_FOLDER      | asset-sg                                                                                   | Folder within the S3 bucket into which objects are stored. |
+| DATABASE_URL         | postgres://asset-swissgeol:asset-swissgeol@my.postgres.example:5432/postgres?schema=public | PostgreSQL access URL.                                     |
+| GOTRUE_JWT_SECRET    | 18af41574b30be7539d8c3e45ccdeea9431cff6419cdce5cabc5f28cfb73e15c                           | JWT secret key for the GoTrue server.                      |
+| OCR_URL              |                                                                                            | Leave empty.                                               |
+| OCR_CALLBACK_URL     |                                                                                            | Leave empty.                                               |
 
 
+## Development Services Configuration
+The file `development/.env` configures secrets for the services used in local development.
+An empty template for the file can be found in [`development/.env.template`](development/.env.template).
 
-### Commands in development environment
+> Make sure that your passwords have a minimal length of 8 and contain at combination of
+> upper, lower and special characters. Some of the passwords will be checked for validity during startup.
 
-| Action | Command                              | Description                               |
-| :------| :------------------------------------| :-----------------------------------------|
-| start  | `web-asset-swissgeol> npm run start` | Starts asset-client and asset-api.        | 
-| build  | `web-asset-swissgeol> npm run build` | Builds asset-client and asset-api.        | 
-| test   | `web-asset-swissgeol> npm run test`  | Executes tests all apps and libs.         | 
-| lint   | `web-asset-swissgeol> npm run lint`  | Analyzes the code from all apps and libs. | 
+| Variable          | Wert     | Beschreibung                             |
+|-------------------|----------|------------------------------------------|
+| STORAGE_USER      | _custom_ | Username for the MinIO container.        |
+| STORAGE_PASSWORD  | _custom_ | Password for the MinIO container.        |
+| DB_USER           | postgres | Username for the PostgreSQL container.   |
+| DB_PASSWORD       | _custom_ | Password for the PostgreSQL container.   |
+| PGADMIN_EMAIL     | _custom_ | Email for the PgAdmin container.         |
+| PGADMIN_PASSWORD  | _custom_ | Password for the PgAdmin container.      |
+| GOTRUE_JWT_SECRET | _custom_ | JWT Secret Key for the GoTrue container. |
diff --git a/apps/server-asset-sg/.env.template b/apps/server-asset-sg/.env.template
@@ -1,10 +1,12 @@
-AUTH_URL=
-FRONTEND_URL=
-S3_REGION=
-S3_ENDPOINT=
-S3_ACCESS_KEY_ID=
-S3_SECRET_ACCESS_KEY=
-S3_BUCKET_NAME=
-S3_ASSET_FOLDER=
-DATABASE_URL=
-GOTRUE_JWT_SECRET=
+AUTH_URL=docker gotrue url 8866 (wird ersetzt durch IAM)
+FRONTEND_URL=ng url
+S3_REGION=local
+S3_ENDPOINT=storage url 9000
+S3_BUCKET_NAME=asset-sg
+S3_ASSET_FOLDER=asset-sg
+S3_ACCESS_KEY_ID=selber generieren in minio
+S3_SECRET_ACCESS_KEY=selber generieren in minio
+DATABASE_URL=db/01_roles + access url postgres
+GOTRUE_JWT_SECRET=selber generieren, gleich wie in development
+OCR_URL=
+OCR_CALLBACK_URL=
diff --git a/development/.env.template b/development/.env.template
@@ -4,4 +4,4 @@ DB_USER=
 DB_PASSWORD=
 PGADMIN_EMAIL=
 PGADMIN_PASSWORD=
-GOTRUE_JWT_SECRET=
+GOTRUE_JWT_SECRET=
diff --git a/development/docker-compose.yaml b/development/docker-compose.yaml
@@ -24,7 +24,7 @@ services:
         environment:
             POSTGRES_USER: ${DB_USER}
             POSTGRES_PASSWORD: ${DB_PASSWORD}
-            POSTGRES_DB: postgres      
+            POSTGRES_DB: postgres
         volumes:
             - ./init/db:/docker-entrypoint-initdb.d
             - ./volumes/db/data/postgres:/var/lib/postgresql/data
@@ -42,14 +42,14 @@ services:
                 condition: service_healthy
         restart: unless-stopped
         ports:
-            - 5051:5050        
+            - 5051:5050
         environment:
             PGADMIN_DEFAULT_EMAIL: ${PGADMIN_EMAIL}
             PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_PASSWORD}
             PGADMIN_LISTEN_PORT: 5050
         volumes:
             - ./volumes/pgadmin/data:/var/lib/pgadmin
-        
+
     gotrue:
         container_name: swissgeol-assets-gotrue
         image: supabase/gotrue@sha256:794d7df376329df93c0d9ed9237868363de5c858301147e08e6620fb67d1a2c6
@@ -80,11 +80,13 @@ services:
             GOTRUE_MAILER_URLPATHS_EMAIL_CHANGE: /auth/v1/verify
             GOTRUE_MAILER_TEMPLATES_INVITE: http://localhost:4200/assets/email-templates/invite.html
             GOTRUE_MAILER_TEMPLATES_RECOVERY: http://localhost:4200/assets/email-templates/recover.html
-            
+
     elasticsearch:
         container_name: swissgeol-assets-elasticsearch
         image: docker.elastic.co/elasticsearch/elasticsearch:8.12.1
         restart: unless-stopped
+        entrypoint: ["/bin/tini", "--"]
+        command: ["sh", "-c", "/init/init.sh & /usr/local/bin/docker-entrypoint.sh"]
         ports:
             - 9200:9200
         environment:
@@ -93,7 +95,8 @@ services:
             discovery.type: single-node
             cluster.routing.allocation.disk.threshold_enabled: false
         volumes:
-            - ./volumes/elasticsearch/data:/usr/share/elasticsearch/data        
+            - ./init/elasticsearch/:/init/:ro
+            - ./volumes/elasticsearch/data:/usr/share/elasticsearch/data
 
     kibana:
         container_name: swissgeol-assets-kibana
@@ -104,7 +107,7 @@ services:
         ports:
             - 5601:5601
         environment:
-            ELASTICSEARCH_HOSTS: http://elasticsearch:9200        
+            ELASTICSEARCH_HOSTS: http://elasticsearch:9200
 
     smtp4dev:
         container_name: swissgeol-assets-smtp
@@ -149,4 +152,4 @@ services:
                 }
         volumes:
             - ./init/oidc/oidc-mock-clients.json:/tmp/config/clients-config.json:ro
-            - ./init/oidc/oidc-mock-users.json:/tmp/config/users-config.json:ro            
+            - ./init/oidc/oidc-mock-users.json:/tmp/config/users-config.json:ro            
diff --git a/development/init/db/01_roles.sql b/development/init/db/01_roles.sql
@@ -1,2 +1,2 @@
 CREATE ROLE "asset-swissgeol" WITH LOGIN PASSWORD 'asset-swissgeol';
-CREATE ROLE rdsadmin WITH LOGIN PASSWORD 'rdsadmin';
+
diff --git a/development/init/elasticsearch/init.sh b/development/init/elasticsearch/init.sh
@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# Elasticsearch URL
+ELASTICSEARCH_URL="http://localhost:9200"
+
+# Check if Elasticsearch is ready.
+check_elasticsearch() {
+  response=$(curl -s -o /dev/null -w "%{http_code}" "${ELASTICSEARCH_URL}/_cat/health")
+  if [ "$response" == "200" ]; then
+    return 0
+  else
+    return 1
+  fi
+}
+
+# Create an index and define its mapping if it does not exist yet.
+init_index() {
+  local INDEX_NAME="$1"
+
+  response=$(curl -s -o /dev/null -w "%{http_code}" "${ELASTICSEARCH_URL}/${INDEX_NAME}")
+
+  # If index does not exist (HTTP response code 404)
+  if [ "$response" != "404" ]; then
+    echo "Index ${INDEX_NAME} already exists, skipping initialization"
+    return 1
+  fi
+
+  echo "Initializing index ${INDEX_NAME}..."
+
+  curl --silent -XPUT "${ELASTICSEARCH_URL}/${INDEX_NAME}"
+  curl --silent -XPUT "${ELASTICSEARCH_URL}/${INDEX_NAME}/_mapping" -d "@/init/mappings/${INDEX_NAME}.json"
+
+  echo "Successfully initialized index ${INDEX_NAME}"
+}
+
+# Wait until ElasticSearch is ready.
+until check_elasticsearch; do
+  echo "Elasticsearch is not yet ready, retrying initialization in 5 seconds..."
+  sleep 5
+done
+
+echo "Elasticsearch is up and running, initializing indexes..."
+init_index "swissgeol_asset_asset"
+echo "Successfully initialized all indexes"
diff --git a/development/init/elasticsearch/index → ...earch/mappings/swissgeol_asset_asset.json b/development/init/elasticsearch/index → ...earch/mappings/swissgeol_asset_asset.json
@@ -1,6 +1,3 @@
-PUT /swissgeol_asset_asset
-
-PUT /swissgeol_asset_asset/_mapping
 {
   "properties": {
     "assetId": {
@@ -58,4 +55,4 @@ PUT /swissgeol_asset_asset/_mapping
       "type": "keyword"
     }
   }
-}
+}