-
diff --git a/get-started/automate-data-ingestion.html b/get-started/automate-data-ingestion.html
index 9ef9e0fd..6d685e0b 100644
--- a/get-started/automate-data-ingestion.html
+++ b/get-started/automate-data-ingestion.html
@@ -2,7 +2,7 @@
-
+
@@ -477,19 +477,19 @@
Create a bucket in MinIO
- The first thing we need to do is to setup where we are going to store our data. Till now, we’ve been storing the data in our Jupyter lab space, but we need a place where we can securely save our datasets and access them through a standard API that is widely supported by other tools and libraries.
-First, we are going to launch the MinIO API. This will forward the connection from our localhost:9000 to the MinIO API server in our cluster. This API allows us to interact with the MinIO storage. In our case, we will use the command line with the MinIO client mc
. mc
is a command line tool similar to mag
, but specific for MinIO. It is installed during the magasin installation.
+The first thing we need to do is to setup where we are going to store our data. Till now, we’ve been storing the data in our Jupyter lab space, but we need a place where we can securely save our datasets and access them through a standard API that is widely supported by other tools and libraries. MinIO is the component that will give us those capabilities.
+First, we are going to launch the MinIO API. This will forward a connection to our localhost:9000
from the MinIO API server in our cluster. This API allows us to interact with the MinIO storage accounts. In our case, we will use the MinIO client mc
command line . mc
is a tool similar to mag
, but specific for MinIO. It is installed during the magasin installation.
-Keep this running and in another terminal run this:
+Keep this running. In another terminal run this command:
mc alias set myminio http://localhost:9000 minio minio123
# mc alias set <alias-name> <endpoint> <access-key> <secret-key>
Added ` myminio ` successfully.
-Now you can stop using Control + c
the mag minio api
command.
+Now, stop mag minio api
command using Control + C
.
Once we have configured the alias, we just need to create the bucket:
mag minio add bucket --bucket-name magasin
-Note that the default alias that mag
asumes is myminio
. If you used another alias you can use --alias
option. For example:
+Note that the default alias that mag
asumes will be myminio
. If you used another alias you can use --alias
option. For example:
mag minio add bucket --bucket-name magasin --alias myalias
-Now, we have a bucket in MinIO which allows us to store files and access them through standard APIs.
+At this point, we have a bucket in MinIO which allows us to store files and access them through standard APIs.
+As a result you will have the following dashboard that collects data from an external datasource, and is automatically updated through an schedule:
+
Step 1: Exploratory data analysis in a Jupyter Notebook
Generally, before you start automating any process of regularly using some data you want to do some initial research and analysis, you want to discover and understand how the data is shaped and what kind of insights can be extract from the data you have. For that, Jupyter Notebook is a great tool. It allows you to run small pieces of code (usually Python or R) interactively in a user friendly interface.
@@ -855,7 +862,7 @@ What’s next?<
-
diff --git a/images/install/install-docker-resources-settings.png b/images/install/install-docker-resources-settings.png
new file mode 100644
index 00000000..112b5b4a
Binary files /dev/null and b/images/install/install-docker-resources-settings.png differ
diff --git a/index.html b/index.html
index 4a5707be..171c8af9 100644
--- a/index.html
+++ b/index.html
@@ -2,7 +2,7 @@
-
+
diff --git a/index.yaml b/index.yaml
deleted file mode 100644
index ccb01680..00000000
--- a/index.yaml
+++ /dev/null
@@ -1,314 +0,0 @@
-apiVersion: v1
-entries:
- dagster:
- - apiVersion: v2
- appVersion: 1.6.4
- created: "2024-02-09T09:47:54.033054788Z"
- dependencies:
- - condition: dagster-user-deployments.enableSubchart
- name: dagster-user-deployments
- repository: ""
- version: 1.6.4
- - condition: postgresql.enabled
- name: postgresql
- repository: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
- version: 8.1.0
- - condition: rabbitmq.enabled
- name: rabbitmq
- repository: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
- version: 6.16.3
- - condition: redis.internal
- name: redis
- repository: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
- version: 12.7.4
- description: The data orchestration platform built for productivity.
- digest: e8622db2a1bd9590024108a3f1f674dac38212147691a93fa280dec6c9eb740e
- icon: https://dagster.io/images/brand/logos/dagster-primary-mark.png
- keywords:
- - analytics
- - data-orchestrator
- - data-pipelines
- - etl
- - workflow
- kubeVersion: '>= 1.18.0-0'
- maintainers:
- - email: support@dagsterlabs.com
- name: Dagster Labs
- url: https://dagster.io/about
- name: dagster
- sources:
- - https://github.com/dagster-io/dagster/tree/master/helm/dagster
- type: application
- urls:
- - https://github.com/unicef/magasin/releases/download/v0.2.0-charts/dagster-1.6.4.tgz
- version: 1.6.4
- - apiVersion: v2
- appVersion: 1.6.0
- created: "2024-02-02T15:14:32.240273332Z"
- dependencies:
- - condition: dagster-user-deployments.enableSubchart
- name: dagster-user-deployments
- repository: ""
- version: 1.6.0
- - condition: postgresql.enabled
- name: postgresql
- repository: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
- version: 8.1.0
- - condition: rabbitmq.enabled
- name: rabbitmq
- repository: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
- version: 6.16.3
- - condition: redis.internal
- name: redis
- repository: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
- version: 12.7.4
- description: The data orchestration platform built for productivity.
- digest: 94b7b3fdd8ab14fc7bfde84afcf0324fcb151a44b8867ce29c6d0d03b6415469
- icon: https://dagster.io/images/brand/logos/dagster-primary-mark.png
- keywords:
- - analytics
- - data-orchestrator
- - data-pipelines
- - etl
- - workflow
- kubeVersion: '>= 1.18.0-0'
- maintainers:
- - email: support@dagsterlabs.com
- name: Dagster Labs
- url: https://dagster.io/about
- name: dagster
- sources:
- - https://github.com/dagster-io/dagster/tree/master/helm/dagster
- type: application
- urls:
- - https://github.com/unicef/magasin/releases/download/v0.1.0-charts/dagster-1.6.0.tgz
- version: 1.6.0
- - apiVersion: v2
- appVersion: 1.5.9
- created: "2023-11-29T10:16:06.344653394Z"
- dependencies:
- - condition: dagster-user-deployments.enableSubchart
- name: dagster-user-deployments
- repository: ""
- version: 1.5.9
- - condition: postgresql.enabled
- name: postgresql
- repository: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
- version: 8.1.0
- - condition: rabbitmq.enabled
- name: rabbitmq
- repository: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
- version: 6.16.3
- - condition: redis.internal
- name: redis
- repository: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
- version: 12.7.4
- description: The data orchestration platform built for productivity.
- digest: 1fa81a21539df12b8b0443387a453e367721c157aa20227f3421ab6d1d20cbf3
- icon: https://dagster.io/images/brand/logos/dagster-primary-mark.png
- keywords:
- - analytics
- - data-orchestrator
- - data-pipelines
- - etl
- - workflow
- kubeVersion: '>= 1.18.0-0'
- maintainers:
- - email: support@dagsterlabs.com
- name: Dagster Labs
- url: https://dagster.io/about
- name: dagster
- sources:
- - https://github.com/dagster-io/dagster/tree/master/helm/dagster
- type: application
- urls:
- - https://github.com/unicef/magasin/releases/download/v0.0.1-charts/dagster-1.5.9.tgz
- version: 1.5.9
- daskhub:
- - apiVersion: v2
- appVersion: jh3.2.1-dg2023.9.0
- created: "2024-02-09T09:47:54.038334167Z"
- dependencies:
- - import-values:
- - child: rbac
- parent: rbac
- name: jupyterhub
- repository: https://jupyterhub.github.io/helm-chart/
- version: 3.2.1
- - name: dask-gateway
- repository: https://helm.dask.org/
- version: 2023.9.0
- description: Multi-user JupyterHub and Dask deployment.
- digest: 366ab1e6884532740afb95a49e114cb4070e2ddfaf709eb2357b2d6e075679cd
- icon: https://avatars3.githubusercontent.com/u/17131925?v=3&s=200
- maintainers:
- - email: jtomlinson@nvidia.com
- name: Jacob Tomlinson (Nvidia)
- - email: jhamman@ucar.edu
- name: Joe Hamman (NCAR)
- - email: erik@sundellopensource.se
- name: Erik Sundell
- - email: tom.w.augspurger@gmail.com
- name: Tom Augspurger
- name: daskhub
- urls:
- - https://github.com/unicef/magasin/releases/download/v0.2.0-charts/daskhub-2024.1.0.tgz
- version: 2024.1.0
- - apiVersion: v2
- appVersion: jh2.0.0-dg2023.1.1
- created: "2023-11-29T10:16:06.351722903Z"
- dependencies:
- - import-values:
- - child: rbac
- parent: rbac
- name: jupyterhub
- repository: https://jupyterhub.github.io/helm-chart/
- version: 2.0.0
- - name: dask-gateway
- repository: https://helm.dask.org/
- version: 2023.1.1
- description: Multi-user JupyterHub and Dask deployment.
- digest: 0419f358edea2e5a1e9005f1fbe6e1b60dcf964b6fd9f3745e06d54c4fb33e00
- icon: https://avatars3.githubusercontent.com/u/17131925?v=3&s=200
- maintainers:
- - email: jtomlinson@nvidia.com
- name: Jacob Tomlinson (Nvidia)
- - email: jhamman@ucar.edu
- name: Joe Hamman (NCAR)
- - email: erik@sundellopensource.se
- name: Erik Sundell
- - email: tom.w.augspurger@gmail.com
- name: Tom Augspurger
- name: daskhub
- urls:
- - https://github.com/unicef/magasin/releases/download/v0.0.1-charts/daskhub-2023.1.0.tgz
- version: 2023.1.0
- drill:
- - apiVersion: v2
- appVersion: 1.21.1-3.9.1
- created: "2024-02-09T09:47:54.038852893Z"
- description: Helm charts for deploying Apache Drill on a Kubernetes cluster
- digest: d9a2f30cfa6e38dbd63fd7a79854b3e08ae9a097867b7cf2bf6d71ea8bf25b56
- home: http://drill.apache.org/
- keywords:
- - apache
- - drill
- - zookeeper
- - magasin
- - unicef
- maintainers:
- - email: merlos@users.noreply.github.com
- name: merlos
- name: drill
- sources:
- - https://github.com/unicef/magasin-drill
- type: application
- urls:
- - https://github.com/unicef/magasin/releases/download/v0.2.0-charts/drill-0.6.1.tgz
- version: 0.6.1
- operator:
- - apiVersion: v2
- appVersion: v5.0.11
- created: "2024-02-09T09:47:54.039854619Z"
- description: A Helm chart for MinIO Operator
- digest: e8f79cba2d56fdfc40a645bd97d8e9618558f1b918e3c49dd9979aac27926a3c
- home: https://min.io
- icon: https://min.io/resources/img/logo/MINIO_wordmark.png
- keywords:
- - storage
- - object-storage
- - S3
- maintainers:
- - email: dev@minio.io
- name: MinIO, Inc
- name: operator
- sources:
- - https://github.com/minio/operator
- type: application
- urls:
- - https://github.com/unicef/magasin/releases/download/v0.2.0-charts/operator-5.0.11.tgz
- version: 5.0.11
- superset:
- - apiVersion: v2
- appVersion: 3.1.0
- created: "2024-02-09T09:47:54.048858841Z"
- dependencies:
- - condition: postgresql.enabled
- name: postgresql
- repository: https://charts.bitnami.com/bitnami
- version: 12.1.6
- - condition: redis.enabled
- name: redis
- repository: https://charts.bitnami.com/bitnami
- version: 17.9.4
- description: Apache Superset is a modern, enterprise-ready business intelligence
- web application
- digest: 3b09b5aa9eff983f1f9ef76c08d1a43671b2bb8b839c7c509dc4233c5eb67e7c
- home: https://superset.apache.org/
- icon: https://artifacthub.io/image/68c1d717-0e97-491f-b046-754e46f46922@2x
- keywords:
- - business intelligence
- - data science
- maintainers:
- - email: craig@craigrueda.com
- name: craig-rueda
- url: https://github.com/craig-rueda
- name: superset
- sources:
- - https://github.com/apache/superset
- urls:
- - https://github.com/unicef/magasin/releases/download/v0.2.0-charts/superset-0.12.1.tgz
- version: 0.12.1
- - apiVersion: v2
- appVersion: 3.0.1
- created: "2023-11-29T10:16:06.36911367Z"
- dependencies:
- - condition: postgresql.enabled
- name: postgresql
- repository: https://charts.bitnami.com/bitnami
- version: 12.1.6
- - condition: redis.enabled
- name: redis
- repository: https://charts.bitnami.com/bitnami
- version: 17.9.4
- description: Apache Superset is a modern, enterprise-ready business intelligence
- web application
- digest: afb72faf3e0748348bd8dccfd70c99294d12d58ccec1271eb20bcbd276491c93
- home: https://superset.apache.org/
- icon: https://artifacthub.io/image/68c1d717-0e97-491f-b046-754e46f46922@2x
- keywords:
- - business intelligence
- - data science
- maintainers:
- - email: craig@craigrueda.com
- name: craig-rueda
- url: https://github.com/craig-rueda
- name: superset
- sources:
- - https://github.com/apache/superset
- urls:
- - https://github.com/unicef/magasin/releases/download/v0.0.1-charts/superset-0.10.15.tgz
- version: 0.10.15
- tenant:
- - apiVersion: v2
- appVersion: v5.0.11
- created: "2024-02-09T09:47:54.049719567Z"
- description: A Helm chart for MinIO Operator
- digest: d0b1363d8227e0cc081e37eef667f7f20bdefecd5794c44066267bcf7c06a045
- home: https://min.io
- icon: https://min.io/resources/img/logo/MINIO_wordmark.png
- keywords:
- - storage
- - object-storage
- - S3
- maintainers:
- - email: dev@minio.io
- name: MinIO, Inc
- name: tenant
- sources:
- - https://github.com/minio/operator
- type: application
- urls:
- - https://github.com/unicef/magasin/releases/download/v0.2.0-charts/tenant-5.0.11.tgz
- version: 5.0.11
-generated: "2024-02-09T09:47:54.024813733Z"
diff --git a/install-magasin.sh b/install-magasin.sh
deleted file mode 100755
index 349660a1..00000000
--- a/install-magasin.sh
+++ /dev/null
@@ -1,653 +0,0 @@
-#!/bin/bash
-
-# This script checks and installs kubectl, helm, and Homebrew (if needed) on your system.
-# It supports automatic installation with the `-y` flag and provides
-# detailed user prompts for confirmation. Additionally, it verifies
-# the functionality of all tools once installed.
-
-
-
-# Magasin base url
-BASE_URL=http://unicef.github.io/magasin
-
-# Values folder. Use -f to overwrite
-VALUES_FOLDER='./'
-
-# Helm repo URL (-u to overwrite)
-MAGASIN_DEFAULT_HELM_REPO=$BASE_URL
-MAGASIN_HELM_REPO=$BASE_URL
-# If -u is set overwritten
-CUSTOM_HELM_REPO=false
-
-# Link to documentation on how to install magasin manually
-MANUAL_INSTALL_LINK=$BASE_URL/install/manual-installation.html
-UNINSTALL_MAGASIN_LINK=$BASE_URL/install/uninstall.html
-GET_STARTED_LINK=$BASE_URL/get-started/tutorial-overview.html
-
-# Skip prompting the user?
-AUTO_INSTALL=false
-
-# Only install local dependencies. Skip installing magasin
-# in the kubernetes cluster
-ONLY_LOCAL_INSTALL=false
-
-# Only check if there is missing stuff
-ONLY_CHECK=false
-
-# Debug mode
-DEBUG=false
-
-# Default REALM
-REALM_ARG='magasin' # default
-REALM_PREFIX='magasin'
-REALM_POSTFIX=''
-
-# Get PLATFORM
-PLATFORM=$(uname)
-LINUX="Linux"
-MACOS="Darwin"
-
-# Function to display messages in red
-echo_debug() {
- if [ "$DEBUG" = true ]; then
- printf "\033[38;5;208m%s\033[0m\n" "$@"
- fi
-}
-
-# Function to display a line of dashes with the width of the terminal window.
-echo_line() {
- local width=$(tput cols) # Get the width of the terminal window
- printf "%${width}s\n" | tr ' ' '-' # Print dashes to fill the width
-}
-
-# Function to display messages prepending [ v ] (v = check)
-echo_success() {
- printf "\033[32m[ \xE2\x9C\x93 ]\033[0m %s\n" "$@"
-}
-
-# Information message prepended by [ i ]
-echo_info() {
- printf "\033[34m[ i ]\033[0m %s\n" "$@"
-}
-
-
-# Function to display failure to comply with a condition.
-# Prepends and x.
-echo_fail() {
- printf "\033[31m[ \xE2\x9C\x97 ]\033[0m %s\n" "$@" # \e[31m sets the color to red, \e[0m resets the color
-}
-
-# Function to display warning messages.
-# Prepends two !! in orangish color.
-echo_warning() {
- printf "\033[38;5;208m[ W ]\033[0m %s\n" "$@"
-}
-
-# Function to display error messages in red. Prepends ERROR
-echo_error() {
- printf "\033[31mERROR:\033[0m %s\n" "$@"
-}
-
-# Exit displaying how to debug
-exit_error() {
- local code=$1
- echo_error "$code"
- echo_error "You may get more information about the issue by running the script including the debug option (-d):"
- echo_error " $script_name -d "
- echo ""
- exit $code
-}
-
-echo_magasin() {
-
-echo ""
-echo " Welcome to the world of tomorrow "
-printf "\033[31m"
-printf " ▄ \n"
-printf " ███ \n"
-printf " ▀ \033[0m\n"
-echo " ▐█▙█▖ ▟██▖ ▟█▟▌ ▟██▖▗▟██▖ ██ ▐▙██▖ "
-echo " ▐▌█▐▌ ▘▄▟▌▐▛ ▜▌ ▘▄▟▌▐▙▄▖▘ █ ▐▛ ▐▌ "
-echo " ▐▌█▐▌▗█▀▜▌▐▌ ▐▌▗█▀▜▌ ▀▀█▖ █ ▐▌ ▐▌ "
-echo " ▐▌█▐▌▐▙▄█▌▝█▄█▌▐▙▄█▌▐▄▄▟▌▗▄█▄▖▐▌ ▐▌ "
-echo " ▝▘▀▝▘ ▀▀▝▘ ▞▀▐▌ ▀▀▝▘ ▀▀▀ ▝▀▀▀▘▝▘ ▝▘ "
-echo " ▜█▛▘ "
-echo ""
-
-}
-
-function usage {
- echo "Usage: $1 [-y] [-c] [-i] [-r realm_prefix-realm_postfix (magasin)] [-f values_folder (./)] [-d] [-h]"
- echo ""
- echo "This script checks dependencies and installs magasin components"
- echo "Each component is installed within its own namespace."
- echo ""
- echo "Options:"
- echo " -y Skip prompting questions during installation"
- echo " -c Only check if all pre-requisites are installed in the local machine."
- echo " -i Only install all pre-requisites in the local machine. Does not install magasin in Kubernetes"
- echo " -r Realm prefix and suffix (default: magasin). Prefix and suffix are separated by '-'."
- echo " If more than one '-', the last one will be used as separator."
- echo " The realm 'magasin-new-dev' will set 'magasin-new' as prefix and 'dev' as suffix."
- echo " -f Folder with custom values.yaml files (default: ./)."
- echo " Files within the folder shall have the same name as the component. Example:"
- echo " drill.yaml, dagster.yaml, superset.yaml, daskhub.yaml"
- echo " -u URL/path to the magasin's helm repository (default: https://unicef.github.io/magasin/)"
- echo " "
- echo " -d Enable debug mode (displays all commands run)."
- echo " -h Display this help message and exit."
- echo " "
- echo "Examples:"
- echo " - Only check if all requirements are installed"
- echo " $1 -c "
- echo " - Setup the realm 'test'. Will use test- as namespaces"
- echo " $1 -r test"
- echo " - Enable debug mode, skip being promted, and setup the realm 'magasin-dev'"
- echo " (which results in magasin--dev as namespaces)"
- echo " $1 -d -y -r magasin-dev"
- exit 0
-}
-
-script_name=$(basename "$0")
-
-while getopts ":f:u:r:yichd" opt; do
- case $opt in
- y)
- AUTO_INSTALL=true
- ;;
- c)
- ONLY_CHECK=true
- ;;
- i)
- ONLY_LOCAL_INSTALL=true
- ;;
- d)
- DEBUG=true
- ;;
- u)
- CUSTOM_HELM_REPO=true
- MAGASIN_HELM_REPO=$OPTARG
- ;;
- f)
- # Check if the folder exists.
- if [ -d $OPTARG ]; then
- echo_debug "Values folder exists $OPTARG"
- VALUES_FOLDER=$OPTARG
- # Check if hte folder name ends with
- if [[ "$VALUES_FOLDER" != */ ]]; then
- echo_debug "Adding slash to values folder $VALUES_FOLDER"
- VALUES_FOLDER="$VALUES_FOLDER/"
- fi
- echo_info "Folder with value files exists ($VALUES_FOLDER)."
- else
- echo_error "Folder $OPTARG does not exist."
- exit 101
- fi
-
- ;;
- r)
- argument=$OPTARG
- # Extracting prefix and postfix
- last_dash=$(echo "$argument" | grep -o '[^-]*$')
- if [[ "$last_dash" == "$argument" ]]; then
- REALM_PREFIX=$argument
- REALM_POSTFIX=""
- else
- REALM_PREFIX=${argument%-$last_dash}
- REALM_POSTFIX=$last_dash
- fi
- REALM_ARG=$argument
- echo_info "Magasin realm set:"
- echo_info " Realm: $REALM_ARG"
- echo_info " Realm prefix '$REALM_PREFIX'"
- echo_info " Realm suffix '$REALM_SUFFIX'"
-
- ;;
- h)
- usage $script_name
- ;;
- \?)
- echo_error "Invalid option: -$OPTARG"
- echo ""
- usage $script_name
- exit 102
- ;;
- :)
- echo_error "Option -$OPTARG requires an argument." >&2
- exit 103
- ;;
- esac
-done
-
-# This is addded in helm command line.
-HELM_DEBUG_FLAG=''
-# If debug display all commands
-if [ "$DEBUG" = true ]; then
- echo_info Setting DEBUG mode ON
- # This will enable debug in helm commands. It is added in all helm command calls
- HELM_DEBUG_FLAG=" --debug"
- set -x
-fi
-
-echo_magasin
-echo "Launching installer..."
-sleep 4
-
-
-# Display vars
-echo "-----------"
-echo_info "BASE_URL: $BASE_URL"
-echo_info "MAGASIN_HELM_REPO: $MAGASIN_HELM_REPO"
-echo_info "CUSTOM_HELM_REPO: $CUSTOM_HELM_REPO"
-echo_info "MANUAL_INSTALL_LINK: $MANUAL_INSTALL_LINK"
-echo_info "REALM_ARG: $REALM_ARG"
-echo_info "REALM_PREFIX: $REALM_PREFIX"
-echo_info "REALM_POSTFIX: $REALM_POSTFIX"
-echo_info "AUTO_INSTALL: $AUTO_INSTALL"
-echo_info "ONLY_CHECK: $ONLY_CHECK"
-echo_info "ONLY_LOCAL_INSTALL: $ONLY_LOCAL_INSTALL"
-echo_info "PLATFORM: $PLATFORM"
-echo_info "PATH: $PATH"
-echo_info "HELM_DEBUG_FLAG: $HELM_DEBUG_FLAG"
-echo "-----------"
-# Initialize report variables
-declare -A install_status
-
-# if a command does not exist this variable is set to true
-command_missing=false
-
-# Checks if a command exists. Returns "installed" and "not installed"
-# if the command does not exist sets $command_missing to true
-function command_exists {
- local command="$1"
- if ! command -v "$command" &> /dev/null; then
- command_missing=true
- echo_fail "$command not installed"
- install_status["$command"]="not installed"
- return 1
- fi
- command_path="$(command -v "$command")"
- echo_success "$command installed ($command_path)"
- install_status["$command"]="installed"
- return 0
-}
-
-# Checks if all the required commands to perform the installation are in linux
-function check_linux_requirements_status {
- echo_info "** magasin installer for a GNU/Linux system (Linux)"
- command_exists "kubectl"
- command_exists "helm"
- command_exists "mc"
- command_exists "mag"
- # only check if any of the previous does not exist
- if [[ "$command_missing" == true ]]; then
- command_exists "apt-get" # debian like package installer
- fi
-}
-
-# Checks if all the required commands are available for MacOS
-function check_macos_requirements_status {
- echo_info "magasin installer for a MacOS system (Darwin)"
- command_exists "kubectl"
- command_exists "helm"
- command_exists "pip3"
- command_exists "mc"
- command_exists "mag"
-
- if [[ "$command_missing" == true ]]; then
- command_exists "brew"
- fi
-}
-
-
-# Check based on system
-echo ""
-if [[ $PLATFORM == $LINUX ]]; then
- check_linux_requirements_status
-
- # Only debian / apt-get systems supported
-
- if [ "${install_status["apt-get"]}" = "not installed" ]; then
- echo ""
- echo_error "apt-get is not installed. Are you in a Debian like system?"
- echo_info "This installation script only works on Debian GNU/Linux like systems (f.i. Debian, Ubuntu, raspbian, Kali...)."
- echo_info "Please read the "Manual Installation" section in:"
- echo ""
- echo_info " $MANUAL_INSTALL_LINK"
- echo ""
- exit 1
- fi
-
-elif [[ $PLATFORM == $MACOS ]]; then
- check_macos_requirements_status
-else
- echo ""
- echo_error Platform = $PLATFORM
- echo_error "This system is not supported by this installation script."
- echo_info " Please visit $MANUAL_INSTALL_LINK"
- echo ""
- exit 2
-fi
-
-
-echo_debug "Is there any command missing?"
-if [[ "$command_missing" == true ]]; then
- echo ""
- echo_fail "There are missing dependencies."
-else
- echo ""
- echo_success "All dependencies are installed."
-fi
-
-# If -c option is set, then end.
-echo_debug "Only check? $ONLY_CHECK"
-if [[ "$ONLY_CHECK" == true ]]; then
- echo_debug "ONLY_CHECK=true"
- exit 0
-fi
-
-#
-# Install missing pre-requisites
-#
-# Is there a command missing?
-
-echo_debug "Is there any dependency missing missing? $command_missing"
-if [[ "$command_missing" == true ]]; then
- # Perform installation based on report
- if [[ $PLATFORM == $LINUX ]]; then
- echo_debug "Install missing commands in $LINUX"
- if [[ "$AUTO_INSTALL" == false ]]; then
- # If not auto install Prompt for installation if any tool is missing and -y flag is not provided
- echo ""
- read -r -p "Do you want to install the missing tools (y/N)?" response
- if [[ ! $response =~ ^[Yy]$ ]]; then
- echo_fail "Installation aborted."
- echo_info "For more information on how install manually: $MANUAL_INSTALL_LINK"
- exit 0
- fi # response
- fi # auto_install
- echo_info "Installing pre-requisites for GNU/Linux.."
- if [ "${install_status["kubectl"]}" == "not installed" ]; then
- # https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/
- echo_info "Installing kubectl..."
- echo_line
- sudo apt-get update
- # apt-transport-https may be a dummy package; if so, you can skip that package
- sudo apt-get install -y apt-transport-https ca-certificates curl
- # Note: In releases older than Debian 12 and Ubuntu 22.04, /etc/apt/keyrings does not exist by default, and can be created using sudo mkdir -m 755 /etc/apt/keyrings
- sudo mkdir -m 755 /etc/apt/keyrings
- curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
- # This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
- echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
- sudo apt-get update
- sudo apt-get install -y kubectl
- echo_line
- fi
-
- if [ "${install_status["helm"]}" == "not installed" ]; then
- echo_info "Installing helm..."
- echo_line
- curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
- sudo apt-get install apt-transport-https --yes
- echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
- sudo apt-get update
- sudo apt-get install helm
- echo_line
- fi
-
- if [ "${install_status["pip3"]}" == "not installed" ]; then
- echo_info "Installing pip..."
- echo_line
- sudo apt-get install python3 python3-pip --yes
- echo_line
- fi
-
- if [ "${install_status["mc"]}" == "not installed" ]; then
- echo_info "Installing mc at /usr/local/bin/mc..."
- echo_line
- sudo curl https://dl.min.io/client/mc/release/linux-amd64/mc \
- --create-dirs \
- -o /usr/local/bin/mc
- sudo chmod +x /usr/local/bin/mc
- echo_line
- fi
-
- if [ "${install_status["mag"]}" == "not installed" ]; then
- echo_info "Installing mag CLI..."
- echo_info "Running: sudo pip install mag"
- echo_line
- sudo pip install mag-cli
- echo_line
- fi
-
-
- elif [[ $PLATFORM == $MACOS ]]; then
- if [[ "$AUTO_INSTALL" == false ]]; then
- # If not auto install Prompt for installation if any tool is missing and -y flag is not provided
- # zsh read format
- echo ""
- read "response?Do you want to install the missing tools (y/N)?"
- if [[ ! $response =~ ^[Yy]$ ]]; then
- echo_fail "Installation aborted."
- echo_info "For more information on how install manually: $MANUAL_INSTALL_LINK"
- exit 0
- fi # response
- fi # auto install
- echo_info "Installing pre-requisites for MacOS..."
- # If brew does not exist => install it
- if [[ "${install_status["brew"]}" == "not installed" ]]; then
- echo_info "Installing brew..."
- echo_line
- /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
- echo_line
- fi
- echo_info "Running: brew install kubectl helm python minio/stable/mc"
- echo_line
- brew install kubectl helm python minio/stable/mc
- echo_line
-
- if [[ "${install_status["mag"]}" == "not installed" ]]; then
- echo_info "Installing mag CLI..."
- echo_info "Running pip install mag-cli"
- echo_line
- pip install mag-cli
- echo_line
- fi
-
- else
- # this probably will never be reached
- echo_error "System not supported ($PLATFORM)."
- exit 1
- fi # else
-fi # command missing
-
-# Verify installations
-
-# Verify kubectl functionality
-not_working=false
-
-
-echo ""
-echo_info "Verifying commands are working..."
-if ! kubectl &> /dev/null; then
- echo_error "The kubectl command ($(command -v "kubectl")) is not working properly."
- echo_error "Installation documentation:"
- echo_error " - For Linux: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/"
- echo_error " - For macOS: https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/"
- not_working=true
-else
- echo_success "kubectl is working"
-fi
-
-# Verify helm functionality
-if ! helm version &> /dev/null; then
- echo_error "The helm command ($(command -v "helm")) is not working properly."
- echo_error "Installation documentation:"
- echo_error " https://helm.sh/docs/intro/install/"
- not_working=true
-
-else
- echo_success "helm is working"
-fi
-
-# Verify helm functionality
-if ! mc --version &> /dev/null; then
- echo_error "The mc command ($(command -v "mc")) is not working properly."
- echo_error "Installation documentation:"
- echo_error " https://min.io/docs/minio/linux/reference/minio-mc.html#install-mc"
- not_working=true
-
-else
- echo_success "helm is working"
-fi
-
-
-if [ "$not_working" = true ]; then
- echo_error "Some of the commands are not working."
- exit_error 10
-fi
-
-if [[ "$ONLY_LOCAL_INSTALL" == true ]]; then
- echo ""
-
- echo_success "All dependencies are installed and working"
- echo_info "Skipping installing magasin in the kubernetes cluster"
- echo ""
- exit 0
-fi
-
-
-# Check if repository already exists
-if helm repo list | grep -q "magasin"; then
- echo_info "magasin helm repository already exists. Resetting it..."
- echo_info "Running: helm repo remove magasin; helm repo add magasin $MAGASIN_HELM_REPO $HELM_DEBUG_FLAG"
- echo_line
- helm repo remove magasin
- helm repo add magasin $MAGASIN_HELM_REPO $HELM_DEBUG_FLAG
- if [[ $? -ne 0 ]]; then
- echo_line
- echo_error "Failed to add magasin repo after removing the existing one."
- exit_error 20
- else
- echo_line
- echo_success "magasin helm repo successfully re-added as 'magasin' pointing to $MAGASIN_HELM_REPO"
- if "$CUSTOM_HELM_REPO" = true; then
- echo_warning "magasin helm repo URL is set to $MAGASIN_HELM_REPO which is not the official one"
- echo_warning "You can set back the official magasin helm repo by running"
- echo_warning " helm repo remove magasin"
- echo_warning " helm repo add magasin $MAGASIN_DEFAULT_HELM_REPO"
- fi
- fi
-else
- # magain helm repo does not exist => add it
- echo_info "Adding magasin helm repo ($MAGASIN_HELM_REPO)..."
- echo_info "Running: helm repo add $MAGASIN_HELM_REPO $HELM_DEBUG_FLAG"
- echo_line
- helm repo add magasin $MAGASIN_HELM_REPO $HELM_DEBUG_FLAG
- if [[ $? -ne 0 ]]; then
- echo_line
- echo_error "Failed to add magasin repo."
- exit_error 30
- else
- echo_line
- echo_success "magasin helm repo successfully added as 'magasin'."
- fi
-fi
-
-
-# check if the realm namespace exits
-if kubectl get namespace "$REALM_ARG" &> /dev/null; then
- echo_error "The realm namespace '$REALM_ARG' exists."
- echo_error " Do you have a magasin instance already installed? You can try: "
- echo_error " 1. Install magasin in another realm: '$script_name -r myrealm'"
- echo_error " 2. Uninstalling '$REALM_ARG' realm instance (see $UNINSTALL_MAGASIN_LINK)"
- echo_error " 3. Remove the namespace: 'kubectl delete namespace $REALM_ARG'"
- exit 60
-fi
-
-
-#
-# Add a configmap with a json with some metadata
-#
-echo_info "Creating the magasin realm namespace."
-echo_info "kubectl create namespace $REALM_ARG"
-kubectl create namespace $REALM_ARG
-
-
-# Flag.Set to true when calling install_chart
-# it fails to install the chart.
-install_chart_failed=false
-
-#
-# Install magasin helm charts in the kubernetes cluster
-# $1 = chart name
-#
-function install_chart {
- local chart=$1
-
- echo_debug "Install_chart $chart"
-
- if [[ -n "$REALM_POSTFIX" ]]; then
- # realm postfix is not empty
- namespace="$REALM_PREFIX-$chart-$REALM_POSTFIX"
- else
- # realm postfix is empty
- namespace="$REALM_PREFIX-$chart"
- fi
-
- echo_info "Installing magasin/$chart in the namespace $namespace."
- values_helm_arg=""
- values_file="$VALUES_FOLDER$chart.yaml"
- echo_debug "values_file = $values_file"
-
- # check if $chart.yaml file exists
- if [ -f "$values_file" ]; then
- echo_success "Custom values file for $chart exists ($values_file)"
- # the -f option in helm allows you to set custom values
- # Include it as part of the
- values_helm_arg="-f $values_file"
- else
- echo_info "Custom values file for $chart does NOT exist ($values_file)"
- fi
- # Check if the namespace already exists if so ask and warn the user.
-
-
- echo_info "helm install $chart magasin/$chart $values_helm_arg --namespace $namespace --create-namespace $HELM_DEBUG_FLAG"
- echo_line
- helm install $chart magasin/$chart $values_helm_arg --namespace $namespace --create-namespace $HELM_DEBUG_FLAG
- if [[ $? -ne 0 ]]; then
- echo_line
- echo_error "Could not install magasin/$chart in the namespace $namespace"
- install_chart_failed=true
- #exit_error 7
- else
- echo_line
- echo_success "magasin/$chart installed in namespace $namespace"
- fi
-
-
-}
-
-
-install_chart drill
-install_chart daskhub
-install_chart dagster
-install_chart superset
-install_chart operator
-install_chart tenant
-
-#
-# Check if the variable is true
-if [ "$install_chart_failed" = true ]; then
- echo_line
- echo_warning "Atention!!"
- echo_warning "Some of the components were not installed successfully"
- echo_warning "Check the messages above. You can try to install the failed charts manually"
- echo_warning "More information about manual installation:"
- echo_warning " $MANUAL_INSTALL_LINK"
-fi
-
-echo_line
-echo_info "Next step start using magasin. Take a look at the tutorial:"
-echo_info " $GET_STARTED_LINK"
-echo_line
diff --git a/install/advanced.html b/install/advanced.html
index 1ad618d9..0735ade6 100644
--- a/install/advanced.html
+++ b/install/advanced.html
@@ -2,7 +2,7 @@
-
+
@@ -984,7 +984,7 @@
-
diff --git a/install/index.html b/install/index.html
index 14c38d8e..57007131 100644
--- a/install/index.html
+++ b/install/index.html
@@ -2,7 +2,7 @@
-
+
@@ -427,10 +427,10 @@ curl -X https://unicef.github.io/magasin/install-magasin.sh | bash
+curl -sSL https://unicef.github.io/magasin/install-magasin.sh | bash
Debian like distributions are Ubuntu, Raspbian, Kali Linux, etc. They use apt-get
as package manager.
Deploying magasin using a Mac OS computer
-curl -X https://unicef.github.io/magasin/install-magasin.sh | zsh
+curl -sSL https://unicef.github.io/magasin/install-magasin.sh | zsh
In both cases, you need a user that can run sudo.
The installer is in beta, in case it fails check the troubleshooting section and, if the problems persist, try the manual installation
@@ -868,7 +868,7 @@
@@ -431,9 +434,19 @@ helm install operator magasin/operator --namespace magasin-operator --create-namespace