Skip to content

Custos Deployment

Jump to bottom
saiprasad-rane edited this page May 6, 2022 · 7 revisions
  1. Create a project named: CustosDeploymentProject
  2. Create 4 instances
Screen Shot 2022-05-06 at 3 50 58 PM Screen Shot 2022-05-06 at 5 21 21 PM
  1. In one instance, install Rancher by following this link.
  2. Remember, you are working on Jetstream 1, so you need to run the following commands: a. sudo adduser b. passwd
  3. Now, we will set up a master-worker K8s cluster using Rancher. Please follow this link.
  4. Choose the calico network when adding nodes to the cluster.
  5. Now, login to the master node.
  6. Create Namespaces : Custos, Keycloak, Vault
  7. Install helm by following this link
  8. Go to https://github.com/airavata-courses/Viyad.git
  9. cd custos/deployment_files/

On all the nodes

sudo mkdir /bitnami

sudo mkdir /bitnami/mysql

sudo mkdir /bitnami/postgresql

sudo mkdir /hashicorp

sudo mkdir /hashicorp/consul

sudo mkdir /hashicorp/consul/data

chmod 777 -R /hashicorp

Make sure all permissions for hashicorp/consul/data directories are changed. A few additional folders will be formed in the following steps, but they will all be created in the master node.

  1. Deploy cert-manager
    cd cert-manager

    kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml

    kubectl get all -n cert-manager

    kubectl apply -f issuer.yaml
Screen Shot 2022-05-06 at 5 37 41 PM
  1. All of the pods should be running at this point like shown above. If not, an error will occur, thus review the kubectl logs to troubleshoot the problem.
  2. Deploy keycloak
    1. kubectl apply -f keycloak/pv.yaml
    2. kubectl apply -f keycloak/pv1.yaml
    3. kubectl apply -f keycloak/pv2.yaml
    4. helm install keycloak-db-postgresql bitnami/postgresql -f postgresql-values.yaml -n keycloak --version 10.12.3

The output for both should be a s shown below:
Screen Shot 2022-05-06 at 5 38 57 PM

Screen Shot 2022-05-06 at 5 40 29 PM
  1. Install OLM

    kubectl create -f https://raw.githubusercontent.com/operator-framework/operator-lifecycle-manager/master/deploy/upstream/quickstart/crds.yaml

    kubectl create -f https://raw.githubusercontent.com/operator-framework/operator-lifecycle-manager/master/deploy/upstream/quickstart/olm.yaml

    git clone https://github.com/keycloak/keycloak-operator

    cp operator.yaml keycloak-operator/deploy/

    cd keycloak-operator

    make cluster/prepare

    kubectl apply -f deploy/operator.yaml -n keycloak

    cd ..

    kubectl apply -f keycloak-db-secret.yaml -n keycloak

    kubectl apply -f custos-keycloak.yaml -n keycloak

    kubectl apply -f ingress.yaml -n keycloak (Replace hostname in ingress.yaml)
Screen Shot 2022-05-06 at 5 42 55 PM
  1. Get the password by using following commands:
    kubectl get secret credential-custos-keycloak -o yaml -n keycloak
  1. Deploy Vault
    sudo mkdir -P /hashicorp/consul/data
    chmod -R 777 /hashicorp

    
     kubectl apply -f vault/pv.yaml
     kubectl apply -f vault/pv1.yaml
     kubectl apply -f vault/storage.yaml

     helm install consul hashicorp/consul --version 0.31.1 -n vault --values vault/consul-config.yaml

     helm install vault hashicorp/vault --namespace vault -f values.yaml --version 0.10.0
Screen Shot 2022-05-06 at 5 51 08 PM
  1. Deploy MySQL
   kubectl apply -f custos/pv.yaml
   kubectl apply -f custos/pv1.yaml
   kubectl apply -f custos/pv2.yaml
   helm install mysql bitnami/mysql -f values.yaml -n custos --version 8.8.8
Screen Shot 2022-05-06 at 5 53 37 PM
  1. Deploy custos Clone Custos Repository
    First, we need to clone the custos repository
git clone https://github.com/apache/airavata-custos.git
cd airavata-custos
git checkout develop

Modify root pom.xml
In root pom.xml do the following:

 <profile>
	<id>container</id>
	<properties>
		<spring.profiles.active>dev</spring.profiles.active>
		<vault.token>{vault-root-token}</vault.token>
		<vault.scheme>http</vault.scheme>
		<vault.host>vault.vault.svc.cluster.local</vault.host>
		<vault.port>8200</vault.port>
		<vault.uri>http://vault.vault.svc.cluster.local:8200</vault.uri>

		<iam.dev.username>admin</iam.dev.username>
		<iam.dev.password>{keycloak-admin-password}</iam.dev.password>
		<iam.staging.username>admin</iam.staging.username>
		<iam.staging.password>{keycloak-admin-password}</iam.staging.password>


		<custos.email.password>custos</custos.email.password>

		<spring.datasource.username>root</spring.datasource.username>
		<spring.datasource.password>{mysql-root-password}</spring.datasource.password>
	</properties>
	...
	...
 </profile>
<host>{hostname}</host>

<ssh.privatekey>/path/to/privatekey/to/master</ssh.privatekey>

<ssh.passphrase>{passphrase if any}</ssh.passphrase>
<dozer.version>5.5.1</dozer.version>

<kafka-clients.version>1.0.0</kafka-clients.version>

<email.version>1.5.0-b01</email.version>
<ssh.username>{username}</ssh.username>
<project.version>1.0</project.version>
<clusterIssuer>letsencrypt-prod</clusterIssuer>

<docker.image.prefix>docker.io/{yourdockerid}</docker.image.prefix>
<docker.image.repo>docker.io/{yourdockerid}</docker.image.repo>

Modify helm values.yaml
Modify src/main/helm/values.yaml in every project to point to your docker repo as below. replace all instances of airavatacustos with your docker id

image:
  repository: {your-docker-id}/${artifactId}

Modify custos-configuration-service pom.xml path: ./airavata-custos/custos-core-services/utility-services/custos-configuration-service/pom.xml

Modify true to false under maven-antrun-plugins configuration

Modify all dev and staging properties in custos-configuration-service path:

.airavata-custos/custos-core-services/utility-services/custos-configuration-service/src/main/resources/*-dev.properties

.airavata-custos/custos-core-services/utility-services/custos-configuration-service/src/main/resources/*-staging.properties

Modify iam.server.url=https://{hostname}:{nodeport}/auth/

Comment out the some code to avoid errors path: .airavata-custos/custos-integration-services/tenant-management-service-parent/tenant-management-service/src/main/java/tasks/TenantActivationTask.java

action: comment lines 225-249

Prepare the master node

mkdir -P ~/custos/artifacts
chmod -R 777 ~/custos

Prepare your local machine Add the following to your maven settings under

<server>
	<id>docker.io</id>
	<username>{dockerhub_user}</username>
	<password>{dockerhub_password}</password>
	<configuration>
		<email>[email protected]</email>
	</configuration>
</server>

Deploy

mvn clean install -P container
mvn -s dockerfile:push -P container
mvn -s antrun:run -P scp-to-remote

Last Step

cd ~/custos/artifacts

helm install cluster-management-core-service cluster-management-core-service-1.1.tgz -n keycloak
  1. Challenge Faced

Deployment of Custos on Jetstream2 was failing multiple times, hence we had to switch to Jetstream1 for deployment.

  1. Credit: Thanks to Team Terra and ADS Team
Clone this wiki locally