Skip to content

Latest commit

 

History

History
196 lines (133 loc) · 9.97 KB

README.md

File metadata and controls

196 lines (133 loc) · 9.97 KB
Raw

RAD Lab Launcher

The RAD Lab Launcher will guide you through the process of launching modules in your Google Cloud environment.

Installation

  1. Download the content to your local machine. Alternatively, you can check it out directly into Google Cloud Shell by clicking the button below. NOTE: You will need to follow these steps to set up a GitHub Personal Access Token.

Open in Cloud Shell

NOTE: If you are using Windows OS make sure to deploy from Command Prompt and Run as Adminstrator.

  1. Decompress the download:

    unzip rad-lab-main.zip

  2. You will need CURL & BASH. These come pre-installed in most linux terminals.

  3. Navigate to the radlab-launcher folder:

    cd ./rad-lab-main/radlab-launcher

  4. Run a script to install the prerequisites:

    python3 installer_prereq.py

    NOTE: Currently the deployment is supported for Python 3.7.3 and above.

    This will install:

  5. Verify the Terraform installation by running:

    terraform -help

    This should produce instructions on running terraform. If you get a command not found message, there was an error in the installation.

Launch Preparation

  1. To launch infrastructure in Google Cloud, a user must have the appropriate IAM permissions. Each module's README.md will list the permissions needed to launch the infrastructure. You can use the Google Cloud Console to view or change IAM permissions.

  2. The following information about your Google Cloud Platform (GCP) environment is typically needed to launch RAD Lab modules:

    The launcher can create one for you if you do not have one already.

  3. Set the default project id where the GCS bucket exists, to store Terraform configs/state. You can use gcloud config set with project property to set the project.

gcloud config set project <myProject>

NOTE: If the default project id is not set then in the guided setup you will need to manually enter the GCS bucket name (you would like to create) where you would like to store Terraform configs/state for RAD Lab configs and also the project ID where the GCS bucket will be created.

Deploy a RAD Lab Module

If you encounter errors during deployment, please see Troubleshooting Common Problems section for a list of common problems and fixes. If you don't see a solution listed, please create an Issue.

  1. Navigate to the RAD Lab Launcher folder from the main directory:

    cd ./radlab-launcher/

  2. Start the guided setup:

    python3 radlab.py

  3. To set any module specific variables, use --varfile argument while running radlab.py and pass a file with variables content:

% python3 radlab.py -h
usage: radlab.py [-h] [--varfile FILE]

optional arguments:
  -h, --help      show this help message and exit
  --varfile FILE  Input file (with complete path) for terraform.tfvars contents
%

NOTE: When the above parameter is not passed then the modules are deployed with module's default variable values.

Example :

python3 radlab.py --varfile /<path_to_file>/<file_with_terraform.tfvars_contents>

NOTE: Save the deployment_id from the output for future reference. It is used to make updates or delete the RAD Lab module deployment.

Example Launch of Data Science Module

  1. Select the RAD Lab modules you would like to deploy
List of available RAD Lab modules:
[1] # RAD Lab Application Mordernization Module (w/ Elasticsearch) (app_mod_elastic)
[2] # RAD Lab Data Science Module (data_science)
[3] Exit
Choose a number for the RAD Lab Module:
  1. Select the Action you want to perform for the corresponding RAD Lab Model:
Action to perform for RAD Lab Deployment ?
[1] Create New
[2] Update
[3] Delete
[4] List
Choose a number for the RAD Lab Module Deployment Action: 1

NOTE: If you are selecting Update/Delete action for RAD Lab Model then you will be required to provide the deployment id which is provided as the output of successfully newly created a RAD Lab Module deployment.

Enter RAD Lab Module Deployment ID (example 'ioi9' is the id for project with id - radlab-ds-analytics-ioi9):

  1. If you selected Create New/Update of RAD Lab deployment, follow the guided set up and provide user inputs like - Organization ID, Billing Account, Folder ID for the RAD Lab Module which you are setting up.

  2. If you selected Create New of RAD Lab deployment, follow the guided setup and provide the name of the GCS bucket and its Project ID where you would like to store the terraform deployment state. Keep in mind you cannot use UPPER case characters, spaces, underscore ** _ ** or contain the word "google". See Bucket Naming Guidelines for a full list of bucket namming guidelines.

Enter the GCS Bucket name where Terraform States will be stored:

NOTE: There should be a Billing associated to the above selected project for the successful creation of the GCS bucket.

  1. [Can be Skipped when running on Cloud Shell] Set up Application Default Credentials. When you run deployment on cloud shell you see a Warning about using the same credentials as the service credentials associated to the GCE VM where cloud shell is running. If you are using the same Cloud Admin account for RAD Lab deployment with which you have logged in to the cloud shell then you may enter N or n or No, etc.

NOTE: There wont be any issues if you enter Y or y or yes, etc even if you are using the same accounts as you will again get to authenticate in next step as part of guided set up.

You are running on a Google Compute Engine virtual machine.
The service credentials associated with this virtual machine
will automatically be used by Application Default
Credentials, so it is not necessary to use this command.

If you decide to proceed anyway, your user credentials may be visible
to others with access to this virtual machine. Are you sure you want
to authenticate with your personal account?

**Do you want to continue (Y/n)?**

  1. [Can be Skipped when running on Cloud Shell] Authenticate as the user of whom you want to have the Application Default Credentials configured to the cloud shell. These credentials should be of the Cloud Admin.

    • Follw the link (from the cloud shell terminal) in your browser.

    • Complete the OAuth flow by Authenticating as Cloud Admin user.

    • Copy the verification code from the browser and paste it on the Cloud Shell terminal waiting for the input.

Go to the following link in your browser:  https://accounts.google.com/o/oauth2/auth?....

Enter verification code:

  1. This is where the Terraform RAD Lab module (example: Data Science module) will kick in and Terraform config scripts will be deployed which will spin up respective projects/services/sources, etc.

  2. Once the RAD Lab deployment is completed it will throw the below Outputs on the Cloud Shell Terminal for the Cloud Admin to share with the RAD Lab users.

Outputs:

deployment_id = "ioi9"
notebooks-instance-names = "notebooks-instance-0"
project-radlab-ds-analytics-id = "radlab-ds-analytics-ioi9"
user-scripts-bucket-uri = "https://www.googleapis.com/storage/v1/b/user-scripts-notebooks-instance-ioi9"


GCS Bucket storing Terrafrom Configs: my-sample-bucket

TERRAFORM DEPLOYMENT COMPLETED!!!

NOTE: Save the deployment_id for future reference, for making any updates or delete the RAD Lab Module created.

NOTE: If you see any errors on your deployment run please follow the Troubleshooting doc to lookup for errors and corresponding solutions.

Troubleshooting Common Problems

  1. Project quota exceeded
  2. Unable to modify Organization Policy Constraints
  3. Local Terraform Deployment ID Directory Already Exists
  4. Timeout when Destroying the deployment