This project contains all the source code for the Global Azure Bootcamp 2017 Science Lab. Created by David Rodriguez (@davidjrh), Adonai Suárez (@adonaisg), Martin Abbott (@martinabbott) and Wesley Cabus (@WesleyCabus) for the Global Azure Bootcamp 2017 Science Lab running Sebastian Hidalgo's Star Formation History SELIGA algorithm at the Instituto de Astrofisica de Canarias.
See more at http://global.azurebootcamp.net
Follow these steps to deploy the GAB Science Lab:
In order to participate on the GAB Science Lab you will need:
- An active Azure subscription. You will need to login as owner of the subscription to setup the needed credentials. You can signup for a free subscription here.
- You must have sufficient permissions to register an application with your Azure AD tenant, and assign the application to a role in your Azure subscription. For more information, check this link.
- Available quota for the following resources:
- 1 Batch Account for the deployment location (you can't have more than 1 batch account per location)
- 1 core available (you will need to check the core quotas for the subscription and batch accounts)
Download from this location the Science Lab scripts and unzip them in a local folder. You will see the following files:
- Setup-GABLabAccount.ps1
- Deploy-GABLabClient.ps1
- GABClient.json
- GABClient.parameters.json
The GAB Science Lab uses Azure Automation to feed the Azure Batch service with inputs to process. In order to automate the process, an Application must be setup as Service Principal. To make this process easier, follow these steps:
Requisites:
- Windows 10 is needed to run this step. If you don't have Windows 10, ask a friend or your local event staff to help you on this one.
- Install the latest version of Microsoft Azure PowerShell through the Web Platform Installer or downloading from here.
- Open a PowerShell console as Administrator
- Change the scripts execution policy to Unrestricted by executing:
Set-ExecutionPolicy -ExecutionPolicy Unrestricted- Execute the Setup-GABLabAccount.ps1 script. These parameters are required:
- SubscriptionId: The Id of the subscription you want use to deploy the Lab Client. Follow this link to get your Azure Subscription Id
- ResourceGroupName: The name of the Resource Group that will be created. (e.g. GABLab2017)
- Location: An Azure Location where the Resource Group will be created. The resource group will contain, between other resources an Automation Account that is not available yet in all the locations. IMPORTANT: You need to specify one of these locations: "East US 2" "South Central US" "West Central US" "North Europe" "West Europe" "Southeast Asia" "Japan East" "Australia Southeast" "Central India" "Canada Central" "UK South" The lab will use an Azure Batch account provisioned on Step 3. Note that you can provision only ONE Azure Batch account per location
- ApplicationDisplayName: The name of the Application (e.g. GABLab2017)
- SelfSignedCertPlainPassword: A plain text password for the Self-signed Certificate. Ensure to use double quotes enclosing the password, and don't use an empty password or the script will fail
.\Setup-GABLabAccount.ps1 -SubscriptionId "00000000-0000-0000-0000-000000000000" -ResourceGroupName "GABLab2017" -Location "North Europe" -ApplicationDisplayName "GABLab2017" -SelfSignedCertPlainPassword "abc"- A credential login prompt will be appear. You must login with your Azure credentials.
If the process finished without errors, this step is completed.
There are two ways to do this step.
- Click on the button:
- Fill the form. You can get info about each field if you hold the cursor over the info icon.
- Choose the same subscription and resource group you specified on Step 2
- Email, FullName, TeamName, CompanyName: fill with your personal info. It be displayed on the global dashboards (e-mail will not)
- CountryCode: the 2 character ISO2 country code. Find your code at Wikipedia
- LabKeyCode: Is a predefined string with your location LAB Key. Ask admin staff at your location for the code.
- VirtualMachinesSku: Size of the virtual machines for the Azure Batch pool. We recommend select one of these (1 core Skus available for batch pools):
- Standard_A1
- Standard_D1
- Standard_D1_v2
- Standard_F1
- InstanceCount: Number of virtual machine instances (20 or less). Check the available cores/quotas in your subscription before setting up a big number. You can start with 1 or 2 virtual machines and change the number of them later on the portal.
Now relax and wait for the green check. Will take around 10 minutes to complete.
- Open GABClient.parameters.json with a text editor.
- Change the value of the parameters. Before do this, read the next notes:
- Email, FullName, TeamName, CompanyName: fill with your personal info. It be displayed on the global dashboards (e-mail will not)
- CountryCode: the 2 character ISO2 country code. Find your code at Wikipedia
- LabKeyCode: Is a predefined string with your location LAB Key. Ask admin staff at your location for the code.
- VirtualMachinesSku: Size of the virtual machines for the Azure Batch pool. We recommend select one of these (1 core Skus available for batch pools):
- Standard_A1
- Standard_D1
- Standard_D1_v2
- Standard_F1
- InstanceCount: Number of virtual machine instances (20 or less). Check the available cores/quotas in your subscription before setting up a big number. You can start with 1 or 2 virtual machines and change the number of them later on the portal.
- Execute the Deploy-GABLabClient.ps1 script in a PowerShell. These parameters are required:
- ResourceGroupName: use the SAME resource group name you used on step 2 when setting up the GAB Lab account
- ResourceGroupLocation: use the SAME location you used on step 2 when setting up the GAB Lab account
.\Deploy-GABLabClient.ps1 -ResourceGroupName "GABLab2017" -ResourceGroupLocation "North Europe"
Now wait for the deployment. After around 10 minutes, you should start seeing the things happening on the batch account.
Once the lab has been deployed, you will see a set of resources under the resource group:
Your Resource Group must be similar to this screenshot. It must have an Azure Storage account, an Azure Automation account, an Azure Batch account and 3 runbooks.
Check if there are 3 runbooks, 36 assets and the status of the automation account is active.
Check if the runbook Create-BatchPool and Setup-JobSchedules have been executed and their status is Completed (these only run one time during the initial deployment). In this screen, you can also see a few runs of the Get-NewBatch runbook. This one runs every 5min to gather new inputs to process.
Open your Azure Batch account and check if one pool of nodes was created.
In the pool sumary, you will see one square for each node you have deployed. On the right, you can see the status guide of the nodes. Probably, your nodes will be in RUNNING state, depending on when you are checking this. Maybe, they are in state IDLE, CREATING, STARTING or WAITING FOR STAR TASK. It is correct. But, if the state is START TASK FAILED, OFFLINE, UNKNOWN or UNUSABLE, something is wrong with the deployment. In that case, please contact your local staff.
Well, since the lab is working correctly, you maybe want play with it. Here are some things you can try:
Do you want to appear in the Global Dashboard? Then do it by donating more computing time! ;) If you increase the number of VMs running tasks, the probability to improve your scores is higher. So, go to the Azure Portal and scale your Azure Batch pool:
Go to the overview tab of you Azure Batch pool and click "Scale" button on the top.
Now, fill the form:
- Mode: Ensure that the selected mode is Fixed.
- Target dedicated: Set the number of nodes after the scale. Keep in mind the availabilty of cores in your subscription (step 3). You can't normally go higher than 20 without contacting Microsoft support, that normally takes some working days, so please, instead of filling MS support inbox with quota requests, if you want to deploy more than 20 cores, try to deploy another GAB Science lab instance in another location.
- Current task action: Probably, your nodes are running a task, ensure that "Requeue" is selected. With this option, the data of the task will not be lost.
- Resize timeout: You can use the default value.
On the picture below you can check the global science lab architecture. The elements you have deployed are inside the "User deployment" box.
Your lab deployment will continue working processing inputs until April 23rd 23:59 or until you delete the deployment resources. Of course, you don't need to wait until that time, but the lab won't continue gathering inputs after that date. In order to delete your deployment:
- Select the Resource Group containing your science lab deployment
- Click on Delete and confirm by typing your resource group name
- Navigate to your Azure Active Directory and delete the App Registration for GABLab2017.
Thanks for your support on Global Azure Bootcamp 2017 Science Lab. Live Long and Prosper!
- How much will cost?
The lab uses an Azure Automation Basic account and a Batch Account. The cost of Automation Account would be less than $1 for a full day. The Azure Batch service is free, Microsoft only charges for the underlying Virtual Machines used on the batch pool. So for example, if you deploy the science lab with 2 instances of A1 Linux VMs during 12 hours, the costs would be under $2. For more information about pricing:
- When deploying the lab you get a "Conflict" error on the CreateBatchPoolJob step Check that the selected VMs are available in your subscription for the desired location. Also check that you have remaining cores available on your subscription. We have observed that in some locations, certain VM Skus are not available for batch. Deploying A1 or D1 VMs should work everywhere. As a workaround, you can go and manually run the runbook "Create-BatchPool" with a different VM Sku