From f1116d4a5dc03bcc1c40a41eea1ab6239755dffa Mon Sep 17 00:00:00 2001 From: skirui-source Date: Wed, 2 Aug 2023 18:49:29 -0700 Subject: [PATCH] update notebook --- .../notebook.ipynb | 65 +++++++++++-------- 1 file changed, 37 insertions(+), 28 deletions(-) diff --git a/source/examples/xgboost-azure-mnmg-daskcloudprovider/notebook.ipynb b/source/examples/xgboost-azure-mnmg-daskcloudprovider/notebook.ipynb index c89fd616..1c97befb 100644 --- a/source/examples/xgboost-azure-mnmg-daskcloudprovider/notebook.ipynb +++ b/source/examples/xgboost-azure-mnmg-daskcloudprovider/notebook.ipynb @@ -1,5 +1,26 @@ { "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Multi-Node Multi-GPU XGBoost example on Azure using dask-cloudprovider \n", + "\n", + "[Dask Cloud Provider](https://cloudprovider.dask.org/en/latest/) is a native cloud intergration library for Dask. It helps manage Dask clusters on different cloud platforms. In this notebook, we will look at how we can use this package to set-up an Azure cluster and run a multi-node multi-GPU (MNMG) example with [RAPIDS](https://rapids.ai/). RAPIDS provides a suite of libraries to accelerate data science pipelines on the GPU entirely. This can be scaled to multiple nodes using Dask as we will see in this notebook. \n", + "\n", + "For the purposes of this demo, we will use a part of the NYC Taxi Dataset (only the files of 2014 calendar year will be used here). The goal is to predict the fare amount for a given trip given the times and coordinates of the taxi trip. We will download the data from [Azure Open Datasets](https://docs.microsoft.com/en-us/azure/open-datasets/overview-what-are-open-datasets), where the dataset is publicly hosted by Microsoft.\n", + "\n", + "---\n", + "\n", + "**NOTE:** In this notebook, we will explore two possible ways to use `dask-cloudprovider` to run our workloads on Azure VM clusters:\n", + "\n", + "1. [Option 1](#-Option-1:-Use-an-Azure-marketplace-VM.-): Using an [Azure Marketplace image](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/nvidia.ngc_azure_17_11?tab=overview) made available for free from NVIDIA. The RAPIDS container will be subsequently downloaded once the VMs start up.\n", + "2. [Option 2](#-Option-2:-Set-up-an-Azure-Customized-VM.-): Using [`packer`](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/build-image-with-packer) to create a custom VM image to be used in the cluster. This image will include the RAPIDS container, and having the container already inside the image should speed up the process of provisioning the cluster.\n", + "\n", + "#### You can either use Option 1 or use Option 2.\n", + "---" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -35,19 +56,9 @@ "outputs": [], "source": [ "# # Uncomment the following and install some libraries at the beginning.\n", - "# If adlfs is not present, install adlfs to read from Azure data lake.\n", + "# # If adlfs is not present, install adlfs to read from Azure data lake.\n", "! pip install adlfs\n", - "! pip install dask-cloudprovider[all]\n", - "#! pip install \"dask-cloudprovider[azure]\" --upgrade" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "!pip show dask_cloudprovider" + "! pip install \"dask-cloudprovider[azure]\" --upgrade" ] }, { @@ -94,23 +105,18 @@ "source": [ "location = \"West US 2\"\n", "resource_group = \"rapidsai-deployment\"\n", - "vnet = \"rapidsai-deployment-vnet\"\n", - "security_group = (\n", - " \"skirui-test-mnmg-daskcloudprovider\" # \"rapidsaiclouddeploymenttest-nsg\"\n", - ")\n", + "vnet = \"skirui-test-mnmg-daskcloudprovider\" # \"rapidsaiclouddeploymenttest-nsg\"\n", "vm_size = \"Standard_NC12s_v3\" # or choose a different GPU enabled VM type\n", "\n", "docker_image = \"rapidsai/rapidsai:23.06-cuda11.8-runtime-ubuntu22.04-py3.10\"\n", - "docker_args = (\"--shm-size=256m -e DISABLE_JUPYTER=true\",)\n", + "docker_args = \"--shm-size=256m -e DISABLE_JUPYTER=true\"\n", "worker_class = \"dask_cuda.CUDAWorker\"\n", "worker_options = {\"rmm-managed-memory\": True}" ] }, { "cell_type": "markdown", - "metadata": { - "tags": [] - }, + "metadata": {}, "source": [ "---\n", "### Option 1: Use an Azure Marketplace VM image \n", @@ -140,11 +146,10 @@ " \"logging.distributed\": \"info\",\n", " \"cloudprovider.azure.azurevm.marketplace_plan\": {\n", " \"publisher\": \"nvidia\",\n", - " \"name\": \"ngc-base-version-23_03_0\",\n", + " \"name\": \"ngc-base-version-23_03_02\",\n", " \"product\": \"ngc_azure_17_11\",\n", " \"version\": \"23.03.0\",\n", " },\n", - " \"cloudprovider.azure.azurevm.disk_size\": 200,\n", " }\n", ")\n", "vm_image = \"\"\n", @@ -165,7 +170,7 @@ "metadata": {}, "outputs": [], "source": [ - "! az vm image terms accept --urn \"nvidia:ngc_azure_17_11:ngc-base-version-23_03_0:23.03.0\" --verbose" + "! az vm image terms accept --urn \"nvidia:ngc_azure_17_11:ngc-base-version-23_03_02:23.03.0\" --verbose" ] }, { @@ -453,7 +458,9 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "tags": [] + }, "source": [ "### Step 2.1: Start the VM Cluster in Azure \n", "\n", @@ -602,7 +609,9 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "tags": [] + }, "source": [ "### Step 3.a: Set up the workers for reading parquet files from Azure Data Lake endpoints " ] @@ -1321,9 +1330,9 @@ ], "metadata": { "kernelspec": { - "display_name": "rapids-23.06", + "display_name": "Python 3 (ipykernel)", "language": "python", - "name": "rapids-23.06" + "name": "python3" }, "language_info": { "codemirror_mode": { @@ -1335,7 +1344,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.12" + "version": "3.9.7" } }, "nbformat": 4,