update notebook

source/examples/xgboost-azure-mnmg-daskcloudprovider/notebook.ipynb

@@ -1,5 +1,26 @@
 {
  "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# <span style=\"color:#8735fb; font-size:24pt\"> Multi-Node Multi-GPU XGBoost example on Azure using dask-cloudprovider </span>\n",
+    "\n",
+    "[Dask Cloud Provider](https://cloudprovider.dask.org/en/latest/) is a native cloud intergration library for Dask. It helps manage Dask clusters on different cloud platforms. In this notebook, we will look at how we can use this package to set-up an Azure cluster and run a multi-node multi-GPU (MNMG) example with [RAPIDS](https://rapids.ai/). RAPIDS provides a suite of libraries to accelerate data science pipelines on the GPU entirely. This can be scaled to multiple nodes using Dask as we will see in this notebook. \n",
+    "\n",
+    "For the purposes of this demo, we will use a part of the NYC Taxi Dataset (only the files of 2014 calendar year will be used here). The goal is to predict the fare amount for a given trip given the times and coordinates of the taxi trip. We will download the data from [Azure Open Datasets](https://docs.microsoft.com/en-us/azure/open-datasets/overview-what-are-open-datasets), where the dataset is publicly hosted by Microsoft.\n",
+    "\n",
+    "---\n",
+    "\n",
+    "**NOTE:**  In this notebook, we will explore two possible ways to use `dask-cloudprovider` to run our workloads on Azure VM clusters:\n",
+    "\n",
+    "1. [Option 1](#-Option-1:-Use-an-Azure-marketplace-VM.-): Using an [Azure Marketplace image](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/nvidia.ngc_azure_17_11?tab=overview) made available for free from NVIDIA. The RAPIDS container will be subsequently downloaded once the VMs start up.\n",
+    "2. [Option 2](#-Option-2:-Set-up-an-Azure-Customized-VM.-): Using [`packer`](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/build-image-with-packer) to create a custom VM image to be used in the cluster. This image will include the RAPIDS container, and having the container already inside the image should speed up the process of provisioning the cluster.\n",
+    "\n",
+    "#### You can either use Option 1 or use Option 2.\n",
+    "---"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -35,19 +56,9 @@
    "outputs": [],
    "source": [
     "# # Uncomment the following and install some libraries at the beginning.\n",
-    "# If adlfs is not present, install adlfs to read from Azure data lake.\n",
+    "# # If adlfs is not present, install adlfs to read from Azure data lake.\n",
     "! pip install adlfs\n",
-    "! pip install dask-cloudprovider[all]\n",
-    "#! pip install \"dask-cloudprovider[azure]\" --upgrade"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "!pip show dask_cloudprovider"
+    "! pip install \"dask-cloudprovider[azure]\" --upgrade"
    ]
   },
   {
@@ -94,23 +105,18 @@
    "source": [
     "location = \"West US 2\"\n",
     "resource_group = \"rapidsai-deployment\"\n",
-    "vnet = \"rapidsai-deployment-vnet\"\n",
-    "security_group = (\n",
-    "    \"skirui-test-mnmg-daskcloudprovider\"  # \"rapidsaiclouddeploymenttest-nsg\"\n",
-    ")\n",
+    "vnet = \"skirui-test-mnmg-daskcloudprovider\"  # \"rapidsaiclouddeploymenttest-nsg\"\n",
     "vm_size = \"Standard_NC12s_v3\"  # or choose a different GPU enabled VM type\n",
     "\n",
     "docker_image = \"rapidsai/rapidsai:23.06-cuda11.8-runtime-ubuntu22.04-py3.10\"\n",
-    "docker_args = (\"--shm-size=256m -e DISABLE_JUPYTER=true\",)\n",
+    "docker_args = \"--shm-size=256m -e DISABLE_JUPYTER=true\"\n",
     "worker_class = \"dask_cuda.CUDAWorker\"\n",
     "worker_options = {\"rmm-managed-memory\": True}"
    ]
   },
   {
    "cell_type": "markdown",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "source": [
     "---\n",
     "### <span style=\"color:#8735fb; font-size:20pt\"> Option 1: Use an Azure Marketplace VM image </span>\n",
@@ -140,11 +146,10 @@
     "        \"logging.distributed\": \"info\",\n",
     "        \"cloudprovider.azure.azurevm.marketplace_plan\": {\n",
     "            \"publisher\": \"nvidia\",\n",
-    "            \"name\": \"ngc-base-version-23_03_0\",\n",
+    "            \"name\": \"ngc-base-version-23_03_02\",\n",
     "            \"product\": \"ngc_azure_17_11\",\n",
     "            \"version\": \"23.03.0\",\n",
     "        },\n",
-    "        \"cloudprovider.azure.azurevm.disk_size\": 200,\n",
     "    }\n",
     ")\n",
     "vm_image = \"\"\n",
@@ -165,7 +170,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "! az vm image terms accept --urn \"nvidia:ngc_azure_17_11:ngc-base-version-23_03_0:23.03.0\" --verbose"
+    "! az vm image terms accept --urn \"nvidia:ngc_azure_17_11:ngc-base-version-23_03_02:23.03.0\" --verbose"
    ]
   },
   {
@@ -453,7 +458,9 @@
   },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "source": [
     "### <span style=\"color:#8735fb; font-size:20pt\"> Step 2.1: Start the VM Cluster in Azure </span>\n",
     "\n",
@@ -602,7 +609,9 @@
   },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "source": [
     "### <span style=\"color:#8735fb; font-size:18pt\"> Step 3.a: Set up the workers for reading parquet files from Azure Data Lake endpoints </span>"
    ]
@@ -1321,9 +1330,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "rapids-23.06",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
-   "name": "rapids-23.06"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -1335,7 +1344,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
+   "version": "3.9.7"
   }
  },
  "nbformat": 4,