cuDF-XGBoost-singleGPU.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align=center>Accelerating Data Science Workflows with RAPIDS</h1>\n",
    "\n",
    "[RAPIDS](https://rapids.ai/) is a suite of open source software libraries that gives you the ability to accelerate and execute end-to-end data science workflows entirely on GPUs. RAPIDS relies on NVIDIA CUDA® primitives for low-level compute optimization, GPU parallelism, and high-bandwidth memory speed through user-friendly Python interfaces and APIs that are familiar to users of Pandas, Scikit-learn and Dask."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to cuDF and XGBoost\n",
    "\n",
    "In this lab we will discuss couple of packages in RAPIDS such as cuDF (DataFrame library interoperable with Pandas) and GPU accelerated XGBoost.You will work through a series of exercises to port and refactor CPU code onto GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"prerequisites\"></a>\n",
    "## Prerequisites\n",
    "\n",
    "This lab is not an introduction to Data Science. We'll assume that you have background in Data Science and experience with the following programming tools and techniques:\n",
    "\n",
    "- [Python 3 programming language](https://docs.python.org/)\n",
    "- [Pandas Data Analysis Library](https://pandas.pydata.org/)\n",
    "- [NumPy Library for Numerical Programming](http://www.numpy.org/)\n",
    "- Machine learning model training with [XGBoost](https://xgboost.readthedocs.io/)\n",
    "- Python plotting with [Matplotlib](https://matplotlib.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Install RAPIDS libraries: cuDF, cuML, cuGraph, XGBoost:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://github.com/zronaghi/Clemson-workshop/raw/master/utils/rapids-install.sh\n",
    "!bash rapids-install.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we begin, let's check out our hardware setup by running the `nvidia-smi` command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also check the CUDA version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvcc --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"libraries\"></a>\n",
    "## Load Libraries\n",
    "\n",
    "Let's load some of the RAPIDS libraries that we'll be using and check versions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cudf; print('cuDF version:', cudf.__version__)\n",
    "import xgboost as xgb; print('XGBoost version:', xgb.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additional libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np; print('numpy version:', np.__version__)\n",
    "import pandas as pd; print('pandas version:', pd.__version__)\n",
    "import sklearn; print('Scikit-learn version:', sklearn.__version__)\n",
    "\n",
    "from sklearn import datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score\n",
    "\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"generate\"></a>\n",
    "## Generate Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use sklearn.datasets to simulate data and build synthetic sub-datasets (SwissRolls and Blobs), combine these sub-datasets and then use a trained model to determine sample's sub-dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#number of total samples\n",
    "nSamples = 10000000\n",
    "\n",
    "SamplesPerDatas = nSamples//2\n",
    "\n",
    "swissrolls= datasets.make_swiss_roll( n_samples = SamplesPerDatas, noise = .005)[0]\n",
    "\n",
    "blobs = datasets.make_blobs( n_samples = SamplesPerDatas, centers = 5,  n_features = 3, cluster_std = 0.25,  random_state = 0)[0] + [0, 1.5, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#features \n",
    "X = np.vstack([blobs, swissrolls])\n",
    "\n",
    "#generate labels \n",
    "blobsLabels = np.zeros(blobs.shape[0])\n",
    "rollsLabels = 1 * np.ones(swissrolls.shape[0])\n",
    "\n",
    "y = np.hstack( [blobsLabels, rollsLabels])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train and Test Data\n",
    "\n",
    "We'll split our dataset into a 75% training dataset and a 25% validation dataset:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Train Data (75% of total data) - Use to optimize model's parameters\n",
    "- Test Data (25% of total data) - Use to evaluate trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split( X, y, test_size = 0.25, random_state = 0, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the dimensions of these dataets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('X_train: ', X_train.shape, X_train.dtype, 'y_train: ', y_train.shape, y_train.dtype)\n",
    "print('X_test', X_test.shape, X_test.dtype, 'y_validation: ', y_test.shape, y_test.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"visualize\"></a>\n",
    "## Visualize Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define a function for plotting using matplotlib:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_data( data, colorStack = 'green',  \n",
    "                  ax3D = False, markerScale=1):\n",
    "    \n",
    "    ax3D = plt.figure(figsize=(12,12)).gca(projection='3d')\n",
    "        \n",
    "    ax3D.scatter(data[0:10000,0], \n",
    "                 data[0:10000,1], \n",
    "                 data[0:10000,2], s = 20*markerScale, c=colorStack, depthshade=False)\n",
    "    \n",
    "    ax3D.view_init(elev=10, azim=95)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_data(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ETL\n",
    "\n",
    "Let's write the dataset to disk (as a comma separated file - CSV) and demonstrate data loading:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "pd.DataFrame(data = X_train).to_csv('X_train.csv', index = False)\n",
    "pd.DataFrame(data = X_test).to_csv('X_test.csv', index = False)\n",
    "pd.DataFrame(data = y_train).to_csv('y_train.csv', index = False)\n",
    "pd.DataFrame(data = y_test).to_csv('y_test.csv', index = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#check size of data on disk\n",
    "!du -h *csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Data CPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "startTime = time.time()\n",
    "\n",
    "pd_X_train = pd.read_csv('X_train.csv',  delimiter=',')\n",
    "pd_X_test = pd.read_csv('X_test.csv',  delimiter=',')\n",
    "pd_y_train = pd.read_csv('y_train.csv',  delimiter=',')\n",
    "pd_y_test = pd.read_csv('y_test.csv',  delimiter=',')\n",
    "\n",
    "PandasIngestion = time.time() - startTime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Data GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RAPIDS enables reading data from disk directly to GPU memory using cuDF (DataFrame manipulation library) with a similar API to Pandas. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise:\n",
    "Use cuDF to load data onto GPU memory, [cuDF API Reference](https://rapidsai.github.io/projects/cudf/en/latest/api.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read csv files into cudf_X_train, cudf_X_test, cudf_y_train, cudf_y_test "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Data load on GPU is {:.2f}x faster than CPU\".format(PandasIngestion/cuDFIngestion))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Solution](#solution1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Training with XGBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's convert our DataFrames to a DMatrix object for XGBoost training. We can instantiate an object of the xgboost.DMatrix by passing in the feature matrix as the first argument followed by the label vector using the label keyword argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "startTime = time.time()\n",
    "\n",
    "train_DataAndLabelsCPU = xgb.DMatrix(pd_X_train, label=pd_y_train)\n",
    "test_DataAndLabelsCPU = xgb.DMatrix(pd_X_test, label=pd_y_test)\n",
    "\n",
    "CPUDMatrix = time.time() - startTime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "startTime = time.time()\n",
    "\n",
    "train_DataAndLabelsGPU = xgb.DMatrix(cudf_X_train, label=cudf_y_train)\n",
    "test_DataAndLabelsGPU = xgb.DMatrix(cudf_X_test, label=cudf_y_test)\n",
    "\n",
    "GPUDMatrix = time.time() - startTime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"DMatrix conversion on GPU is {:.2f}x faster than CPU\".format(CPUDMatrix/GPUDMatrix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"parameters\"></a>\n",
    "## Set Parameters\n",
    "\n",
    "There are a number of parameters that can be set before training XGBoost model:\n",
    "\n",
    "* General parameters relate to which booster we are using, commonly tree or linear model\n",
    "* Booster parameters depend on which booster you have chosen\n",
    "* Learning task parameters decide on the learning scenario\n",
    "\n",
    "For all available options execute the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#?xgb.XGBClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CPU Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nCores = !nproc --all\n",
    "nCores = int(nCores[0])\n",
    "print(nCores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# instantiate params\n",
    "paramsCPU = {}\n",
    "\n",
    "# booster params\n",
    "booster_params = {\n",
    "    'max_depth': 6,\n",
    "    'num_class': 3,\n",
    "    'tree_method':'hist',\n",
    "    'random_state': 0,\n",
    "    'n_jobs': nCores\n",
    "}  \n",
    "paramsCPU.update(booster_params)\n",
    "\n",
    "# learning task params\n",
    "learning_task_params = {\n",
    "    'objective': 'multi:softmax'\n",
    "}\n",
    "paramsCPU.update(learning_task_params)\n",
    "\n",
    "print(paramsCPU)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GPU Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using XGBoost to train models on the GPU is very similar to CPU, we need to change couple of parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "paramsGPU = {}\n",
    "\n",
    "booster_params = {\n",
    "    'max_depth': 6,\n",
    "    'num_class': 3,\n",
    "    'tree_method':'gpu_hist',\n",
    "    'random_state': 0,\n",
    "    'n_gpus': 1\n",
    "}  \n",
    "paramsGPU.update(booster_params)\n",
    "\n",
    "learning_task_params = {\n",
    "    'objective': 'multi:softmax'\n",
    "}\n",
    "paramsGPU.update(learning_task_params)\n",
    "\n",
    "print(paramsGPU)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"train\"></a>\n",
    "## Train XGBoost Classification Model\n",
    "\n",
    "Now it's time to train our model! We can use the `xgboost.train` function and pass in the parameters, training dataset, the number of boosting iterations, and the list of items to be evaluated during training. \n",
    "The wall time output indicates how long it took to train an XGBoost model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model training settings\n",
    "num_round = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train on CPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "startTime = time.time()\n",
    "\n",
    "xgBoostModelCPU = xgb.train( dtrain = train_DataAndLabelsCPU, params = paramsCPU, num_boost_round = num_round )\n",
    "\n",
    "CPUXGB = time.time() - startTime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train on GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use GPU DMatrix and parameters to train the model on GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Solution](#solution2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Training GPU is {:.2f}x faster than CPU\".format(CPUXGB/GPUXGB))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"predict\"></a>\n",
    "## Evaluate Model\n",
    "\n",
    "Generate predictions and evaluate model based on accuracy score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hint: use predict from xgboost and accuracy_score from sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Solution](#solution3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "In this notebook, we showed how to use GPU DataFrames and XGBoost in RAPIDS.\n",
    "\n",
    "To learn more about RAPIDS check out: \n",
    "\n",
    "* [RAPIDS Website](http://rapids.ai)\n",
    "* [RAPIDS on GitHub](https://github.com/rapidsai/)\n",
    "* [NVIDIA Data Science Webpage](https://www.nvidia.com/en-us/deep-learning-ai/solutions/data-science/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='solution1'></a>\n",
    "#### Solution 1: Load Data GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "startTime = time.time()\n",
    "\n",
    "cudf_X_train = cudf.read_csv('X_train.csv', delimiter=',')\n",
    "cudf_X_test = cudf.read_csv('X_test.csv', delimiter=',')\n",
    "cudf_y_train = cudf.read_csv('y_train.csv', delimiter=',')\n",
    "cudf_y_test = cudf.read_csv('y_test.csv', delimiter=',')\n",
    "\n",
    "cuDFIngestion = time.time() - startTime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='solution2'></a>\n",
    "#### Solution 2: Train XGBoost on GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "startTime = time.time()\n",
    "\n",
    "xgBoostModelGPU = xgb.train( dtrain = train_DataAndLabelsGPU, params = paramsGPU, num_boost_round = num_round )\n",
    "\n",
    "GPUXGB = time.time() - startTime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='solution3'></a>\n",
    "#### Solution 3: Evaluate Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "yPredTrainGPU = xgBoostModelGPU.predict(train_DataAndLabelsGPU)\n",
    "yPredTestGPU = xgBoostModelGPU.predict(test_DataAndLabelsGPU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( 'GPU test accuracy: {0:.6f} '.format( accuracy_score(pd_y_test, yPredTestGPU) ))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}