Intro-to-cuDF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Intro to cuDF\n",
    "=======================\n",
    "\n",
    "cuDF is a Python GPU DataFrame library (built on the Apache Arrow columnar memory format) for loading, joining, aggregating, filtering, and manipulating data. We'll start by introducing the pandas library, and move on to cuDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import math\n",
    "np.random.seed(12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"pandas\"></a>\n",
    "## Pandas\n",
    "\n",
    "Data scientists typically work with two types of data: unstructured and structured. Unstructured data often comes in the form of text, images, or videos. Structured data - as the name suggests - comes in a structured form, often represented by a table or CSV. We'll focus the majority of these tutorials on working with structured data.\n",
    "\n",
    "Pandas represents data in a table and allows a data scientist to manipulate the data to perform a number of useful operations such as filtering, transforming, aggregating, merging, visualizing and many more. More information on Pandas: http://pandas.pydata.org/pandas-docs/stable/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd; print('Pandas Version:', pd.__version__)\n",
    "\n",
    "\n",
    "# here we create a Pandas DataFrame with\n",
    "# two columns named \"key\" and \"value\"\n",
    "df = pd.DataFrame()\n",
    "df['key'] = [0, 0, 2, 2, 3]\n",
    "df['value'] = [float(i + 10) for i in range(5)]\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can perform many operations on this data. For example, let's say we wanted to sum all values in the `value` column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aggregation = df['value'].sum()\n",
    "print(aggregation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"cudf\"></a>\n",
    "## cuDF\n",
    "\n",
    "Pandas is fantastic for working with small datasets that fit into your system's memory and workflows that are not computationally intense. However, datasets are growing larger and data scientists are working with increasingly complex workloads - the need for accelerated computing is increasing rapidly.\n",
    "\n",
    "cuDF is a package within the RAPIDS ecosystem that allows data scientists to easily migrate their existing Pandas workflows from CPU to GPU, where computations can leverage the immense parallelization that GPUs provide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cudf; print('cuDF Version:', cudf.__version__)\n",
    "\n",
    "\n",
    "# here we create a cuDF DataFrame with\n",
    "# two columns named \"key\" and \"value\"\n",
    "df = cudf.DataFrame()\n",
    "df['key'] = [0, 0, 2, 2, 3]\n",
    "df['value'] = [float(i + 10) for i in range(5)]\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take this cuDF DataFrame and perform a `sum` operation over the `value` column. The key difference is that any operations we perform using cuDF use the GPU instead of the CPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aggregation = df['value'].sum()\n",
    "print(aggregation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DataFrame Basics with cuDF\n",
    "\n",
    "In the following tutorial, you'll get a chance to familiarize yourself with cuDF. For those of you with experience using pandas, this should look nearly identical.\n",
    "\n",
    "Portions of this were borrowed from the 10 Minutes to cuDF guide."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Object Creation\n",
    "---------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a `cudf.Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = cudf.Series([1,2,3,None,4])\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a `cudf.DataFrame` by specifying values for each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.DataFrame([('a', list(range(20))),\n",
    "('b', list(reversed(range(20)))),\n",
    "('c', list(range(20)))])\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a `cudf.DataFrame` from a `pd.Dataframe`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf = pd.DataFrame({'a': [0, 1, 2, 3],'b': [0.1, 0.2, None, 0.3]})\n",
    "gdf = cudf.DataFrame.from_pandas(pdf)\n",
    "print(gdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Viewing Data\n",
    "-------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Viewing the top rows of a GPU dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.head(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sorting by values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.sort_values(by='b'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selection\n",
    "------------\n",
    "\n",
    "## Getting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selecting a single column, which initially yields a `cudf.Series` (equivalent to `df.a`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df['a'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selection by Label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selecting rows from index 2 to index 5 from columns `a` and `b`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.loc[2:5, ['a', 'b']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selection by Position"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selecting via integers and integer slices, like numpy/pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.iloc[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.iloc[0:3, 0:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also select elements of a `DataFrame` or `Series` with direct index access."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df[3:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(s[3:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1\n",
    "\n",
    "Try to select only the rows at index `4` and `9` from `df`.\n",
    "\n",
    "<details><summary><b>Solution</b></summary>\n",
    "   <pre>\n",
    "    <br>print(df.iloc[[4,9]])\n",
    "   </pre>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Boolean Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selecting rows in a `DataFrame` or `Series` by direct Boolean indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df[df.b > 15])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selecting values from a `DataFrame` where a Boolean condition is met, via the `query` API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.query(\"b == 3\"))  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val = 3\n",
    "df.query(\"b == @val\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also pass local variables to cuDF queries, via the `local_dict` keyword or `@` operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cudf_comparator = 3\n",
    "print(df.query(\"b == @cudf_comparator\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Supported logical operators include `>`, `<`, `>=`, `<=`, `==`, and `!=`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2\n",
    "\n",
    "Try to select only the rows from `df` where the value in column `b` is greater than the value in column `c` + 6.\n",
    "\n",
    "<details><summary><b>Solution</b></summary>\n",
    "   <pre>\n",
    "    <br>print(df.query(\"b > c + 6\"))\n",
    "   </pre>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Missing Data\n",
    "------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Missing data can be replaced by using the `fillna` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(s.fillna(999))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Operations\n",
    "------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculating descriptive statistics for a `Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = cudf.Series(np.arange(10)).astype(np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(s.mean(), s.var())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applymap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Applying functions to a `Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_ten(num):\n",
    "    return num + 10\n",
    "\n",
    "print(s.applymap(add_ten))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## String Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like pandas, cuDF provides string processing methods in the `str` attribute of `Series`. Full documentation of string methods is a work in progress. Please see the cuDF and [nvStrings API](https://docs.rapids.ai/api/nvstrings/nightly/) documentation for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = cudf.Series(['A', 'B', 'C', 'Aaba', 'Baca', None, 'CABA', 'dog', 'cat'])\n",
    "print(s.str.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3\n",
    "\n",
    "Try to convert all the strings to uppercase. Take a look at the nvStrings API documentation linked above if you need some help.\n",
    "\n",
    "<details><summary><b>Solution</b></summary>\n",
    "   <pre>\n",
    "    <br>print(s.str.upper())\n",
    "   </pre>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Concatenating `Series` and `DataFrames` row-wise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = cudf.Series([1, 2, 3, None, 5])\n",
    "print(cudf.concat([s, s]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Append"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Appending values from another `Series` or array-like object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(s.append(s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Join"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Performing SQL style merges. Note that the dataframe order is not maintained, but may be restored post-merge by sorting by the index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_a = cudf.DataFrame()\n",
    "df_a['key'] = ['a', 'b', 'c', 'd', 'e']\n",
    "df_a['vals_a'] = [float(i + 10) for i in range(5)]\n",
    "\n",
    "df_b = cudf.DataFrame()\n",
    "df_b['key'] = ['a', 'c', 'e']\n",
    "df_b['vals_b'] = [float(i+100) for i in range(3)]\n",
    "\n",
    "merged = df_a.merge(df_b, on=['key'], how='left')\n",
    "print(merged)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4\n",
    "\n",
    "Using the DataFrames we created above, try to do an `inner` join using `merge`.\n",
    "\n",
    "<details><summary><b>Solution</b></summary>\n",
    "   <pre>\n",
    "    <br>print(df_a.merge(df_b, on=['key'], how='inner'))\n",
    "   </pre>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grouping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like pandas, cuDF supports the [Split-Apply-Combine](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html) groupby paradigm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['agg_col1'] = [1 if x % 2 == 0 else 0 for x in range(len(df))]\n",
    "df['agg_col2'] = [1 if x % 3 == 0 else 0 for x in range(len(df))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grouping and then applying the `sum` function to the grouped data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.groupby('agg_col1').sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grouping and applying statistical functions to specific columns, using `agg`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.groupby('agg_col1').agg({'a':'max', 'b':'mean', 'c':'sum'}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5\n",
    "\n",
    "We can also group by multiple columns at once, which we call grouping hierarchically. Try to group `df` by `agg_col1` and `agg_col2` and calculate the mean of column `a` and minimum of column `b`.\n",
    "\n",
    "<details><summary><b>Solution</b></summary>\n",
    "   <pre>\n",
    "    <br>print(df.groupby(['agg_col1', 'agg_col2']).agg({'a':'mean', 'b':'min'}).to_pandas())\n",
    "   </pre>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time Series\n",
    "------------\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DataFrames` supports `datetime` typed columns, which allow users to interact with and filter data based on specific timestamps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "date_df = cudf.DataFrame()\n",
    "date_df['date'] = pd.date_range('11/20/2018', periods=72, freq='D')\n",
    "date_df['value'] = np.random.sample(len(date_df))\n",
    "print(date_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6\n",
    "\n",
    "Try to use `query` to filter `date_df` to only those row with a date before `2018-11-23`. This is a bit trickier than the prior exercises. As a hint, you'll want to explore the `to_datetime` function from the `pandas` library.\n",
    "\n",
    "<details><summary><b>Solution</b></summary>\n",
    "   <pre>\n",
    "    <br>\n",
    "    search_date = pd.to_datetime('2018-11-23')\n",
    "    print(date_df.query(\"date &lt= @search_date\"))\n",
    "            </br>\n",
    "   </pre>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also interact with datetime columns to extract things like the day, hour, minute, and more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "date_df['minute'] = date_df.date.dt.minute\n",
    "print(date_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Converting Data Representation\n",
    "--------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Converting a cuDF `DataFrame` to a pandas `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.head().to_pandas())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Converting a cuDF `DataFrame` to a numpy `ndarray`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.as_matrix())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Converting a cuDF `Series` to a numpy `ndarray`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df['a'].to_array())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting Data In/Out\n",
    "------------------------\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CSV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Writing to a CSV file, by first sending data to a pandas `Dataframe` on the host."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists('example_output'):\n",
    "    os.mkdir('example_output')\n",
    "    \n",
    "df.to_pandas().to_csv('example_output/foo.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading from a csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.read_csv('example_output/foo.csv')\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it! You've got the basics of cuDF down! Let's talk a little bit about the computational performance of cuDF and GPUs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Performace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the primary reasons to use cuDF over pandas is performance. For some workflows, the GPU can be **much** faster than the CPU. Let's illustrate this by starting with a small example: creating a DataFrame and calculating the sum of a column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.random.rand(10000000) # 10 million values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf = pd.DataFrame()\n",
    "cdf = cudf.DataFrame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time pdf['a'] = a\n",
    "%time cdf['a'] = a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit\n",
    "pdf['a'].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit\n",
    "cdf['a'].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A More Realistic Example: Sensor Data Analytics\n",
    "\n",
    "To get a more realistic sense of how powerful cuDF and GPUs can be, let's imagine you had a fleet of sensors that collect data every millisecond. These sensors could be measuring pressure, temperature, or something else entirely.\n",
    "\n",
    "Let's imagine we want to analyze one day's worth of sensor data. We'll assign random values for the sensor `value` to use for this example. We'll start by creating the data, and generating some datetime related features like we learned about in the above tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "date_df = pd.DataFrame()\n",
    "date_df['date'] = pd.date_range(start='2019-07-05', end='2019-07-06', freq='ms')\n",
    "date_df['value'] = np.random.sample(len(date_df))\n",
    "\n",
    "date_df['hour'] = date_df.date.dt.hour\n",
    "date_df['minute'] = date_df.date.dt.minute\n",
    "\n",
    "print(date_df.shape)\n",
    "date_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From our sensor data, we want to get the maximum sensor value for each minute. Since we don't want to combine values in the same minute of different hours, we'll need to do a hierarchical groupby."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time results = date_df.groupby(['hour', 'minute']).agg({'value':'max'})\n",
    "results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is fairly slow! Imagine if we had a fleet of sensors. Time would become a serious concern.\n",
    "\n",
    "Let's try this in cuDF now, using the GPU DataFrame. We'll run the same code as above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "date_df = cudf.DataFrame()\n",
    "date_df['date'] = pd.date_range(start='2019-07-05', end='2019-07-06', freq='ms')\n",
    "date_df['value'] = np.random.sample(len(date_df))\n",
    "\n",
    "date_df['hour'] = date_df.date.dt.hour\n",
    "date_df['minute'] = date_df.date.dt.minute\n",
    "date_df['second'] = date_df.date.dt.second\n",
    "\n",
    "print(date_df.shape)\n",
    "print(date_df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time results = date_df.groupby(['hour', 'minute', 'second']).agg({'value':'max'})\n",
    "results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the results may vary slightly, it should be clear that GPU acceleration can make a significant difference. We can get much faster results with the same code!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 7\n",
    "\n",
    "Play around with some more pandas and cuDF operations and compare the performance between them. What operations can you find that gives the highest performance ratio? \n",
    "\n",
    "You can start a cell with `%%time` to time the cell, or with `%%timeit`, which runs the cell multiple times and gives an average. `%%timeit` gives a more accurate benchmark but takes longer to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# User Defined Functions with cuDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, the built-in methods of cudf.DataFrame don't do exactly what we want. We need to write a custom function (also known as a user defined function) to apply over the DataFrame.\n",
    "\n",
    "cuDF’s DataFrame class has two primary methods that let users run custom Python functions on GPUs: `apply_rows` and `apply_chunks`. We’ll walk through how to use `apply_rows` to create your own UDFs and show how you can implement a GPU-accelerated windowing function.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `apply_rows`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`apply_rows` processes each of the DataFrame rows independently in parallel. Under the hood, the `apply_rows` method will optimally divide the long columns into chunks, and assign chunks into different GPU blocks for parallel computation. \n",
    "\n",
    "In order to use `apply_rows`, we need to write a kernel function. A kernel function is a function that will be executed on each row of the DataFrame set the output value for each row. **The execution order of rows is arbitrary, so each execution of the function MUST be independent of other execution.**\n",
    "\n",
    "At a very high level, in cuDF's user defined functions:\n",
    "\n",
    "\n",
    "- Compute is spread across multiple \"blocks\", which have access to both global memory but also their own in-block memory \n",
    "- Within each block, many \"threads\" operate independently and can quickly access data in their block-specific shared memory \n",
    "\n",
    "\n",
    "As a result, the loop in the example function below resembles serial code, but executes in parallel in multiple threads on the GPU. When `kernel` is invoked, the function arguments corresponding to the input/output are strided so as to improve GPU parallelism. The kernel function is compiled to the GPU using `numba.cuda`, so the kernel function must only use Python features/functions that are [supported](https://numba.pydata.org/numba-doc/dev/cuda/cudapysupported.html) by Numba for CUDA."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A kernel function takes the form\n",
    "\n",
    "```python\n",
    "def kernel(in1, in2, in3, ..., out1, out2, ..., kwarg1, kwarg2, ...):\n",
    "    for i, (x, y, z, ...) in enumerate(zip(in1, in2, in3, ...)):\n",
    "        out1[i] = ...\n",
    "        out2[i] = ...\n",
    "```\n",
    "\n",
    "`in1, in2, in3, ...` are the input columns. `out1, out2, ...` are the output columns. The kernel function should not return a result. Instead, output columns are passed as arguments and the result is written to them. Each thread writes a result to a specific index in the output column, which is why write `out1[i] = ...`.\n",
    "\n",
    "Additional keyword arguments can be passed (`kwarg1, kwarg2, ...`). Inside the kernel function, [standard numba.cuda attributes](https://numba.pydata.org/numba-doc/dev/cuda/kernels.html#thread-positioning) like `numba.cuda.threadIdx` can be used to access things like the thread or block indices. We'll explain a little bit more about this below.\n",
    "\n",
    "To execute this function on our DataFrame, we use `apply_rows`. `apply_rows` is called like:\n",
    "\n",
    "```python\n",
    "df = df.apply_rows(kernel\n",
    "                   incols=['in1', 'in2', 'in3', ...],\n",
    "                   outcols={'out1': np.float64, 'out2': np.int8, ...},\n",
    "                   kwargs={'kwarg1': val1, 'kwarg2': val2, ...})\n",
    "```\n",
    "\n",
    "`incols` is a list of the arguments for our `kernel` function representing the columns in the DataFrame. As a result, `in1`, `in2`, etc. must match the names of columns in the DataFrame that we intend to use.\n",
    "\n",
    "`outcols` is a dictionary mapping the output column names to their dtype. If we intend to generate two output columns, we need to `outcols` needs to contain two keys (`out1` and `out2`).\n",
    "\n",
    "`kwargs` is a dictionary mapping the keyword argument parameters to their values. If our `kernel` function needs additional arguments contained in our DataFrame, we can pass them in here. `kwargs` can be an empty dictionary if there are no keyword arguments.\n",
    "\n",
    "After calling `apply_rows` as above, `df` would have extra columns `out1`, `out2`, ... with the output results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Haversine distance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example below, we create a DataFrame representing pairs of latitude and longitude points. We use `apply_rows` to calculate the [Haversine distance](https://en.wikipedia.org/wiki/Haversine_formula) between two points in the input arrays.\n",
    "\n",
    "$$\n",
    "d = 2r \\arcsin\\left(\\sqrt{\\sin^2\\left(\\frac{\\varphi_2 - \\varphi_1}{2}\\right) + \\cos(\\varphi_1) \\cos(\\varphi_2)\\sin^2\\left(\\frac{\\lambda_2 - \\lambda_1}{2}\\right)}\\right)\n",
    "$$\n",
    "\n",
    "where $\\varphi_1,\\varphi_2$ are the latitudes and $\\lambda_1,\\lambda_2$ are the longitudes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import cos, sin, asin, sqrt, pi, atan2\n",
    "\n",
    "import time\n",
    "import cudf\n",
    "import numpy as np\n",
    "from numba import cuda"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(12)\n",
    "data_length = 1000\n",
    "\n",
    "df = cudf.DataFrame()\n",
    "df['lat1'] = np.random.normal(10, 1, data_length)\n",
    "df['lon1'] = np.random.normal(10, 1, data_length)\n",
    "df['lat2'] = np.random.normal(10, 1, data_length)\n",
    "df['lon2'] = np.random.normal(10, 1, data_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def haversine_distance_kernel(lat1, lon1, lat2, lon2, out):\n",
    "    \"\"\"Haversine distance formula taken from Michael Dunn's StackOverflow post:\n",
    "    https://stackoverflow.com/questions/4913349/haversine-formula-in-python-bearing-and-distance-between-two-gps-points\n",
    "    \"\"\"\n",
    "    for i, (x_1, y_1, x_2, y_2) in enumerate(zip(lat1, lon1, lat2, lon2)):\n",
    "        print('thread id:', cuda.threadIdx.x, 'block id:', cuda.blockIdx.x,\n",
    "              'array size:', lat1.size, 'block threads:', cuda.blockDim.x)\n",
    "\n",
    "        x_1 = pi/180 * x_1\n",
    "        y_1 = pi/180 * y_1\n",
    "        x_2 = pi/180 * x_2\n",
    "        y_2 = pi/180 * y_2\n",
    "        \n",
    "        dlon = y_2 - y_1\n",
    "        dlat = x_2 - x_1\n",
    "        a = sin(dlat/2)**2 + cos(x_1) * cos(x_2) * sin(dlon/2)**2\n",
    "        \n",
    "        c = 2 * asin(sqrt(a)) \n",
    "        r = 6371 # Radius of earth in kilometers\n",
    "        \n",
    "        out[i] = c * r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.apply_rows(haversine_distance_kernel,\n",
    "                   incols=['lat1', 'lon1', 'lat2', 'lon2'],\n",
    "                   outcols=dict(out=np.float64),\n",
    "                   kwargs=dict())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example above, the printed output from applying our `haversine_distance_kernel` function shows some informative information. If you were to look at the entire printed output, you'd notice a few things:\n",
    "\n",
    "- The processing was spread across 15 CUDA blocks\n",
    "- Within each block, 64 separate threads were used for computation.\n",
    "- In this case, most threads in a block handled one element from the input array, but some threads have to deal with two elements, because there are 1000 rows and 960 threads (15 blocks * 64 threads per block)\n",
    "\n",
    "`apply_rows` handled all of this for us!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Exercise 8**\n",
    "\n",
    "Modify the above example to pass in the radius of the earth `r` as a keyword argument to the kernel.\n",
    "\n",
    "<details><summary><b>Solution</b></summary>\n",
    "   <pre>\n",
    "def haversine_distance_kernel(lat1, lon1, lat2, lon2, out, r):\n",
    "    \"\"\"Haversine distance formula taken from Michael Dunn's StackOverflow post:\n",
    "    https://stackoverflow.com/questions/4913349/haversine-formula-in-python-bearing-and-distance-between-two-gps-points\n",
    "    \"\"\"\n",
    "    for i, (x_1, y_1, x_2, y_2) in enumerate(zip(lat1, lon1, lat2, lon2)):\n",
    "        print('thread_id:', cuda.threadIdx.x, 'bid:', cuda.blockIdx.x,\n",
    "              'array size:', lat1.size, 'block threads:', cuda.blockDim.x)\n",
    "\n",
    "        x_1 = pi/180 * x_1\n",
    "        y_1 = pi/180 * y_1\n",
    "        x_2 = pi/180 * x_2\n",
    "        y_2 = pi/180 * y_2\n",
    "        \n",
    "        dlon = y_2 - y_1\n",
    "        dlat = x_2 - x_1\n",
    "        a = sin(dlat/2)**2 + cos(x_1) * cos(x_2) * sin(dlon/2)**2\n",
    "        \n",
    "        c = 2 * asin(sqrt(a)) \n",
    "        \n",
    "        out[i] = c * r\n",
    "   </pre>\n",
    "    <pre>\n",
    "df = df.apply_rows(haversine_distance_kernel,\n",
    "                   incols=['lat1', 'lon1', 'lat2', 'lon2'],\n",
    "                   outcols=dict(out=np.float64),\n",
    "                   kwargs=dict(r=6371))\n",
    "print(df.head()\n",
    "</pre>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}