01-MNIST-Data-Exploration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Exploration\n",
    "\n",
    "In this notebook, we'll take an opportunity to explore the data we'll be using.\n",
    "Data exploration and understanding is a fundamental step in Machine Learning.\n",
    "\n",
    "In this lab, the data we'll be looking at is the MNIST data set. It consists of 60,000 examples of handwritten digits, with a label corresponding to what the intended digit is. It's worth mentioning that in ML, 60,000 examples is considered a fairly small dataset. Creating an AI to label handwritten digits from the MNIST dataset is a very common, almost \"hello world\" style of AI program\n",
    "\n",
    "It's helpful to know a few characteristics about the dataset in this course in particular. Specifically how it's laid out. In the MNIST dataset we have provided we have done some preprocessing of the data. The data you'll be provided is in CSV format, with the first element corresponding to the actual value, and the rest of the elements corresponding to a 28x28 pixel image of the digit. That 28x28 pixels is flattened in the dataset into 784 individual pixel elements. Each pixel element is in the range of \\[0,255\\] which corresponds to a greyscale. In the case of a color image, each element might be a triple which would correspond to an (R,G,B) value.\n",
    "\n",
    "This dataset will be used to train the model to recognize the relationship between the pixel values of the 28x28 image, and the digit that we tell the model that the pixel values correspond to. The goal of the neural network that you develop over the course of this lab will be to take in an image of a digit, and correctly classify it by outputting the correct digit. No neural network is going to be perfect, but we hope to achieve high accuracy.\n",
    "\n",
    "Accuracy in this lab is defined specifically as the number of correct predictions divided by the total number of predictions.\n",
    "\n",
    "### $ Accuracy = {Correct\\_Predictions \\over Total\\_Predictions} $\n",
    "\n",
    "Note that this metric isn't always the optimal metric for every situation, and some ML systems will need different metrics, but for the purposes of this course, it will work well enough\n",
    "\n",
    "At the end of this notebook, we'll really understand what our data looks like, and that will help us when constructing the AI in the future notebook\n",
    "\n",
    "To start, let's load the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some common imports\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas is a column-oriented data manipulation library, it's very commonly used, and has a parallel in Spark in their dataframe API\n",
    "\n",
    "One of the convenient things is that Pandas can load many formats of data easily"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train_df: pd.DataFrame = pd.read_csv('Resources/mnist_train.csv', header=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Printing a DataFrame will let us examine the data. Helpfully (or unhelpfully) it will truncate the output to roughly match the size of the terminal. In this case we see pretty much all zeros. This might lead us to wonder if there's any data in there at all, so we'll look at a few more ways to look at the data in the coming cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take a look at only a few rows with `head()` or `tail()`. It accepts an argument for how many rows 5 is the default, which is probably what we want\n",
    "\n",
    "Note that when looking at the data, column 0 appears to have our label, while the rest appear to be zeros. If we were to look at a bit more expanded dataframe, we would see that there's some padding around the digits. We'll examine it a bit closer in the next few steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also see some basic statistics about the data in the DataFrame with the `describe()` method\n",
    "This is useful as a basic sanity check that the data corresponds to what we expect\n",
    "\n",
    "These statistics aren't particularly useful for this dataset, but it's helpful to see some of them nonetheless\n",
    "We can see that in columns 775-780 there are some non-zero pixels. That at least tells us some images extend that far"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, this helps, but doesn't really show us the meat of what we need to see to understand the data.\n",
    "\n",
    "Let's now look at a single example. We use the `loc` field to extract a single row and we can look at the field 'values' to get the underlying numpy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.loc[0].values\n",
    "\n",
    "# That looks better, at least now we're sure there's data in there"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have some idea of what the data in the DataFrame looks like, we could proceed, but it's also good to build just a bit more intuition about it since we can see the values but they don't yet really give us a clear picture of what we're looking at\n",
    "\n",
    "In this case, the best way would be to look at them graphically, excluding the first element which is the label\n",
    "\n",
    "To do that, we bust out our trusty pyplot library. Graphical representations of data often are the most intuitive\n",
    "way to examine data, and especially image data\n",
    "\n",
    "To get a good look at our data, let's take that first example and see that the label does correspond to the image in the way we expect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "selected_feature_row = 0\n",
    "# Get the element at [0, 0] (note that this supports the exact same python slicing that you're used to)\n",
    "label = train_df.loc[selected_feature_row, 0]\n",
    "print(\"Label: %d\" % label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that our label for row 0 is 5\n",
    "Now let's look at the rest of it graphically"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We take the rest of row 0 and call it \"features\", which is an ML term that roughly refers to the characteristics of a piece of data corresponding to the label above.\n",
    "\n",
    "ML is all about generating or predicting a \"label\" given a set of \"features\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = train_df.loc[selected_feature_row, 1:].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we could look at this as a single dimensional array when we pass it to the next bit, but instead since it's an image, we want to reshape it into the proper 2d shape. That will give us the most clarity about what we want to see. Note that reshape takes a single tuple, rather than a set of integer arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reshaped_features = features.reshape((28, 28))\n",
    "# Set this to show the full line width, the notebook will truncate it, but it helps show the data better\n",
    "np.set_printoptions(linewidth=150)\n",
    "reshaped_features\n",
    "\n",
    "# We can see it's got some pretty regular structure here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`pyplot.imshow()` takes in an array of values (or n-dimensional array) and prints an image based on their values.\n",
    "\n",
    "It's useful for things like displaying images from pixels, or heatmaps, etc. We're using it for the former, but it has many other uses\n",
    "\n",
    "In our case, we pass in the reshaped 28x28 array from above, and tell it to use the 'greyscale' color map. You can check the documentation for pyplot to learn more about the features and other keywords it supports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Label: %d\" % label)\n",
    "plt.imshow(reshaped_features, cmap='Greys')\n",
    "\n",
    "# Printing this out, we see a pixellated 5, that tells us pretty much what we want to know about the way\n",
    "# the data looks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the below cell to look at other examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "selected_feature_row = np.random.randint(0, 59999)\n",
    "label = train_df.loc[selected_feature_row, 0]\n",
    "features = train_df.loc[selected_feature_row, 1:].values\n",
    "print(\"Label: %d\" % label)\n",
    "plt.imshow(features.reshape((28, 28)), cmap='Greys')"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}