scikit_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A first example of machine learning\n",
    "==\n",
    "In this notebook we'll apply a scikit-learn pipeline to a simple dataset (the listing of apartments in Airbnb of Berlin), and see how overfitting looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T21:59:01.195267Z",
     "start_time": "2019-01-05T21:59:01.003209Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can invoke system commands by prepending them with a `!`, commands like `head`, `tail`, `wc` can be useful to quickly inspect a text file. Most of them are not present on Windows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T21:59:01.345919Z",
     "start_time": "2019-01-05T21:59:01.197499Z"
    }
   },
   "outputs": [],
   "source": [
    "!head listings.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "numpy provides the function `loadtxt` to load simple CSV files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T21:59:01.355499Z",
     "start_time": "2019-01-05T21:59:01.349177Z"
    }
   },
   "outputs": [],
   "source": [
    "#np.loadtxt('listings.csv', delimiter=',', usecols=(54, 59, 48, 49, 79 ), skiprows=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It does not work because this file contains newlines inside the fields. Luckily the Python CSV module can still process it.\n",
    "\n",
    "This code loads some columns from the CSV into separate numpy arrays.\n",
    "\n",
    "First, we create plain Python lists, then replace them with proper arrays (faster and smaller).\n",
    "\n",
    "Don;t worry: with Pandas this kind of operation becomes much easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T21:59:03.589649Z",
     "start_time": "2019-01-05T21:59:01.360913Z"
    }
   },
   "outputs": [],
   "source": [
    "from csv import DictReader\n",
    "\n",
    "review_scores_rating = []\n",
    "price = []\n",
    "latitude = []\n",
    "longitude = []\n",
    "bathrooms = []\n",
    "\n",
    "for l in DictReader(open('listings.csv')):\n",
    "    price.append(l['price'])\n",
    "    review_scores_rating.append(l['review_scores_rating'])\n",
    "    latitude.append(l['latitude'])\n",
    "    longitude.append(l['longitude'])\n",
    "    bathrooms.append(l['bathrooms'])\n",
    "\n",
    "latitude = np.array([float(l) for l in latitude])\n",
    "longitude = np.array([float(l) for l in longitude])\n",
    "price = np.array([float(l[1:].replace(',', '')) for l in price])\n",
    "\n",
    "# We assume the rating is 1 if not specified\n",
    "review_scores_rating = np.array([int(l) if l != '' else 0 for l in review_scores_rating])\n",
    "\n",
    "# We assume there's 1 bathroom if not stated otherwise\n",
    "bathrooms = np.array([float(l) if l != '' else 1 for l in bathrooms])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's very useful to have a look at the shape of the numpy arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T21:59:03.596806Z",
     "start_time": "2019-01-05T21:59:03.591360Z"
    }
   },
   "outputs": [],
   "source": [
    "print(latitude.shape)\n",
    "print(bathrooms.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T21:59:05.079158Z",
     "start_time": "2019-01-05T21:59:03.601693Z"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "# change the figure size\n",
    "from matplotlib.pyplot import figure\n",
    "figure(num=None, figsize=(8, 6), dpi=80)\n",
    "\n",
    "# reshape is needed to create a second dimension of size 1\n",
    "X = price.T.reshape(-1, 1)\n",
    "Y = review_scores_rating.T\n",
    "model = LinearRegression()\n",
    "model.fit(X, Y)\n",
    "model.score(X,Y)\n",
    "\n",
    "\n",
    "plt.scatter(X, Y, marker='X')\n",
    "\n",
    "x_plot = np.linspace(0, 9000, 200)\n",
    "y_plot = model.predict(x_plot.reshape(-1, 1))\n",
    "plt.plot(x_plot, y_plot, color='red')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turns out there are prices much much greater than the rest, making the visualization and the model less effective. Let's ignore them by placing a cap of 500 on the price.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T21:59:05.091151Z",
     "start_time": "2019-01-05T21:59:05.081299Z"
    }
   },
   "outputs": [],
   "source": [
    "too_high = np.argwhere(price > 500)\n",
    "print(f'shape before: {price.shape}')\n",
    "Ylow = np.delete(Y, too_high)\n",
    "Xlow = np.delete(price, too_high).reshape(-1, 1)\n",
    "print(f'shape after: {Xlow.shape}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T21:59:05.486107Z",
     "start_time": "2019-01-05T21:59:05.098969Z"
    }
   },
   "outputs": [],
   "source": [
    "model = LinearRegression()\n",
    "model.fit(Xlow, Ylow)\n",
    "model.score(Xlow, Ylow)\n",
    "\n",
    "\n",
    "plt.scatter(Xlow, Ylow, marker='X')\n",
    "\n",
    "x_plot = np.linspace(0, 500, 200)\n",
    "y_plot = model.predict(x_plot.reshape(-1, 1))\n",
    "plt.plot(x_plot, y_plot, color='red')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-04T14:40:29.991524Z",
     "start_time": "2019-01-04T14:40:29.977525Z"
    }
   },
   "source": [
    "In scikit you can combine models using `make_pipeline`, in this case we combine `PolynomialFeatures` with `LinearRegression`, to run a linear regression on the features generated by the first step, which are the original ones multiplied and to various powers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T22:00:05.692525Z",
     "start_time": "2019-01-05T21:59:05.488667Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "X = np.vstack((latitude, longitude, bathrooms)).T\n",
    "print(f'the shape of X is {X.shape}')\n",
    "Y = review_scores_rating.T\n",
    "print(f'the shape of Y is {Y.shape}')\n",
    "\n",
    "\n",
    "for degree in range(1, 20):\n",
    "    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())\n",
    "    model.fit(X, Y)\n",
    "    score = model.score(X, Y)\n",
    "    print(f'with degree {degree} the score was {score:.5f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-04T15:50:18.004263Z",
     "start_time": "2019-01-04T15:50:17.991237Z"
    }
   },
   "source": [
    "The model reaches the best score at degree 11 (notice it could change with other cities). This seems the best result, but what is happening here is that we have overfitting. The dataset we use to check the model is the same we used to train it.\n",
    "\n",
    "Let's try instead by partitioning the data in train and test datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T22:00:58.558258Z",
     "start_time": "2019-01-05T22:00:05.694891Z"
    }
   },
   "outputs": [],
   "source": [
    "train_X = X[:21000,:]\n",
    "test_X = X[21000:,:]\n",
    "\n",
    "train_Y = Y[:21000]\n",
    "test_Y = Y[21000:]\n",
    "\n",
    "for degree in range(1, 20):\n",
    "    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())\n",
    "    model.fit(train_X, train_Y)\n",
    "    score = model.score(test_X, test_Y)\n",
    "    print(f'with degree {degree} the score was {score}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T22:00:59.052579Z",
     "start_time": "2019-01-05T22:00:58.560697Z"
    }
   },
   "outputs": [],
   "source": [
    "# change the figure size\n",
    "from matplotlib.pyplot import figure\n",
    "figure(num=None, figsize=(8, 6), dpi=80)\n",
    "\n",
    "# reshape is needed to create a second dimension of size 1\n",
    "X = price.T.reshape(-1, 1)\n",
    "\n",
    "\n",
    "\n",
    "model = make_pipeline(PolynomialFeatures(20), LinearRegression())\n",
    "#model = LinearRegression()\n",
    "model.fit(X, Y)\n",
    "model.score(X,Y)\n",
    "\n",
    "\n",
    "\n",
    "plt.scatter(X, Y, marker='X')\n",
    "\n",
    "x_plot = np.linspace(0, 9000, 200)\n",
    "y_plot = model.predict(x_plot.reshape(-1, 1))\n",
    "plt.plot(x_plot, y_plot, color='red')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turns out there are prices much much greater than the rest, making the visualization and the model pointless. Let's ignore them by placing a cap of 500 on the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T22:00:59.627490Z",
     "start_time": "2019-01-05T22:00:59.055636Z"
    }
   },
   "outputs": [],
   "source": [
    "figure(num=None, figsize=(8, 6), dpi=80)\n",
    "\n",
    "\n",
    "model = make_pipeline(PolynomialFeatures(30), LinearRegression())\n",
    "#model = LinearRegression()\n",
    "model.fit(Xlow, Ylow)\n",
    "model.score(Xlow, Ylow)\n",
    "\n",
    "\n",
    "plt.scatter(Xlow, Ylow, marker='X')\n",
    "\n",
    "x_plot = np.linspace(0, 500, 200)\n",
    "y_plot = model.predict(x_plot.reshape(-1, 1))\n",
    "plt.plot(x_plot, y_plot, color='red')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just for fun, let's draw a map of prices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-05T22:01:00.875236Z",
     "start_time": "2019-01-05T22:00:59.631315Z"
    }
   },
   "outputs": [],
   "source": [
    "figure(num=None, figsize=(9, 7), dpi=80)\n",
    "\n",
    "plt.scatter(latitude, longitude, c=review_scores_rating, marker='.', cmap=plt.cm.get_cmap('inferno'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}