Regression and decision trees coursework.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from collections import defaultdict\n",
    "import sys\n",
    "import seaborn as sn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "SEED_VALUE = 15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1: Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both the regression test and train data are loaded using numpy, and a random seed value is set to ensure that the data is shuffled consistently, allowing our results to be replicable when  the function is run several times. The input data is shuffled to remove any preexisting patterns in the dataset that might act as irrelevant predictors of the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(SEED_VALUE)\n",
    "regression_train_data = np.loadtxt(\"regression_train.csv\", delimiter=',')\n",
    "regression_test_data = np.loadtxt(\"regression_test.csv\", delimiter=',')\n",
    "np.random.shuffle(regression_train_data)\n",
    "np.random.shuffle(regression_test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task 1.1: Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def standardize(data, mean=[], sigma=[]):\n",
    "    if len(mean) == 0:\n",
    "        mean = np.mean(data, axis=0)\n",
    "    if len(sigma) == 0:\n",
    "        sigma = np.std(data, axis=0)\n",
    "    \n",
    "    data_std = np.zeros(data.T.shape)\n",
    "    \n",
    "    for i in range(data.shape[1]):\n",
    "        if sigma[i] == 0:\n",
    "            data_std[i] = data.T[i]\n",
    "        else:\n",
    "            data_std[i] = (data.T[i] - mean[i])/ sigma[i]\n",
    "    \n",
    "    data_std = data_std.T\n",
    "#     data_std = (data - mean) / sigma\n",
    "    return data_std, mean, sigma\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "For this task, we obtain a linear regression model to predict the median value of owner-occupied homes in USD 1000's as your target variable using all the other features as predictors. By using the statistical interpretation of linear regression, we fnid the parameters $\\beta_{\\text ML}$ that maxmize the likelihood:\n",
    "$$\n",
    "p(\\mathcal Y | \\mathcal X, \\boldsymbol\\beta) = \\prod_{n=1}^N p(y_n | \\boldsymbol x_n, \\boldsymbol\\beta)\\,.\n",
    "$$\n",
    "\n",
    "These parameters allow us to formulate our regression line, which we then use to predict values given data points from a test set. The accuracy of our model can be calculated using the mean square error.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "standard deviation of first column:  0.0\n",
      "First column of X_train [1. 1. 1. 1. 1.]\n",
      "First column of X_test [1. 1. 1. 1. 1.]\n"
     ]
    }
   ],
   "source": [
    "X_train = regression_train_data[:, :-1]\n",
    "y_train = regression_train_data[:, -1]\n",
    "X_test = regression_test_data[:, :-1]\n",
    "y_test = regression_test_data[:, -1]\n",
    "\n",
    "# We standardise both the training dataset and the test dataset with the same mean and standard deviation parameters, so that both datasets have predictor features with the same range.\n",
    "X_train, mean_train, sigma_train = standardize(X_train)\n",
    "X_test, _, _ = standardize(X_test, mean_train, sigma_train)\n",
    "\n",
    "print(\"standard deviation of first column: \", sigma_train[0])\n",
    "print(\"First column of X_train\", X_train[:5, 0])\n",
    "print(\"First column of X_test\", X_test[:5, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By inspection of both the test and train dataset, we can see that the first few values of the first column of both X_train and X_test are 1s. The standard deviation of the first column being 0 implies that the first column of both X_train and X_test are filled with 1s. This means that the bias term has been included in the dataset and we do not need to further augment our dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From our lectures, the maximum likelihood estimator is given by the formula:\n",
    "$$\n",
    "\\boldsymbol\\beta_{\\text ML} = (\\boldsymbol X^T\\boldsymbol X)^{-1}\\boldsymbol X^T\\boldsymbol y\\in\\mathbb{R}^D\\,.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def max_likelihood_estimate(X, y):\n",
    "    # resolves the normal equation XT(y - X*beta) = 0\n",
    "    optimal_beta = np.linalg.solve(X.T @ X, X.T @ y)\n",
    "    return optimal_beta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_with_estimate(X_test, beta):\n",
    "    predicted_y = X_test @ beta\n",
    "    return predicted_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#calculate the mean square error of the regression line\n",
    "def mse(y_train, y_predicted):\n",
    "    n = y_train.shape[0]\n",
    "    return np.sum(np.square(y_train - y_predicted), axis=0) / n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The parameters for linear regression, beta_ml is:\n",
      "[ 2.30185644e+01 -6.42428261e-01 -7.47418815e+03 -3.98149204e+04\n",
      "  3.98147514e+04 -4.00063949e+08  9.79120942e+02 -1.63905857e-01\n",
      "  1.28483429e+03 -9.75994730e+02 -4.91540040e-02 -6.10750434e+03\n",
      "  7.07828586e-01 -3.68838142e+00  4.00063947e+08  6.10572590e+03\n",
      "  7.47539451e+03 -1.28838888e+03]\n"
     ]
    }
   ],
   "source": [
    "beta_ml = max_likelihood_estimate(X_train, y_train)\n",
    "print(\"The parameters for linear regression, beta_ml is:\")\n",
    "print(beta_ml)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In Sample MSE is  24.374980456628485\n"
     ]
    }
   ],
   "source": [
    "predicted_y_for_training_data = predict_with_estimate(X_train, beta_ml)\n",
    "in_sample_mse = mse(y_train, predicted_y_for_training_data)\n",
    "print(\"In Sample MSE is \", in_sample_mse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out of Sample MSE is  19.53707302477274\n"
     ]
    }
   ],
   "source": [
    "predicted_y_for_test_data = predict_with_estimate(X_test, beta_ml)\n",
    "out_of_sample_mse = mse(y_test, predicted_y_for_test_data)\n",
    "print(\"Out of Sample MSE is \", out_of_sample_mse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The out of sample MSE of 24.4 is slightly higher as compared to the in sample MSE of 19.6. \n",
    "<br>\n",
    "This is unexpected since models tend to perform better on the training set as compared to the test set. Since our cost function is mean squared error, one reason why our model appears to perform worse on the training set might be due to the larger squared errors from certain data points that are outliers in the data. We plot a graph of squared errors for each input to further investigate this hypothesis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def squared_error(y_train, y_predicted):\n",
    "    n = y_train.shape[0]\n",
    "    return np.square(y_train - y_predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_squared_errors(y_train, y_predicted, title):\n",
    "    errors = squared_error(y_train, y_predicted)\n",
    "    plt.figure()\n",
    "    plt.style.use('seaborn-whitegrid')\n",
    "    plt.title(title)\n",
    "    plt.plot(np.arange(len(y_train)), errors, '+', markersize=8)\n",
    "    plt.xlabel(\"$i$\")\n",
    "    plt.ylabel(\"Squared error of $ith$ data point\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAESCAYAAAAfXrn0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhFElEQVR4nO3de7gU9Z3n8TcYlOggMc7k9sQJo4Gvx6gYjjGOVyYEIzqJrpn4xGguGI1GdyWMiRLEJxfBy24CoxmjMxojCtnNqGF3xSGCRIwQXePxihy/CIibTHSNGhQvEIGzf1Q1Nsfuc+rXXdVV1f15Pc95TnV1ddX3V9Vd3/r9fnUZ0tfXh4iIdLaheQcgIiL5UzIQERElAxERUTIQERGUDEREBCUDEREB3pF3AFIuZnYocBmwJ9HBxO+Ab7r7E7kG1o+Z/TPwgrt/N+9Y6jGzg4DbgA3AZ919fQPzGAkscPdPNBHHYuAL7v7CINMlWqdJ5yfFomQgiZnZLsBC4Bh3fygedxqwyMz+xt235hpg+XwGuNvdz2hiHnsAhzQZx8QmP5/1/KQFlAwkxK7Au4C/qBo3H3gF2AnYambfB04FXgDuBQ529/FmdiOw0t1/AFD92sz+HpgO7Ay8B5jr7heb2XjgSuC1eJkfA44BZsTTvk5UK7nPzHYHrgfGAs8CW4DltQphZhcBnyWq2awHznH3P5jZMuAlYF/gmnia6tcL4v+jgCFxnP/NzEbFZe2N35sQl+dw4E1gHTDZ3V+tiuFU4BxgJzN7p7ufamYXA6fEsa8G/rO7P9c/Lnf/UVVxfgq808weAbqBMfE62zPeJle5+w1m9hfxtKOBbUAPcBbwk3g+d5vZce7+u6oY667TAbbZT6vnF3/2bdPV2i6SL/UZSGLu/ifgAuCXZrbOzG4GJgN3ufufzewkoh3oR4EjgP0Gm6eZDQHOB77s7gcDhwLfNrO/jCfZHzjF3Q8E/hq4FDjO3T8KfA34hZntBnwPeINoh/k5wOos70vAAcAh7n4Q8O9EO7yKP7n7flU73OrX84mO5A8g2tGfZmafj6f7IHCJu48hSgjjgbHu3k2UDA7sty7nA9cCP48TwWRgEvCxuKwrgRsHiKtiMvBGXJYhwK3AtHi5RwPfjJv2/hMwIp7uY/Fn93b3yfHw31UngljNdTrQNqueH/D7etMhhaNkIEHcfTbwXuA8oqPFC4GH47brTwC/cPdX3P1N4F8TzK8P+DTQbWbfAWYT7dR2iyf5nbs/Ew9PBN4PLI2PhOcTHeV+GPgkcJO797n7H4mO4mv5e6Kd0oPxPP4LOyaOe/tNfy9AnHAOB66O436ZaGc9KZ5uC3BfPPw4sBX4P2Z2CXCbu/9mkFUxCfipu78Wv74SmGBmO9eJq5YxwD7ADXHZ7gHeSZSclwMfiWsZ04B/cvc1g8yv5jpNsM0ImU6KQclAEjOzw83sW+6+0d0XuvsFwEeAPqId9RtEP/aKP1cN9/V7b+d4nrsBDwPjgIeAbxE1rVSmfbXqMzsBS939oMof0Y59Zfx+9fy31CnGTsAVVZ8/mGgnX/Fqv+krr4f2m39l3LB4eLO7bwFw9w1EzSPfJEoKPzezc+rEUx1X9Y3ChhI149ZaDwPN4+Ua6+en7v40UdK8DNgduMvMPp1gnm9bpwm2GSHTSTEoGUiIPwIzzOyIqnHvB0YSHQ3fAZxsZnuY2VDgS/0+ezCAmX2AqAkDojbs3YEZ7n47UfPKLkQ7tv6WAseY2b7xfI4DHiM6+l0EfNXMhprZHsAJdcpwJ3BG3B4O8H3g5sEK7u4bgfuBc+Nlj4zLt6T/tHF7+lLgN/GZNzfxVtNMPb8ETo93oBDVvH7t7psH+dwWon6HIYADb8Sd+pjZXkSJstvMvk7UZ7DY3S8kWg/j4nls5a2kVq3eOh1sm1XmF7JtJWdKBpKYu68GTgQujfsMVgH/RtQ56u6+jKh5YznwADvuYH4EvN/MnGin9Kt4/GNEZyg9aWa9RM0Kq4iOYvsvfxVRP8H/MLNHgUuAz8Qds98lOup8EridKDnVcn28vPvN7AmitvyvJFwFpxI13Twel+8X7NiuX7EIeAJYaWYPAocRtb8P5CfAXcAD8XoYFy9vMM/GsTwBjCDaYZ9hZo8Bi4GL3X0FUULaCVhlZj1ECfyqeB63APeY2f795v1daq/TwbbZLURNVNsGmU4KZIhuYS1ZMbN/IDojZnzesYjIwFQzEBER1QxEREQ1AxERQclARERQMhAREUp8b6Kenh51doiIBOru7q550V9pkwFAd3d3Q5/r7e2lq6sr5WiKS+Vtf51WZpW3MT09PXXfUzORiIgoGYiIiJKBiIigZCAiIigZiORqzpLVeYcgAigZiOTqyqVP5R2CCKBkICIiKBmIiAglv+hMpGzmLFn9tqahUdPu2D48ZcJopk4c0+qwRJQMRFpp6sQxO+zsR027g/WXH59jRCIRNROJiIiSgYiIKBmI5GrKhNF5hyACKBmI5EqdxVIUSgYiIqJkICIiSgYiIoKSgYiIoGQgIiIoGYiICEoGIiKCkoGIiKBkICIiKBmIiAhKBiItp+ceSxEpGYi0mJ57LEWkZCAiIkoGIiLSosdemtkw4AZgFLALMBNYBdwI9AErgXPdfZuZnQmcBWwBZrr7wlbEKJIVPfdYyqBVz0A+DXjR3b9oZnsCDwOPADPcfZmZXQucYGb3AecBBwPDgeVmtsTdN7coTpHU6bnHUgatSga3ALdWvd4CdAP3xK8XAccAW4EV8c5/s5mtAQ4EftuiOEVEOlJLkoG7vwpgZiOIksIM4Afu3hdPshEYCewOvFz10cr4mnp7exuKZ9OmTQ1/toxU3uJJO74ylDlNKm/6WlUzwMz2AhYAP3b3n5nZf616ewSwAXglHu4/vqaurq6GYunt7W34s2Wk8hbLlAk70dWVbh9B0cucNpW3MT09PXXfa8nZRGb2XmAxcKG73xCPftjMxsfDk4B7gQeAI81suJmNBLqIOpdF2oY6i6WIWlUzmA7sAVxsZhfH46YAV5nZzkAvcKu7bzWzq4gSw1DgInff1KIYRUQ6Vqv6DKYQ7fz7O7rGtNcB12UelIiIbKeLzkRERMlARESUDEREBCUDERFByUBERAhMBmY2o9/ry9INR0RE8pDo1FIz+ypwBtBlZsfFo3cChgHfzig2ERFpkaTXGcwDlhJdPDYrHrcNeD6LoEREpLUSJYP4LqLrga9lGo2IiOQi6ApkM/sSUbPQLsAQoM/d984iMBERaZ3Q21FcCHwa+F0GsYiISE5Ck8E6d1+TSSQiIpKb0GTwupktInpkZR+Au09POygREWmt0GTw7/1e99WcSkRESiXRRWdmdkY8uC9g8d++8Z+IiJRc0ppBpcP4SWBPogvO+tB1BiIibSFRzcDd74wH3wDOBg4Hvo6aiURE2kLojeqmAuPc/UTgIOC8tAMSEZHWC00G29z9VQB33wjo+cQiIm0g9GyitWb2Q+DXwFHA2vRDEhGRVgutGZwOrAMmxv/PTD0iERFpuaCagbtvMbNrgL8Cnnd3dSCLiLSB0IfbnERUI1gEPGVmEzOJSkREWiq0mehi4BB3H0d0eumsQaYXEZESCE0GL7r78wDu/v+AV9IPSUREWi30bKJXzOxO4B6gG9jVzC4F3bBORKTMQpPB/6oa/o80AxERkfyEnk00N6tARIpszpLVTJ04Ju8wRDIT2mcg0pGuXPpU3iGIZKqpZGBmw9IKRERE8hPUTGRmZwP/GH9uCLAFGJ1BXCIi0kKhHchnAEcDM4BbgG+kHZBIEcxZsvptTUOjpt2xfXjKhNHqQ5C2EpoMXnD3Z81shLsvM7PvZxKVSM6mThyzw85+1LQ7WH/58TlGJJKt0D6Dl83sRKDPzM4iukeRiIiUXGgyOAN4BpgGjCF66pmIiJRcaDPRFHefGQ+fb2aXEV2NnIiZfRy4wt3Hm9k44Hag0jB7jbv/3MzOBM4i6pye6e4LA2MUSd2UCTpPQtpbomRgZl8lqhV0mdlx8eihwM7AtxPO4wLgi8Br8ahxwGx3/2HVNO8jepTmwcBwYLmZLXH3zUmWIZIVdRZLu0taM5gHLAWm89adSrcBzwcsay1wEnBz/LobMDM7gah28A3gEGBFvPPfbGZrgAOB3wYsR0REAiVKBvHOeT3wterxIRedufttZjaqatQDwPXu3mNmFwHfAR4BXq6aZiMwst48e3t7ky5+B5s2bWr4s2Wk8ra/Tiuzypu+0IvOzgLOJ52Lzha4+4bKMPAjomcrj6iaZgSwgTq6uroaWnBvb2/Dny0jlbf9dVqZVd7G9PT01H0v9GyiM4kuOlsETAaeaDws7jSzQ+LhCUAPUW3hSDMbbmYjgS5gZRPLEBGRBEKTwQvu/iwwwt2XAe9uYtlfB/7JzJYRPTVtprs/B1wF3Av8CrjI3Tc1sQwREUkg9NTSpi46c/f1wKHx8EPAYTWmuQ64LjAuERFpQiMXna0nuuhsNLroTESkLSS9zuBpoK9q1JtE1xgcT9SuLyIiJZa0ZrAvsB9wN/B5dzeiawaWZxWYiIi0TqJk4O6b447cfdz9gXjcw4BlGZyIiLRGaAfyBjO7hOgU0L8l6j8QEZGSC+1APhV4DpgU/5+cekQiItJyQTUDd38NuDqjWEREJCehNQMREWlDSgYiIpK8mcjMPkHUZ7CB6H5BjwEr9awBEZHyC+kzmAecG3/mQOBE4CPAh9MPS0REWikkGaxx9wXx8C1ZBCMiIvkYtM/AzG4ys28A95nZ+dmHJCIirZakZjAXGAu8D/iUmZ0HPBr/PebuqiWIiJTcoMnA3ZcCS83sQ+7+jJm9g+g+RWOJnlmsZCAiUnIhfQYLgHHuvoXoTKLHzOzQbMISEZFWGjQZmNnJwDhghJl1AavdfWv89r8SnVkkIiIllqRmsAIYTvRgm9mAmdkG4A/AG9mFJiIirZKkz+A/gJvMbK27rwAws3cDfwM8mXF8IiLSAklOLf1OPNhnZiMA3P0ld++Jb1wnIiIll6SZaHH8/xvA/vHZRKuIO5Hd/daMYhMRkRZJ0kx0X/z/ZAAz24XoNhQHAB8HlAxEREou9ElnxDemeyj+ExGRNpDoFtZmNjLrQEREJD9Jn2dwO4CZXZNhLCIikpOkzURvmNlvgdFmNjYeNwToc/fDsglNRERaJWkymAR8APgX4JzswhERkTwkSgbuvg34PXB8fMHZPsDT7v5ClsGJiEhrBD0D2cw+B9wPXATcb2anZRKViIi0VFAyAP6R6M6lJwIfBaakHpGIiLRcaDLY5u6vArj7RmBT+iGJSKh5j7yUdwhScqEXna01sx8CvwaOAtamH5KIhJr/6AZmnZJ3FFJmoTWD04F1wMT4/5mpRyQiIi0XVDOIn3J2dUaxiIhIToLvTSQi+ZuzZDVXLn1qh3Gjpt2xfXjKhNFMnTim1WFJiQUlAzP7O3e/O6tgRCSZqRPH7LCzHzXtDtZffnyOEUnZhdYMvgc0nAzM7OPAFe4+3sw+DNwI9AErgXPdfZuZnQmcBWwBZrr7wkaXJyIiyYQmgz4zWwA4sA3A3acn+aCZXQB8Eag8HW02MMPdl5nZtcAJZnYfcB5wMNFzl5eb2ZL4ttkiIpKR0GRwQxPLWgucBNwcv+4G7omHFwHHAFuBFfHOf7OZrQEOBH7bxHJF2t6pY9+VdwhScqFnE81t9N5E7n6bmY2qGjXE3fvi4Y3ASGB34OWqaSrja+rt7U26+B1s2rSp4c+Wkcrb/v5h3107qsydto1bUd7QDuSTgZlEz0De38y+6+7zGlz2tqrhEcAG4JV4uP/4mrq6uhpacG9vb8OfLSOVt/11WplV3sb09PTUfS/0orOppHdvoofNbHw8PAm4F3gAONLMhsdPV+si6lwWEZEM5XlvovOB78WdxjsDt7r7c8BVRInhV8BF7q77H4mIZKyl9yZy9/XAofHwauDoGtNcB1wXGJeIiDQhtGYwD92bSESk7YTWDGa4+1GZRCIiIrlp2UVn0nnmLFmt++OIlERoM9Fc4H8CvUQJwdMOSNpH/xupiUhxhdYMvqJmIhGR9qNmIhERaem9iUpF7d3hdI99kfIKvjcRgJnt4e5/yiakYrhy6VPacQXSPfZFyiv03kRHAT8GdjKzW4Bn3P0nmUQmIiItE3o20UyiK4+fAy4Fzkk9IhERabnQPoNt7v6SmfW5+yYz25hJVDlQe3f6pkwYnXcIIpJQaDJYY2aXAXua2TTgmQxiyoXau9On5ClSHqHNRGcTJYDlRI+v1L2JRETaQOjZRFuAazOKRUREchJaM+gYau8WkU6iZFCH2rtFpJMkaiYys6eBvqpRbwLDgM3u3jkPIhURaVNJawb7AvsBdwOfd3cDPkvUkSwiIiWXKBm4++b4WcT7uPsD8biHAcsyOJFOMmfJ6rxDkA4W2mewwcwuMbNPm9mlwPoMYhLpSHr+Q/spU4IPTQanEt2KYlL8f3LqEYmItIkyJfjQ6wxeA67uhLuWioh0ktC7lh4NXI3uWirSNN0PS4ok9N5ElxDdtfQ2oruWrgCUDEQaoPthtZ8yJ3jdtVREJCVlTvChHchte9dSEZFOFpoM/g3dtVQkE7ofVr7KdBpoFkKbiWa4+1GZRCLS4YraltwpsnjueZkSfGgy6DOzBYAD2wDcfXrqUYmItIEyJfjQZHBDJlGIiEiuQi86m1v92syGpRuOiEhrlPk00CyEXnR2FnB+/LkhRLey7py1JSJto8yngWYh9GyiM4GjgUVE9yValXpEIiLScqHJ4AV3fxYY4e7LgHenH5KIiLRaaDJ42cxOJDqr6Czgr9IPqX11+nnMIkVWptNAsxCaDL5K9AyDacBo4Oy0A2pnZbqdrUin6aTO4lpCTy19nLeehTwE+CRwUDMBmNnDwMvxy6eBWcCN8XJWAue6+7ZmliEiIgMLrRnsC3QRPQ/5NKJnIjfMzIYDuPv4+G8yMJvoSucjiRLOCc0sQ0SKQ02lxRV6ncHmqpcr4pvWNWMssKuZLY5jmQ50A/fE7y8CjgEWNLmcXOg8ZpEdZXHLB0lH6HUGl/FWM9H7iW9J0YTXgR8A1xP1QSwChrh7ZRkbgZH1Ptzb29vQQjdt2tTwZ0Mc+0E49st7b389ae46FlW9hq0tiaNV5S2KTisvlKvMacRZpvKmoRXlDe0zeLJq+FHgl00ufzWwJt75rzazF4lqBhUjgA31PtzV1dXQQnt7exv+bHPW5bLc/Mqbj4v++wpmnfLRvMNoqfJs43R+A+UpbzrSKm9PT0/d95q6HUUKTgcOAM4xsw8AuwOLzWx8fB3DJJrsl5DOM//RDcw6Je8oBNRUWiahzUSPEh2tbwKGx6OHAH3uvnfdD9b3E+BGM1tO1Px0OvACcJ2Z7Qz0Arc2MN/MzFmyuuEvb6efxyydR7d8qK2Z/UhWQpuJfgPc5O73mdkBwLdo4gE37v5n4As13jq60XlmrZkOsKJtfBHJRxE70kOTwX7ufh+Auz9uZn/d7wwjkZZTU4RI80KTwQYzuwR4ADgCPQM5dUWsPhadmiLKQ02lxRWaDL4AnAMcS3Rm0UWpR1QwrT7qLGL1USQtnfrdLkPtNVEyMLNxRJ29hwBrgW8SnfK5Hrg9o9gKQUedItKsyj6k8r+I+5Gkt6OYBXzZ3d8EZhKd8vkxohvWiRTKqWPflXcIIm9T9BtVJm0mGuruj8XXAuzm7g8BmJluINekMlQfy+a0g/SYDZFQiZNB/P9Y4C4AM9uF6JqDjpJ2B5iaocKog13aQRE70pMmg7vMbAWwF/AZM9sHuAb4eWaRFZR2RPlSB7uUxWC1fijW/iRRMnD3K8zsfwPPu/uLlWTg7qW8m2gn05F1OWg7lUut7VW2Wn/i5xm4e6+7vxgPr1UiyEbW1ce8OrF0H/swRe9slB21w/YKvc5AMtauR4ONNu+og13y0Ik1MyUDKbSyVbWlPWTRN1XETuNqSgZtLjqyXges2z5OR9bFoxpQuTSyvYq+/ZQM2tzUiWM49oNbtz8Yo1VH1tq5hVENqFzacXspGUgmsvqxFL2qXRad2CY+EB28KBlIybT7D7JVdL3GjgY6eBk17Y6OWFeJTy3tZO10WqSOrMuhE7aTflfJzXvkpUznD0oGiRThHOK0fjh5HeEk+bG0cudQ9B1RJxyJFuF3lZast9f8RzdkOn9QM1FplL1anyT2ZsoY2gZe9vUZSm3iyVUOFKrXTyesKyWDNpVFB2GROx07beceqh3PfslKp64rJYMa2uEoKoudo3a40oykv6siH3S0Sh77ICWDGopwZNAOCWkwrSxjJ6zPokv6u9JBRz77ICWDgipCQspaM2UcfOe+0w7z7oT1GaITzlZKS9brqig1ISWDNhG6c0xnnvlV6wfauff29tLVlf+Pq8iKsPMpi6zXVVFqQkoGCZThKCqLnaOq9ZKlyu9KTXiDa8VzvZUMEkjyRcz66LgMCalZrSxjJ6zPoqv8XtSEN7hWPNdbF52lJOsLaDrhyKiZMobu3Fu5Pot+gVs9ZY27oqjxz1mymlHT7tj+B+zwOq+4VTMogCxqFVkc+WZRrU+r7EVOlmVtRitr3BVFjb+oNSElg4wl2dll8aXN4keQRbW+qD9Y2VErTxJQE14+1EzUoKRVvXa6/0q7aqZaXtSmiLTV+x6nVf7q+VQnnU5Zv0WgmkGD2uGWt3md31yU86ormqmd1PpsWc+OGSzuWtstrZpdvfmEzL8o6z30+12UmlBHJoN5j7zErK78ll+UL21aP+SQL3Ol7NXlL8OOMonKTqCobcKDKfoBzmA72aKs99DfVd7rtaIjk8H8Rzcw65RouJmj1FqfTXrHwyJ8aas1sx5Cv/hXLn1qh53MYGUvWk2inrL2fzTzPW50/o2ot37L8v0ouo7vM2imTb/y2eofTq35TZkwmvWXH8/6y4/P5Uub5MEYoW3CoeObkVZ7dWX6pP09teY/0GcbiSkL1TEkiafW+q18Tyvf28pwpRZ45dKnEp8WWWv+g22DkPXbzG+40e9QI59NS2bL7evrK+Xfgw8+2NeoD124sObw7MXe8HySzL9a9bKSTFNvfL3hWvHMXux9H7pwYd2/yueTxBYyPuly65UxNIZVq1Y1PJ/Q7ZLkuxT6vRpoefWGq8sc+v1OEne9eSZZVr31WC+GeuOrh6f/bHlQ/EnKlUSteBr5ftcTun1DxfvNmvvUQtYMzGyomV1rZveZ2TIz+3Az80t6NNfKM3+ql1Wvzb1ePNXj6w3XMnXimO1HetVHe5XhytFg2uthoOXWa25IK4bQ+SSZvlaNsP/36sqlT21/v9naYKPbe6D5JJmmmbjT2H4Drd/5j24YtJZQHX8z620wA32/Q1sDsoxzMEXtMzgRGO7uf2tmhwI/BE5odGbVbfS1Om+rOzQHan9spOM3SedqXmf0QP024Va1w9ZrAx4otqJ0Mg/U/1GEDtd6Qs8aCv2upLX96q3f6pMQKuXI47vbboqaDI4Afgng7veb2cFpzbiZH3DSjt/qBJDXjj5J0mrkh1ZvnoMtK0SjsfVfVtKdUvWV1Ummr1aE0wKj9bIOWLd93GA77pCzhupNn3R7JDlJoN56rDW+md9w6DYe7LfUioOUwbZvWssd0tfX1/RM0mZm1wO3ufui+PX/BfZ29y2VaXp6evp23XXXhuY/ae46Fn15b+Y98tKAD5o+dey7BrxBVGU+SSRZFqT74OtK/PXinPfIS5x20LsbWg/15plknVSW239cGjGEzieN7ZJ0mqQ3GxsspsGWEfL9hvC401jvoRpdJ0mExFZd9sF+V0mElGuw33MSr7/+Ot3d3UNqvlmvMyHPvzFjxsweM2bMyVWvf99/mmY6kOt1PjXTmRQqi0610I7WLObZzDpJEk+SZSXpTM1iuc10TDYSU/VwkjI3c8JDvelD42xGaId2FrFlUa4k8+/YDmRgBXAcQNxn8HiaM6/O2s1U84vQRJCWIpelyLFVK2s7dej6zXr6euqt37J8P4quqH0GC4CJZvYbYAgwOasFNfMDLsuPP8mDMUJ/aKHjm5HWTiCvnVirVcedxQkModsj699JK299Hrpus5DZcutVGYr+10wzUXWVq1qz54OHSLKstK4zqFfeZmLLWjMx1LvOIK3lpjVNUqHXGTQTU1pxZ/0dqm7qTbLcNK/7yFIz2zeJgZqJCtmBnERPT09fd3d3Q5+NHgOZ482JWkzlbX+dVmaVtzE9PT11O5CL2mcgIiItpGQgIiJKBiIiomQgIiIU9ArkJHp6esoZuIhIjup1IJc2GYiISHrUTCQiIkoGIiJS3NtRpM7MhgI/BsYCm4Ez3H1NvlGlz8yGATcAo4BdgJnAKuBGoA9YCZzr7ttyCjETZvYeoAeYCGyhjctrZt8GPgPsTPSdvof2Lu8wYC7Rd3orcCZtuo3N7OPAFe4+Pn6o1430K6OZnQmcRbQOZrr7wjSW3Uk1gxOJH5gDTCN6YE47Og140d2PBCYB/wzMBmbE44bQxIOCiijeWfwL8EY8qm3La2bjgcOAw4Gjgb1o4/LGjgPe4e6HAd8HZtGGZTazC4DrgeHxqLeV0czeB5xHtP0/BVxmZruksfxOSgY7PDAHSO2BOQVzC3Bx1estQDfR0SPAIuCTrQ4qYz8ArgX+EL9u5/J+iuguvguA24GFtHd5AVYD74hr97sDb9KeZV4LnFT1ulYZDwFWuPtmd38ZWAMcmMbCOykZ7A68XPV6q5m1XTOZu7/q7hvNbARwKzADGOLuldPGNgIjcwswZWb2FeCP7n5n1ei2LS/wl0QHMp8DzgbmA0PbuLwArxI1ET0JXAdcRRtuY3e/jSjRVdQqY//9WGpl76Rk8Aowour10Oonp7UTM9sLuBu42d1/BlS3pY4ANuQRV0ZOJ7rd+TLgIOAm4D1V77dbeV8E7nT3P7u7A5vYcWfQbuUFmEpU5jFEfX5zifpLKtqxzFD7d9t/P5Za2TspGWT6wJyiMLP3AouBC939hnj0w3FbM0T9CPfmEVsW3P0odz/a3ccDjwBfAha1a3mB5cCxZjbEzD4A7AYsbePyAvyJt46GXwKG0cbf6Sq1yvgAcKSZDTezkUAXUedy09qumWQALXtgTs6mA3sAF5tZpe9gCnCVme0M9BI1H7Wz84Hr2rG87r7QzI4i2ikMBc4FnqZNyxubA9xgZvcS1QimAw/S3mWGGt9jd99qZlcRJYahwEXuvimNhekKZBER6ahmIhERqUPJQERElAxERETJQEREUDIQERGUDEREBCUDERFByUAkdWZ2mJl9L+84RELoojMREVHNQCRtZnaLmR2RdxwiIZQMRNK3P216I0RpX0oGIikys+HAsPjBIyKloWQgkq6PED1zWqRUlAxE0nUA8FjeQYiEUjIQSZeSgZSSTi0VERHVDERERMlARERQMhAREZQMREQEJQMREUHJQEREUDIQERGUDEREBPj/iqNM5xjsOR0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAESCAYAAAAfXrn0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA3+UlEQVR4nO3de1wU9d4H8M/CtphcUvJSiRgYKNZBePSIRFJoBpbmpURFqU7aUU830I5iJWhe0Cfz+piSJ8uDFqHY7ZiW8miUF45SZhqEoZn3Uk/GbnL/PX/47LYss7szC7s7wOf9evFiL3P5zm9m5zu/3/xmRiOEECAiolbNw90BEBGR+zEZEBERkwERETEZEBERmAyIiAhMBkREBCYDMnP48GEkJydj2LBhGDp0KCZNmoTjx4+7O6wGXnnlFaxatcrdYdhUXFyM+++/H6NGjcKZM2ccmkZ5eTkee+wxxePl5+dj/vz5Noe5ePEixo4d61BcckRGRtpd7iNHjiA9Pd1pMZAyWncHQOpQVVWFyZMnY/369bjzzjsBAB9++CGeeuop5Ofnw9PT080RNi/5+fmIiorCggULHJ7G1atX8e233yoeb9CgQRg0aJDNYTp37oycnBxHQ2sSP/zwAy5evOjWGOgPTAYEALh27RrKy8vx+++/mz57+OGH4ePjg9raWnh6emLFihX4+OOP0b59e/Tt2xdHjx5FdnY20tLSEBISgokTJwJAvfe7d+9GVlYWqqqqcOXKFYwYMQIpKSkoLCzEggUL0LZtWxgMBuTl5eHLL7/EmjVrUF1djTZt2mDmzJmIjIyEXq/HSy+9hJKSEnTq1Amenp7o06eP5HKsWbMGn332Gerq6tClSxdkZGSgc+fOSE5Oxk033YQTJ05g3Lhx+Oyzz+q9Hzx4MObMmYOzZ89CCIERI0Zg0qRJOHPmDMaPH4/u3bvj7NmzePvtt7F27Vp89dVXuOGGGxAQEIDMzEx4e3ubYvjoo4/w7rvvora2FhUVFXjttdewevVqbNu2DZ6enggKCsLs2bPRsWPHBnElJyebpjNr1ixUVFRg+PDh2Lp1K3r37o1BgwahpKQES5Yswffff4/33nsP1dXVuHr1Kp566ikkJSVh69at+PTTT5GVlYXk5GRERETgq6++wvnz5xEdHY158+bh3LlzGDZsGL7++musWrUKZ8+exS+//IKzZ8+ic+fOePXVV9GpUyccOXIEc+bMQXV1NQIDA3Hu3DmkpaUhKiqqXrkfOnQI8+bNg0ajwZ/+9CfU1dUBAOrq6rBw4UJ88803MBgMEEJg/vz5uO2227By5UqUl5dj1qxZWLBggeRw1tYzOYEg+n/r168X4eHhYuDAgeKFF14QmzdvFr///rsQQohPP/1UPPjgg6K8vFxUVVWJSZMmiQkTJgghhJg5c6b4xz/+YZqO8X1dXZ2YMGGCOHnypBBCiAsXLoiwsDBx+fJlceDAAdGzZ09x5swZIYQQJ0+eFEOHDhVXrlwRQghRWloqYmJihMFgEAsWLBAzZswQdXV14vLlyyI2NlasXLmyQfzvv/++SElJEdXV1UIIIXJycsSkSZOEEEJMmDBBzJo1yzSs5fvx48eL9evXCyGE+O2338SwYcPEv/71L3H69GkRGhoqDh48KIQQ4uDBgyIhIUHU1dUJIYT47//+b1FUVNQglpUrV4q5c+cKIYTYsmWLGDNmjDAYDKbvnnzySck4zJ0+fVpERESY3oeGhor3339fCCGEXq8XiYmJpvL6+uuvTcPm5eWJv/71r6bpP/fcc6K2tlaUl5eLe+65R+zfv7/etFeuXCkGDRokysvLhRBCTJ48WaxYsUJUV1eL2NhYsWfPHiGEEPv37xc9evQQBw4cqBdnZWWluPvuu8W+ffuEEEJ8/PHHIjQ0VJw+fVp89dVX4tlnnxW1tbVCCCGysrLE5MmTG8RpazhyDdYMyOQvf/kLRo8ejYMHD+LgwYNYt24d1q1bhy1btuDAgQMYPHgwfHx8AABjxozBhg0bbE5Po9Fg7dq12LNnD/71r3+hrKwMQghcu3YNAHDrrbeiS5cuAIC9e/fi559/xhNPPFFv/J9++gn79+/Hiy++CI1GA39/fwwePFhyfrt378a3336LRx55BMD1o1LjvACgb9++9YY3vv/999/x1VdfYf369QAAX19fjBo1CgUFBejduze0Wi0iIiIAAKGhofD09MTo0aNxzz33ID4+HuHh4TbLoaCgAKNGjULbtm0BAI899hjWrl2LqqoqybhsMQ7r7e2NtWvX4vPPP8ePP/6IkpKSerU6c3FxcfDw8ICPjw+6deuGq1evIiAgoN4w/fr1M63bXr164erVqygtLQUA3HvvvQCA/v37IyQkpMH0S0tLodVqER0dDQAYOnSo6VxAZGQkbrrpJuTk5OD06dMoLCysV4sykjscOQ9PIBMAoKioCP/4xz/g4+ODuLg4zJgxA9u2bYNGo8HevXvh5eUFYXYbqxtuuMH0WqPR1PuuuroawPWd7MiRI3Hs2DH06tULM2bMgFarNQ1r3DkC13fc0dHR+PDDD01/ubm5pp2P+fStnb+oq6vDpEmTTOPn5eXh3XffNX1vPj/z93V1dfWmb/yspqYGAKDT6aDVXj9u8vPzw4cffoiZM2fC09MTKSkp2LRpk82yraurg0ajkZy2VFy2GIe9cOECRowYgbNnz6JPnz5ISUmxOk6bNm1Mry3Xla1hPD09GwxrrewthzOW1549ezB58mQA189ljBs3TnJ8ucOR8zAZEADA398fa9aswaFDh0yf/fLLL9Dr9QgNDcV9992HHTt24OrVq6irq8MHH3xgGq59+/Y4evQogOu9VP79738DAE6dOgW9Xo+UlBQMHDgQhYWFqKqqMrUnm4uOjsbevXtRVlYGAPj888/x8MMPo6KiAgMGDMCWLVtQV1eHq1evIj8/X3IZ7rnnHmzZsgV6vR4AsGLFCsyYMcPusvv4+KB3796mnXp5eTk++OAD3H333Q2G3b17N5544glERkbi2WefxYgRI0zLbs2AAQOQl5dnOnLPzs7Gn//8Z+h0OpvjabVa1NbWSu68jx49Cn9/f/ztb3/DPffcg927dwMAamtr7S6vXN27d4dOp0NBQQGA671/SktL6yU2AOjRoweEEPj8888BXD95fvXqVQDXa3xxcXFISkrCXXfdhV27dpli9PT0NCVFW8ORa7CZiAAAQUFBWL16NZYtW4YLFy7Ay8sLvr6+WLhwIYKDgxEcHIzHHnsMSUlJ8PLyMjXvAEBycjJeeOEFxMfHIyAgAP379wdwfSdx3333YciQIdDpdAgNDcUdd9yBU6dONdgR3nHHHXjllVcwbdo0CCGg1WqxZs0aeHt749lnn0VGRgaGDBkCf39/hIaGSi7D6NGjcfHiRSQmJkKj0eDWW2/FokWLZC3/kiVL8Morr2Dr1q2oqqrCsGHDMGrUKJw9e7becLGxsSgoKMDQoUPRtm1b3HTTTZg3b57NaT/66KM4f/48Ro8ejbq6OnTr1g1LliyxG1PHjh0RHh6Ohx56qEHtIyYmBlu2bEFCQgI0Gg369esHf39/nDp1StbyyqHVarFq1SpkZGRg6dKluP3229GhQ4d6tQjgei1x9erVmDNnDpYuXYqwsDDcfPPNAICxY8di+vTpGDZsGGpqahATE2M6wR8REYHVq1fjmWeeQWpqqtXhPDx4zOoKGiF12EFkx44dO7Bp0yZkZ2e7OxRyosWLF2PixIno0KEDzp8/j+HDh2PXrl3w8/Nzd2jUxFgzICKrunTpgieeeMJ0rmf+/PlMBC0UawZERMQTyERExGRARERgMiAiIjTjE8hFRUXuDoGIqNmxdr+nZpsMAOsLZU9xcTHCwsKaOJrGU2NcaowJYFxKqTEuNcYEtOy4bB1Es5mIiIiYDIiIiMmAiIjAZEBERGAyICKya9nOUneH4HRMBkREdqzIP+7uEJyOyYCIiJgMiIiomV90RkTkDMt2ljZoGro9bZvp9fODQpA6WPohS80VkwERkYXUwaH1dva3p23Dj4secmNEzsdmIhtaQw8CIiKAycCm1tCDgIgIYDIgIrLr+UEh7g7B6ZgMiIjsaGkni6XwBLKZ1tiDgIgIYDKopzX2ICAiAthMREREYDIgIiIwGdjUGnoQEBEBTAY28WQxEbUWTAZEROS63kRbt27F+++/DwCorKxEcXEx3nnnHSxcuBAajQYhISHIyMiAh4cHcnNzkZOTA61Wi6lTpyIuLs5VYRIRtUouSwajRo3CqFGjAABz587FI488gtWrVyMlJQVRUVFIT09Hfn4+IiIikJ2djby8PFRWViIpKQkxMTHQ6XSuCpWIqNVxeTPRt99+ix9++AFjxozBsWPH0K9fPwBAbGws9u3bhyNHjiAyMhI6nQ6+vr4IDAxESUmJq8O0ijevI6KWyOUXnWVlZeHpp58GAAghoNFoAADe3t4oLy+HXq+Hr6+vaXhvb2/o9XrJaRUXFzsUQ0VFhcPjrsg/gYSAWofGtacxcTmLGmMCGJdSaoxLjTEBrTculyaD3377DSdOnED//v0BAB4ef1RMDAYD/Pz84OPjA4PBUO9z8+RgLiwszKE4iouLHR4XONGIcW1rXFzOocaYAMallBrjUmNMQMuOq6ioyOp3Lm0mOnjwIO6++27T+169eqGwsBAAUFBQgL59+yI8PBxFRUWorKxEeXk5ysrKEBrKLp5ERM7k0prByZMnERAQYHo/c+ZMzJ49G0uXLkVwcDDi4+Ph6emJ5ORkJCUlQQiB1NRUeHl5uTLMenjzOiJqDVyaDCZNmlTvfVBQEDZu3NhguMTERCQmJroqLJt48zoiag140RkRETEZEBERk4FivHkdEbVETAYK8WQxEbVETAZERMRkQERETAZERAQmAyIiApMBERGByYCIiMBkQEREYDIgIiIoTAavv/56vfevvfZakwZDRETuIeuupZs3b8aWLVtQVlaGgoICAEBtbS1qamowffp0pwZIRETOJysZDB8+HNHR0cjKysKUKVMAXH9K2c033+zU4IiIyDVkJQOdToeAgADMmzfP2fEQEZEbKHq4zQcffICsrCxUVVWZHmafn5/vrNiIiMhFFCWDdevWYe3atbj11lsdmllWVhb+93//F9XV1Rg3bhz69euHtLQ0aDQahISEICMjAx4eHsjNzUVOTg60Wi2mTp2KuLg4h+ZHRETyKOpN1LVrV3Tr1g06nc70J1dhYSG+/vprvPvuu8jOzsaFCxeQmZmJlJQUvPPOOxBCID8/H7/88guys7ORk5ODN998E0uXLkVVVZXiBSMiIvkU1QzatGmDSZMmISwsDBqNBgAwbdo0WeN++eWXCA0NxdNPPw29Xo8ZM2YgNzcX/fr1AwDExsZi79698PDwQGRkpCnZBAYGoqSkBOHh4QoXjYiI5FKUDO699956740JQY7//Oc/OHfuHNauXYszZ85g6tSppvMOAODt7Y3y8nLo9Xr4+vqaxvP29oZer1cSJhERKST7OoPRo0fjxIkTpp23+Y5cjnbt2iE4OBg6nQ7BwcHw8vLChQsXTN8bDAb4+fnBx8cHBoOh3ufmycFccXGx7Pmbq6iocHhcZ1JjXGqMCWBcSqkxLjXGBLTeuGQlg1tuuQUAEBwcjF9//RW1tbXQaDSKrjPo06cP/vnPf+Ivf/kLfv75Z1y7dg3R0dEoLCxEVFQUCgoK0L9/f4SHh2P58uWorKxEVVUVysrKEBoq/ajJsLAw2fM3V1xc7PC4zqTGuNQYE8C4lFJjXGqMCWjZcRUVFVn9TlYyGDBgAADAy8sLOTk56N69O0pLS/HMM8/IDiIuLg4HDx7Eo48+CiEE0tPTERAQgNmzZ2Pp0qUIDg5GfHw8PD09kZycjKSkJAghkJqaCi8vL9nzISIi5RSdM9iwYQO2bt1qasd//PHHMWLECNnjz5gxo8FnGzdubPBZYmIiEhMTlYRGRESNoKhrqUajgbe3NwDAx8eHR+xERC2EoppBYGAgFi1ahL59++LQoUMIDAx0VlxERORCimoGCxcuRNeuXbFv3z507dqV9yoiImohFNUMtFotxo0bhytXruDmm29W1LWUiIjUS1HN4LPPPsP999+Pp556Cg888AD27t3rrLiIiMiFFNUMXn/9dWzevBk333wzLl26hClTpiAmJsZZsRERkYsoqhm0a9fOdKFZhw4d4OPj45SgiIjItRTVDHx8fDBx4kT8+c9/xrFjx1BRUYGlS5cCkH/DOiIiUh9FyWDQoEGm1507d27yYIiIyD0UJYORI0c6Kw4iInIjRecMiIioZWpUMqiurm6qOIiIyI0UNRO9++67ePvtt1FTUwMhBLRaLT777DNnxUZERC6iqGawefNmZGdnIzY2FpmZmbjjjjucFRcREbmQomTQvn17dOrUCQaDAVFRUbh69aqz4iIiIhdSlAx8fX2xa9cuaDQa5OTk4MqVK86Ki4iIXEhRMpg/fz5uu+02TJ8+HT/++CPmzp3rrLiIiMiFFCWDf/7zn+jVqxc6deqEtLQ0fPHFF86Ki4iIXEhWb6LNmzdjy5YtKCsrQ0FBAQCgrq4O1dXVmD59uuyZjRgxAr6+vgCAgIAATJkyBWlpadBoNAgJCUFGRgY8PDyQm5uLnJwcaLVaTJ06FXFxcQ4sGhERySUrGQwfPhzR0dHIysrClClTAAAeHh6mm9bJUVlZCQDIzs42fTZlyhSkpKQgKioK6enpyM/PR0REBLKzs5GXl4fKykokJSUhJiYGOp1OyXIREZECspKBTqdDQEBAgyebKbnorKSkBNeuXcOTTz6JmpoaTJs2DceOHUO/fv0AALGxsdi7dy88PDwQGRkJnU4HnU6HwMBAlJSUIDw8XMFiERGREoouOsvJycFbb73l0EVnbdq0wcSJEzF69Gj8+OOPeOqppyCEMD0tzdvbG+Xl5dDr9aamJOPner1eSZhERKSQomSQm5uL7OxsrFmzBgkJCdiwYYPscYOCgtCtWzdoNBoEBQWhXbt2OHbsmOl7g8EAPz8/+Pj4wGAw1PvcPDmYKy4uVhK+SUVFhcPjOpMa41JjTADjUkqNcakxJqD1xqUoGVhedLZy5UrZ427ZsgWlpaWYM2cOLl68CL1ej5iYGBQWFiIqKgoFBQXo378/wsPDsXz5clRWVqKqqgplZWUIDQ2VnGZYWJiS8E2Ki4sdHteZ1BiXGmMCGJdSaoxLjTEBLTuuoqIiq98pSgaNuejs0UcfxaxZszBu3DhoNBosXLgQ7du3x+zZs7F06VIEBwcjPj4enp6eSE5ORlJSEoQQSE1NhZeXl5IwiYhIIUXJYP78+Th9+jSmT5+Ot956S9FFZzqdDq+99lqDzzdu3Njgs8TERCQmJioJjYiIGkFWMhg4cKDpRC8AaLVaVFdXY8+ePdi+fbvTgiMiIteQlQx27NgBIQTmzp2LsWPHIjw8HN999x3eeecdZ8dHREQuIPs6AwA4ffq0qb9/r169cPLkSedFRkRELqP4BPLy5csRHh6Ow4cPo0uXLs6Ki4iIXEjRjeqWLFmCjh07oqCgAB06dEBmZqaz4iIiIhdSVDNo27Ytxo8f76xYiIjITRTVDIiIqGViMiAiIvnNRPv378fHH38MPz8/hISEoEePHggNDeWtpYmIWgDZNYO///3viIuLQ+/evXH69GmsXr0aDz30kDNjIyIiF5FdM+jWrRsGDx4MABgyZIjTAiIiItezWzOYMWMG3n77bURERGD9+vWuiImIiFzMbs1g5MiRKCkpwaVLl/Dll18iOzsbPXr0QM+ePdGjRw/WEoiIWgC7ySA6OhrR0dE4e/YsunTpgpqaGvzwww8oKSnBkSNHmAyIiFoA2SeQn3nmGQDX71jas2dPjBgxAvHx8U4LjFqOZTtL3R0CEdlhNxl88sknWLJkCQwGA8rKylBbW2v6bvbs2U4NjlqGFfnH3R0CEdlht5moT58+qKqqwubNm5GZmYmTJ0/Cz88PnTp1Qps2bVwRIxEROZndZNC5c2eMGDECXbt2RZ8+fQAAv/76K86cOYPg4GCnB0hERM5nNxn8z//8D5555hloNBro9Xr4+PigXbt2aNeuneKZXb58GaNGjcL69euh1WqRlpYGjUaDkJAQZGRkwMPDA7m5ucjJyYFWq8XUqVMRFxfnyHKRGy3bWdqgaej2tG2m188PCkHq4FBXh0VENthNBjExMQCADRs24Pjx46itrUX37t3Ro0cP9OjRAwkJCbJmVF1djfT0dFPTUmZmJlJSUhAVFYX09HTk5+cjIiIC2dnZyMvLQ2VlJZKSkhATE8NbXjQzqYND6+3sb0/bhh8X8Wp1IjWzmwwiIyMBACtWrAAAVFVV4fjx4ygtLcU333wjOxksXrwYY8eOxRtvvAEAOHbsGPr16wcAiI2Nxd69e+Hh4YHIyEjodDrodDoEBgaipKTE9HQ1IiJyDkXPMwCuPwLzzjvvxJ133il7nK1bt8Lf3x8DBgwwJQMhBDQaDQDA29sb5eXl0Ov18PX1NY3n7e0NvV6vNEQiIlJIVjIoLy+vt5NWKi8vDxqNBvv370dxcTFmzpyJK1eumL43GAzw8/ODj48PDAZDvc9tzbe4uNiheCoqKhwe15nUGFdTxDS+d7smXy41lhXAuJRQY0xAK45LyJCUlCSEECI9PV3O4DZNmDBB/PDDD2Ly5MniwIEDQgghZs+eLbZt2yZ+/vlnMXToUFFRUSF+++03ER8fLyoqKiSnc+jQIYdj+O677xwe15nUGJcaYxKCcSmlxrjUGJMQLTsuW/tNWTWDNm3a4JFHHsGpU6fw/fffG5MINBoNcnJyHEpCM2fOxOzZs7F06VIEBwcjPj4enp6eSE5ORlJSEoQQSE1NhZeXl0PTJyIi+WQlg3Xr1uHnn39Geno6MjIyGjXD7Oxs0+uNGzc2+D4xMRGJiYmNmgcRESkj695EHh4euOWWW/DGG2/A29sbly9fxo033oguXbo4Oz4iInIBRc9A3r59O8aMGYO1a9dizJgx+PDDD50VFxERuZCirqVvv/02tm7daury+fjjj2P48OHOio2IiFxEUc1Ao9HA29sbAODj48OTu0RELYSimkFgYCAWLVqEvn374tChQwgMDHRWXESkEst2lvJeUq2AoprBwoUL0bVrV+zbtw9du3bFvHnznBUXEakEn0fROiiqGWi1WowfP95ZsRARkZsoqhkQEVHLpKhmcODAAfTv399ZsRCRCvB5FK2TomSwatUqJgOiFo7Po2idFCUDjUaDp59+GkFBQfDwuN7CNG3aNKcERkRErqMoGTzyyCPOioOIiNxIUTIYOXIkfv31V/z0008ICAiAv7+/s+IiIpV4flCIu0MgF1DUm+iTTz7hvYmIWhmeLG4dFNUMNmzYwHsTERG1QLw3ERER8d5ERESksGbw8MMP895ERNRiLNtZ6u4QVENRzWDNmjXYtGmTQzOqra3Fyy+/jJMnT8LT0xOZmZkQQiAtLQ0ajQYhISHIyMiAh4cHcnNzkZOTA61Wi6lTpyIuLs6heRIR2bIi/zhPkP8/l110tnv3bgBATk4OCgsLTckgJSUFUVFRSE9PR35+PiIiIpCdnY28vDxUVlYiKSkJMTEx0Ol0CheNiIjkUpQMRowYAU9PT4dmdP/99+O+++4DAJw7dw4dOnTAnj170K9fPwBAbGws9u7dCw8PD0RGRkKn00Gn0yEwMBAlJSUIDw93aL5ERGSfomTw/vvvO9xMBFy/BfbMmTOxc+dOrFy5Ert374ZGowEAeHt7o7y8HHq9Hr6+vqZxjN1YiYgaS85N+BICXB2VOrj83kSLFy/GCy+8gMTERFRWVpo+NxgM8PPzg4+PDwwGQ73PzZODueLiYkXzNqqoqHB4XGdSY1xqjAlgXEqpMS53xJQQACQ8Hmx6P2TDCWw3ew/UqrKsAOeXl8vuTfTBBx/g4sWLmDx5Mm688UZoNBrcddddKCwsRFRUFAoKCtC/f3+Eh4dj+fLlqKysRFVVFcrKyhAaKn2CJywszKFYiouLHR7XmdQYlxpjAhiXUmqMSx0xnWgQgzriaqgp4ioqKrL6neJ7EwHA1atXcdNNNykK4oEHHsCsWbMwfvx41NTU4MUXX0T37t0xe/ZsLF26FMHBwYiPj4enpyeSk5ORlJQEIQRSU1N5cRsRkZMpSgYHDx7E3LlzUVtbi4SEBNx2220YPXq0rHHbtm2LFStWNPh848aNDT5LTExEYmKiktCIiBTjTfj+oOiis+XLl2Pjxo3o0KEDpkyZgnfffddZcREROR2vMfiDomTg4eGBdu3aQaPRwMvLy3SfIiIiat4UJYPAwEC89tpr+PXXX/HGG2/gtttuc1ZcRETkQoqSwdy5c3HbbbehT58+uPHGG3lvIiKiFkLRCWStVotx48Y5KxYiInITRTUDIiJqmZgMiIhIXjPRwIEDTfcQAq43F9XU1ECn02H79u1OC46IiFxDVjLYsWMHhBCYO3cuxo4di/DwcHz33Xd45513nB0fERG5gKxkYHyWwOnTp023ku7VqxdOnjzpvMiIiMhlFPUm8vX1xfLlyxEeHo7Dhw+jS5cuzoqLiIhcSNEJ5CVLlqBjx44oKChAhw4dkJmZ6ay4iIjIhRTVDNq2bYvx48c7dNdSIiJSL0XJ4N///jdeeeUVh+5aSkRE6qWomWjFihW8aykRUQvEu5YSUZNYtrPU3SFQI/CupUTUJCwfNE/Ni6JkMGTIEN61lIioBVJ0AnnNmjXYtGmT4plUV1fjxRdfxNmzZ1FVVYWpU6fijjvuQFpaGjQaDUJCQpCRkQEPDw/k5uYiJycHWq0WU6dORVxcnOL5ERHZsmxnKZ9yZkFRMtBoNHj66acRFBQED4/rlYpp06bZHe+jjz5Cu3bt8Oqrr+I///kPRo4ciZ49eyIlJQVRUVFIT09Hfn4+IiIikJ2djby8PFRWViIpKQkxMTGmK6CJSD2W7Sxt0DR0e9o20+vnB4Wodoe7Iv+4amNzF0XJ4JFHHnFoJgkJCYiPjze99/T0xLFjx9CvXz8AQGxsLPbu3QsPDw9ERkZCp9NBp9MhMDAQJSUlpltgEJF6pA4OrbdDvT1tG35c9JAbI6LGUJQMRo4cWe99dXW1rPGMvY70ej2ee+45pKSkYPHixaY7oXp7e6O8vBx6vR6+vr71xtPr9VanW1xcrCR8k4qKCofHdSY1xqXGmADGpZSr4lIyD3eXlbV5uzsua5wdl6JkkJOTg7feegs1NTUQQuCGG27Ap59+Kmvc8+fP4+mnn0ZSUhKGDRuGV1991fSdwWCAn58ffHx8YDAY6n1unhwshYWFKQnfpLi42OFxnUmNcakxJoBxKeWauE4omocry0qqSWvIhhOm1+ZNWi15HRYVFVn9TlEyyM3NRXZ2NtasWYOEhARs2LBB1niXLl3Ck08+ifT0dERHRwO4ftfTwsJCREVFoaCgAP3790d4eDiWL1+OyspKVFVVoaysDKGhbNcjag6eHxTi7hCsYpOWfYq6lrZv3x6dOnWCwWBAVFQUrl69Kmu8tWvX4rfffsPrr7+O5ORkJCcnIyUlBatWrcKYMWNQXV2N+Ph4dOzYEcnJyUhKSsLjjz+O1NRUeHl5ObRgRCTNWReH8YRs86b4Fta7du2CRqNBTk4Orly5Imu8l19+GS+//HKDzzdu3Njgs8TERCQmJioJi4gUYE8akqKoZrBgwQJ06dIF06dPx6lTpzB37lxnxUVE5DRqbtJyF0U1g2HDhpl6AAkhsG/fPnz44YdOCYyopeGFTurB9dCQomSwY8cOANcTwdGjR2X3JCIi9zXPXO9JcwLAH71nmsvFYeQ6ipKB+ZXAffr0wdKlS5s8ICJqWqmDQ5EQUGvqlsieNCRFUTJ47bXXTM1Ev/zyi+mWFERE1LwpSgbBwcGm1z179sSAAQOaPCCilqI537uHWp9G3Y6CiKxT64VO7ElDUhQlg4cffhgGgwFeXl6orKwEcP1kskajQX5+vlMCJGrNnNEDibURkqKo0T8yMhJLlizBJ598gtdffx19+vTBjh07sH37dmfFR+QSan1kI58eRq6iKBmUlZUhMjISANCjRw+cP3/edLtpoubMFTtdNs+Qmim+HcXy5csRHh6OoqIiPgOZSAE2z5Ac7ro4UXHX0nfeeQdffPEFgoODkZqa6qy4iFol9kAid12cKCsZHDt2DC+99BI2b96MwMBArF+/Hr6+vujSpQsGDhzo7BiJmozxqEutO1219kCilk9WMli2bBkWLVqEG264AcuXL8e6devQrVs3TJo0icmAmhXjURd3us0L7+vkfLKSgRACPXv2xMWLF3Ht2jXceeedAMArkInIJSybTlpSclBLLVVWMqirqwMAfPHFF6YnlVVVVdV7RCURNT32QJLWkp7JoJZaqqxkEB0djbFjx+LChQtYs2YNfvrpJ8yZMwcPPvigs+MjahS5R11q3em2lB2ePbaO9NV6DUhLIysZ/PWvf8WgQYPg7++P9u3b46effsK4ceMwePBgZ8dH1Chyj7pay05XraSagWwlcWp6shv9u3fvjvbt2wMAAgMDHUoE33zzDZKTkwEAp06dwrhx45CUlISMjAxTU1Rubi5GjRqFxMRE7N69W/E8iFqC1n40nDo4FD8ueshmc8ntadtMfy2pvNxVS1V0nUFjrFu3Dh999BFuvPFGAEBmZiZSUlIQFRWF9PR05OfnIyIiAtnZ2cjLy0NlZSWSkpIQExPDK5yp1WlJbeJNxTwxtOTeX+5a7y5LBoGBgVi1ahVmzJgB4Pq1C/369QMAxMbGYu/evfDw8EBkZKTpFheBgYEoKSlBeHi4q8KkFk6t5wZaG3tPX5Ni/J7r0Dlclgzi4+Nx5swZ03vj3U4BwNvbG+Xl5dDr9fD19TUN4+3tDb1eb3WaxcXFDsVSUVHh8LjOpMa41BgT4HhcCQGObzdyNGV5NWWczlqPGw9fwYQIf8XjJQQA9425DW3atAEADNlwAtsfD5YcdsiGExjfu53ZfGqxAs5bjy1tm5fLZcnAkvk1CgaDAX5+fvDx8anXXdVgMNRLDpaMj/FTqri42OFxnUltcS3bWYqEgDaqislIbWVl5GhcUidMh2z446i5sX3NnVVemzZsw4JxMQ6NWz+mEzbiO9FgHs8P8kRYmHOaU1ratmWuqKjI6nduSwa9evVCYWEhoqKiUFBQgP79+yM8PBzLly9HZWUlqqqqUFZWhtBQtpu6y4r840iwcrRGTUstfc2bC55PaXpuu4R45syZWLVqFcaMGYPq6mrEx8ejY8eOSE5ORlJSEh5//HGkpqbCy8vLXSEStVqu7p1j6zwAzxG4hktrBgEBAcjNzQUABAUFYePGjQ2GSUxMRGJioivDInIapbdNUMttFqz1ZnLWrRNsjSNnemopt+bMbc1EpD7W262vt13z9snKKe0iahxerUfDam3Oaq5dcdWUxJgMnEhNK1oOqR/69seDVXkyraVrTtsNOU5NSYy3HXWixj5KsSVdVdmSlqUlWraztN4VvUDzu8K3OcSoZqwZqJiajhoaqyUtiy0bD1/Bpg31L56y1aaultsXO9r846zmLHu1aqlyW5F/3PSZsdxcWTtvbi0BlpgMyKrrP/Rad4fRrEyI8K/XJ97eTtWZbfCu2Dk56+S4vYMHqXID0KDsXHkQImdeakn+UpgMmpDcFd1cjiBSB4eq8kpMNVDrOjSPS221MbXFo1RTrHO1noAHmAyalNwVbe1HobSJwVmaYqN39xGQnGVozHKqdcfWVHGZN/+oOfHZ28bkTkfO8ikpW7WWmS1MBjK4asVOiPDHpm9+NSUQJUcNjYnRctym2KE4egRk7yEnTdXMYG0YJfOQM6zSNnW1dCm1tz009W/CkYMH823MOP7zg0JM0zH+tzcdR7d3Oc9caE4Jgb2JZGhsryBXsBejsaeFVI8LNS2frVic0TvL8jPjPOT0TJFT5kp3BkoSkeV742dSPYKGbDhRbxgl05ZivuzWhrfXQ2nj4SumYc2fX2A8aDB/b69cjN9LPQdByXSUsBWzeUyA9TJSS/IHWDNwKuOKtncEERXkj/cmRzs1FuPRj6ubN9RUXZZadmtHvXKevAXUv61yUx1xymGctrW4jIxxXb/JWShuT9vmUI3JnFQClRreXu2wqc5HOWsbc6S2Yi0WW2WkFq06GTh7R2V+tGLtR2Hc4Mw3MvPXrtqZWlZv62/0nrJjGJO1v15ic9UVtXLbj6XumW95n3zL6RjXgZzeK/ZidMa6/HHRQ6ZtynzbkrPjt8bW9iCHKw8CzLcxueeKbG0r43u3w4IwZU2dxgRp3r1VqswaWy7OLNdWnQya+v4rjjYLrMg/jh8XPYTi4mIM2XCiwY9aSmN6Ltk7mgT+2MkZjyjlKjx5RfJza+UyJmt/g3Fs7XisJSk5P1xjOVubj1R5GtugrR2FW5Z5QoDV0J1SU3BWe75lOZm3xZt/bm2eUjUrJRw9eDDfGVvGafxNOKPGYt6DS+r3a4yhsduAM2ubrToZWOPIyU+ppgU5HL1q0l6MxvZaoOGJNPMfi/FHbn406UqFJ6/Ifpyh+XdyklRTPFTd/ByCeZmb1wyM3y3bWVrvXk6OzM8aW8ti+d/8wqv6451oENPzg0Jw4MRluwlZatnNWV7wJfW9rR28ZTJxpBOB+YGVMU5Xdts0zt/IWu3q9rRtpqZhNTWjttpkMCZrf5NOz9qRttTKjgryt1kNN08QxtfWjvKtMf54LZsPrB0xN4ZlXLaWTer8iO0mqoZX7AKweQRuJJUwzadnfsRorwwsy92y14pxXgkBtaZ7OVk7OjQuh5KdgLXkb37UbkxMlgclf5wzCJNc/1JxWr63l9TkbFO2llfugZRlTyHzGI0smyqlOLITlkpmcmvo5usL+KMGreQA0vyEuzO0qmQgt11Zascrd+OR003TfEM1zj8qyL/eBmIZm/HozdEucPa+s7ZD/lNnLww8Y/2cgbGZx9bO1Pzo2VrSMz+as9XEZZyP1EN3zIe3dhRq3uRj3gRk66jX8nNjfLaW2V65Gindxsy3Yaman5LmFVvbuFRZWq4/851bU9TC5DKvAUjNt/DklXpxWW5LxnGsHcWP790OHc40XBdSZWUtSUutJ3sHgOZxmv931fU6rapraerg613BzJ+1atntTOrIw9qRo3nXOSPjyeDb0/64uZecLn2FJ69g++PBDbqpmX9vj3E+lt35pJKLeY3D+GM3n+/zg0Lw46KH8O3FSqzIP261a6oxsVnrEmheLkbG2KQ+Mx/WsszNd3QbD1+R7NVivly2WDsiM5+HsRwsl8lawjGPxzx2OV0PLeOXmq7lztvaBY2AvJvMWXYPNX9v/tpa4rPc7gGYthspljFJrUOljOUh1Z3UfEdpuTyW4xhfPz8oBBMi/O3WFOXWJI37HOM8zFn+PszXgfG/cXxXdJVtVcnAFlsbpVSBj8naL7t5xfwoZEzW/gY7ayNjX3BH4zTvVWG5ARl3XFI9ZqSWz3LZLDdWR2M0kvph2GrfHZO1v16cm775VTIWW811UkfN1q60tSwny8RuTGbmR30r8o/XW4fmXYutsXadg/l3UskxKqj+Q+ifHxTSYEdsvsMwNjEYzyNYO89gbM+Wis2yJmecl3He5jU64+fWdmLAH+vQPBbLBCb1ezH/r+R3K+egbEX+cYeaY5btLDU1PVvGaq9mIHUuz3y65v8tXzcljRBCOGXKTlZUVIQ+ffrIHl6q14olYw3A/D/wR/XYuLHLmZbS+azIP369SeauQNlJpku7Nni0T1cA9Y/MzGM3fm5+vsC4U5daPvN4zJsBzJk3adlaxgMnLgOQV6uRGn9F/nFZ84oK8kf/4JslT2BKnRy1nI+RZU8Qy+RnXj6WZWYep/G1efnZWw7z76XamY3fyzm/YW3ZLHfWUtMxn7ecuAFY3U6khhuTtR/9g2+uN3/zZZI6/2GtOUguR7Yh88+MjGVn3KaM27i1bc9y25H6zPz3ZmS5zm1R2kxka7+pymRQV1eHOXPm4Pvvv4dOp8P8+fPRrVu3esMoTQbmlJw4JGXkbMCOUtP6krsDdNV8XBWPkfm6cMa8zacpp6eb5XU7SubTmEQjNT1bcTbVNuxoLylb+01VNhPt2rULVVVVeO+99zB9+nQsWrTIKfORm1GlmluakmVVWup1c2HZi8O83b2xGvsjkipPy8+acv1aTsuR9Sm3F4/cpgNHYpBq0rN3ot5d7J1TsaQkETTF71Et3UilqLJmkJmZifDwcDz00PXCHzBgAL744ot6wzRFzUAuVx51WR4RNdV8LbuXyplnUx3FKKktuPoIl1ynKbYnJduSnPm5a9u01kTk6LTkJplm10z00ksv4YEHHsC9994LALjvvvuwa9cuaLV/9IQtKipC27ZtZU1v4+Er2PTNr84IlYjI5bZLdK2W4/fff29ezUQ+Pj4wGAym93V1dfUSgVFYWJisvwXjYhr0ZLDXe8Wyi6HUa3PmXdMsP7PHsguj1Pi2uuxZxiv1nbXubVLztHytpCnAWi8hqeWztszW5ie3PG3FZrlMUvFalrVUPPa2CeM0rG1HUqSGtTZtW8PZmo+98je+t9bF1vje3u9IaW8xW90uLedna/rWysFak521uOytc1vbqLVlkorfcnu09Ts3dj0H5O/7LP9sUWUy+K//+i8UFBQAAA4fPozQUNe3sylp2zPfMKyNZ2+Ham9+cr63Ng9H2iktl8mya6pSUjHYuq2AvXmN793ObldRpfFIfaekCi61HSi9ytjWMpn3/lISi635WRve1gVXStk7UHEFpfE39veolCPbS1NTZTIYPHgwdDodxo4di8zMTMyaNcvp81S6URr7VQPyVqDlDs4ZP4Km3JCs7QyaYkfkyA7bsl/9hAh/RQlGzjylhrGVmKR2oON7t5M1XWvbga1lsrbzlppWU+z8LGOzt1O3jEPO78NWQpJLyXhytk2pdShnutbWsyPLZa0sLV83JVXejsLDwwOvvPKK06ZvvrKNfYXt/RjMX0cFSe+IpIY1f29+lCk1/p86e9kc3/K1sY+zrXlbi8NWzOavLcvK+F9q3kbGZTNeZ2A5vtQOwFaZSY9TC6BhX3CpaVmuX7k7LXs7V6lkNSHCHz/85mF1OMt5pQ4ObVCWUmVinL5UrNa2KfPxL136BR06dGzwua2djL3agbXysbWNm6+vS5d+sbpMUvFYloGc8Sxf20uoqYNDUVxse9sybtfmvwNr07W1X7G1w7fcPq7HVWx1mk1BlSeQ5WhMbyLjTbvURo1xqTEmgHEppca41BgT0LLjanbXGRARkWsxGRAREZMBERExGRAREZr5CWQiIlKmWd2OgoiIXIvNRERExGRAREQqvQLZGeQ8MMeVRowYAV9fXwBAQEAApkyZgrS0NGg0GoSEhCAjIwMeHq7L1d988w2WLFmC7OxsnDp1SjKW3Nxc5OTkQKvVYurUqYiLi3NpXMeOHcOUKVNw++23AwDGjRuHBx980KVxVVdX48UXX8TZs2dRVVWFqVOn4o477nBreUnFdMstt7i9rGpra/Hyyy/j5MmT8PT0RGZmJoQQbt+2pOIqLy93e3kZXb58GaNGjcL69euh1WpdV16ilfj000/FzJkzhRBCfP3112LKlClui6WiokIMHz683meTJ08WBw4cEEIIMXv2bPHZZ5+5LJ433nhDDB06VIwePdpqLD///LMYOnSoqKysFL/99pvptSvjys3NFW+++Wa9YVwd15YtW8T8+fOFEEJcuXJF3HvvvW4vL6mY1FBWO3fuFGlpaUIIIQ4cOCCmTJni9rKyFpcayksIIaqqqsTf/vY38cADD4gffvjBpeXVapqJioqKMGDAAABAREQEjh496rZYSkpKcO3aNTz55JN47LHHcPjwYRw7dgz9+vUDAMTGxmLfvn0uiycwMBCrVq0yvZeK5ciRI4iMjIROp4Ovry8CAwNRUlLi0riOHj2KPXv2YPz48XjxxReh1+tdHldCQgKef/5503tPT0+3l5dUTGooq/vvvx/z5s0DAJw7dw4dOnRwe1lZi0sN5QUAixcvxtixY9GpUycArv0ttppkoNfr4ePjY3rv6emJmpoat8TSpk0bTJw4EW+++Sbmzp2LF154AUIIaDQaAIC3tzfKy8tdFk98fHy950VIxaLX603NWsbP9Xq9S+MKDw/HjBkzsGnTJnTt2hWrV692eVze3t7w8fGBXq/Hc889h5SUFLeXl1RMaigrANBqtZg5cybmzZuH+Ph4t5eVtbjUUF5bt26Fv7+/6aAVcO1vsdUkA7kPzHGFoKAgPPzww9BoNAgKCkK7du1w+fIfd/g0GAzw8/NzS2wA6p2rMMZiWX4Gg6HeBukKgwcPxl133WV6/d1337klrvPnz+Oxxx7D8OHDMWzYMFWUl2VMaikr4PrR7qefforZs2ejsrKy3vzduW2Zx3XPPfe4vbzy8vKwb98+JCcno7i4GDNnzsSVK388ktPZ5dVqkoEaHphjtGXLFixatAgAcPHiRej1esTExKCwsBAAUFBQgL59+7otvl69ejWIJTw8HEVFRaisrER5eTnKyspcXoYTJ07EkSNHAAD79+/HnXfe6fK4Ll26hCeffBJ///vf8eijjwJwf3lJxaSGsvrggw+QlZUFALjxxhuh0Whw1113uX3bkorrmWeecXt5bdq0CRs3bkR2djbCwsKwePFixMbGuqy8Ws1FZ8beRKWlpRBCYOHChejevbtbYqmqqsKsWbNw7tw5aDQavPDCC2jfvj1mz56N6upqBAcHY/78+fD09HRZTGfOnMG0adOQm5uLkydPSsaSm5uL9957D0IITJ48GfHx8S6N69ixY5g3bx5uuOEGdOjQAfPmzYOPj49L45o/fz62b9+O4OA/nkH70ksvYf78+W4rL6mYUlJS8Oqrr7q1rH7//XfMmjULly5dQk1NDZ566il0797d7duWVFy33nqr27ctc8nJyZgzZw48PDxcVl6tJhkQEZF1raaZiIiIrGMyICIiJgMiImIyICIiMBkQERGYDIiICEwGREQEJgOiJvfVV19h5cqV7g6DSBFedEZERKwZEDW15557DocOHXJ3GESKMBkQNbHjx4+jR48e7g6DSBEmA6ImVFlZiZqaGpff3puosZgMiJrQ8ePH3XY3XKLGYDIgakKlpaVsIqJmicmAqAkxGVBzxa6lRETEmgERETEZEBERmAyIiAhMBkREBCYDIiICkwEREYHJgIiIwGRAREQA/g/JpWgNZ+Xf1QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_squared_errors(y_test, predicted_y_for_test_data, \"Squared errors for test data\")\n",
    "plot_squared_errors(y_train, predicted_y_for_training_data, \"Squared errors for training data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1.2\n",
    "From the graphs, we can observe that for the training data, there are more outliers with significantly higher squared errors with values in the range of 600 - 700. In contrast, the outliers for the test data have squared errors in the range of 150-250. There are also generally more outliers in the training dataset as compared to the test dataset. This might be one of the reasons that contributed to the higher MSE of the training dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1.2: Ridge Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this task, we use ridge regression to create a model for our dataset. Since we have a hyperparameter, $\\lambda$ for ridge regression that controls how much penalty we apply to large values of $\\beta$ to keep variance low, we use 5 fold cross validation on the training dataset to find the optimal $\\lambda$ for our model. We then find the best estimator for the ridge regression using this $\\lambda$. Since our task is regression, MSE is used to calculate the performance of our model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the lectures, ridge regression is an extension of linear regression with least squares loss function, by including a (usually small) positive penalty term $\\lambda$:\n",
    "$$\n",
    "\\underset{\\boldsymbol\\beta}{\\text{min}} \\| \\mathcal Y - \\mathcal X \\boldsymbol\\beta \\|^2 + \\lambda \\| \\boldsymbol\\beta \\|^2 = \\underset{\\boldsymbol\\beta}{\\text{min}} \\ \\text{L}_{\\text{ridge}} (\\boldsymbol\\beta)\n",
    "$$\n",
    "where $\\text{L}_{\\text{ridge}}$ is the ridge loss function. The solution is\n",
    "$$\n",
    "\\boldsymbol\\beta^{*}_{\\text{ridge}} = (\\boldsymbol X^T\\boldsymbol X + \\lambda I)^{-1}\\boldsymbol X^T\\boldsymbol y \\, .\n",
    "$$\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# shuffles the dataset and splits the dataset into k folds\n",
    "def split_data(data, num_of_folds=10):\n",
    "    fold_size = int(data.shape[0] / num_of_folds)\n",
    "    np.random.seed(SEED_VALUE)\n",
    "    shuffled_data = np.random.permutation(data)\n",
    "    return np.array_split(shuffled_data, num_of_folds)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform cross validation, we split our data set into 5 folds. np.array_split is used to split the dataset into 5 folds, and all data points are divided into the folds. This generates folds of different lengths. I chose not to truncate the last 4 data points to make the folds have equal length, even though the data points lost to truncation would be small. This is because our data set size is relatively small, with 400 data points, and we should consider all the data points when generating models for the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(81, 19)\n",
      "(81, 19)\n",
      "(81, 19)\n",
      "(81, 19)\n",
      "(80, 19)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None, None, None, None, None]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "folds = split_data(regression_train_data, 5)\n",
    "[print(fold.shape) for fold in folds]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generates the training sets and test sets for each cross validation fold\n",
    "def generate_test_train_sets(folds, i):\n",
    "    # drop one fold as test\n",
    "    cv_test_set = folds[i]\n",
    "    test_x = cv_test_set[:, : -1]\n",
    "    #labels for test data\n",
    "    test_y = cv_test_set[:, -1]\n",
    "\n",
    "    # use the rest as training data\n",
    "    cv_train_set = np.vstack([folds[k] for k in range(len(folds)) if k != i])\n",
    "\n",
    "    train_x = cv_train_set[:, :-1]\n",
    "    train_y = cv_train_set[:, -1]\n",
    "    return train_x, train_y, test_x, test_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# finds beta_ML given penalty term\n",
    "def ridge_regression_estimate(X, y, penalty):\n",
    "    R, C = X.shape\n",
    "    I = np.identity(C)\n",
    "    beta_ridge = np.linalg.solve(X.T @ X + penalty * I, X.T @ y)\n",
    "    return beta_ridge\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#plot penalty parameter against mse\n",
    "def plot_penalty_param(params_to_errors_map, title):\n",
    "    plt.scatter(params_to_errors_map.keys(), params_to_errors_map.values(), marker='+')\n",
    "    plt.title(title)\n",
    "    plt.xlabel(\"$\\u03BB$\")\n",
    "    plt.ylabel(\"Mean Square Error\")\n",
    "    plt.show()\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We perform cross validation across the different folds and obtain graphs of mean square errors against the hyperparameter $\\lambda$ for each fold. We then take the fold averaged penalty parameter as our optimal hyperparameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform cross validation across each fold of thee datsset\n",
    "def cross_validation_across_each_fold():\n",
    "    num_of_folds = len(folds)\n",
    "    #initialise list of penalty terms to try out in cross validation\n",
    "    penalty_params = np.linspace(0, 20, num=100)\n",
    "\n",
    "    opt_lambda_in_each_fold = []\n",
    "    \n",
    "    # grid of errors for each parameter in each fold\n",
    "    errors_grid = np.zeros((num_of_folds, len(penalty_params)))\n",
    "    errors_for_ith_fold = {}\n",
    "    lowest_error = 0\n",
    "    \n",
    "    for i in range(num_of_folds):\n",
    "        train_x, train_y, test_x, test_y = generate_test_train_sets(folds, i)\n",
    "        optimal_lambda = None\n",
    "        min_mse = sys.maxsize\n",
    "        \n",
    "        for j, penalty in enumerate(penalty_params):\n",
    "            #train model using function\n",
    "            beta_ridge = ridge_regression_estimate(train_x, train_y, penalty)\n",
    "            \n",
    "            # use the model to predict values for the validation set and calculate the the corresponding mse\n",
    "            predicted_y = predict_with_estimate(test_x, beta_ridge)\n",
    "            # calculate mse on the test set with beta_ridge calculated\n",
    "            error_validation = mse(test_y, predicted_y)\n",
    "        \n",
    "            #add error for the validation set to the grid of errors\n",
    "            errors_grid[i][j] = error_validation\n",
    "            # if mse for current lambda < min_mse so far, replace optimal lambda with current lambda\n",
    "            if min_mse > error_validation:\n",
    "                min_mse = error_validation\n",
    "                optimal_lambda = penalty\n",
    "        \n",
    "        lowest_error += min_mse\n",
    "        opt_lambda_in_each_fold.append(optimal_lambda)\n",
    "\n",
    "    # plot mse against various penalty parameters for each fold\n",
    "    for i in range(len(errors_grid)):\n",
    "        param_error_map = {}\n",
    "        for j in range(len(penalty_params)):\n",
    "            param_error_map[penalty_params[j]] = errors_grid[i, j]\n",
    "        plot_penalty_param(param_error_map, \"Errors for each \\u03BB for Fold \" + str(i))\n",
    "    \n",
    "    # find average lowest mse across all 5 folds\n",
    "    lowest_error /= num_of_folds\n",
    "    # get min lambda for each fold and get the average of these lambda across all 5 folds\n",
    "    average_optimal_lambda = sum(opt_lambda_in_each_fold) / len(opt_lambda_in_each_fold)\n",
    "    \n",
    "    # retrain the ridge regression model on the entire training dataset with the average penalty \n",
    "    optimal_beta_ridge = ridge_regression_estimate(X_train, y_train, average_optimal_lambda)\n",
    "    return average_optimal_lambda, lowest_error, optimal_beta_ridge, opt_lambda_in_each_fold\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAESCAYAAAD38s6aAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmEklEQVR4nO3de1RU5foH8O/IVcRLxNFDooIliFlmlNdUVMRLgAQqcgg1rYOmef+JoAaKSl7S0NJwLfV4yEKPmvdLLtKwRDI8muZkKeRRSEQRkTsM7+8Pz8xhYIZhgLny/azVgtl7z97P7JkeH5797nckQggBIiIyOy0MHQAREekGEzwRkZligiciMlNM8EREZooJnojITDHBExGZKSZ4UuLu7g4/Pz+MHTtW6b+7d+8aJJ7o6GgMGzYMGzduNMjxa9q8eTNWrFhRr203btyImJiYOrfZunUrvLy8EBkZ2aiY+vXrV+s9u3r1ap3PGzZsmMptTp48ibCwMJXPOXv2LPz8/DBy5EjMnj0bhYWFDY6bdM/S0AGQ8dm1axccHBwMHQYAYM+ePTh79iz++te/GjoUrY0fPx5vvfUWFi1aBDs7O5Xb7Nu3D+vXr8drr73WqGONGTMGH374YaP2oUleXh4iIyPx1VdfwcXFBevWrcP69es1/iNGhsMET/WWlpaGVatWwc7ODkVFRVi0aBHWrVuneLx//358/fXXSExMRIsWLeDo6Ihly5bB1dUVixcvRn5+Pu7cuQMvLy8MHToUH330EaqqqgAA4eHhGDlypNLx/va3v0EIgffeew/R0dFo27YtVqxYgfz8fEgkEkydOhUBAQG14tq/fz+sra0V+8nJycGKFSvw559/oqKiAm+++SamT58OAPj888+RnJyM0tJSlJSUICIiAiNGjEBlZSXWrVuHs2fPwsLCAr1790Z0dDQAICMjA2FhYcjNzYWjoyM2bNiA9u3b1zpfHTt2hI2NDY4dO4bx48fXWj937lzk5ORgyZIlmDNnDl599VXExMQgKysLQggEBATg3Xffxd27dxEaGornn38eWVlZSExMVHk8VSoqKvDRRx8hNTUVFhYWePnllxEZGQl7e3ul7eLj43HkyBG0a9cOXbp0Ubmv77//Hi+99BJcXFwAACEhIRg7diyio6MhkUjqFQ/pmSCqxs3NTfj6+gp/f3/Ff++//74QQogLFy6I7t27i7t376p8fP78eeHt7S0ePnwohBBi//79YvTo0aKqqkpERESIyZMnK44zadIkcfToUSGEEFKpVMTExKiN5+HDh6KiokIMHz5cnDp1SgghxL1798SgQYPEpUuXasVRU1hYmEhOThZCCFFaWirCwsLEsWPHxN27d0VYWJgoKSkRQghx9OhR4evrK4QQYteuXSI0NFSUlJQImUwm5syZI77++muxadMmMWzYMMVrnDFjhvj0009VHve7774Tnp6eIigoSO35Hjp0qPj555+FEEKEhoaKHTt2CCGEKCgoEH5+fuLo0aPizp07ws3NTVy8eFHlPjZt2iT69u2r9J5t3rxZCCFEfHy8mDVrligvLxcymUwsXrxYLFu2TOnYp0+fFmPGjBFPnjwRFRUV4u9//7t4++23ax0nISFB8VwhhKioqBBubm7iyZMnal8fGRYreKqlrhaNk5MTOnbsqPLxuXPnMGbMGMVzAwMDsWrVKkX/3tPTU/G80aNHY8WKFfj2228xYMAAzJ8/v86Y/vjjD5SVlcHHxwcA0KFDB/j4+ODcuXPo27dvrbjkiouLcfHiRTx+/Bjx8fGKZb/++ivGjBmDtWvX4siRI7h9+zauXLmCoqIiAMD58+cxduxY2NraAgA++eQTAE/73QMHDlS8xu7duyMvL09lzElJSViwYAE+//xzXL9+HT169FD7+oqLi3Hp0iXs2LEDANC6dWsEBgYiJSUFvXr1gqWlJV555RW1z1fXoklJScG8efNgZWUFAAgLC8PMmTOVtklNTcWIESMUVX1QUBASExNr7auqqkplpd6iBS/lGSu+M6SVmr3k6o/l7ZbqhBCorKyste3EiRNx+PBhDBw4EN9//z38/f1RVlam9rgymaxWclG37+qqqqoghEBSUhIOHTqEQ4cOYc+ePQgPD8cvv/yC4OBgFBYWYuDAgXj33XcVz7O0VK59Hjx4gPv379daJ5FIIFRM55STk4OffvoJ/v7+CAoKwp49e9S+tupx1lwmf33W1ta1YqqPmkm5qqoKFRUVtbarfmwLCwuV+3JyclKcA+Dpa2zbtq3ac0+GxwRPTWbQoEE4fvy4oqLdv3+/2p7uxIkTIZVKERgYiNjYWBQUFCA3N1ftvrt27QpLS0t88803AJ4ml1OnTmHAgAF1xmRvb49XXnkFO3fuBAAUFBQgJCQEycnJuHjxInr27Il33nkHffr0QXJyMmQyGQCgf//+OHr0KMrLy1FVVYWYmBgcO3as3udi7969GDVqFFq1aoVx48bh+PHjKC4urjPOXr16Yffu3QCAJ0+e4ODBgxpfnyaDBg3CV199hYqKClRVVWH37t0YOHCg0jaDBw/GyZMnUVBQgKqqKhw6dEjlvt544w1cuXIFf/zxB4Cnf6EMHz68UfGRbrFFQ7VMnjy51p/d8+fPV7Qr1Bk4cCCmTJmCyZMno6qqCg4ODkhISFD5J/zChQuxevVqfPLJJ5BIJJg1axacnZ3V7tvKygpbtmzBypUrsXnzZshkMsycORP9+vVDWlpanXGtX78esbGx8PPzQ3l5OXx9feHv748HDx7gm2++wejRo1FVVYWhQ4fi8ePHKCwsxMSJE5GVlYXAwEAIIdCnTx+EhYVh69atdR4LePrXxr/+9S9s2bIFAPDcc8+hV69eai+2Vo9zxYoVOHDgAMrLy+Hn54fAwEBkZWVpPKY6M2bMwJo1axAQEIDKykq8/PLLWLZsmdI2Q4YMwY0bNxAUFIQ2bdqge/fuePToUa19Pfvss4iLi8Ps2bNRUVGBzp07Y82aNQ2OjXRPIlT9fUlERCaPLRoiIjPFBE9EZKaY4ImIzBQTPBGRmWKCJyIyU0Y1TDI9Pd3QIRARmaTqd4rLGVWCB1QHWR9SqRQeHh5NHE3jMS7tMC7tMC7tGGtcQONiU1ccs0VDRGSmmOCJiMwUEzwRkZligiciMlNM8EREZsroRtEQETU3wQmpKC4uxpEmHuHDCp6IyEyxgici0rPghFSlx2mZeUrL94T3b5LjsIInIjJTrOCJiPREXqHLK/a+rg6Kn8XFxU1Wucuxgici0oPghFRc/7NAr8dkBU9EpCc9nNpgT3h/lb12qVTa5Mdjgici0qGabRl5Jd/DqY3Oj80WDRGRjqhry8greV1jBU9EpEN1tWV0jQmeiKgJVW/BGKItUx1bNEREeqCvtkx1rOCJiJpAzYupANDa1tIgiV2OFTwRUSMZYox7fbCCJyJqAoa8mKoOEzwRUQMY08VUddiiISJqQobsudfECp6ISAvGeDFVHVbwRET1ZKwXU9VhBU9EpAVjvJiqDhM8EVEdTOFiqjps0RARNYAx9txrYgVPRKSCKV1MVYcVPBFRDaZ2MVUdVvBERCqY0sVUdZjgiYj+y5DfvqQLbNEQEcHw376kCzqp4GUyGZYuXYrMzExYWFggLi4ORUVFiI6OhoWFBVxcXLBq1Sq0aMF/X4jIeJhDW6Y6nST4M2fOAACSkpKQlpaGuLg4tGjRAjNnzsSQIUOwYMECnD17FsOGDdPF4YmI6sWUx7jXh04SvLe3N7y8vAAA2dnZcHR0RIcOHZCfnw8hBIqKimBpyfY/ERknU27LVCcRQghd7TwiIgKnT5/Gpk2bkJ+fjxUrVsDBwQGtW7fGF198ARsbG6Xt09PTYWdn16BjlZaWwtbWtinCblKMSzuMSzuMSzvyuBadzAYAXM0pBQC81MEWGXll6Opgg7WjnjNobA1RXFwMT0/PWst1muABIDc3FxMmTEBJSQkSExPRrVs37N69Gzdv3kR0dLTStunp6SqDrA+pVAoPD4+mCLlJMS7tMC7tMC7tSKVSxKTk12rL9HV1UCwzVOXemHOmLnfqpE9y8OBB5OTkIDw8HC1btoREIkG7du1gb28PAGjfvj0uXbqki0MTEWlkbhdT1dFJgvfx8UFkZCRCQ0NRWVmJqKgotGvXDvPmzYOlpSWsrKwQGxuri0MTEdUiv3Dq0tZS0ZYxp4up6ugkwdvZ2SE+Pr7W8qSkJF0cjoioQczlYqo6HMpCRGZH3nqRU0wY1tbS5CYMawzeaUREZKZYwROR2ag5l0xfVwelnzGD2xnl6B5dYQVPRGbBXKb4bUqs4InIbGga/iiVSg0RlsEwwRORyTL3uWQaiy0aIjI7zWWUjCas4InI5JjD96XqAyt4IjIpvJhaf6zgicjkNJe5ZBqLCZ6IjB4vpjYMWzREZLLYc68bK3giMlq8mNo4rOCJyCjxYmrjsYInIqPFi6mNwwRPREaDF1ObFls0RGT02HNvGFbwRGRwvJiqG6zgicigeDFVd1jBE5HB8WKqbjDBE5He8WKqfrBFQ0RGgz33psUKnoj0QlXVDvBiqi6xgiciMlOs4IlIpzgE0nA0VvAXLlzQRxxEZIY4BNKwNFbwmzdvRr9+/fQRCxGZIQ6BNByNCV4ikWDmzJlwdXVFixZPC/758+frPDAiMk3BCam4dvcRejrncwikgWlM8EFBQfqIg4jMHHvu+qcxwfv5+WHPnj24efMmXFxcEBISoo+4iMiEcAikcdJ4kfXDDz/EnTt3MHDgQGRlZWHp0qX6iIuIiBpJYwV/+/Zt7N69GwDg7e2NiRMnatypTCbD0qVLkZmZCQsLC8TFxWHjxo148OABACArKwu9evXCxo0bGxk+ERmSuiGQLm0tWbUbAY0JvqysDCUlJWjZsiVKS0shk8k07vTMmTMAgKSkJKSlpSEuLg5bt24FADx+/BiTJk1CZGRkI0MnIkPihVPjpzHBT548GWPHjkW3bt1w8+ZNzJ49W+NOvb294eXlBQDIzs6Go6OjYt3mzZvx9ttvo3379g2PmoiMgrohkFKp1JBh0X9JhBCirg0uXLiA7t27486dO3B2dsYzzzxT751HRETg9OnT2LRpE9544w08fPgQkyZNwuHDh2FhYVFr+/T0dNjZ2Wn/KgCUlpbC1ta2Qc/VJcalHcalHX3GtehkttLjqzmlAICXOtgiI68MXR1ssHbUc3qPSxvGGhfQuNiKi4vh6elZa3m9bnTavXs32rVrp/VB16xZg4ULF2LChAk4duwYTp48CV9fX5XJXc7Dw0Pr4wBPK4aGPleXGJd2GJd29BmXXUp+jSVPE7ydnR162tkp9dx5vrTXmNjS09NVLtfJjU4HDx5ETk4OwsPD0bJlS0gkElhYWCA1NRUzZsxoQPhEZAiqhj/2dXVQ+smLqcZLY4IPCAios+JWxcfHB5GRkQgNDUVlZSWioqJgY2ODzMxMdOrUqcHBEhFR/WlM8MePH8eOHTu02qmdnR3i4+NrLT927JhW+yEiw+AMkOZBY4Jv3bo1kpOT4eLiomjRuLq66jwwIjIMDn80HxoTfF5eHv7xj38oHkskEvzzn//UZUxEZGCcAdI8aEzwiYmJSo/Lysp0FgwRGQa/BNs8qZ2LZu7cuYrfq/fg33vvPZ0GRETGgz1306a2gn/48KHi97Nnz2Lq1KkAAA33RRGRieAMkOavXl+6XT2pSyQSnQVDRERNR20FXz2RM6kTmQdW7c2L2gR/8+ZNLFiwAEIIpd9v3bqlz/iIiKiB1Cb4Tz75RPF79Tng6zMfPBEZF9641DypTfB9+vTRZxxEpCMc7th8aRwHT0SmjzcuNU9M8ERmiDcuEVCPBJ+Tk4N169bh0aNHGDlyJNzd3dGrVy99xEZETYw99+ZFY4JftmwZ3nnnHWzZsgWvvfYaFi9ejL179+ojNiLSAodAUk0ab3QqKytD//79IZFI0LVrV9jY2OgjLiIiaiSNFby1tTXOnTuHqqoqXL58GdbW1vqIi4jqYdHJbPyx5z+s2kkljRV8bGwsDhw4gEePHmHHjh2IiYnRQ1hERNRYGiv4nTt3YuPGjfqIhYjqST7c8WpOqWIZq3aqSWMFf+vWLRQUFOgjFiKqB/nFVCJNNFbwt27dQt++feHg4KCYdOz777/XeWBEpJ68UvfbmAw7OztW7aSSxgR/5swZfcRBRHVQd+NSRl4ZetrZGTg6MlYaE/zly5dx4MABVFRUAADu37+P7du36zwwItKsq4MNq3dSS2OCX7lyJaZMmYJTp07Bzc0N5eXl+oiLqNmrz41LUqnUgBGSsdN4kbVNmzbw9fWFvb09PvjgA+Tk5OgjLiIiaiSNFbxEIsHvv/+OkpISZGRkIDc3Vx9xETU78qGPcrxxiRpLYwW/ePFi/P777wgLC8PChQsREhKij7iIiKiRNFbw3bp1Q7du3QAABw4c0HlARM2Jqj57X1cHpZ+s2qmhNCb4N954Q/F7fn4+OnXqhBMnTug0KCIiajyNCb76TU1ZWVn49NNPdRoQkbnjtL6kLxp78NV17NgRGRkZuoqFiIiakMYKfv78+YopCu7fv49nn31W405lMhmWLl2KzMxMWFhYIC4uDq1atcLSpUtRUFAAmUyGtWvXonPnzo1/BUQmQj5KhlU76YvGBD9x4kTF7zY2NujZs6fGncqnN0hKSkJaWhri4uLQtm1b+Pn5YcyYMbhw4QIyMjKY4KnZ4PehkiFoTPDZ2dlKjzMzMxW/BwQEqHyOt7c3vLy8FM93dHREWloa3N3dMWXKFHTs2BFLlixpeNREJkheqcsreVbtpGsSIYSoa4PZs2ejuLgYffr0wbVr1/DgwQN4enpCIpFg/vz5de48IiICp0+fxqZNmxAeHo4VK1YgKCgIn376KWQyGebMmaO0fXp6OuwaOHFSaWkpbG1tG/RcXWJc2jGnuBadzEZGXhm6Otgo5m1/qYOtYtnaUc8ZJC59YFzaa0xsxcXF8PT0rLVcYwVfWFiI7du3K/rwU6dOxYIFC+p10DVr1mDhwoWYMGECWrdujWHDhgEAhg0bpvZLRDw8POq175qkUmmDn6tLjEs75hSXXUo+Wjyu/G/R8jTB29nZoWcTTu9rTudLH4w1LqBxsaWnp6tcrjHB5+Xl4cmTJ2jTpg3y8vJQWFio8WAHDx5ETk4OwsPD0bJlS0gkEvTp0wffffcdAgICcPHiRbzwwgvavwoiI8chkGRMNCb46dOnY+zYsWjXrh2Ki4uxfPlyjTv18fFBZGQkQkNDUVlZiaioKHh4eGDp0qVISkqCvb09Pv744yZ5AUREpJrGBD9q1Ch4e3vj8ePH9RoiCTz9MzQ+Pr7W8p07d2ofIZGRY9VOxkrtjU537tzBzJkzUVlZicuXL8Pf3x8jRozA5cuX9RgeERE1lNoKfvXq1Rg3bhwsLS0RFxeHtWvX4oUXXsDChQuRmJiozxiJjA6rdjIFahN8eXk5hg8fjkePHuHevXsYOHAgAKCqqkpvwRERUcNp7MGnpqaiX79+AJ4m9ydPnug8KCJjxKqdTI3aBN+tWzcsWLAA165dQ2xsLO7fv48NGzYokj0RERk3tQk+IiICKSkpCA8Ph5ubG27cuIHu3bsjLCxMn/ERGZS8andpa6m4GxVg1U6mQW2Cl0gkGDJkiOKxu7s73N3d9RIUERE1nsYePFFzo7LX3taSVTuZHK2+8IOIiExHvSYbS0lJQXl5uWKZummCiUyVphEyMYPbGe0kVUTqaEzw77//Ptq3bw8nJycAUMwqSURExk1jghdCYP369fqIhUivtBnXLpVKDREiUaNo7MG7u7vjypUrKC8vV/xHRETGT2MF/+OPP+Lbb79VPJZIJEhOTtZpUES6wrtRqTnRmOAPHz6sjziIiKiJaUzwycnJ+PLLL1FRUQEhBPLz83HkyBF9xEbUaPIvuJZj1U7NicYe/GeffYZZs2bByckJb731Ftzc3PQRFxERNZLGCv6ZZ55B7969kZSUhMDAQBw4cEAfcRE1mKo+e19XB6WfrNqpOdBYwVtZWeHixYuorKzEuXPnkJubq4+4iIiokTRW8MuXL0dGRgZmzJiB+Ph4zJ49Wx9xEWmFo2OIatOY4Dt06ICMjAxcunQJM2fOhKurqz7iIiKiRtKY4Dds2IB79+7h1q1bsLKywrZt27BhwwZ9xEZUJ1btRHXT2INPT0/H2rVrYWdnh7feegt3797VR1xERNRIGit4mUyGsrIySCQSyGQytGjBGYbJcFi1E9WfxgQ/efJkBAYGIi8vD+PHj8eUKVP0EBYRETWWxgQ/evRoDBgwALdv34azszMcHBz0EReRAqt2ooZRm+AjIyPVPikuLk4nwRDJyacYYPImaji1Cf7atWsoLS2Fv78/evfuDSGEPuMiwvU/CxCckMqqnaiB1Cb4I0eO4LfffsPhw4exbds2vP766/D390eXLl30GR81I8EJqbh29xF6Oucrkvr1PwsMHBWR6aqzB+/m5oaFCxcCAC5evIiPP/4Y9+7dw969e/USHFEPpzaK/jurdiLt1OtLt0+fPo2jR4+ipKQE/v7+Gncqk8mwdOlSZGZmwsLCAnFxcXjy5AmmT58OFxcXAEBISAjGjBnT6BdApq0+F1BrTvlLRPWjNsGfOHECx44dQ3Z2Nnx8fLB8+XI4OzvXa6dnzpwBACQlJSEtLQ1xcXEYNmwY3nnnHUydOrVpIqdmg5U7UcOoTfDz5s1D165d0b17d/z222/YuHGjYt3HH39c5069vb3h5eUFAMjOzoajoyOuXbuGzMxMJCcno0uXLoiKioK9vX3TvAoyKXVV7S5tLZnQiZqIRKgZHvPjjz+qfVKfPn3qtfOIiAicPn0amzZtQk5ODtzd3dGzZ09s3boVBQUFiIiIUNo+PT0ddnZ2WoT/P6WlpbC1tW3Qc3WJcdW26GQ2MvLK0NXBBldzSgEAL3WwRUZeGVzaWWH9mPr9pahPfB+1w7i015jYiouL4enpWWu52gTfVHJzczFhwgQkJSWhQ4cOAICbN28iNjYWu3btUto2PT1dZZD1IZVK4eHh0eh4mxrjekrdl3DUvIDK86UdxqUdY40LaFxs6nKnTiaWOXjwIBISEgAALVu2hEQiwaxZs/Dzzz8DAFJTU/Hiiy/q4tBERPRfGkfRNISPjw8iIyMRGhqKyspKREVFwcnJCbGxsbCysoKjoyNiY2N1cWgyIpxigMiwdJLg7ezsEB8fX2t5UlKSLg5HRoRTDBAZD50keGreOMUAkXFggqdGqXkTEqcYIDIeTPCkE5xigMjwmOBJa+qGPFb/ySkGiAyPCZ50hpU7kWExwVO9cMgjkenhN2gTEZkpVvCkFqt2ItPGCp6UBCek8uIokZlgBU+18EYlIvPABE8qWzG8UYnI9DHBk0q8UYnI9DHBN1OLTmbjjz3/4XehEpkxJvhmRNuZHlm5E5k2JvhmRn4BVf5VeQAvoBKZKyZ4M8cLqETNFxN8M9TDqQ2u3X2Ens7PsGonMmNM8GaoPneg+m1MNmCERKQPTPBmQtsLqGtHPWe03y5PRE2DCd6M8A5UIqqOCd6E8QIqEdWFCd7EaGrF8A5UIpLjbJImqHor5klpJYCnrZi+rg7YE94fPZzaGDhCIjIGrOBNgLatGFbuRAQwwZsstmKISBMmeCNUc5IvTgZGRA3BBG8ktB3Hru22RNT8MMEbkZrj2Pu6Oij9ZEInIm0wwRsQx7ETkS4xwesZx7ETkb7oZBy8TCZDZGQkJk6ciNDQUPznP/9RrDty5AiCg4N1cViTwXHsRKQPOqngz5w5AwBISkpCWloa4uLisHXrVkilUuzbtw9CiCY9XnBCKoqLi3HESCfP4jh2IjIEnSR4b29veHl5AQCys7Ph6OiIR48eYf369YiKisKyZct0cVijIm/FxAxup3I9WzFEpGs668FbWloiIiICp0+fRnx8PJYsWYKoqCjY2NjU+TypVFrvYyw6mQ0Aiq+fk89xvnbUcw2MuukUFxcjI68MC48X4pfccsWyVlYSdHWwQczgdlh0shjFxcVaveamUlpaapDjasK4tMO4tGOscQG6iU0imrpfUkNubi6GDx8OR0dHdOzYEWVlZbh58yaCgoKwZMkSpW3T09Ph6elZ733Lq+SawwoNVRGrasW0spKgqEIo4jOWql0qlRrlfPCMSzuMSzvGGhfQuNjU5U6dVPAHDx5ETk4OwsPD0bJlSzg6OuLEiROwsbHB3bt3MX/+/FrJvSHkSVLegzdE0tQ0Kqargw3+eFxpFEmdiJoXnYyi8fHxwfXr1xEaGopp06bVqzVjyuoaFbN21HMcFUNEBqGTCt7Ozg7x8fEq1zk7O2Pv3r1Nerw94f312lfjqBgiMgW80ameGnqDkrFe0CEi88cv/KhDcEKq0oyNvEGJiEwJK3gNak4AxlYMEZkKJvgaNPXXeYMSEZkKtmhQuxVTUw+nNmzFEJHJaZYVvKoLpjVbMYD6b09i5U5EpqBZJni5mnfCsr9OROak2SR4Vb311rbKL5/9dSIyJ2ad4Oszdr06fpE1EZkTs0vwNZO6pt569edUfx4RkakzuwQPaDd2HWBSJyLzZDYJftHJbNil5HPsOhHRf5ntOHiOXSei5s5sKvi1o56Dh4eHUg+evXUias7MJsGrwqRORM2Z2SV4JnUioqfMtgdPRNTcMcETEZkpJngiIjPFBE9EZKaY4ImIzBQTPBGRmZIIIYShg5BLT083dAhERCbJ09Oz1jKjSvBERNR02KIhIjJTTPBERGbK5KYqqKqqQkxMDG7cuAFra2usXLkSXbp0Uaz/9ttv8dlnn8HS0hJBQUGYMGGCXuKqqKhAVFQUsrKyUF5ejhkzZmD48OGK9Tt37sS+ffvg4OAAAFi+fDm6du2ql9gCAgLQunVrAICzszPi4uIU6wx1vg4cOICvv/4aAFBWVgapVIoffvgBbdo8nfXTEOfrypUrWL9+PRITE3H79m0sXrwYEokE3bp1Q3R0NFq0+F89pOlzqKu4pFIpYmNjYWFhAWtra6xZswaOjo5K29f1fusqrl9++QXTp0+Hi4sLACAkJARjxoxRbGuo8zVv3jw8ePAAAJCVlYVevXph48aNStvr+nypyg0vvPCCfj5fwsScOnVKRERECCGE+Pe//y2mT5+uWFdeXi68vb1Ffn6+KCsrE4GBgeL+/ft6iWvfvn1i5cqVQggh8vLyxJAhQ5TWL1iwQFy9elUvsVRXWloqxo4dq3KdIc9XdTExMSIpKUlpmb7P17Zt24Svr68YP368EEKI8PBwceHCBSGEEMuWLRPffPON0vZ1fQ51GVdoaKi4fv26EEKIr776SqxevVpp+7reb13GtXfvXrF9+3a12xvqfMnl5+cLf39/kZOTo7RcH+dLVW7Q1+fL5Fo06enpGDRoEADglVdewbVr1xTrbt26hc6dO6Nt27awtraGp6cnfvrpJ73ENWrUKMyZM0fx2MLCQmn9L7/8gm3btiEkJAQJCQl6iQkAfv31V5SUlGDq1KmYNGkSLl++rFhnyPMld/XqVdy8eRPBwcFKy/V9vjp37ozNmzcrHb9Pnz4AgMGDB+P8+fNK29f1OdRlXBs2bICHhwcAQCaTwcbGRmn7ut5vXcZ17do1nD17FqGhoYiKikJhYaHS9oY6X3KbN2/G22+/jfbt2yst18f5UpUb9PX5MrkEX1hYCHt7e8VjCwsLVFZWKtbJ/9QCgFatWtX6oOlKq1atYG9vj8LCQsyePRtz585VWv/mm28iJiYGu3btQnp6Os6cOaOXuGxtbTFt2jRs374dy5cvx8KFC43ifMklJCRg5syZtZbr+3yNHDkSlpb/61gKISCRSAA8PS9PnjxR2r6uz6Eu45InqEuXLuGLL77AlClTlLav6/3WZVwvv/wyFi1ahN27d6NTp0747LPPlLY31PkCgIcPHyI1NRWBgYG1ttfH+VKVG/T1+TK5BG9vb4+ioiLF46qqKsUbWnNdUVGRUgLTtT///BOTJk3C2LFj4efnp1guhMDkyZPh4OAAa2trDBkyBNevX9dLTK6urvD394dEIoGrqyvatWuH3NxcAIY/XwUFBcjIyEC/fv2UlhvyfMlV74cWFRUprg3I1fU51LXjx48jOjoa27ZtU1yjkKvr/dalESNGoGfPnorfa75fhjxfJ0+ehK+vb62/qgH9na+auUFfny+TS/CvvvoqUlJSAACXL1+Gm5ubYt3zzz+P27dvIz8/H+Xl5fjpp5/Qu3dvvcT14MEDTJ06Ff/3f/+HcePGKa0rLCyEr68vioqKIIRAWlqa4n8GXdu3bx8++ugjAEBOTg4KCwvxl7/8BYBhzxcAXLx4EQMGDKi13JDnS65Hjx5IS0sDAKSkpOC1115TWl/X51CXDh06hC+++AKJiYno1KlTrfV1vd+6NG3aNPz8888AgNTUVLz44otK6w11vuTxDB48WOU6fZwvVblBX58vkxtFM2LECPzwww+YOHEihBBYvXo1jhw5guLiYgQHB2Px4sWYNm0ahBAICgpChw4d9BLX559/joKCAmzZsgVbtmwBAIwfPx4lJSUIDg7GvHnzMGnSJFhbW6N///4YMmSIXuIaN24cIiMjERISAolEgtWrV+PEiRMGP18AkJmZCWdnZ8Xj6u+joc6XXEREBJYtW4YNGzaga9euGDlyJABg0aJFmDt3rsrPoa7JZDKsWrUKTk5O+OCDDwAAr7/+OmbPnq2IS9X7rY9KOSYmBrGxsbCysoKjoyNiY2MBGPZ8yWVmZtb6x1Cf50tVbliyZAlWrlyp888X72QlIjJTJteiISKi+mGCJyIyU0zwRERmigmeiMhMMcETEZkpJngiIjPFBE9EZKaY4Ik0uHHjBoYOHYovv/zS0KEQaYUJnkgDd3d3bNiwAYcOHTJ0KERaYYInqodnn30Wv//+u6HDINIKEzxRPaxfvx7l5eXIysoydChE9cYET6RBSkoKSkpK4OXlhZs3bxo6HKJ6Y4InqkNZWRnWrVuH6OhouLm54bfffjN0SET1xgRPVIctW7YgICAAzs7OcHNzYx+eTAoTPJEaGRkZOH/+PCZPngwATPBkcjgfPBGRmWIFT0RkppjgiYjMFBM8EZGZYoInIjJTTPBERGaKCZ6IyEwxwRMRmSkmeCIiM/X/XNyQqmejv5gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAESCAYAAAAbq2nJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtgElEQVR4nO3de1zUZb4H8M84Cr7QdJbIToRb4BGlbfNCXshVMzPIFRW8gpGG7TEiCLNQERDFSMEwlw1Rj7aFpJjaES9ZHd2WNPIoZmnOmrdcRUIRDIfhMgzP+cNldgbmwsDcYD7v16tX8+P3PL/5zm/G5/t7nud3kQghBIiIyCl1sXcARERkP0wCREROjEmAiMiJMQkQETkxJgEiIifGJEBE5MSYBMgsAwYMQHBwMKZMmaLz3/Xr1+0Sz/Lly/HMM89g3bp1dnn/5rKysrBy5cpWlV23bh1SUlKMltmwYQOefvppLF26tF0xjRw5ssV3dubMGaP1nnnmGb1lDh06hIiICIP1hBBYvHgxtmzZ0uaYyXa62jsA6ng+/PBDuLu72zsMAEB+fj6++uor/Md//Ie9QzHbjBkzEBISgvj4eLi5uekts2vXLqxduxZPPvlku95r4sSJSE5Obtc2WuPSpUtYsWIFfvjhB/j6+lr9/aj9mATIYo4fP463334bbm5uqK6uRnx8PDIyMjTLu3fvxqefforc3Fx06dIFHh4eSEpKgre3N5YsWYI7d+7g2rVrePrppzFu3DisXr0ajY2NAIAFCxYgMDBQ5/3Cw8MhhMCf/vQnLF++HL1798bKlStx584dSCQSREZGYurUqS3i2r17N1xcXDTbKSsrw8qVK1FaWgqVSoU//vGPeOWVVwAAOTk5OHz4MGpra1FTU4PFixdjwoQJaGhoQEZGBr766itIpVIMGTIEy5cvBwBcvnwZERERuHXrFjw8PJCZmYk+ffq02F8PP/wwXF1dceDAAcyYMaPF+ri4OJSVlWHZsmV4/fXXMXToUKSkpKCkpARCCEydOhUvv/wyrl+/jjlz5qBfv34oKSlBbm6u3vfTR6VSYfXq1SgqKoJUKsUTTzyBpUuXomfPnjrl1q9fj3379kEmk+GRRx4xuL28vDzMmDEDnp6erXp/cgCCyAy+vr5i0qRJYvLkyZr/Xn31VSGEEN9++60YOHCguH79ut7lb775Rjz77LPi9u3bQgghdu/eLZ5//nnR2NgoFi9eLObOnat5nxdffFHs379fCCGEXC4XKSkpBuO5ffu2UKlUYvz48eLzzz8XQgjxyy+/iNGjR4tTp061iKO5iIgIcfjwYSGEELW1tSIiIkIcOHBAXL9+XURERIiamhohhBD79+8XkyZNEkII8eGHH4o5c+aImpoaoVarxeuvvy4+/fRT8ec//1k888wzms8YFRUl/vKXv+h937///e/C399fTJs2zeD+HjdunPjhhx+EEELMmTNHbN26VQghRFVVlQgODhb79+8X165dE76+vuLEiRN6t/HnP/9ZjBgxQuc7y8rKEkIIsX79evHaa6+J+vp6oVarxZIlS0RSUpLOe3/55Zdi4sSJ4u7du0KlUon/+q//Ei+88ILBmIUQYvHixeK///u/jZYhx8CeAJnN2HDQQw89hIcffljv8tdff42JEydq6oaGhuLtt9/WzCf4+/tr6j3//PNYuXIljhw5gqeeegpvvPGG0Zh+/vln1NXV4bnnngMAPPjgg3juuefw9ddfY8SIES3iaqJUKnHixAn8+uuvWL9+veZv//jHPzBx4kSkp6dj3759uHr1Kr7//ntUV1cDAL755htMmTIF3bt3BwC89957AO6Nv48aNUrzGQcOHIiKigq9Me/YsQOLFi1CTk4Ozp07h8cee8zg51MqlTh16hS2bt0KALjvvvsQGhqKwsJCDBo0CF27dsXgwYMN1jc0HFRYWIiFCxeiW7duAICIiAhER0frlCkqKsKECRM0vYNp06YhNzfX4HtRx8KJYbKo5mPb2stNQzvahBBoaGhoUXb27NkoKCjAqFGjcPToUUyePBl1dXUG31etVkMikbRq29oaGxshhMCOHTuwd+9e7N27F/n5+ViwYAF+/PFHzJo1CwqFAqNGjcLLL7+sqde1q+7xU3l5OW7evNlinUQigdBze66ysjKcPHkSkydPxrRp05Cfn2/ws2nH2fxvTZ/PxcWlRUyt0djYqLPfGhsboVKpWpTTfm+pVGr2+5DjYhIgmxk9ejQOHjyoOTLevXu3wTHm2bNnQy6XIzQ0FKmpqaiqqsKtW7cMbtvHxwddu3bFF198AeBeI/v555/jqaeeMhpTz549MXjwYHzwwQcAgKqqKoSFheHw4cM4ceIEHn/8cbz00ksYPnw4Dh8+DLVaDQAICAjA/v37UV9fj8bGRqSkpODAgQOt3hc7d+5EUFAQevTogenTp+PgwYNQKpVG4xw0aBDy8vIAAHfv3sX//M//mPx8powePRrbt2+HSqVCY2Mj8vLyMGrUKJ0yY8aMwaFDh1BVVYXGxkbs3bu3Xe9JjoXDQWS2uXPnoksX3eOHN954QzM0YsioUaMwb948zJ07F42NjXB3d8fGjRtbbAsA3nzzTaSlpeG9996DRCLBa6+9Bi8vL4Pb7tatG7Kzs7Fq1SpkZWVBrVYjOjoaI0eOxPHjx43GtXbtWqSmpiI4OBj19fWYNGkSJk+ejPLycnzxxRd4/vnn0djYiHHjxuHXX3+FQqHA7NmzUVJSgtDQUAghMHz4cERERGDDhg1G3wu412v55JNPkJ2dDQDw9PTEoEGDDE4Qa8e5cuVK7NmzB/X19QgODkZoaChKSkpMvqchUVFRWLNmDaZOnYqGhgY88cQTSEpK0ikzduxYnD9/HtOmTUOvXr0wcOBAVFZWtvk9ybFIhL6+KhEROQUOBxEROTEmASIiJ8YkQETkxJgEiIicGJMAEZETs/gpoiqVCgkJCSgpKUF9fT2ioqLg6emJ1NRUSKVSuLi4YM2aNfDw8NCps2TJEpSUlKBLly5ITU1Fv379Wmy7uLjY0uESETkF7SvydVj6PhS7du0Sq1atEkIIUVFRIcaOHSvmzJkjzp07J4QQYvv27SItLU2nzpdffiliY2OFEEIcPXpUvPbaa3q3ffLkyXbF1hSDo2Fc5mFc5mFc5umMcRlrOy3eEwgKCtK526NUKtW5i6JarYarq6tOHW9vb6jVajQ2NkKhULTp8nciIjKf1S4WUygUiIqKwsyZMxEcHAwAOHXqFJYtW4a8vDydG5CVlpbi1VdfhVKpRGVlJXJycjB06NAW2ywuLjZ4D5jWqK2tNXlVqz0wLvMwLvMwLvN0xriUSqXthoOEEOLGjRsiJCREfPLJJ5q/HThwQEyaNEn885//bFE+LS1NrF27VlN3woQJora2tkU5DgfZFuMyD+MyD+MyT4cZDiovL0dkZCSSk5MREBAAAJo7M+bm5kImk7Wo06tXL82tbHv37o2GhgbNjbqIiMh6LJ4EcnJyUFVVhezsbGRnZ0OtVuPChQvw9PRETEwMAGDYsGGIjY1FfHw84uLiMG/ePCQkJCA8PBwqlQoLFy5s17APERG1jsWTQGJiIhITE1tVNj09XfO66YEeRESka9bGIiiVSuzz87P4tnmxGBGRE+O5mEREDmrWxiIAwPErFTrL+QsCLPYe7AkQETkx9gSIiBxU0xF/05yAJXsATdgTICJyQLM2FmmGf6yJPQEiIgeXvyAAcrncKttmEiAiciCGJoNTxsis8n4cDiIicmLsCRAROQjtU0Cbnw5qreEg9gSIiJwYewJERHZmi4vCDGFPgIjIibEnQERkZ9oXhWkv2wJ7AkREdmSri8IMYU+AiMhB2LIH0IRJgIjIDuw5GayNw0FERE6MPQEiIjuw52SwNvYEiIhszN6TwdrYEyAisiN79QCaMAkQEdmIo0wGa7N4ElCpVEhISEBJSQnq6+sRFRUFT09PpKamQiqVwsXFBWvWrIGHh4dOvY0bN+LIkSNQqVQICwvDjBkzLB0aERE1Y/EkUFBQAJlMhoyMDFRWViIkJAReXl5ISkqCn58fduzYgc2bN2Pp0qWaOsePH8d3332H7du3o6amBlu3brV0WEREdmXsDqH2ZPEkEBQUhMDAQM2yVCpFZmYm+vTpAwBQq9VwdXXVqXP06FH4+voiOjoaCoUC8fHxlg6LiIj0kAghhDU2rFAoEBUVhZkzZyI4OBgAcOrUKSxbtgx5eXlwd3fXlE1MTMSNGzeQk5OD69evIyoqCocOHYJEItHZZnFxMdzc3NocU21tLbp3797m+tbCuMzDuMzDuMxj6bjiD90AAJwpqwUA/P7Be9tOD/K0WVxKpRL+/v5611llYri0tBTR0dEIDw/XJICDBw9iw4YN2LRpk04CAACZTAYfHx+4uLjAx8cHrq6uqKiowP33399i235+fm2OSy6Xt6u+tTAu8zAu8zAu81g6LrfCO/96dS8JNB3Imvse7YmruLjY4DqLJ4Hy8nJERkYiOTkZAQH3xrv27t2L/Px85ObmQiaTtajj7++Pjz76CC+99BJu3ryJmpoaveWIiDoaR7kozBCLJ4GcnBxUVVUhOzsb2dnZUKvVuHDhAjw9PRETEwMAGDZsGGJjYxEfH4+4uDiMGzcOJ06cwPTp0yGEQHJyMqRSqaVDIyKyGUdt9JuzeBJITExEYmJiq8qmp6drXnMymIg6M0dNBrxYjIjIghzxgjBjeO8gIiInxp4AEZGFOOoFYcawJ0BE5MTYEyAiaqeONg+gjT0BIiInxp4AEVE7dMR5AG3sCRAROTH2BIiI2qAjzwNoY0+AiMiJsSdARNQGjn5juNZiT4CIyEyzNhZpGv+Ojj0BIqJ26Kg9gCZMAkRErdRZJoO1cTiIiMiJsSdARNQKHf2iMEPYEyAicmLsCRARGdEZ5wG0sSdAROTE2BMgIjKgs84DaGNPgIjIiVm8J6BSqZCQkICSkhLU19cjKioKnp6eSE1NhVQqhYuLC9asWQMPD48WdW/fvo3Q0FBs3boV/fr1s3RoRESt0tnnAbRZPAkUFBRAJpMhIyMDlZWVCAkJgZeXF5KSkuDn54cdO3Zg8+bNWLp0qU49lUqF5ORkdO/e3dIhERGRARZPAkFBQQgMDNQsS6VSZGZmok+fPgAAtVoNV1fXFvXWrFmD2bNnY9OmTZYOiYjILJ3l5nCtYfEk0KNHDwCAQqFAbGws4uLiNAng1KlT2LZtG/Ly8nTq7NmzB+7u7hg9erTJJCCXy9scW21tbbvqWwvjMg/jMg/jMs+bB6+jy6EbSA/yhFKpBNC+dsdSrLa/hBXcuHFDhISEiE8++UTztwMHDohJkyaJf/7zny3Kh4eHizlz5ogXXnhB+Pv7i2nTpombN2+2KHfy5Ml2xXXu3Ll21bcWxmUexmUexmWeSZn/K2bmfGPvMFpoz/4y1nZavCdQXl6OyMhIJCcnIyDgXhdq7969yM/PR25uLmQyWYs62j2DiIgIpKSk4IEHHrB0aEREBjUN/ZwpqwVQ6xRDQYAVhoNycnJQVVWF7OxsZGdnQ61W48KFC/D09ERMTAwAYNiwYYiNjUV8fDzi4uLg6elp6TCIiKgVLJ4EEhMTkZiY2Kqy6enpLf6Wm5tr6ZCIiIzSPuoPXncYbm5unb4H0IQXixEROTGTPYFvv/0WI0eOtEUsREQ25UwXhRlisieQlZVliziIiMgOTPYEJBIJoqOj4e3tjS5d7uWMN954w+qBERFZk6GbwznCNQG2ZDIJTJs2zRZxEBGRHZhMAsHBwcjPz8fFixfx6KOPIiwszBZxERFZBecBdJmcE0hOTsa1a9cwatQolJSUtPr0TyIicnwmewJXr17VXNH77LPPYvbs2VYPiojIGpzhITHmMtkTqKurQ01NDYB7NzBSq9VWD4qIiGzDZE9g7ty5mDJlCvr374+LFy8iNjbWFnEREVkM5wEMM5kEHnjgAezcuRPXrl2Dl5cXfvOb39giLiIisgGTSSArKwt5eXl67/5JROToOA9gHC8WIyJyYiaTwNSpUyGVSm0RCxGRxXAeoHVMJoGDBw9i69attoiFiIhszGQSuO+++3D48GE8+uijmuEgb29vqwdGRNRWnAdoPZNJoKKiAn/96181yxKJBB999JE1YyIiIhsxmQSaP+mrrq7OasEQEbUH5wHMZ/CK4bi4OM1r7TmBP/3pT1YNiIiIbMdgErh9+7bm9VdffaV5LYSwakBERG2hfdQ/wtsdI7zdkb8ggL0AE1r1jGHthl8ikVgtGCIisi2DcwLajb05Db9KpUJCQgJKSkpQX1+PqKgoeHp6IjU1FVKpFC4uLlizZg08PDyM1hk/fnwbPxIRORPOA7SPwSRw8eJFLFq0CEIIndeXLl0yusGCggLIZDJkZGSgsrISISEh8PLyQlJSEvz8/LBjxw5s3rwZS5cuNVqHSYCIjGlq7Kl9DCaB9957T/Na+xkCpp4nEBQUhMDAQM2yVCpFZmYm+vTpAwBQq9VwdXU1WYeIqDWajvjZA2gbibDSTK9CoUBUVBRmzpyJ4OBgAMCpU6ewbNky5OXlwd3dvVV1tBUXF8PNza3NMdXW1qJ79+5trm8tjMs8jMs8nS2u+EM3AABnymoBAL9/UHcb6UGedonL2toTl1KphL+/v951Jq8TaIvS0lJER0cjPDxc05gfPHgQGzZswKZNm/QmAH119PHz82tzXHK5vF31rYVxmYdxmaezxeVWeOdfr+4lgaYDQ0v1ADrb/gLuHUAbYvEkUF5ejsjISCQnJyMg4N6XsnfvXuTn5yM3N1fvLan11SEiao63g7A8k0mgrKxMM2EbGBiIAQMGYNCgQQbL5+TkoKqqCtnZ2cjOzoZarcaFCxfg6emJmJgYAMCwYcMQGxuL+Ph4xMXFYevWrTp1AGDz5s0O2SUjIupMTCaBpKQkvPTSS8jOzsaTTz6JJUuWYOfOnQbLJyYmIjExsVVvnp6ebnYdInI+PA3Uelr1oPmAgABIJBL4+Pi0OLOHiIg6LpM9ARcXF3z99ddobGzE6dOn4eLiYou4iIgAcB7A2kz2BFJTU7Fnzx5UVlZi69atSElJsUFYRERkCyZ7Ah988AHWrVtni1iIiDQ4D2AbJnsCly5dQlVVlS1iISIiGzPZE7h06RJGjBgBd3d3zY3kjh49avXAiMh5cR7Adkwmgb/97W+2iIOIiOzAZBI4ffo09uzZA5VKBQC4efMmtmzZYvXAiMj5cB7A9kzOCaxatQrDhw+HQqGAp6en3ts+EBG116yNRThXyvlHWzOZBHr16oVJkyahZ8+eiImJQVlZmS3iIiIn9NhDvfh4SBszORwkkUhw4cIF1NTU4PLly7h165Yt4iIiJ6FvCOhcaRUee6iXPcNyGiZ7AkuWLMGFCxcQERGBN998E2FhYbaIi4icWFOPgKzPZE+gf//+6N+/PwBgz549Vg+IiJwHTwW1P5NJ4A9/+IPm9Z07d9C3b1989tlnVg2KiIhsw2QS0L4wrKSkBH/5y1+sGhARdX6zNhZBqVRqHhHJXoD9mJwT0Pbwww/j8uXL1oqFiJwATwV1LCZ7Am+88YbmdhE3b97E/fffb/WgiKhze+yhXkgZI0PKv54XzB6A/ZhMArNnz9a8dnV1xeOPP27VgIioc2p+Kmi8Uomff23gqaB2ZjIJ3LhxQ2f5ypUrmtdTp061eEBE5Dx4Kqj9mUwCR44cgVKpxPDhw3H27FmUl5fD399fM0RERGSKvlNBU8bI4OfnZ8+wCK1IAgqFAlu2bNE0+pGRkVi0aJHB8iqVCgkJCSgpKUF9fT2ioqLg6emJ1NRUSKVSuLi4YM2aNfDw8NDUaWxsREpKCs6fPw8XFxesWrUKjzzyiAU+HhERGWMyCVRUVODu3bvo1asXKioqoFAojJYvKCiATCZDRkYGKisrERISAi8vLyQlJcHPzw87duzA5s2bsXTpUk2d//3f/0V9fT3y8/Nx+vRprF69Ghs2bGj/pyMiuzJ2V1C5XG63uOjfTCaBV155BVOmTIFMJoNSqcSKFSuMlg8KCkJgYKBmWSqVIjMzE3369AEAqNVquLq66tQpLi7G6NGjAQCDBw/G2bNnzf4gRORYeA+gjsFkEggKCsKzzz6LX3/9tVWnh/bo0QPAvWGk2NhYxMXFaRLAqVOnsG3bNuTl5enUUSgU6Nmzp2ZZKpWioaEBXbu2DK89Rw+1tbUOefTBuMzDuMxjr7iUSiUe7d0VKWNkiFcqAdybBwDu/Tvm/jKPteIymASuXbuG1atXY/369Th9+jRef/11uLm5ISMjA4MHDza60dLSUkRHRyM8PBzBwcEAgIMHD2LDhg3YtGkT3N3ddcr37NkT1dXVmuXGxka9CQBAuyaS5HK5Q05EMS7zMC7z2DqupiGfpquBUwrvaE4F1Y6D+8s87YmruLjY4DqDVwynpaUhNDQUXbt2xTvvvIP09HRs27YN7777rtE3Ky8vR2RkJN566y1Mnz4dALB3715s27YNubm56Nu3b4s6Q4cORWFhIYB7TzLz9fVt1QcjIsdi6GpgngrquAz2BOrr6zF+/HhUVlbil19+wahRowDcO0o3JicnB1VVVcjOzkZ2djbUajUuXLgAT09PxMTEAACGDRuG2NhYxMfHIy4uDhMmTMCxY8cwe/ZsCCGQlpZmwY9IRLbU1ODzfkAdg8k5gaKiIowcORLAvQRw9+5do+UTExORmJjYqjdPT0/XvF65cmWr6hCR4+GDYToug0mgf//+WLRoEc6ePYvU1FTcvHkTmZmZmoRARGQMh4A6BoNJYPHixSgsLMSCBQvg6+uL8+fPY+DAgYiIiLBlfETk4PhgmI7NYBKQSCQYO3asZnnAgAEYMGCATYIiIiLbMDknQESkj7GrganjMOuhMkREAB8M05m06gZyhYWFqK+v1/yNt5AmIp4K2jmYTAKvvvoq+vTpg4ceeggAeAtpIifGU0E7H5NJQAiBtWvX2iIWInJghhp8ngrasZlMAgMGDMD333+vc88KFxcXqwZFRI6JQ0Cdj8kk8H//9384cuSIZlkikeDw4cNWDYqIHAeHgDo3k0mgoKDAFnEQkQPiEFDnZzIJHD58GB9//DFUKhWEELhz5w727dtni9iIyAFwCKhzM5kE3n//fSQlJWHHjh0YMWIEjh07Zou4iMiOOATkPExeLPab3/wGQ4YMAQCEhoairKzM6kERkX3M2likSQDNcQioczLZE+jWrRtOnDiBhoYGfP3117h165Yt4iIiO2pq7DkE1PmZTAIrVqzA5cuXERUVhfXr1yM2NtYWcRGRDRm6DxB1fiaTwIMPPojLly/j1KlTiI6Ohre3ty3iIiIbMTbezx5A52cyCWRmZuKXX37BpUuX0K1bN2zatAmZmZm2iI2IbIRnADkvkxPDxcXFSE9Ph5ubG0JCQnD9+nVbxEVEVtY0CXz8SgWOX6ngnUGdlMkkoFarUVdXB4lEArVajS5dePdpoo7OUIPPM4Ccj8nhoLlz5yI0NBQVFRWYMWMG5s2bZ4OwiMjaOAREQCuSwPPPP4+nnnoKV69ehZeXF9zd3Y2WV6lUSEhIQElJCerr6xEVFYXx48cDANLS0uDt7Y2wsLAWdZYsWYKSkhJ06dIFqamp6NevXzs+FhHpw4vAqDmDSWDp0qUGK73zzjsG1xUUFEAmkyEjIwOVlZUICQnBkCFDEB8fj59//hnz589vUefvf/87GhoasGPHDhw7dgzvvfcesrKyzPwoRGQM7wNE+hhMAmfPnkVtbS0mT56MIUOGQAjRqg0GBQUhMDBQsyyVSlFdXY2YmBgUFhbqrePt7Q21Wo3GxkYoFAp07cpHHxNZA4eAqDmJMNK6//TTTygoKMAPP/yAYcOGYfLkyXjkkUdatWGFQoGoqCjMnDkTwcHBAICsrCx4eHi0GA4qLS3Fq6++CqVSicrKSuTk5GDo0KEttllcXAw3NzdzPp+O2tpadO/evc31rYVxmYdxtV78oRu4VFGHfu6uOFNWCwD4/YPdcbmiDj7urkgP8rRbbI64v4DOGZdSqYS/v7/edUYPuX19ffHmm28CAE6cOIF3330Xv/zyC3bu3Gn0DUtLSxEdHY3w8HBNAjDmr3/9K/7whz9g0aJFKC0txdy5c7Fv3z64urq2KKv9cBtzyeXydtW3FsZlHsbVem6FdyCpqPvXwdO9JODm5obH3dzs3gtwxP0FdM64iouLDa5r1YPmv/zyS+zfvx81NTWYPHmy0fLl5eWIjIxEcnIyAgJa9yPr1asXunXrBgDo3bs3GhoaoFarW1WXiFpqPgEMAPd178rxf2rBYBL47LPPcODAAdy4cQPPPfccVqxYAS8vL5MbzMnJQVVVFbKzs5GdnQ0A2Lx5s95uTHx8POLi4jBv3jwkJCQgPDwcKpUKCxcubNewD5Ez4xk/ZA6DSWDhwoXw8fHBwIED8dNPP2HdunWade+++67BDSYmJiIxMVHvupiYGJ3l9PR0zev169e3OmgiMk57AlipVPLonwwymAQ++ugjW8ZBRO1k6BqAR3vzbDsyzOCvY/jw4baMg4jawdg1ACljZPYJijoEHiIQdRKGrgGQy+X2DIscHJMAUQelffTP20BQW/GWoESdDE8DJXOwJ0DUwfAaALIk9gSIOhA++IUsjT0Bog6GN4EjS2ISIHJwnAAma+JwEFEHxTkAsgT2BIgcFCeAyRbYEyByQJwAJlthT4DIQXECmGyBSYDIQXACmOyBw0FEDo5zAGRN7AkQ2ZG+o3+AE8BkO+wJEBE5MfYEiOyAp3+So2ASILKhpsafyFEwCRDZQdPRPk//JHtjEiCyMkOnfhI5AosnAZVKhYSEBJSUlKC+vh5RUVEYP348ACAtLQ3e3t4ICwtrUW/jxo04cuQIVCoVwsLCMGPGDEuHRuRw2AMge7N4EigoKIBMJkNGRgYqKysREhKCIUOGID4+Hj///DPmz5/fos7x48fx3XffYfv27aipqcHWrVstHRaRTfHUT+ooLJ4EgoKCEBgYqFmWSqWorq5GTEwMCgsL9dY5evQofH19ER0dDYVCgfj4eEuHRUREelg8CfTo0QMAoFAoEBsbi7i4OPTt2xd9+/Y1mAQqKytx48YN5OTk4Pr164iKisKhQ4cgkUhalJXL5W2Orba2tl31rYVxmcdR43rz4HX8/PEV+Li74kxZLQBAqVSiRzcJfNxdkTJGBkAGoH2/Y3M56v5iXOaxVlxWmRguLS1FdHQ0wsPDERwcbLK8TCaDj48PXFxc4OPjA1dXV1RUVOD+++9vUdbPz6/Nccnl8nbVtxbGZR5HjavLoRvoIpXCzc0NwL0k4Obmhi6/NsDNzc1uMTvq/mJc5mlPXMXFxQbXWTwJlJeXIzIyEsnJyQgIaN24p7+/Pz766CO89NJLuHnzJmpqaiCTySwdGpHFaY/9Nx39Axz7p47D4reNyMnJQVVVFbKzsxEREYGIiAjU1tbqLRsfH48bN25g3Lhx8PPzw/Tp0xEVFYXk5GRIpVJLh0ZERM1YvCeQmJiIxMREvetiYmJ0ltPT0zWvORlMHYWhM396dJPgca/f8OifOhTeQI6oFWZtLOIFXtQp8YpholY6V1qFWRuLDJ7376gTikTGOE0SmLWxCEqlEvv4j5RaSd+wD5/7S52N0yQBIkt47KFemsTAsX/qDDp9Emh+33betZGMac3tHjg3QJ1Jp08CRKaYe2DAAwjqTDp9EtC+b7tSqdQcyc3aWMR/zKRhatKXqLPq9EmASB9O+hLd4zRJIH9BAILXHdY52tNuCHi0R5z0JWfkNEmgNThp3Llx0peoJadKAulBnvDz89M0BgBwt7YBx69U6DQQ1Hlw0pfIOKdKAsacK63SmxDYKHQ8zRt+TvoSGeaUSUD7H76hIQJtHCbqeJo3/Jz0JdLPKZNAc9rjwRwm6niaj+Pra/g56Uukn9MnAVMNAoeJHFNremfNG35O+hK15PRJQBuHiRxb0/6+96zelkM+I7zddf7fvOHn90TUEpOAEeYMEzEhWIe+Sd74Q0rNoxxNjfXz+yAyjknAgLYOEwFMCO1l6uyey2q1pizH+onah08Wa4X8BQE4kxKI/AUBGOHtrjm9sIm+hNCET6Rqneb7Sbvhv1vboLNPfdxdcV/3rhjh7Y78BQGctCdqByaBNmg66mRCaB9zGv7HHuqlafjTgzx19nn+ggD2AojayOJJQKVS4a233kJ4eDimT5+Ow4cPa9alpaVh+/btBuvevn0bY8eOxaVLlywdlsU0b3AslRA6c4LQ/mxtbfibH/Gz4SeyDIsngYKCAshkMnz88cfYvHkzUlNTUVFRgZdffhlHjhwxWE+lUiE5ORndu3e3dEhWY6mE0FxHTQjGGnttbPiJHIfFJ4aDgoIQGBioWZZKpaiurkZMTAwKCwsN1luzZg1mz56NTZs2WTokm2jeODU/s6j5LYubEsLvUz5Ho1qNapUAoHtqatNy0/YdZcK5tTE1n9D9fcrnuFvboFnXxNj5/Pb+rESdncV7Aj169EDPnj2hUCgQGxuLuLg49O3bF4MGDTJYZ8+ePXB3d8fo0aMtHY5dmNNDaK61PQZDR92WfB1/6EaLvxuKuamxb0ps+o7ytfGIn8gxWOUU0dLSUkRHRyM8PBzBwcEmy+/evRsSiQRFRUWQy+VYvHgxNmzYgAceeKBFWblc3ua4amtr21W/LZoubJLL5VAqlXi0d1ekjJEhXqkEcO/OprW1tUj+qgKXK+rwaO+uOPOvo+Wz1ytRrRI4fqUCv0s6qOktBK87jMsVdfBxd9VsV/s9LPW6sbFR5+/B6+7N7zSdo68d09nrlZrP3Kh1CuejvbvisloNH3dXpAd5ahJLyhgZ4g8poVQqIZfLdfaTKfb4HluDcZmHcZnHWnFJhBDCkhssLy9HREQEkpOTERCgezSXlZUFDw8PhIWFGawfERGBlJQU9OvXr8W64uJi+Pv7tzk2uVwOPz+/Nte3JO1hFLlcjpTCO5plfUNI93XvqhlKsedrAHrXjfB21zukY+x1WznS96iNcZmHcZmnPXEZazst3hPIyclBVVUVsrOzkZ2dDQDYvHmz3gnf+Ph4xMXFwdPT09JhOLzmjaCxOYWm9a25lYU1NR/GMtTAN6f92TjMQ+RYLJ4EEhMTkZiYqHddTEyMznJ6enqLMrm5uZYOqcMxdVO01h5pt/e1UqnEvoXjWxy9s7En6jx42wgH54gNqyPGRERtwyTQQRlqiC35umkSig09UefF20YQETkxJgEiIifGJEBE5MSYBIiInBiTABGRE2MSICJyYha/bYQ1FRcX2zsEIqIOydBtIzpUEiAiIsvicBARkRNjEiAicmKd7rYRjY2NSElJwfnz5+Hi4oJVq1bhkUce0aw/cuQI3n//fXTt2hXTpk3DzJkzbRKXSqVCQkICSkpKUF9fj6ioKIwfP16z/oMPPsCuXbvg7u4OAFixYgV8fHxsEtvUqVNx3333AQC8vLzwzjvvaNbZa3/t2bMHn376KQCgrq4Ocrkcx44dQ69e9+5kao/99f3332Pt2rXIzc3F1atXsWTJEkgkEvTv3x/Lly9Hly7/PqYy9Tu0VlxyuRypqamQSqVwcXHBmjVr4OHhoVPe2Pdtrbh+/PFHvPLKK3j00UcBAGFhYZg4caKmrL3218KFC1FeXg4AKCkpwaBBg7Bu3Tqd8tbeX/rahv/8z/+03e9LdDKff/65WLx4sRBCiO+++0688sormnX19fXi2WefFXfu3BF1dXUiNDRU3Lx50yZx7dq1S6xatUoIIURFRYUYO3aszvpFixaJM2fO2CQWbbW1tWLKlCl619lzf2lLSUkRO3bs0PmbrffXpk2bxKRJk8SMGTOEEEIsWLBAfPvtt0IIIZKSksQXX3yhU97Y79Cacc2ZM0ecO3dOCCHE9u3bRVpamk55Y9+3NePauXOn2LJli8Hy9tpfTe7cuSMmT54sysrKdP5ui/2lr22w5e+r0w0HFRcXax5TOXjwYJw9e1az7tKlS/jtb3+L3r17w8XFBf7+/jh58qRN4goKCsLrr7+uWZZKpTrrf/zxR2zatAlhYWHYuHGjTWICgH/84x+oqalBZGQkXnzxRZw+fVqzzp77q8mZM2dw8eJFzJo1S+fvtt5fv/3tb5GVlaXz/sOHDwcAjBkzBt98841OeWO/Q2vGlZmZqXnwiFqthqurq055Y9+3NeM6e/YsvvrqK8yZMwcJCQlQKBQ65e21v5pkZWXhhRdeQJ8+fXT+bov9pa9tsOXvq9MlAYVCgZ49e2qWpVIpGhoaNOuaunXAvechN/8xWou+Zy9r++Mf/4iUlBR8+OGHKC4uxt/+9jebxNW9e3fMnz8fW7ZswYoVK/Dmm286xP5qsnHjRkRHR7f4u633V2BgILp2/ffoqRACEokEwL39cvfuXZ3yxn6H1oyrqRE7deoUtm3bhnnz5umUN/Z9WzOuJ554AvHx8cjLy0Pfvn3x/vvv65S31/4CgNu3b6OoqAihoaEtyttif+lrG2z5++p0SaBnz56orq7WLDc2Nmq+9ObrqqurdRo5aystLcWLL76IKVOm6Dx7WQiBuXPnwt3dHS4uLhg7dizOnTtnk5i8vb0xefJkSCQSeHt7QyaT4datWwDsv7+qqqpw+fJljBw5Uufv9txfTbTHZ6urqzVzFU2M/Q6t7eDBg1i+fDk2bdqkmTNpYuz7tqYJEybg8ccf17xu/n3Zc38dOnQIkyZNatE7B2y3v5q3Dbb8fXW6JDB06FAUFhYCAE6fPg1fX1/Nun79+uHq1au4c+cO6uvrcfLkSQwZMsQmcZWXlyMyMhJvvfUWpk+frrNOoVBg0qRJqK6uhhACx48f1/yDsbZdu3Zh9erVAICysjIoFAo88MADAOy7vwDgxIkTeOqpp1r83Z77q8ljjz2G48ePAwAKCwvx5JNP6qw39ju0pr1792Lbtm3Izc1F3759W6w39n1b0/z58/HDDz8AAIqKivC73/1OZ7299ldTPGPGjNG7zhb7S1/bYMvfV6c7O2jChAk4duwYZs+eDSEE0tLSsG/fPiiVSsyaNQtLlizB/PnzIYTAtGnT8OCDD9okLn3PXp4xYwZqamowa9YsLFy4EC+++CJcXFwQEBCAsWPH2iSu6dOnY+nSpQgLC4NEIkFaWho+++wzu+8vALhy5Qq8vLw0y9rfo732V5PFixcjKSkJmZmZ8PHxQWBgIIB/Pzdb3+/Q2tRqNd5++2089NBDmke5Dhs2DLGxsZq49H3ftjjiTklJQWpqKrp16wYPDw+kpqYCsO/+anLlypUWCdOW+0tf27Bs2TKsWrXKJr8vXjFMROTEOt1wEBERtR6TABGRE2MSICJyYkwCREROjEmAiMiJMQkQETkxJgEiIifGJEDUTufPn8e4cePw8ccf2zsUIrMxCRC104ABA5CZmYm9e/faOxQiszEJEFnA/fffjwsXLtg7DCKzMQkQWcDatWtRX1+PkpISe4dCZBYmAaJ2KiwsRE1NDZ5++mlcvHjR3uEQmYVJgKgd6urqkJGRgeXLl8PX1xc//fSTvUMiMguTAFE7ZGdnY+rUqfDy8oKvry/nBajDYRIgaqPLly/jm2++wdy5cwGASYA6JD5PgIjIibEnQETkxJgEiIicGJMAEZETYxIgInJiTAJERE6MSYCIyIkxCRAROTEmASIiJ/b/NNjlsJhR3cMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAESCAYAAAAbq2nJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAqZElEQVR4nO3df1xUdb4/8Nc4CIRIRP64bLiKrvjjummamnnJbYt+mCKhXuC6aOmalWmIXsC84CiIgoa5bJLslbaIXTGha1pp5Y/FVTLDS2WOmqJ+ExMRROTnAPP5/uHlxMgMw8CcMzDzej4ePZyZc2bmxYHO+3x+nHNUQggBIiJySD1sHYCIiGyHRYCIyIGxCBAROTAWASIiB8YiQETkwFgEiIgcmJOtA1D3MmzYMPj5+aFHD8Pjh7fffhs+Pj6K51m9ejWOHDmC6dOnY9myZYp//91SU1Nx8+ZNxMXFmV138+bNuHXrFjQajcl10tLSkJ2djUmTJmH9+vUdzpSVlYX+/fsbvJ6QkIDf/va3Jt/3+9//Hlu2bGm1zr59+5CVlYXMzMxW79m9eze2b98OlUqFe+65B6tWrWrzO8j2WATIYu+99x68vLxsHQMAkJ2djcOHD+Nf/uVfbB3FYrNnz8bzzz+PqKgouLm5GV1n165d2LRpEx5++OFOfdfUqVPbVZg6o6ioCBs3bkRubi769euHf/zjH1iyZAkOHz4s6/dS57AIkNUcP34c69atg5ubG6qrqxEVFYWNGzdKz3NycvDRRx8hMzMTPXr0QJ8+fRAbGwtfX1/ExMSgoqICP/30E373u9/h8ccfx4YNG6DX6wEAixYtwtNPP23wff/xH/8BIQQWLlyI1atX495778XatWtRUVEBlUqF+fPnIygoqFWunJwcODs7S59TUlKCtWvX4ueff0ZDQwOee+45vPzyywCAd955BwcOHEBdXR1qa2sRHR2NgIAANDY2YuPGjTh8+DDUajUeeughrF69GsCdnWF4eDhKS0vRp08fpKSkoF+/fq221wMPPAAXFxd88sknmD17dqvlERERKCkpwapVq/D6669j7Nix0Gg0KC4uhhACQUFB+OMf/4grV65gzpw5GDJkCIqLi5GZmWn0+4xpaGjAhg0bkJ+fD7VajQcffBArV66Eu7u7wXpbtmzBnj174OnpiYEDBxr9LGdnZyQkJEjfPWrUKNy4cQM6nc5ge1MXI4gs4OfnJ6ZNmyYCAwOl/1599VUhhBBfffWVGD58uLhy5YrR58eOHRNPPvmkKCsrE0IIkZOTI5599lmh1+tFdHS0mDdvnvQ9c+fOFXv37hVCCKHVaoVGozGZp6ysTDQ0NIgnnnhC7N+/XwghxLVr14S/v784efJkqxx3Cw8PFwcOHBBCCFFXVyfCw8PFJ598Iq5cuSLCw8NFbW2tEEKIvXv3imnTpgkhhHjvvffEnDlzRG1trWhqahKvv/66+Oijj8Sf/vQn8fvf/176GV955RXx5z//2ej3/uMf/xDjxo0TM2fONLm9H3/8cfHdd98JIYSYM2eOyMjIEEIIUVlZKaZPny727t0rfvrpJ+Hn5ydOnDhh9DP+9Kc/iYkTJxr8zlJTU4UQQmzZskW89tprQqfTiaamJhETEyNiY2MNvvuLL74QU6dOFbdv3xYNDQ3ipZdeEn/4wx9MZhZCCL1eL5YvXy6WLFnS5npke2wJkMXa6g7y9vbGAw88YPT5kSNHMHXqVOm9wcHBWLduHa5cuQIAGDdunPS+Z599FmvXrsXBgwfx6KOPIjIyss1Mly5dQn19PZ566ikAQP/+/fHUU0/hyJEjmDhxYqtczWpqanDixAncunULW7ZskV47c+YMpk6diuTkZOzZsweXL1/Gt99+i+rqagDAsWPHMGPGDLi6ugIA3nrrLQB3+t8nT54s/YzDhw9HeXm50cw7duzA8uXL8c477+D06dMYOXKkyZ+vpqYGJ0+eREZGBgCgd+/eCA4ORl5eHkaPHg0nJyeMGTPG5PtNdQfl5eVh2bJl6NmzJwAgPDwcixcvNlgnPz8fAQEBUutg5syZRscDWmaNiYnBtWvX8N///d8m16OugbODyKru7ttu+by5a6clIQQaGxtbrRsaGoqPP/4YkydPxj//+U8EBgaivr7e5Pc2NTVBpVK167Nb0uv1EEJgx44d2L17N3bv3o3s7GwsWrQIP/zwA0JCQlBVVYXJkyfjj3/8o/Q+JyfD46cbN27g+vXrrZapVCoII5fnKikpwTfffIPAwEDMnDkT2dnZJn+2ljnvfq3553N2dm6VqT30er3BdtPr9WhoaGi1XsvvVqvVJj/v6tWrCA0NhVqtxvvvvw8PDw+LM5GyWARIMf7+/vj000+lI+OcnByTfcyhoaHQarUIDg5GfHw8KisrUVpaavKzBw8eDCcnJ3z++ecA7uxk9+/fj0cffbTNTO7u7hgzZgzeffddAEBlZSXCwsJw4MABnDhxAqNGjcKLL76ICRMm4MCBA2hqagIATJo0CXv37oVOp4Ner4dGo8Enn3zS7m2xc+dOPPPMM+jVqxdmzZqFTz/9FDU1NW3mHD16NLKysgAAt2/fxv/8z/+Y/fnM8ff3x9///nc0NDRAr9cjKysLkydPNljnsccew759+1BZWQm9Xo/du3cb/ayqqiqEh4fjqaeewubNm6VWEnVt7A4ii82bN6/VFNHIyEiz/9NPnjwZL7zwAubNmwe9Xg8vLy9s27at1WcBwIoVK5CYmIi33noLKpUKr732WptTUHv27ImtW7ciISEBqampaGpqwuLFi/HII4/g+PHjbebatGkT4uPjMX36dOh0OkybNg2BgYG4ceMGPv/8czz77LPQ6/V4/PHHcevWLVRVVSE0NBTFxcUIDg6GEAITJkxAeHg40tLS2vwu4E6r5cMPP8TWrVsBAL/61a8wevRokwPELXOuXbsWubm50Ol0mD59OoKDg1FcXGz2O0155ZVXkJSUhKCgIDQ2NuLBBx9EbGyswTpTpkzB2bNnMXPmTHh4eGD48OG4efNmq8/KysrC1atX8cUXX+CLL76QXv/rX/+K++67r8MZSV4qYaytSkREDoHdQUREDoxFgIjIgbEIEBE5MBYBIiIHxiJAROTAutUU0YKCAltHICLqllqekd9StyoCgOkfpD20Wi1GjBhhxTTWwVyWYS7LMJdl7DFXWwfQ7A4iInJgLAJERA6MRYCIyIGxCBAROTAWASIiB9btZgcRETmakG35qKmpwR4ZZi2xJUBE5MDYEiAi6qJCtuUDAI5fLDd4nr1oktW+gy0BIqIuKGRbPk7/XCn797AlQETURY309kD2oknSmIA1WwDNWASIiLoQY11Ap3+uxKB75dldszuIiKiLMNUFNNLbA8nP/EqW72RLgIioC2nZBQT8Mgis1Wpl+T7ZWgJlZWWYMmUKLly4IL22Z88ehISEtFpXr9cjLi4OISEhCA8Px+XLl+WKRUTU5YRsy0fItnwcv1iO4xfLFRsUBmQqAg0NDYiLi4Orq6v0mlarxa5duyCEaLX+l19+CZ1Oh+zsbCxfvhwbNmyQIxYRUZfTVheQHAPBd5OlCCQlJSE0NBT9+vUDANy8eRObNm3CG2+8YXT9goIC+Pv7AwDGjBmDU6dOyRGLiKhLat7hT/T1wkRfL2QvmqRIAQBkGBPIzc2Fl5cX/P39kZ6eDr1ej1WrVuGNN96Ai4uL0fdUVVXB3d1deq5Wq9HY2Agnp9bxOtMvVldXJ1u/Wmcwl2WYyzLMZRmlckXtuwoA+L6kDgAwffMBFJXXY7CXi9HvlyuX1YtATk4OVCoV8vPzodVqMX36dPj4+ECj0aC+vh7nz5/HunXrsGrVKuk97u7uqK6ulp7r9XqjBQBAp+74Y493DJITc1mGuSzjyLlCtuXj0q1GjPT2AHCnCLi5uWGUm5vJFoBcdxazehHIysqSHoeHh0Oj0WDIkCEAgCtXriAyMtKgAADA2LFjcejQIUydOhWFhYXw8/Ozdiwioi7F1Cwgpdl0imhUVBQiIiIQEBCAo0ePIjQ0FEIIJCYm2jIWEZEsTJ0IdqdFYBuyFoHMzEyD5z4+Pti5c6f0PDk5WXq8du1aOaMQEdmUqR2+UrOATOHJYkRECukqXUAtsQgQEcmgeUffrCt1AbXEawcRESnM1l1ALbElQERkRXcP/k709TL4t6vs/JuxJUBEZCVKXvPHWtgSICKyoq44+NsWFgEiok7qivP/24vdQUREnWDrq4B2FlsCRESd1N26gFpiESAi6oDu3AXUEruDiIgs1N27gFpiS4CIqAO6cxdQSywCRETtZC9dQC2xO4iIqB3sqQuoJbYEiIjayV66gFpiESAiaoM9dgG1xO4gIiIT7LULqCW2BIiI2mCPXUAtsQgQEbUQte8qLmX/P4z09rDbLqCW2B1ERNQO9tQF1BJbAkRE+GUA+PuSOum13q5Odrvzb8aWABE5vO54MxhrYUuAiAi/dPdM33wAbm5udn303xKLABE5LGPnABSV12OUm5stYymK3UFE5JBMdQEN9nJxmFYAwJYAETkwY+cAaLVaG6dSFosAETmMlvP9HeEcgPZgdxAROTx7nwbaFtlaAmVlZQgODkZGRgaEEIiNjYUQAsOHD0dsbCzUarXB+kFBQejduzcAwMfHB+vXr5crGhE5mLsHgAHHOAegPWQpAg0NDYiLi4OrqysAICUlBZGRkRg/fjxiYmJw8OBBBAQESOvX19cDADIzM+WIQ0QOzNG7e8yRpQgkJSUhNDQU6enpAIDU1FSo1WrodDqUlpbi/vvvN1j/zJkzqK2txfz589HY2IjIyEiMGTNGjmhE5IDs/SJwnWH1IpCbmwsvLy/4+/tLRUCtVqO4uBgvvvgi3N3d4evra/AeV1dXLFiwALNnz8alS5ewcOFC7Nu3D05OreN1ZuS+rq6uS478M5dlmMsyjporat9VAL9cBmL65gMoKq/HYC+XNr/X0baXSgghrPmBc+bMgUqlgkqlglarxaBBg5CWloa+ffsCAD788EN88803SEpKkt6j0+mg1+ul7qNZs2YhNTUV3t7eBp9dUFCAcePGdTibVqvFiBEjOvx+uTCXZZjLMo6Yy9gsoIm+XgDMtwLscXu1te+0eksgKytLehweHg6NRoPY2FjExMRg0KBB6NWrF3r0MJyUtGvXLpw7dw4ajQYlJSWoqqqSigYRUUewC6h9FDlP4KWXXkJMTAx69uyJe+65BwkJCQCAqKgoREREYNasWVi5ciXCwsKgUqmQmJhotCuIiMgUngPQMbLuaVvO9tmxY0er5cnJydLjN998U84oROSgOA20bTzcJqJujecAdA6LABF1S807f+ocFgEi6taaj/Y5ANwxLAJE1K0YuwcAdRyLABF1G23N9mELoGNYBIioW+H8f+tiESCiLo3z/+XF+wkQUbfEKaDWYbYl8NVXX+GRRx5RIgsRkYTz/5VhtiWQmpqqRA4iIompm8CT9ZltCahUKixevBi+vr7Shd8iIyNlD0ZEjo0DwMowWwRmzpypRA4icnAcALYNs91B06dPR01NDb777jtUVlbiueeeUyIXEREADgDLzWxLIC4uDh4eHpg8eTK+/vpr/Nd//ZfB1T+JiDqDA8C2ZbYIXL58WbpRzJNPPonQ0FDZQxGRY2B3j+2ZLQL19fWora3FPffcg7q6OjQ1NSmRi4gcBAeAbctsEZg3bx5mzJiBoUOH4vz581i6dKkSuYjITjUf/Q+610m6CTxbBLZjtgj07dsXO3fuxE8//QQfHx/cd999SuQiIgfDMQDbMFsEUlNTkZWVBU9PTwXiEJG9ajUAfK8TB4C7AJ4sRkSyY3dP12W2CAQFBUGtViuRhYjs2N0DwJrHPDFixAgbpyKzReDTTz9FRkaGElmIyI7wDODuwewZw71798aBAwdw4cIFXLx4ERcvXlQiFxHZKY4BdC1mWwLl5eX461//Kj1XqVR4//335cxERN0YzwDuXswWgczMTIPn9fX1soUhou6N3T3dj8nuoIiICOlxyzGBhQsXyhqIiLq35iP+ib5emOjrhexFk/C95mm2Arooky2BsrIy6fHhw4cxf/58AIAQQv5URNRtcAC4e2vXPYZb7vhVKpVsYYjIfnAMoHsw2RJoubPvyI6/rKwMwcHByMjIgBACsbGxEEJg+PDhiI2NNTj3QK/XQ6PR4OzZs3B2dkZCQgIGDhxo8XcSkTKMHf0DHADujkwWgfPnz2P58uUQQhg8vnDhgtkPbWhoQFxcHFxdXQEAKSkpiIyMxPjx4xETE4ODBw8iICBAWv/LL7+ETqdDdnY2CgsLsWHDBqSlpVnhxyMioraYLAJvvfWW9LjlPQTacz+BpKQkhIaGIj09HcCd6w+p1WrodDqUlpbi/vvvN1i/oKAA/v7+AIAxY8bg1KlTFv0QRKQMTv+0PyaLwIQJEzr0gbm5ufDy8oK/v79UBNRqNYqLi/Hiiy/C3d0dvr6+Bu+pqqqCu7u79FytVqOxsRFOTq3jabXaDuUCgLq6uk69Xy7MZRnmsoy1ckXtu4qi8noM9nKRXqupqYG+qQk1NTUWf4e9by9rkyuX2fMELJWTkwOVSoX8/HxotVpER0cjLS0NDzzwAD7//HN8+OGH2LBhA5KSkqT3uLu7o7q6Wnqu1+uNFgAAnbrWiFar7ZLXKmEuyzCXZayVyy2vAqPc3Kx2Axh7317W1plcBQUFJpdZvQg034oSAMLDw6HRaBAbG4uYmBgMGjQIvXr1kq5G2mzs2LE4dOgQpk6disLCQvj5+Vk7FhF1AKd/2j+zRaCkpAQbN27EzZs38fTTT2PYsGEYPXq0RV/y0ksvISYmBj179sQ999yDhIQEAEBUVBQiIiIQEBCAo0ePIjQ0FEIIJCYmduynISJFcAzAfpgtArGxsXjxxRexdetWPPzww4iJicHOnTvb9eEtLzmxY8eOVsuTk5Olx2vXrm3XZxKRfJq7eZpxANj+mT1ZrL6+HpMmTYJKpcLgwYPh4uJi7i1ERNRNmG0JODs748iRI9Dr9SgsLISzs7MSuYhIQXdP/Zzo62XwL4/+7ZfZlkB8fDxyc3Nx8+ZNZGRkQKPRKBCLiJTSPNBLjslsS+Ddd9/F5s2blchCRDZy960feeTvOMwWgQsXLqCyshIeHpwORmQvOPWTmrWrCEycOBFeXl7SheT++c9/yh6MiJTH2T+Ox2wROHTokBI5iEhmvPInGWO2CBQWFiI3NxcNDQ0AgOvXr2P79u2yByMiIvmZLQIJCQl44YUXsH//fvj5+UGn0ymRi4isIGRbPk5duYlRPhU8+iejzE4R9fDwwLRp0+Du7o4lS5agpKREiVxERKQAsy0BlUqFH3/8EbW1tSgqKkJpaakSuYioE3jdf2ovsy2BmJgY/PjjjwgPD8eKFSsQFhamRC4i6iCe/EWWMNsSGDp0KIYOHQrgzg1jiKjra3nyV01NDY/+ySSzReDf/u3fpMcVFRUYMGAAPvvsM1lDEZFl2jr5a9C9Vr9tCNkRs38dLU8MKy4uxp///GdZAxGR9Yz09oDmMU9bx6AuzKJDhAceeABFRUVyZSEiC7T35K+ueL9c6jrMFoHIyEjpchHXr1/H/fffL3soIiJShtkiEBoaKj12cXHBqFGjZA1ERKbx0g9kbWaLwNWrVw2eX7x4UXocFBRk9UBERKQcs0Xg4MGDqKmpwYQJE3Dq1CncuHED48aNk7qIiEhePPonOZktAlVVVdi+fbu0058/fz6WL18uezAiIpKf2SJQXl6O27dvw8PDA+Xl5aiqqlIiF5FD49E/KcVsEXj55ZcxY8YMeHp6oqamBmvWrFEiFxERKcBsEXjmmWfw5JNP4tatW5weSiQjHv2TLZi8gNxPP/2ExYsXo7GxEYWFhQgMDERAQAAKCwsVjEdERHIy2RJITEzErFmz4OTkhPXr1yM5ORm/+c1vsGLFCmRmZiqZkchu8eifbM1kEdDpdHjiiSdw8+ZNXLt2DZMnTwYA6PV6xcIREZG8zI4J5Ofn45FHHgFwpwDcvn1b9lBE9qr5Zi/NePRPtmayCAwdOhTLly/HqVOnEB8fj+vXryMlJUUqCOaUlZUhODgYGRkZ0Ol0iI+Ph1qthrOzM5KSktCnTx+D9YOCgtC7d28AgI+PD9avX9+JH4uIiNrDZBGIjo5GXl4eFi1aBD8/P5w9exbDhw9HeHi42Q9taGhAXFwcXF1dAQDr1q1DbGwsRowYgR07duAvf/kLVq5cKa1fX18PABxrILtkrN9/oq+Xwb88+idbMVkEVCoVpkyZIj0fNmwYhg0b1q4PTUpKQmhoKNLT0wEAKSkp6NevHwCgqakJLi4uBuufOXMGtbW1mD9/PhobGxEZGYkxY8ZY+rMQEZGFVEIIYc0PzM3NxbVr1/Dqq68iPDwcGo0GQ4YMAQCcPHkSq1atQlZWFry8vKT3nD17Ft9++y1mz56NS5cuYeHChdi3bx+cnAxrVEFBAdzc3Dqcra6uTmqddCXMZZnukCtq31UUlddjsJcLvi+pAwD8tr+r9FryM7+ySa6uhLks05lcNTU1GDdunNFlVr/vXE5ODlQqFfLz86HVahEdHY20tDScOHECaWlpSE9PNygAAODr64uBAwdCpVLB19cXnp6eKC0thbe3d6vPHzFiRIezabXaTr1fLsxlme6Qyy2vAj1uNf7fQcudIuDm5ia9pmT+7rC9uhJ7zFVQUGByWbsuIJeXlwedTie91tYlpLOysqTHzS2BY8eOITs7G5mZmfD09Gz1nl27duHcuXPQaDQoKSlBVVUV+vbtay4aUZcSte8qLmX/P875p27FbBF49dVX0a9fP+mo3NJLSOv1eqxbtw7e3t5YsmQJAGD8+PFYunQpoqKiEBERgVmzZmHlypUICwuDSqVCYmJiq64gIiKyPrN7WiEENm3a1KEPb57t8/XXXxtdnpycLD1+8803O/QdRLbUcuZPc98/wKN/6j5MXjuo2bBhw/Dtt99Cp9NJ/xERkX0w2xL4+uuvcfDgQem5SqXCgQMHZA1F1JWZut5Pr54qjPK5j0f/1K2YLQIff/yxEjmIiMgGzBaBAwcO4G9/+xsaGhoghEBFRQX27NmjRDaiLqM9V/vsqlMLidpidkzg7bffxmuvvQZvb288//zz8PPzUyIXEREpwGxL4L777sNDDz2EHTt2IDg4GLm5uUrkIrI5XuufHIHZlkDPnj1x4sQJNDY24siRIygtLVUiF5FNhGzLb3W5ZyJ7ZrYlsGbNGhQVFeGVV17Bli1bsHTpUiVyEdnM6Z8rEbItn0f/5BDMFoH+/fujqKgIJ0+exOLFi+Hr66tELiLFGOv2Of1zpY1TESnDbBFISUnBtWvXcOHCBfTs2RPp6elISUlRIhuRzYz09pAKA4/+yZ6ZHRMoKChAcnIy3Nzc8Pzzz+PKlStK5CKSVci2fPxWs1/q9rld1wjgTrfPRF8vZC+ahJHeHjZOSSQ/sy2BpqYm1NfXQ6VSoampCT16mK0bRHaBLQByBGaLwLx58xAcHIzy8nLMnj0bL7zwggKxiKyPUz6JWjNbBJ599lk8+uijuHz5Mnx8fFrdEIaoK2ue7skdPJFxJotAyxvB3239+vWyhCGSA6d8EplmsgicOnUKdXV1CAwMxEMPPQQr34qYSDac8knUfiaLwJ49e3Du3Dl8/PHHSE9Px/jx4xEYGIiBAwcqmY/IKjjlk8i4NscE/Pz8sGLFCgDAiRMn8Oabb+LatWvYuXOnIuGI2uPuyzyY6vbh5SCIWmvXjea/+OIL7N27F7W1tQgMDFQiF1GbOjLgyxYAUWsmi8Bnn32GTz75BFevXsVTTz2FNWvWwMfHR8lsRG26e8B3oq+Xwb/c6ROZZ7IILFu2DIMHD8bw4cNx7tw5bN68WVrGm8KTLTQP+A6610m6qTsHfIk6x2QReP/995XMQdQhHPAl6hyTRWDChAlK5iAyyuhZvvc6ccCXyErMDgwTKc3SQV+2AIg6jkWAuqS2zvLVPObJG7oTWQmLAHUJPMuXyDZYBMhmzHX7mBr01Wq1iuQjcgS8OQDZVMtuH97YhUh5srUEysrKEBwcjIyMDOh0OsTHx0OtVsPZ2RlJSUno06ePtK5er4dGo8HZs2fh7OyMhIQEXqPITlna7cNBXyJ5yVIEGhoaEBcXB1dXVwDAunXrEBsbixEjRmDHjh34y1/+YnCp6i+//BI6nQ7Z2dkoLCzEhg0bkJaWJkc0soGOdvsQkfxk6Q5KSkpCaGgo+vXrB+DOzeqbZ3M0NTXBxcXFYP2CggL4+/sDAMaMGYNTp07JEYtsiN0+RF2T1VsCubm58PLygr+/P9LT0wFAKgYnT57EBx98gKysLIP3VFVVwd3dXXquVqvR2NgIJ6fW8TozKFhXV9clBxXtMVfUvqsoKq/HYC8X6RIPp67clJbX1NRA39SEmpoaaLVaaB7zBNC+3689bi85MZdlHC2X1YtATk4OVCoV8vPzodVqER0djbS0NJw4cQJpaWlIT09vdYtKd3d3VFdXS8/1er3RAgCgU/PDtVptl5xfbi+5Wnb7uOVVoMetRri5uQG4UwRG+dxnlW4fe9leSmEuy9hjroKCApPLrF4EWh7lh4eHQ6PR4NixY8jOzkZmZiY8PT1bvWfs2LE4dOgQpk6disLCQvj5+Vk7Fsnk7v5+c7dy5CUeiLoW2c8T0Ov1WLduHby9vbFkyRIAwPjx47F06VJERUUhIiICAQEBOHr0KEJDQyGEQGJiotyxyIru3vFztg9R9yFrEcjMzAQAfP3110aXJycnS4/Xrl0rZxSyInPTPDnbh6j74MliZFbItnyz3TgjvT0424eoG+JlI8iokG35qKmpwZ4WA1HNhaA9/f1sARB1DywCZFJReb1BX39v17b/XLjjJ+p+WARI0p6+/pa40yfq/jgm4ODM9fff3dfPHT+RfWFLwAG1d27/oHudjM7tZyEgsh8sAg7C3I6fc/uJHBOLgB2zZMdvbG5/V7x+ChFZF8cE7NzdV++8e8fPuf1Ejo0tATtj6dm8nNtP5NhYBOyApTdt4Y6fiJqxCHRTnbl6J3f8RNSMRaAb4QwfIrI2FoEurrMzfIiI2sLZQV3Q3WfxcoYPEcmFLYEuojNH/OzvJ6KOYhGwEWMzerjjJyKlsQjIrOXO3tiOv/m6/d+X3LkZO3f8RKQkFgEZtDVv/+6j/d6uTtA3NUnLueMnIiWxCFiJuT7932r243Zdo7Ss2UhvD9TU1MDNzU16P3f8RKQUFoFO6Oi8/buP9rVaLTR5FdJy7viJSCksAu3QVr9+RwZzeY1+IuoqWARMsKRfv72DuS1xp09EXYHDFwFLjvLb6tdvz2Aud/xE1NU4ZBHo6FF+S5zFQ0T2wCGLAND5o3zu+InIHjhUEYjadxVueRWdOspviTt+IuruHKoI3K0jR/nc8RORPZGtCJSVlSE4OBgZGRkYMmQIACAxMRG+vr4ICwtrtX5QUBB69+4NAPDx8cH69eutnin5mV9hxIgRbU7VbMadPRE5AlmKQENDA+Li4uDq6goAKC8vR1RUFC5duoQFCxa0Wr++vh4AkJmZKUecNvEon4gcmSz3E0hKSkJoaCj69esHAKiursaSJUswY8YMo+ufOXMGtbW1mD9/PubOnYvCwkI5YkmyF03iDp+ICDK0BHJzc+Hl5QV/f3+kp6cDAAYMGIABAwYgLy/P6HtcXV2xYMECzJ49G5cuXcLChQuxb98+ODm1jqfVajucra6urlPvlwtzWYa5LMNclnG0XFYvAjk5OVCpVMjPz4dWq0V0dDTS0tLQt29fk+/x9fXFwIEDoVKp4OvrC09PT5SWlsLb27vVuiNGjOhwNq1W26n3y4W5LMNclmEuy9hjroKCApPLrF4EsrKypMfh4eHQaDRtFgAA2LVrF86dOweNRoOSkhJUVVWZfQ8REXWeTe8xHBUVhatXr2LWrFm4ffs2wsLCsGzZMiQmJhrtCiIiIuuSdU9792yfJUuWGDxPTk6WHr/55ptyRiEiIiNs2hIgIiLbcpg+l+Z7+e7pggM+RES2wpYAEZEDs/uWQPNlIZovGtfWZaSJiBwNWwJERA7M7lsCzUf8zWMCbAEQEf2CLQEiIgdm9y2BZtmLJnXJ64EQEdkSWwJERA6MRYCIyIGxCBAROTAWASIiB8YiQETkwFgEiIgcmEoIIWwdor3aujsOERGZNm7cOKOvd6siQERE1sXuICIiB8YiQETkwOzushF6vR4ajQZnz56Fs7MzEhISMHDgQGn5wYMH8fbbb8PJyQkzZ87Ev//7vyuSq6GhAW+88QaKi4uh0+nwyiuv4IknnpCWv/vuu9i1axe8vLwAAGvWrMHgwYMVyRYUFITevXsDAHx8fLB+/Xppma22V25uLj766CMAQH19PbRaLY4ePQoPDw8Attle3377LTZt2oTMzExcvnwZMTExUKlUGDp0KFavXo0ePX45pjL3dyhXLq1Wi/j4eKjVajg7OyMpKQl9+vQxWL+t37dcuX744Qe8/PLLGDRoEAAgLCwMU6dOlda11fZatmwZbty4AQAoLi7G6NGjsXnzZoP15d5exvYNv/nNb5T7+xJ2Zv/+/SI6OloIIcT//u//ipdffllaptPpxJNPPikqKipEfX29CA4OFtevX1ck165du0RCQoIQQojy8nIxZcoUg+XLly8X33//vSJZWqqrqxMzZswwusyW26sljUYjduzYYfCa0tsrPT1dTJs2TcyePVsIIcSiRYvEV199JYQQIjY2Vnz++ecG67f1dyhnrjlz5ojTp08LIYT4+9//LhITEw3Wb+v3LWeunTt3iu3bt5tc31bbq1lFRYUIDAwUJSUlBq8rsb2M7RuU/Puyu+6ggoIC+Pv7AwDGjBmDU6dOScsuXLiAX//617j33nvh7OyMcePG4ZtvvlEk1zPPPIPXX39deq5Wqw2W//DDD0hPT0dYWBi2bdumSCYAOHPmDGprazF//nzMnTsXhYWF0jJbbq9m33//Pc6fP4+QkBCD15XeXr/+9a+Rmppq8P0TJkwAADz22GM4duyYwfpt/R3KmSslJQUj/u8Wqk1NTXBxcTFYv63ft5y5Tp06hcOHD2POnDl44403UFVVZbC+rbZXs9TUVPzhD39Av379DF5XYnsZ2zco+fdld0WgqqoK7u7u0nO1Wo3GxkZpWXOzDgB69erV6o9RLr169YK7uzuqqqqwdOlSREREGCx/7rnnoNFo8N5776GgoACHDh1SJJerqysWLFiA7du3Y82aNVixYkWX2F7Ntm3bhsWLF7d6Xent9fTTT8PJ6ZfeUyEEVCoVgDvb5fbt2wbrt/V3KGeu5p3YyZMn8cEHH+CFF14wWL+t37ecuR588EFERUUhKysLAwYMwNtvv22wvq22FwCUlZUhPz8fwcHBrdZXYnsZ2zco+fdld0XA3d0d1dXV0nO9Xi/90u9eVl1dbbCTk9vPP/+MuXPnYsaMGZg+fbr0uhAC8+bNg5eXF5ydnTFlyhScPn1akUy+vr4IDAyESqWCr68vPD09UVpaCsD226uyshJFRUV45JFHDF635fZq1rJ/trq6WhqraNbW36HcPv30U6xevRrp6enSmEmztn7fcgoICMCoUaOkx3f/vmy5vfbt24dp06a1ap0Dym2vu/cNSv592V0RGDt2LPLy8gAAhYWF8PPzk5YNGTIEly9fRkVFBXQ6Hb755hs89NBDiuS6ceMG5s+fj//8z//ErFmzDJZVVVVh2rRpqK6uhhACx48fl/6HkduuXbuwYcMGAEBJSQmqqqrQt29fALbdXgBw4sQJPProo61et+X2ajZy5EgcP34cAJCXl4eHH37YYHlbf4dy2r17Nz744ANkZmZiwIABrZa39fuW04IFC/Ddd98BAPLz8/Gv//qvBstttb2a8zz22GNGlymxvYztG5T8+7K72UEBAQE4evQoQkNDIYRAYmIi9uzZg5qaGoSEhCAmJgYLFiyAEAIzZ85E//79Fcn1zjvvoLKyElu3bsXWrVsBALNnz0ZtbS1CQkKwbNkyzJ07F87Ozpg0aRKmTJmiSK5Zs2Zh5cqVCAsLg0qlQmJiIj777DObby8AuHjxInx8fKTnLX+PttpezaKjoxEbG4uUlBQMHjwYTz/9NAAgKioKERERRv8O5dbU1IR169bB29sbS5YsAQCMHz8eS5culXIZ+30rccSt0WgQHx+Pnj17ok+fPoiPjwdg2+3V7OLFi60KppLby9i+YdWqVUhISFDk74tnDBMROTC76w4iIqL2YxEgInJgLAJERA6MRYCIyIGxCBAROTAWASIiB8YiQETkwFgEiDrp7NmzePzxx/G3v/3N1lGILMYiQNRJw4YNQ0pKCnbv3m3rKEQWYxEgsoL7778fP/74o61jEFmMRYDICjZt2gSdTofi4mJbRyGyCIsAUSfl5eWhtrYWv/vd73D+/HlbxyGyCIsAUSfU19dj48aNWL16Nfz8/HDu3DlbRyKyCIsAUSds3boVQUFB8PHxgZ+fH8cFqNthESDqoKKiIhw7dgzz5s0DABYB6pZ4PwEiIgfGlgARkQNjESAicmAsAkREDoxFgIjIgbEIEBE5MBYBIiIHxiJAROTAWASIiBzY/wcS18QOCqbhQwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAESCAYAAAD5d3KwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAv00lEQVR4nO3deVhTZ74H8G9EUREZaq1eFbE47rhVvLhVO0UplooLaEERF2yvOjoVl4I6qFGsWHCpg0stT7UdtAWuyxXUYjt0FKeil+JFZemiUEbQUhUxQmQ/9w8nGQ5JCAFOEuD7eZ4+T86W880xzY/3vOc9RyYIggAiIqJ/aWPqAEREZF5YGIiISISFgYiIRFgYiIhIhIWBiIhEWBiIiEikrakDUMswcOBADBgwAG3aiP/WOHDgAOzs7IyeZ8uWLbh8+TI8PDywevVqo++/toiICDx+/BibN2/Wu+7evXvx5MkTyOVynescOnQIMTExGDduHEJDQxuc6fjx4+jevbto/vbt2zFs2DCd27m4uGDfvn0a6yQkJOD48eOIiorS2ObYsWP48ssvIZPJ0Lt3b2zfvh0vvvhig3KT9FgYqMl8/vnn6NKli6ljAABiYmJw8eJF/Md//Iepoxhszpw5mDVrFgIDA2FlZaV1nRMnTmDXrl0YPXp0o/bl7u5er2LVGOnp6Thy5AjOnDmDzp0748MPP8S+ffuwbds2SfdLDcfCQJK7du0aPvjgA1hZWaGkpASBgYEIDw9XT588eRKnT59GVFQU2rRpg65du2LTpk1wcHDA+vXrUVRUhLt37+IPf/gDXn/9dezcuRPV1dUAgKVLl8LNzU20v3nz5kEQBLz77rvYsmULfve732Hbtm0oKiqCTCaDv78/Zs6cqZHr5MmTsLS0VL9PQUEBtm3bhvv376OiogJvvfUWli1bBgD4+OOPkZiYiNLSUjx79gxBQUFwdXVFZWUlwsPDcfHiRVhYWOCVV17Bli1bAADZ2dnw8/PDgwcP0LVrV+zZswfdunXTOF69evVC+/btce7cOcyZM0djeUBAAAoKCvDnP/8Zq1atwqhRoyCXy5Gfnw9BEDBz5ky88847yMvLg6+vL37/+98jPz8fUVFRWvenTUVFBXbu3Ink5GRYWFhg+PDh2LBhA6ytrUXr7du3D/Hx8bC1tUWfPn20vtfQoUNx4cIFtGvXDmVlZSgoKDBJK5IMIBA1gQEDBgjTpk0Tpk+frv7vj3/8oyAIgnD16lVh0KBBQl5entbpK1euCFOmTBEePXokCIIgnDx5UnjzzTeF6upqISgoSFi4cKF6PwsWLBDOnj0rCIIgZGVlCXK5XGeeR48eCRUVFcLkyZOFCxcuCIIgCL/++qswceJE4fr16xo5avPz8xMSExMFQRCE0tJSwc/PTzh37pyQl5cn+Pn5Cc+ePRMEQRDOnj0rTJs2TRAEQfj8888FX19f4dmzZ0JVVZWwatUq4fTp08Jf/vIXwcXFRf0Zly9fLuzfv1/rfi9duiQ4OTkJXl5eOo/366+/Lty8eVMQBEHw9fUVjhw5IgiCICgUCsHDw0M4e/ascPfuXWHAgAFCSkqK1vf4y1/+IowZM0b0bxYRESEIgiDs27dPWLlypVBeXi5UVVUJ69evFzZt2iTa9zfffCO4u7sLT58+FSoqKoT/+q//EubPn68z8zfffCM4OzsLr776qpCTk6NzPTI9thioydR1KqlHjx7o1auX1unLly/D3d1dva2npyc++OAD5OXlAQCcnJzU27355pvYtm0bvv32W4wfPx5r1qypM9Mvv/yCsrIyvPHGGwCA7t2744033sDly5cxZswYjVwqSqUSKSkpePLkCfbt26ee98MPP8Dd3R1hYWGIj49Hbm4ubty4gZKSEgDAlStXMGPGDHTo0AEA8NFHHwF4fj5/woQJ6s84aNAgFBYWas0cHR2NtWvX4uOPP0ZmZiaGDBmi8/MplUpcv34dR44cAQB07twZnp6eSEpKwogRI9C2bVuMHDlS5/a6TiUlJSVh9erVaNeuHQDAz88PK1asEK2TnJwMV1dXdSvCy8tLa/+CypQpUzBlyhTExsZiyZIl+OabbzT6pMg88F+FjKL2ufKa06rTQjUJgoDKykqNdX18fBAXF4cJEybgH//4B6ZPn46ysjKd+62qqoJMJqvXe9dUXV0NQRAQHR2NM2fO4MyZM4iJicHSpUuRkZEBb29vFBcXY8KECXjnnXfU27VtK/5b6+HDh/jtt980lslkMghablNWUFCA77//HtOnT4eXlxdiYmJ0fraaOWvPU30+S0tLjUz1UV1dLTpu1dXVqKio0Fiv5r4tLCy0vldubi6+//579bSXlxfu3buHJ0+eGJyLjIOFgUxu4sSJOH/+vPov6JMnT+o8Z+3j44OsrCx4enoiJCQECoUCDx480Pneffv2Rdu2bfH1118DeP7De+HCBYwfP77OTNbW1hg5ciSOHj0KAFAoFJg7dy4SExORkpKCoUOHYvHixXB2dkZiYiKqqqoAAOPGjcPZs2dRXl6O6upqyOVynDt3rt7HIjY2FlOnTkWnTp0we/ZsnD9/Hkqlss6cI0aMwPHjxwEAT58+xf/8z//o/Xz6TJw4EV9++SUqKipQXV2N48ePY8KECaJ1Jk2ahISEBCgUClRXV+PMmTNa3+vBgwdYs2aN+t83Pj4e/fv3xwsvvNCojCQdnkqiJrNw4UKNUwNr1qxRn1bRZcKECVi0aBEWLlyI6upqdOnSBYcPH9Z6mmHdunXYsWMHPvroI8hkMqxcubLOjsx27drh4MGD2L59OyIiIlBVVYUVK1Zg7NixuHbtWp25du3ahZCQEHh4eKC8vBzTpk3D9OnT8fDhQ3z99dd48803UV1djddffx1PnjxBcXExfHx8kJ+fD09PTwiCAGdnZ/j5+eHQoUN17gt43rr57//+bxw8eBAA0LNnT4wYMUJnJ3TNnNu2bcOpU6dQXl4ODw8PeHp6Ij8/X+8+dVm+fDk+/PBDzJw5E5WVlRg+fDg2bdokWue1117Djz/+CC8vL9jY2GDQoEF4/PixxnuNHj0ay5Ytw4IFC2BhYYFu3brhwIEDDc5G0pMJ2tqzRETUavFUEhERibAwEBGRCAsDERGJsDAQEZEICwMREYm0iMtVU1NTTR2BiKhZqnlnAZUWURgA7R+uPrKysjB48OAmTtN4zGUY5jIMcxnGXHMBjcum649qyQrDjRs3sGvXLkRFRWH16tV4+PAhACA/Px8jRozA3r171etWVFRg48aNyM/PR3l5OZYvX47JkycjIyMDy5Ytw8svvwwAmDt3Ltzd3aWKTEREkKgwREZGIi4uDh07dgQAdRF48uQJFixYgA0bNojWj4uLg62tLcLDw/H48WPMmjULkydPRmZmJhYvXgx/f38pYhIRkRaSdD7b29sjIiJCY35ERATmz5+vcU/4qVOnYtWqVepp1c240tPTcfHiRfj6+mLjxo0oLi6WIi4REdUg2S0x8vLysGbNGsTGxgIAHj16hAULFiAuLk7nXRiLi4uxfPlyvP322/Dw8MDJkycxcOBADB06FIcOHYJCoUBQUJDGdqmpqTrvkqlPaWmp3nv5mAJzGYa5DMNchjHXXEDjsimVStN2PickJGDatGk6i8L9+/exYsUKzJs3Dx4eHgAAV1dX2NjYqF+HhITofP+Gdr6Ya6cScxmGuQzDXIYx11yANJ3PRhvHkJycjEmTJmld9vDhQ/j7++P999/H7Nmz1fOXLFmCmzdvqrd3dHSULJ/34WR4H06W7P2JiJoLo7UYcnJy0Lt3b9G8wMBABAQE4MiRI1AoFDh48KD6lsORkZGQy+UICQlBu3bt0LVr1zpbDERE1DQkKwx2dnbq/gUAWh9WEhYWBgAIDg5GcHCwxnJHR0dER0dLFREAEJhwD1ZJRbiW8/whIqpWQ8zScZLul4jIXPGWGEREJNLqC0PY1J6IWToOYxy6YIxDF3VLgf0NRNRatfrCQEREYiwM/1KzpXAtpxDXcgp5pRIRtUosDEREJMLCUEPM0nHsbyCiVo+FgYiIRFgYtGB/AxG1ZiwMREQk0mKe4NbUavcvcCQ0EbUWbDEYgKeTiKg1YItBD7YUiKi1YWGoB1UrgTfaI6LWgKeSiIhIhC2GetDVEc2WAxG1RGwxEBGRCFsMBqjdUmCfAxG1RGwxEBGRCAtDA/Bme0TUkrEwEBGRCAtDI/Bme0TUErEwEBGRiGSF4caNG/Dz8wMArF69Gn5+fvDz84OLiwtWr14tWre6uhqbN2+Gt7c3/Pz8kJubCwDIzc3F3LlzMW/ePGzZsgXV1dVSxW0w9jcQUUsjSWGIjIxEcHAwysrKAAB79+5FVFQU9u/fj86dO2PDhg2i9f/2t7+hvLwcMTExWLt2LXbu3AkACA0NRUBAAL744gsIgoDExEQp4hIRUQ2SFAZ7e3tERERozI+IiMD8+fPRrVs30fzU1FRMnDgRADBy5Eikp6cDADIyMuDs7AwAmDRpEq5cuSJF3CbB/gYiaikkGeDm5uaGvLw80bxHjx4hOTlZo7UAAMXFxbC2tlZPW1hYoLKyEoIgQCaTAQA6deqEp0+f6txnVlZWg7KWlpY2eNvalEql1umGvH9T5mpKzGUY5jIMcxlOimxGG/mckJCAadOmwcLCQmOZtbU1SkpK1NPV1dVo27Yt2rT5d4OmpKQENjY2Ot9/8ODBDcqVlZXV4G1ri//X+9QcCe19OBnypCKDR0U3Za6mxFyGYS7DMJfhGpMtNTVV63yjXZWUnJyMSZMmaV02atQoJCUlAQDS0tIwYMAAAMCQIUNw7do1AEBSUhJGjx5tnLBERK2Y0QpDTk4OevfuLZoXGBiIe/fuwdXVFZaWlvDx8UFoaKj6dFNQUBAiIiLg7e2NiooKuLm5GStuo7C/gYiaM8lOJdnZ2SE2NlY9fe7cOY11wsLC1K+3bdumsdzBwQHHjh2TJiAREWnFAW4S4fgGImquWBiIiEiEhUFi7G8gouaGhYGIiERYGIyA/Q1E1JywMBARkQgLgxGxv4GImgMWBiIiEmFhMDL2NxCRuWNhICIiERYGE2F/AxGZKxYGIiISYWEwIfY3EJE5YmEgIiIRFgYzoK2/ITDhHlsORGQSLAxERCTCwmAmavc3hE3tCYD9DURkfCwMREQkwsJgZlT9DYEJ9zi+gYhMgoWBiIhEWBjMUMzScQib2pPjG4jIJFgYiIhIpK1Ub3zjxg3s2rULUVFRePToEYKDg6FQKFBVVYWwsDDY29ur1z116hROnz4NACgrK0NWVha+++473L17F8uWLcPLL78MAJg7dy7c3d2limx2YpaOU/cvXMspBPDvVoOqFUFE1NQkKQyRkZGIi4tDx44dAQDh4eHw8PCAu7s7rl69iuzsbFFh8PT0hKenJwBg69at8PLygo2NDTIzM7F48WL4+/tLEbPZyryvgPfhZBYHIpKEJKeS7O3tERERoZ6+fv06CgoKsGjRIsTHx8PZ2Vnrdrdu3cLt27fh7e0NAEhPT8fFixfh6+uLjRs3ori4WIq4Zk3b/ZSG9LAxdSwiasFkgiAIUrxxXl4e1qxZg9jYWDg6OmLbtm3w8vLC/v37UVVVhVWrVmlss3LlSsyfPx9jx44FAJw8eRIDBw7E0KFDcejQISgUCgQFBWlsl5qaCisrqwblLC0tRYcOHRq0rZRq5wpMuIfswjL07dIetwpKAQDDuj9frhoMZ4pc5oK5DMNchjHXXEDjsimVSjg5OWnMl6yPoSZbW1u4uLgAAFxcXLB3716NdRQKBbKzs9VFAQBcXV1hY2Ojfh0SEqJzH4MHDzY4l/fhZCiVSsSvnmzwtlLLysoSfab4wYNrXJX0vDCoimFDPntT5TIXzGUY5jKMueYCGpctNTVV63y9p5KuXr3aoB3W5OTkhEuXLgEAUlJS0K9fP411UlJSMH78eNG8JUuW4ObNmwCA5ORkODo6NjpLc8bbdBORMehtMURERIj+im+IoKAgBAcHIzo6GtbW1ti9ezcAIDAwEAEBAejZsydycnJgZ2cn2k4ulyMkJATt2rVD165d62wxGEL1Q8orfYiINOktDDKZDCtWrICDgwPatHnewFizZo3eN7azs0NsbCwAoFevXjh69KjGOmFhYerX77zzjsZyR0dHREdH691Xa8PLWIlISnoLg5eXlzFyGFXNUzBKpZI/pkRENejtY/Dw8IBSqcTNmzehUCjw1ltvGSMX6cH+BiKSit7CsHnzZty9excTJkxAfn4+goODjZHLKFT3JCIion/TeyopNzcXx48fBwBMmTIFPj4+koei+mN/AxE1Nb0thrKyMjx79gzA84EUVVVVkoeipsHnOBBRQ+htMSxcuBAzZsxA//79cfv2bbz33nvGyEUGqN2/wP4GImoMvYXhpZdeQmxsLO7evQs7Ozu88MILxshFjcBxGkTUGHpPJUVERMDW1hbDhg1jUTBzqiuViIgaQ7IBbmQ62k4tqfobWDiISB+9hWHmzJmwsLAwRhYiIjIDegvD+fPnceTIEWNkoSbGS1mJqCH09jF07twZiYmJuHPnDnJycpCTk2OMXCQh1RPgiIi00dtiKCwsxGeffaaelslk+Otf/yplJmpCuvobiIh00VsYoqKiRNNlZWWShSFpqVoKPK1ERHXReSopICBA/bpmH8O7774raSCShq5nRfO0EhHVprMwPHr0SP364sWL6tcSPSKajEDbHVm1FQsiat3q9cznmsVAJpNJFoaMh6eViEgXnS2GmgWAxaBlYUuBiOqis8Vw+/ZtrF27FoIgiF7fuXPHmPlIIhwdTUS66CwMH330kfp1zWcw8HkMREQtm87C4OzsbMwcZCIcHU1Etekd+UytEy9jJWq96nVVUkPcuHEDu3btQlRUFB49eoTg4GAoFApUVVUhLCwM9vb2ovVnzpyJzp07AwDs7OwQGhqK3NxcrF+/HjKZDP3798eWLVvUd3ilpsPR0URUk97CUFBQgPDwcDx+/Bhubm4YOHAgRowYUec2kZGRiIuLQ8eOHQEA4eHh8PDwgLu7O65evYrs7GxRYVCNpq49yjo0NBQBAQEYM2YMNm/ejMTERLi6uhr8Ian+eBkrEen983vTpk3w8vJCeXk5Ro8ejQ8++EDvm9rb2yMiIkI9ff36dRQUFGDRokWIj4/X6L/44Ycf8OzZM/j7+2PBggVIS0sDAGRkZKjXnTRpEq5cuWLIZyMDcXQ0EQH1aDGUlZVh3LhxOHToEPr27Yv27dvrfVM3Nzfk5eWpp/Pz82FjY4PPPvsM+/fvR2RkJFatWqVe3qFDByxZsgRz5szBL7/8gnfffRcJCQkQBEE9hqJTp054+vSpzn1mZWXpzaVNaWlpg7eVkqlyySfZAgAClUr1dGCCEkqlEllZWTxeBmIuwzCX4aTIprcwWFpa4vLly6iurkZaWhosLS0N3omtrS1cXFwAAC4uLti7d69ouYODA/r06QOZTAYHBwfY2triwYMHov6EkpIS2NjoHpQ1ePBgg3MBzwtKQ7eVkqlzWSUVIfO+AvKkItwqKAUAyJOKoFQqEb96ssly6WLq46ULcxmGuQzXmGypqala5+s9lRQSEoJTp07h8ePHOHLkCORyucE7d3JywqVLlwAAKSkp6Nevn2j5iRMnsHPnTgDP+zSKi4vx0ksvYciQIbh27RoAICkpCaNHjzZ439Qwuk4rZReW8bQSUQunt8Vw9OhRjb/wDRUUFITg4GBER0fD2toau3fvBgAEBgYiICAAs2fPxoYNGzB37lzIZDLs2LEDbdu2RVBQEDZt2oQ9e/agb9++cHNza1QOMoy2q5U89iaaMhIRGYHewnDnzh0oFIo6T+NoY2dnh9jYWABAr169cPToUY11wsLC1K9VxaImBwcHHDt2zKD9kjRUHdDPTyuV8molohZM76mkO3fuYMyYMZgwYQJeffVVvPrqq8bIRWaEVysRtS56Wwx///vfjZGDzJyqZeCxNxFWVlYcBEfUguktDGlpaTh16hQqKioAAL/99hs+/fRTyYOR+eIgOKKWTe+ppO3bt8PZ2RnFxcXo2bMnbG1tjRCLzFXY1J58lgNRC6e3MNjY2GDatGmwtrbGn/70JxQUFBgjF5kxbY8IBcBTS0QthN7CIJPJ8PPPP+PZs2fIzs7GgwcPjJGLiIhMRG9hWL9+PX7++Wf4+flh3bp1mDt3rjFyUTNQs6VwLacQ13IK1c92IKLmS2/nc//+/dG/f38AwKlTpyQPRM2fqnOandFEzZPewlBz3EJRURF69+6Nr776StJQ1HzwWQ5ELY/ewvCPf/xD/To/Px/79++XNBA1X7yMlahlMOhxaL169UJ2drZUWagZ4+hoopZDb4thzZo16mci/Pbbb3jxxRclD0XNE08rEbUMeguDj4+P+nX79u0xdOhQSQNRy6DttFLmfQWG9LDhqSUiM6e3MNy7d080nZOTo349c+bMJg9EzR9bCkTNm97C8O2330KpVMLZ2Rnp6el4+PAhnJyc1KeXiLSpeVop874CAPC0tFI91qHmOkRkXvQWhuLiYnz66afqQuDv74+1a9dKHoxaNo51IDJfegtDYWEhnj59ChsbGxQWFqK4uNgYuaiFqPnDz05pouZBb2FYtmwZZsyYAVtbWyiVSmzdutUYuaiF4lgHIvOntzBMnToVU6ZMwZMnT3ipKjVKXS0FFggi86FzgNvdu3exYsUKVFZWIi0tDdOnT4erqyvS0tKMGI9aGm237GYxIDIvOlsMO3bswOzZs9G2bVuEhoYiLCwM/fr1w7p16xAVFWXMjNSCqVoKPLVEZD50thjKy8sxefJkPH78GL/++ismTJiA7t27o7q62pj5qIWqq6XA22gQmZbeeyUlJydj7NixAIDq6mo8ffq0Xm9848YN+Pn5AQAePXqE5cuXw9fXFz4+PvjnP/8pWreiogLvv/8+5s2bh9mzZyMxMREAkJGRgYkTJ8LPzw9+fn44f/68QR+OzJ+2U0t8dCiRaek8ldS/f3+sXbsW6enpCAkJwW+//YY9e/aoi0RdIiMjERcXh44dOwIAwsPD4eHhAXd3d1y9ehXZ2dmwt7dXrx8XFwdbW1uEh4fj8ePHmDVrFiZPnozMzEwsXrwY/v7+TfBRydzxiiUi86CzxRAUFITp06cjIiICzs7OePz4MQYNGoSgoCC9b2pvb4+IiAj19PXr11FQUIBFixYhPj4ezs7OovWnTp2KVatWqactLCwAAOnp6bh48SJ8fX2xceNGjqFowXh3ViLzIRMEQZDijfPy8rBmzRrExsbC0dER27Ztg5eXF/bv34+qqipRIVApLi7G8uXL8fbbb8PDwwMnT57EwIEDMXToUBw6dAgKhUJrYUpNTYWVlVWDcpaWlqJDhw4N2lZKrTlXYMLz+3OFTe0pem3qXA3BXIZhLsM1JptSqYSTk5PGfL3jGJqCra0tXFxcAAAuLi7Yu3evxjr379/HihUrMG/ePHh4eAAAXF1dYWNjo34dEhKicx+DBw9uULasrKwGbyul1pzLKqkImfcVkCcV4VZBKQBA/q95uu7O2pqPV0Mwl2HMNRfQuGypqala5xv0oJ6GcnJywqVLlwAAKSkp6Nevn2j5w4cP4e/vj/fffx+zZ89Wz1+yZAlu3rwJ4HknuKOjozHikomxA5rItOp1E72kpCSUl5er5xl6u+2goCAEBwcjOjoa1tbW2L17NwAgMDAQAQEBOHLkCBQKBQ4ePIiDBw8CeN6BLZfLERISgnbt2qFr1651thioZeHdWYlMR29h+OMf/4hu3bqhR48eAFDv223b2dkhNjYWwPNHgh49elRjnbCwMABAcHAwgoODNZY7OjoiOjq6Xvuj1oV3ZyWSjt7CIAgCdu3aZYwsRBp4d1Yi49NbGAYOHIgbN26IOjcsLS0lDUWki66xDvJJtiZMRdSy6C0M//u//4tvv/1WPS2TydQjk4mMSVdLIfO+AoEJSsSb6VUjRM2N3sIQFxdnjBxE9VKzU1o17X04GUql0pSxiFoUvYUhMTERX3zxBSoqKiAIAoqKihAfH2+MbER10nZaqa6xDkRUP3rHMRw4cAArV65Ejx49MGvWLAwYMMAYuYjqxLEORNLRWxheeOEFvPLKKwAAT09PFBQUSB6KqD5q3pm1U7vnl1HXHOvAK5eIGkZvYWjXrh1SUlJQWVmJy5cv48GDB8bIRdRovAEfUcPoLQxbt25FZWUlli9fjtjYWLz33nvGyEVUbzFLx+HEPAc+14GoiejtfO7evTuys7Nx/fp1rFixAg4ODsbIRdRg7JQmahy9LYY9e/bg9OnTiImJQVZWFjZs2GCMXEQNwpYCUePpLQypqakICwuDlZUVZs2ahby8PGPkImqwmp3SnTs8bxTX7JQeJr/AvgeiOugtDFVVVSgrK4NMJkNVVRXatDHKnbqJiMhE9PYxLFy4EJ6enigsLMScOXOwaNEiI8QiarzaN+Dj7buJ6kdvYXjzzTcxfvx45Obmws7ODl26dDFGLiKj4O27iTTpLAx1dTKHhoZKEoZIKrx9N1H96SwM6enpKC0txfTp0/HKK69AEARj5iKSFC9pJdJNZ09yfHw8Dhw4gLKyMnzyySdIS0uDvb09Jk6caMx8RE2Ol7QS1a3OPoYBAwZg3bp1AICUlBTs3r0bv/76q/qRnUTNlb5nSrP1QK2Z3s7n4uJifPPNNzh79iyePXuG6dOnGyMXERGZiM7C8NVXX+HcuXO4d+8e3njjDWzduhV2dnbGzEYkOV7SSqRJZ2FYvXo1+vbti0GDBuGnn37C3r171ct2795tlHBE5oIFgloTnYXhr3/9a6Pe+MaNG9i1axeioqLw6NEjBAcHQ6FQoKqqCmFhYbC3t1evW11dDblcjh9//BGWlpbYvn07+vTpg9zcXKxfvx4ymQz9+/fHli1bOPKaJKPrktaa00Stgc7C4Ozs3OA3jYyMRFxcHDp27AgACA8Ph4eHB9zd3XH16lVkZ2eLCsPf/vY3lJeXIyYmBmlpadi5cycOHTqE0NBQBAQEYMyYMdi8eTMSExPh6ura4FxEhlIVBF7WSq2JJH9+29vbIyIiQj19/fp1FBQUYNGiRYiPj9coOqmpqerLYEeOHIn09HQAQEZGhnrdSZMm4cqVK1LEJdKguhEfUWuk96qkhnBzcxPdhTU/Px82Njb47LPPsH//fkRGRmLVqlXq5cXFxbC2tlZPW1hYoLKyEoIgQCZ7/sjGTp064enTpzr3mZWV1aCspaWlDd5WSsxlGKlyySfZAgAClUpkF5ZBqVSqO6Y99iYiu7AMfbu0R9jUnkbN1VjMZRhzzQVIk02SwlCbra0tXFxcAAAuLi6ijmwAsLa2RklJiXq6uroabdu2FfUnlJSUwMZG96CkwYMHNyhbVlZWg7eVEnMZRupcVklFaPOkElZWVgBKn8+zslLP07Xv1nq8Goq5DNeYbKmpqVrnG6Un18nJCZcuXQLwfKBcv379RMtHjRqFpKQkAEBaWhoGDBgAABgyZAiuXbsGAEhKSsLo0aONEZdIQ8zScbgld+NzHqhVMEphCAoKwpkzZ+Dj44PLly9j2bJlAIDAwEDcu3cPrq6usLS0hI+PD0JDQ9U38AsKCkJERAS8vb1RUVEBNzc3Y8QlImrVJDuVZGdnp751Rq9evXD06FGNdcLCwtSvt23bprHcwcEBx44dkyoiUYPUZ1Cc6solVR8FUXPCQQFERCRilM5nopZKX+shUKnELzH/5LgHalbYYiAiIhG2GIiaiCF9D2w9kDlji4GIiETYYiCSgKpFkJWVBXlSEVsP1KywxUBkBrwPJ3NwHJkNthiIJGZI3wOROWBhIDKxzPsKnmIis8LCQGRE2loPQ3rYqJ/3QGQO2MdAZEKqlgFvzEfmhC0GIhPhqSIyVywMRGaAg+PInPBUElEzxMtbSUpsMRCZGV7eSqbGwkDUzPDyVpIaCwORGePlrWQKLAxEzYiqZaDrFFNNbEFQQ7EwEDUTDfmhVxULFgkyBAsDUTNU+xRTTarTTOykpoZiYSBqodhJTQ3FwkDUzNX+oa9PJ3Vgwj1YJRWxSJBWkg1wu3HjBvz8/AAAGRkZmDhxIvz8/ODn54fz58+L1j116pR62dtvv41hw4ZBoVDo3Y6ItNN3D6bswjL1uhwsR7VJ0mKIjIxEXFwcOnbsCADIzMzE4sWL4e/vr3V9T09PeHp6AgC2bt0KLy8v2NjY6N2OiDTpawVk3legpELgYDnSSZIWg729PSIiItTT6enpuHjxInx9fbFx40YUFxdr3e7WrVu4ffs2vL29DdqOiLSLWToOt+RuotZDzSJQux+Cd3MlAJAJgiBI8cZ5eXlYs2YNYmNjcfLkSQwcOBBDhw7FoUOHoFAoEBQUpLHNypUrMX/+fIwdOxYA6r1damoqrKysGpSztLQUHTp0aNC2UmIuwzCXfoEJ9wAAYVN7Yt35PPxSVIG+XdrjVkEpAGBY9w7ILixD3y7tETa1p2h9YzGn41WTueYCGpdNqVTCyclJY75ROp9dXV1hY2Ojfh0SEqKxjkKhQHZ2troo1Hc7lcGDBzcoW1ZWVoO3lRJzGYa59IuvkaNNwj0MtXtBNFjOysoKJQWluFVQCnlSEX55UokhPWwwePBgo42HMKfjVZO55gIaly01NVXrfKPcXXXJkiW4efMmACA5ORmOjo4a66SkpGD8+PEGb0dEhgub2rPOH3ltl7qqsLO65TNKi0EulyMkJATt2rVD165d1X/5BwYGIiAgAD179kROTg7s7OzqtR0RNR1992PSNR6CWi7JCoOdnR1iY2MBAI6OjoiOjtZYJywsTP36nXfe0Viuazsikk7t+zHVp0hwPETLwgf1EJFazNJxoh/52uMh6mop8BRTy8HCQERa6SsSgObDg1RYJJo33hKDiPSqz6A5Xf0QvMNr88PCQEQGaWxnNQuF+eOpJCJqlLr6IWoXidqnl3jKyTyxxUBEDVb7r359VzQBz/slAM3nRXgfToZSqRQNxCPTYGEgoiZRV5Goqa5TTi//7vlPUs3TTTz1ZHwsDETU5LT9iKvm1dUvcau0ss5BdCwSxsHCQESS0vYjXt9BdMPkF+o89aTr/alx2PlMREZV1/iITu1kOgfR1ff+TezQbjwWBiIymdpFom+X9uoiMcahC27J3ep1tVPNIlETi0TDsDAQkVmIWTpO57Mf6ntJ7DD5BVzLKWSropHYx0BEZqdmK8KQS2Jrqu9dYeu6Ako1LZ9k2ySfq7lgYSCiZkPfJbGN7dCui64C0hI7wVkYiKhZquuHuCGtipoFY5j8AoB/D8YLVCrVT7SrS32KR1O9Vu1PikGBLAxE1OzV99STarm+glFbdmEZSioErQWkIa0Pc8fCQEQtVn1bFap1tf1FrhqRfaugtM591dX6kOI1IL69iL7PawgWBiJqFepqVdRlSA8byCfZQp5UpN62Ma2P5oCFgYhatboKhmo6KyurzvfQ1/qQuo+hqTu+WRiIiOpBVwFpSVcjqbAwEBE1kfoUj6Z6rZrW15ppCI58JiIiEclaDDdu3MCuXbsQFRWFjIwMLFu2DC+//DIAYO7cuXB3dxetP3PmTHTu3BkAYGdnh9DQUOTm5mL9+vWQyWTo378/tmzZgjZtWMuIiKQkSWGIjIxEXFwcOnbsCADIzMzE4sWL4e/vr3X9srIyAEBUVJRofmhoKAICAjBmzBhs3rwZiYmJcHV1lSIyERH9iyR/ftvb2yMiIkI9nZ6ejosXL8LX1xcbN25EcXGxaP0ffvgBz549g7+/PxYsWIC0tDQAQEZGBpydnQEAkyZNwpUrV6SIS0RENUjSYnBzc0NeXp56evjw4ZgzZw6GDh2KQ4cO4cCBAwgKClIv79ChA5YsWYI5c+bgl19+wbvvvouEhAQIggCZTAYA6NSpE54+fapznw3tgCktLZWk86axmMswzGUY5jKMueYCpMlmlKuSXF1dYWNjo34dEhIiWu7g4IA+ffpAJpPBwcEBtra2ePDggag/oaSkRP0e2gxu4L1CsrKyGrytlJjLMMxlGOYyjLnmAhqXLTU1Vet8oxSGJUuWYNOmTRg+fDiSk5Ph6OgoWn7ixAn89NNPkMvlKCgoQHFxMV566SUMGTIE165dw5gxY5CUlISxY8fq3IeuD1gfjdlWSsxlGOYyDHMZxlxzAU2fTSYIgtCk7/gveXl5WLNmDWJjY5GRkYGQkBC0a9cOXbt2RUhICKytrREYGIiAgAB07doVGzZswL179yCTybBu3TqMGjUKOTk52LRpEyoqKtC3b19s374dFhYWUsQlIqJ/kawwEBFR88RBAUREJNJqbolRXV0NuVyOH3/8EZaWlti+fTv69OmjXv7tt9/iwIEDaNu2Lby8vPD2228bJVdFRQU2btyI/Px8lJeXY/ny5Zg8ebJ6+dGjR3HixAl06dIFALB161b07dvXKNm0DTpUMdXxOnXqFE6fPg3g+fiXrKwsfPfdd+oLE0xxvGoO5tQ3KFPf91CqXFlZWQgJCYGFhQUsLS3x4YcfomvXrqL16/r3liqXvsGvpjpeq1evxsOHDwEA+fn5GDFiBPbu3StaX+rjpe23oV+/fsb5fgmtxIULF4SgoCBBEATh//7v/4Rly5apl5WXlwtTpkwRioqKhLKyMsHT01P47bffjJLrxIkTwvbt2wVBEITCwkLhtddeEy1fu3atcOvWLaNkqam0tFSYMWOG1mWmPF41yeVyITo6WjTP2Mfrk08+EaZNmybMmTNHEARBWLp0qXD16lVBEARh06ZNwtdffy1av67voZS5fH19hczMTEEQBOHLL78UduzYIVq/rn9vKXPFxsYKn376qc71TXW8VIqKioTp06cLBQUFovnGOF7afhuM9f1qNaeSUlNTMXHiRADAyJEjkZ6erl52584d2Nvb43e/+x0sLS3h5OSE77//3ii5pk6dilWrVqmna3euZ2Rk4JNPPsHcuXNx+PBho2QCdA86BEx7vFRu3bqF27dvw9vbWzTf2Mer9mBOfYMy6/oeSplrz5496ksaq6qq0L59e9H6df17S5lL3+BXUx0vlYiICMyfPx/dunUTzTfG8dL222Cs71erKQzFxcWwtrZWT1tYWKCyslK9TNUkBJ4Ppqv9BZVKp06dYG1tjeLiYrz33nsICAgQLX/rrbcgl8vx+eefIzU1FX//+9+Nkks16PDTTz/F1q1bsW7dOrM4XiqHDx/GihUrNOYb+3i5ubmhbdt/n5EV9AzKrOt7KGUu1Q/b9evXcezYMSxatEi0fl3/3lLmGj58OAIDA3H8+HH07t0bBw4cEK1vquMFAI8ePUJycjI8PT011jfG8dL222Cs71erKQzW1tYoKSlRT1dXV6u/CLWXlZSUiH74pHb//n0sWLAAM2bMgIeHh3q+IAhYuHAhunTpAktLS7z22mvIzMw0SiYHBwdMnz5dY9AhYPrjpVAokJ2drTGuxZTHS0XfoMy6vodSO3/+PLZs2YJPPvlE3QejUte/t5RcXV0xdOhQ9eva/16mPF4JCQmYNm2a1kvkjXW8av82GOv71WoKw6hRo5CUlAQASEtLw4ABA9TLfv/73yM3NxdFRUUoLy/H999/j1deecUouR4+fAh/f3+8//77mD17tmhZcXExpk2bhpKSEgiCgGvXrqn/J5LaiRMnsHPnTgAQDToETHu8ACAlJQXjx4/XmG/K46WiGpQJAElJSRg9erRoeV3fQymdOXMGx44dQ1RUFHr37q2xvK5/byktWbIEN2/eBACtg19NdbxUeSZNmqR1mTGOl7bfBmN9v1rNVUmurq747rvv4OPjA0EQsGPHDsTHx0OpVMLb2xvr16/HkiVLIAgCvLy80L17d6Pk+vjjj6FQKHDw4EEcPHgQADBnzhw8e/YM3t7eWL16NRYsWABLS0uMGzcOr732mlFyzZ49Gxs2bMDcuXMhk8mwY8cOfPXVVyY/XgCQk5MDOzs79XTNf0dTHS+VoKAgbNq0CXv27EHfvn3h5uYGAOrBnNq+h1KrqqrCBx98gB49euBPf/oTAOA///M/8d5776lzafv3NsZf5nK5XGPwK2Da46WSk5OjUUSNeby0/Tb8+c9/xvbt2yX/fnGAGxERibSaU0lERFQ/LAxERCTCwkBERCIsDEREJMLCQEREIiwMREQkwsJAREQiLAxEEvnxxx/x+uuv44svvjB1FCKDsDAQSWTgwIHYs2cPzpw5Y+ooRAZhYSCS0Isvvoiff/7Z1DGIDMLCQCShXbt2oby8HPn5+aaOQlRvLAxEEklKSsKzZ8/whz/8Abdv3zZ1HKJ6Y2EgkkBZWRnCw8OxZcsWDBgwAD/99JOpIxHVGwsDkQQOHjyImTNnws7ODgMGDGA/AzUrLAxETSw7OxtXrlzBwoULAYCFgZodPo+BiIhE2GIgIiIRFgYiIhJhYSAiIhEWBiIiEmFhICIiERYGIiISYWEgIiIRFgYiIhL5f0P9yUqWjT9EAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAESCAYAAAAbq2nJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAr2ElEQVR4nO3de1xUZf4H8M8IogEpS6U/C2+s4uWXqeiC5oppEWiCN1wQI1vb9bK0hqGIxk1RTPCnuWyIdttCLV21n7fSNjQxJTJKU5ytlPKlYHgBRBguA/P8/vDH7AzMMAzMOQPM5/16+XLOdb5z5vB8z/M85zyjEEIIEBGRTepk7QCIiMh6mASIiGwYkwARkQ1jEiAismFMAkRENoxJgIjIhjEJkFkGDRqEgIAATJs2Te/f9evXrRJPfHw8Jk2ahM2bN1vl/RtKTU3FmjVrmrXu5s2bkZCQ0OQ6W7duxVNPPYWVK1e2KqYxY8Y0+s4uXLjQ5HaTJk0yuM7Ro0cRFhbW5Laff/45Ro4c2eKYST721g6A2p/3338frq6u1g4DALB792588cUX+K//+i9rh2K22bNnY8aMGYiKioKjo6PBdfbu3YuNGzdi9OjRrXqvKVOmIC4urlX7aK5ffvkFGzZskOW9qPWYBMhicnJysG7dOjg6OqKiogJRUVFISUnRTu/btw8ff/wxMjIy0KlTJzz88MOIjY1F//79ER0djdLSUly7dg1PPfUUJk6ciNdffx0ajQYAsHDhQvj5+em9X2hoKIQQ+POf/4z4+Hh0794da9asQWlpKRQKBebPn4/p06c3imvfvn1wcHDQ7qeoqAhr1qzBjRs3oFar8dxzz2HRokUAgPT0dGRmZqKqqgqVlZVYsWIFfH19UVtbi5SUFHzxxRews7PDyJEjER8fDwDIz89HWFgYbt26hYcffhibNm1Cjx49Gh2vxx57DF26dMGRI0cwe/bsRssjIiJQVFSE1157Da+88go8PT2RkJCAgoICCCEwffp0/OlPf8L169cxd+5c/Pa3v0VBQQEyMjIMvp8harUar7/+OrKzs2FnZ4cnnngCK1euhLOzs956W7ZswaFDh+Di4oK+ffsa3V9lZSWWL1+O6OhoLFu2rFkxkJUJIjN4eHiIqVOnisDAQO2/v/zlL0IIIb766isxePBgcf36dYPTZ86cEc8884y4c+eOEEKIffv2icmTJwuNRiNWrFgh5s2bp32fF154QRw+fFgIIYRSqRQJCQlG47lz545Qq9Xi6aefFseOHRNCCPHrr7+K8ePHi2+//bZRHA2FhYWJzMxMIYQQVVVVIiwsTBw5ckRcv35dhIWFicrKSiGEEIcPHxZTp04VQgjx/vvvi7lz54rKykpRV1cnXnnlFfHxxx+Lv/3tb2LSpEnaz7h48WLx97//3eD7njx5UowaNUrMmjXL6PGeOHGi+P7774UQQsydO1e8++67QgghysrKREBAgDh8+LC4du2a8PDwEGfPnjW4j7/97W/C29tb7ztLTU0VQgixZcsW8fLLL4uamhpRV1cnoqOjRWxsrN57/+tf/xJTpkwR9+7dE2q1WixYsEA8//zzBt9r2bJl4p///Ke4du2aGDFihNHPRW0HawJktqaag3r16oXHHnvM4PSpU6cwZcoU7bYzZ87EunXrtP0Jo0aN0m43efJkrFmzBsePH8eTTz6JV199tcmYfvnlF1RXV+PZZ58FAPTs2RPPPvssTp06BW9v70Zx1VOpVDh79izu3r2LLVu2aOf9+9//xpQpU5CcnIxDhw7h6tWrOH/+PCoqKgAAZ86cwbRp09C1a1cAwBtvvAHgfvv7uHHjtJ9x8ODBKC4uNhjzRx99hMjISKSnp+PSpUsYOnSo0c+nUqnw7bff4t133wUAPPjgg5g5cyaysrIwfPhw2NvbY8SIEUa3N9YclJWVhaVLl6Jz584AgLCwMISHh+utk52dDV9fX23tYNasWcjIyGi0r507d8Le3h5BQUFW6yMi87FjmCyqYdu27nR9044uIQRqa2sbrRsSEoKDBw9i3Lhx+PLLLxEYGIjq6mqj71tXVweFQtGsfevSaDQQQuCjjz7CgQMHcODAAezevRsLFy5EXl4egoODUV5ejnHjxuFPf/qTdjt7e/3rp9u3b+PmzZuNlikUCggDw3MVFRXhm2++QWBgIGbNmoXdu3cb/Wy6cTacV//5HBwcGsXUHBqNRu+4aTQaqNXqRuvpvrednZ3BfX388ce4cOECpk2bhgULFqCqqgrTpk1DUVGR2XGRfJgESDbjx4/HJ598or0y3rdvn9E25pCQECiVSsycOROJiYkoKyvDrVu3jO7b3d0d9vb2+OyzzwDcL2SPHTuGJ598ssmYnJ2dMWLECLz33nsAgLKyMsyZMweZmZk4e/YsHn/8cfzxj3+El5cXMjMzUVdXBwAYO3YsDh8+jJqaGmg0GiQkJODIkSPNPhZ79uyBv78/nJycEBQUhE8++QQqlarJOIcPH46dO3cCAO7du4f//d//Nfn5TBk/fjw+/PBDqNVqaDQa7Ny5E+PGjdNbx8fHB0ePHkVZWRk0Gg0OHDhgcF979+7F4cOHceDAAWzfvh1du3bFgQMH0LNnz1bFSNJicxCZbd68eejUSf/64dVXX9U2jRgzbtw4vPjii5g3bx40Gg1cXV2xbdu2RvsCgGXLliEpKQlvvPEGFAoFXn75Zbi5uRndd+fOnZGWloa1a9ciNTUVdXV1CA8Px5gxY5CTk9NkXBs3bkRiYiICAgJQU1ODqVOnIjAwELdv38Znn32GyZMnQ6PRYOLEibh79y7Ky8sREhKCgoICzJw5E0IIeHl5ISwsDFu3bm3yvYD7tZZ//vOfSEtLAwA8+uijGD58uNEOYt0416xZg/3796OmpgYBAQGYOXMmCgoKTL6nMYsXL8aGDRswffp01NbW4oknnkBsbKzeOhMmTMAPP/yAWbNmoVu3bhg8eDBKSkpa/J7UtiiEoboqERHZBDYHERHZMCYBIiIbZvE+AbVajVWrVqGgoAA1NTVYvHgxBgwYgOjoaCgUCgwcOBDx8fGN2oG3bduG48ePQ61WY86cOU22jRIRkWVYPAkcPHgQLi4uSElJQUlJCWbMmIHBgwcjIiIC3t7eiIuLQ2ZmJnx9fbXb5OTk4LvvvsOHH36IyspK7b3QREQkLYsnAX9/f73H++3s7JCXlwcvLy8A9283O336tF4S+PLLL+Hh4YHw8HCUl5cjKirK0mEREZEBFk8CTk5OAIDy8nIsWbIEERER2LBhg/aBFCcnJ9y7d09vm5KSEhQWFiI9PR3Xr1/H4sWLcfTo0UYP/+Tm5lo6XCIim6D7RL4uSZ4TuHHjBsLDwxEaGoqAgACkpKRol1VUVKBbt25667u4uMDd3R0ODg5wd3dHly5dUFxcjIceeqjRvo19kOZQKpUYMmRIi7eXCuMyD+MyD+MyT0eMq6kLaIvfHXT79m3Mnz8fy5cvR1BQEABg6NCh2gd2srKyGg2LO2rUKJw6dQpCCBQVFaGyshIuLi6WDo2IiBqweE0gPT0dZWVlSEtL0z4R+dprr2Ht2rXYtGkT3N3dtX0GUVFRiIiIwMSJE3H27FkEBQVBCIG4uDij45MQEZHlWDwJxMTEICYmptH8HTt2NJqXnJysfc3OYCIi+fFhMSIiG8YkQERkw2xmFNHgbdlQqVQ41AZ7/YmIrIU1ASIiG9bhawLB27IBADk/F+tN71441moxERG1FawJEBHZsA5fE6i/4q/vE2ANgIjoP1gTICKyYR2+JlBv98KxUCqV1g6DiKhNYU2AiMiGMQkQEdkwJgEiIhvGJEBEZMOYBIiIbBiTABGRDWMSICKyYUwCREQ2jEmAiMiGMQkQEdkwJgEiIhvGJEBEZMOYBIiIbBiTABGRDbN4ElCr1Vi+fDlCQ0MRFBSEzMxMXL16FXPmzEFoaCji4+Oh0WgMbnvnzh1MmDABV65csXRYRERkgMV/T+DgwYNwcXFBSkoKSkpKMGPGDAwePBgRERHw9vZGXFwcMjMz4evrq7edWq1GXFwcunbtaumQiIjICIvXBPz9/fHKK69op+3s7JCXlwcvLy8AgI+PD86cOdNouw0bNiAkJAQ9evSwdEhERGSExZOAk5MTnJ2dUV5ejiVLliAiIgJCCCgUCu3ye/fu6W2zf/9+uLq6Yvz48ZYOh4iImqAQQghL7/TGjRsIDw/X9gv4+PggKysLAPD555/jzJkziIuL064/d+5cKBQKKBQKKJVK9OvXD1u3bsUjjzyit9/c3Fw4Ojq2OK6qqqo22dzEuMzDuMzDuMzTEeNSqVQYNWqU4YXCwm7duiX8/f3FmTNntPMWLlwovvrqKyGEELGxseLIkSNGt3/++efF5cuXDS775ptvWhXbpUuXWrW9VBiXeRiXeRiXeTpiXE2VnRZvDkpPT0dZWRnS0tIQFhaGsLAwREREIDU1FcHBwVCr1fDz8wMAREVFobCw0NIhEBFRM1n87qCYmBjExMQ0mr9jx45G85KTkxvNy8jIsHRIRERkBB8WIyKyYUwCREQ2jEmAiMiGMQkQEdkwJgEiIhvGJEBEZMOYBIiIbBiTABGRDWMSICKyYUwCREQ2jEmAiMiGMQkQEdkwJgEiIhvGJEBEZMOYBIiIbBiTABGRDWMSICKyYUwCREQ2jEmAiMiGMQkQEdkwJgEiIhvGJEBEZMOYBIiIbBiTABGRDbO39A7VajVWrVqFgoIC1NTUYPHixRgwYACio6OhUCgwcOBAxMfHo1OnTk1u8/TTT1s6NCIiasDiSeDgwYNwcXFBSkoKSkpKMGPGDAwePBgRERHw9vZGXFwcMjMz4evr2+Q2TAJERNKzeBLw9/eHn5+fdtrOzg55eXnw8vICAPj4+OD06dN6ScDQNkREJD2LJwEnJycAQHl5OZYsWYKIiAhs2LABCoVCu/zevXsmtzFGqVS2OLaqqqpWbS8VxmUexmUexmUeW4vL4kkAAG7cuIHw8HCEhoYiICAAKSkp2mUVFRXo1q2byW2MGTJkSIvjUiqVrdpeKozLPIzLPIzLPB0xrtzcXKPLLH530O3btzF//nwsX74cQUFBAIChQ4ciJycHAJCVlYXRo0eb3IaIiKRn8SSQnp6OsrIypKWlISwsDGFhYYiIiEBqaiqCg4OhVqu17f9RUVEoLCw0uE1VVZWlQyMiogZMNgd99dVXGDNmTLN3GBMTg5iYmEbzd+zY0WhecnJyk9sQEZG0TNYEUlNT5YiDiIiswGRNQKFQIDw8HP3799c+4PXqq69KHhgREUnPZBKYNWuWHHEQEZEVmGwOCggIgEqlwvfff4+ysjI899xzcsRFREQyMJkE4uLicO3aNYwbNw4FBQXswCUi6kBMNgddvXoVO3fuBAA888wzCAkJkTwoIiKSh8maQHV1NSorKwHcf2y5rq5O8qCIiEgeJmsC8+bNw7Rp0zBw4EBcvnwZS5YskSMuIiKSgckk8Mgjj2DPnj24du0a3Nzc8Jvf/EaOuIiISAbNeljMxcUFw4YNYwIgIupg+LAYEZENM5kEpk+fzh95ISLqoEwmgU8++QTvvvuuHLEQEZHMTCaBBx98EJmZmejXr5+2Oah///6SB0ZERNIzmQSKi4vxj3/8QzutUCjwwQcfSBkTERHJxGQSyMjI0Juurq6WLBgiIpKX0VtEdX/sXbdP4M9//rOkARERkXyMJoE7d+5oX3/xxRfa10IISQMiIiL5NOs3hnULfoVCIVkwREQkL6NJQLewZ8FPRNQxGe0Yvnz5MiIjIyGE0Ht95coVOeMjIiIJGU0Cb7zxhva17m8I8PcEiIg6DqNJwMvLS844iIjICkw+J2AutVqNVatWoaCgADU1NVi8eDEGDBiA6OhoKBQKDBw4EPHx8dqnjwFAo9EgISEBP/zwAxwcHLB27Vr07dvX0qEREVEDzbo7yBwHDx6Ei4sLdu3ahbfeeguJiYlYv349IiIisGvXLgghkJmZqbfN559/jpqaGuzevRuRkZF4/fXXLR0WEREZYLImUFRUhJSUFJSUlMDPzw+DBg3C8OHDja7v7+8PPz8/7bSdnR3y8vK0zUs+Pj44ffo0fH19tevk5uZi/PjxAIARI0bg4sWLLf5ARETUfCaTQGxsLP74xz8iLS0No0ePRnR0NPbs2WN0fScnJwBAeXk5lixZgoiICGzYsEF7m6mTkxPu3bunt015eTmcnZ2103Z2dqitrYW9fePwlEpl8z6ZAVVVVa3aXiqMyzyMyzyMyzy2FpfJJFBdXY2xY8di69atcHd3R5cuXUzu9MaNGwgPD0doaCgCAgKQkpKiXVZRUYFu3brpre/s7IyKigrttEajMZgAAGDIkCEm398YpVLZqu2lwrjMw7jMw7jM0xHjys3NNbrMZJ+Ag4MDTp06BY1Gg3PnzsHBwaHJ9W/fvo358+dj+fLlCAoKAgAMHToUOTk5AICsrCyMHj1abxtPT09kZWUBAM6dOwcPDw9TYRERkQWYTAKJiYnYv38/SkpK8O677yIhIaHJ9dPT01FWVoa0tDSEhYUhLCwMERERSE1NRXBwMNRqtbbPICoqCoWFhfD19YWDgwNCQkKwfv16rFy50iIfjoiImmayOei9997D5s2bm73DmJgYxMTENJq/Y8eORvOSk5O1r9esWdPs9yAiIsswWRO4cuUKysrK5IiFiIhkZrImcOXKFXh7e8PV1VV7h8+XX34peWBERCQ9k0ngxIkTcsRBRERWYDIJnDt3Dvv374darQYA3Lx5E++8847kgRERkfRM9gmsXbsWXl5eKC8vx6OPPgoXFxcZwiIiIjmYTALdunXD1KlT4ezsjL/+9a8oKiqSIy4iIpKBySSgUCjw008/obKyEvn5+bh165YccRERkQxMJoHo6Gj89NNPCAsLw7JlyzBnzhw54iIiIhmY7BgeOHAgBg4cCADYv3+/5AEREZF8TCaB3//+99rXpaWl6N27Nz799FNJgyIiInmYTAK6D4YVFBTg73//u6QBERGRfMz6ecnHHnsM+fn5UsVCREQGBG/LhkqlwiEJhrg2mQReffVV7XARN2/exEMPPWTxIIiIyDpMJoGQkBDt6y5duuDxxx+XNCAiIroveFs2ACDn52K96d0Lx1rsPUwmgcLCQr3pn3/+Wft6+vTpFguEiIj+I3hbNi7dKMPQXt1Mr9wKJpPA8ePHoVKp4OXlhYsXL+L27dsYNWqUtomIiIikMbRXN+xeOFbbJ2DJGkA9k0mgvLwc77zzjrbQnz9/PiIjIy0eCBERGW4CunSjDP26m3UfT7OZfGK4uLgY9+7d074uLy+XJBAiIltXX+A3NLRXNyT7PyrJe5pMLYsWLcK0adPg4uIClUqF1atXSxIIERHpNwEB/+kEViqVkryfySTg7++PZ555Bnfv3uXtoUREEjDWBCR1pzDQRHPQtWvXEB4ejtraWpw7dw6BgYHw9fXFuXPnJA+KiMhWNNUEJEVHcENGawJJSUkICgqCvb091q9fj+TkZAwYMADLli1DRkaG5IEREdkKY01AcjCaBGpqavD000+jpKQEv/76K8aNGwcA0Gg0sgVHRNRRWbMJSJfJu4Oys7MxZswYAPcTQP2dQqacP38eYWFhAIC8vDwEBQUhNDQUiYmJjRKJWq1GZGQkQkJCEBoaiitXrpj7OYiI2g1rNwHpMpoEBg4ciMjISGzZsgXBwcG4efMmVq1apU0ITXnrrbcQExOD6upqAEBsbCxWrVqFXbt2wdnZGYcOHdJb/+TJk6itrcVHH32E8PBwvPHGG637VEREbVx9ge/d3xXe/V2xe+FY2RMA0EQSWLFiBQIDA5GamgovLy+UlJRg8ODBWLFihcmd9unTB6mpqdrpoqIieHp6AgA8PT2Rm5urt37//v1RV1cHjUaD8vJy2NtL81AEEZE1BW/LRvC2bOT8XIycn4uN1gjkZLS0VSgUmDBhgnZ60KBBGDRoULN26ufnh+vXr2une/fuja+//hpeXl44ceIEKisr9dZ3dHREQUEBJk+ejJKSEqSnpxvdd2vula2qqpLsXtvWYFzmYVzmYVzmkSquqKOFyC+uhrtrF+08lUqFft3tkeDjYvI9pYpLlkvupKQkrFu3Dm+//TaGDRsGBwcHveX/+Mc/8Pvf/x6RkZG4ceMG5s2bh0OHDqFLly6N9jWkFeNpK5XKVm0vFcZlHsZlHsZlHqnicswqxeOOji2+C6g1cTVsfdElSxI4efIkkpKS0LNnTyQmJsLHx0dvebdu3dC5c2cAQPfu3VFbW4u6ujo5QiMiklRbuQvImGYNIJeVlYWamhrtPHOHkO7bty8WLFiABx54AN7e3tpmpqioKERERODFF1/EqlWrEBoaCrVajaVLl8LR0dG8T0JE1IbUF/6GWOMuIGNMJoG//OUv6NGjB3r16gUAzR5C2s3NDXv27AEATJo0CZMmTWq0TnJysvb1li1bmrVfIqL2pL6wt8aDYM1hMgkIIbBx40Y5YiEiaveM/RpYW2UyCQwaNAjnz5/X65Bo2LFLRERNt/e3tRpAPZNJ4Ouvv8bx48e10wqFApmZmZIGRUTUXllzHKCWMJkEDh48KEccRETtVlu/A6gpJpNAZmYmdu3aBbVaDSEESktLGw37QERkq4wV+G3pDqCmmBxA7s0338TLL7+MXr16YcaMGfDw8JAjLiKidqOtjAPUEiaTwG9+8xuMHDkSADBz5kwUFRVJHhQRUVvXFscBagmTSaBz5844e/YsamtrcerUKdy6dUuOuIiI2qy2NBR0a5nsE1i9ejXy8/OxePFibNmyBUuWLJEjLiKiNq293QVkjMkk0LNnT+Tn5+Pbb79FeHg4+vfvL0dcRERtTnu+C8gYk0lg06ZN+PXXX3HlyhV07twZ27dvx6ZNm+SIjYiozWjvdwEZY7JPIDc3F8nJyXB0dMSMGTP0fieAiMiWtOe7gIwxWROoq6tDdXU1FAoF6urq0KmTybzR5rX3Njwikofu1X9HagLSZTIJzJs3DzNnzkRxcTFmz56NF198UYawiIjarvbeBKTLZBKYPHkynnzySVy9ehVubm5wdXWVIy5JRB0thGNWaaPR/TrKl0lEltGwAxgAHuxq36EK/3pGk8DKlSuNbrR+/XpJgrGGSzfKELwtu8N9sUTUMlFHC/HL3doO09xjitEkcPHiRVRVVSEwMBAjR46EEELOuCSR7P8ohgwZolcDaOtjfROR/DrKMwDNYTQJHDp0CD/++CMOHjyI7du343e/+x0CAwPRt29fOeOTTH0NgE1DRAT8pwy4UFQFoKrDdQAb0+StPh4eHli2bBk++OADjBkzBv/zP/+DP/zhD3LFJpndC8ca/GLrEwMR2ZaONAyEuZr1Q/P/+te/cPjwYVRWViIwMFCOuCRn6Hc/mQCIbFd9gR+wOROOjo4dvvCvZzQJfPrppzhy5AgKCwvx7LPPYvXq1XBzc5MzNtmwaYjI9hh7BiC/uBqPOzpaOTr5GG0OWrp0KfLz89GvXz/8+OOP2Lx5MyIjIxEZGSlnfJJj0xAR6XJ37WJTF4BGawIffPCBnHFYFZuGiGyHqWcAlEqltUKzCqNJwMvLq1U7Pn/+PDZu3IiMjAzk5eUhPj4eDg4OGDJkCF577bVGw09s27YNx48fh1qtxpw5czB79uxWvX9LsWmIqOOylTt+zCHJQEBvvfUWYmJiUF1dDQCIjY3FqlWrsGvXLjg7Ozf6jeKcnBx89913+PDDD5GRkYFff/1VirBMYtMQUcdnaBC4Cwl+NnuhJ0kS6NOnD1JTU7XTRUVF8PT0BAB4enoiNzdXb/0vv/wSHh4eCA8Px6JFi/DUU09JEVaz1I8KqHuC8KqBqP0K3paNYQnHOsRPQUrB5C2iLeHn56c35HTv3r3x9ddfw8vLCydOnEBlZaXe+iUlJSgsLER6ejquX7+OxYsX4+jRo1AoFI323Zr2uqqqqmZvr1KpkF9cjYDNmf//8AgQsDkT+cXVcHftgmT/R1scR2vikhPjMg/jMo9ccalUKmjq6qBSqfTm9etujwQfl0Yx2NrxkiQJNJSUlIR169bh7bffxrBhw+Dg4KC33MXFBe7u7nBwcIC7uzu6dOmC4uJiPPTQQ432NWTIkBbHoVQqm739IZ3hJYD7ScDR0RGd7tbC0dGxVXG0Ji45MS7zMC7zSBmXbtt//UWco6MjHuxaa/IBsI54vBq2vuiS5ccBTp48iaSkJGzfvh2lpaUYN26c3vJRo0bh1KlTEEKgqKgIlZWVcHFxkSO0Juk2DT3Y9X6+vFdVq61Osp+AiNo7WWoCffv2xYIFC/DAAw/A29sbEyZMAABERUUhIiICEydOxNmzZxEUFAQhBOLi4mBnZydHaK3CEUiJ2hZbGgLaUiRLAm5ubtizZw8AYNKkSZg0aVKjdZKTk7Wvo6KipAql1XRPHj5LQNQ28fbPlpGlJtDRGHqWoP7k49UGkfXY0hDQlsIkYCbWAIjahoZ/hx31N4ClxiTQArrDTNTfa6zbYay7DhHJj7Xy5mMSkAA7jImkYWjkT+/+rnr/8+/OPEwCrcAOYyJq75gELIgdxkSWZ+jqH+Ctn5Yiy8NitoBjDBFRe8SagAWxw5jIMnj1Lx/WBGTEIamJqK1hTUAC7DAmahkO+yA/JgEZsMOYyDQ+5GUdTAISYw2AqPk47IP8mARkYKrDWKVSwTGrVG9dIltgqAOYNQJ5MQkQkazqr/ITfFyMrsOmUvkwCcjIUIdxPfYXkC25dKMMUUdV2l/9AtgBbC1MAkQkOUPNPvl1dVaOigAmAavRvdpRKpVIyCo12F/AWgF1VO6uXfDLXdO/+UvSYhIgIkmYeuo3wccFCf9/QwRZD5NAG9Gwv4C1ArIFPJetj0mAiCzGnDF/lEqlNUKkBpgE2iDWCohILkwCRNQqHPGzfWMSaONYK6C2iMM6dBySDSV9/vx5hIWFAQDy8vIQFBSE0NBQJCYmQqPRGNzmzp07mDBhAq5cuSJVWB1e8LZsjlVEstAdGPFeVS2A+1f/3v1dsXvhWFxI8GOSaAckqQm89dZbOHjwIB544AEAQGxsLGJiYuDp6YnNmzfj0KFDmDZtmt42arUacXFx6Nq1qxQhdQjm1AqILM1Qs0/9OUjtlyQ1gT59+iA1NVU7XVRUBE9PTwCAp6cncnNzG22zYcMGhISEoEePHlKEZBMu3SjTSwjDEo6xVkCtYqpmObRXN179t3OS1AT8/Pxw/fp17XTv3r3x9ddfw8vLCydOnEBlZaXe+vv374erqyvGjx+P7du3N7nv1txWVlVV1SZvS2tJXPcH33IBAESpVMgvrka/7va48P/VcpVKBU1dHVQqVYs/c0c6XnLoiHGp/v/cCticqR3nR6VSwamzAu6uXZDg44Koo6oWnWcd8XhJSaq4ZOkYTkpKwrp16/D2229j2LBhcHBw0Fu+b98+KBQKZGdnQ6lUYsWKFdi6dSseeeSRRvsaMmRIi+NQKpWt2l4qrY3LMasUjzs6asdhv3SjDI6OjqgoqsKFoirtkBTmdh531OMllY4QV8Or/vqC/5e7tdp5jo6O6HS3Fo6OjhgyZAgOtfAzd4TjJafWxGWo9aWeLEng5MmTSEpKQs+ePZGYmAgfHx+95Tt37tS+DgsLQ0JCgsEEQIaZW/3mnR2kqznnw9Be3XgXWgclyw/N9+3bFwsWLEBISAicnZ0xYcIEAEBUVBQKCwvlCMFm6LbLevd3xYNd7+f5hp3HRLp07/TRvdffu7+rtr2fNxx0TJLVBNzc3LBnzx4AwKRJkzBp0qRG6yQnJzeal5GRIVVIhMadx7y6s00tudOH50jHxIfFOjBDt5Q2fKpTF5uJOjZT3y+bfGyTLM1B1DbU/3E3p5koeFs2oo6yqa69a/g9mnrAi00+toc1ARth6srOUDNRv+73Tw/WENqXht9XfnG1tuAHmm724Xdse5gEbJCpZqL6hHChqpZPIbcTDQt+3Sv++ul6bPYhXUwCpC0MmkoI9TUEXSxArKs1BT+fJKd6TAI2rmFBrltIXLxe0mRHMsCmIrm1pODv192+UcHP74vqMQmQVsOCwd21i8GrxvoCp2FTEROCNFp7xa9SqfS2J9LFu4PIoN0LxyLZ/9Em1zHUmVxPd+AxDm9tvobHrOFdPQ0LfmN3+DTneyTbxpoAmdTwCrI5fQfGOpNZWzDOUm38PLZkDtYEqEUaPnOgW+DrJoRhCce0QxE0VVuwFQ0/syWv+Fn4U0swCZDZGhY4TSUEXbbUfGTO52HBT9bE5iBqlabuLqpfbqnmo/rX939LoW3QjS/qaCEcs0oNFswNm3aGJRwDAO1Tu2zqIWthEiCLMTUUcXMSwrCEY9qC0dzkUP/+xpZJ8bohQ4W9oYLe0PFhwU/WwCRAktAtuJqqLTQ1qJ2x5BClUuGXu7VWfYq5JYW9oSd1DSWX+mkiOTAJkKxa2nykK7+4GhVq0Sg5NGxiabjM0q+bW9gbK+hNHRsiOTAJkNU0t/mofl3dge3qf/bQmhoW9gGbM+Go8zOfhjT8zCz4ydqYBKhNaKr5SNfQXt2Q4OOChKxS7brW7BNoTmHPgp7aMiYBatMMFaZKpdJa4TSiG1+y/6Nt8gfKiZrCJEDtUlNX2saWSfGaqL3jw2JERDaMSYCIyIYxCRAR2TAmASIiG8YkQERkw5gEiIhsmEIIIawdRHPl5uZaOwQionZp1KhRBue3qyRARESWxeYgIiIbxiRARGTDOtywERqNBgkJCfjhhx/g4OCAtWvXom/fvtrlx48fx5tvvgl7e3vMmjULf/jDH2SJS61WY9WqVSgoKEBNTQ0WL16Mp59+Wrv8vffew969e+Hq6goAWL16Ndzd3WWJbfr06XjwwQcBAG5ubli/fr12mbWO1/79+/Hxxx8DAKqrq6FUKnH69Gl063b/NwSscbzOnz+PjRs3IiMjA1evXkV0dDQUCgUGDhyI+Ph4dOr0n2sqU+ehVHEplUokJibCzs4ODg4O2LBhAx5++GG99Zv6vqWKKy8vD4sWLUK/fv0AAHPmzMGUKVO061rreC1duhS3b98GABQUFGD48OHYvHmz3vpSHy9DZcOAAQPkO79EB3Ps2DGxYsUKIYQQ3333nVi0aJF2WU1NjXjmmWdEaWmpqK6uFjNnzhQ3b96UJa69e/eKtWvXCiGEKC4uFhMmTNBbHhkZKS5cuCBLLLqqqqrEtGnTDC6z5vHSlZCQID766CO9eXIfr+3bt4upU6eK2bNnCyGEWLhwofjqq6+EEELExsaKzz77TG/9ps5DKeOaO3euuHTpkhBCiA8//FAkJSXprd/U9y1lXHv27BHvvPOO0fWtdbzqlZaWisDAQFFUVKQ3X47jZahskPP86nDNQbm5uRg/fjwAYMSIEbh48aJ22ZUrV9CnTx90794dDg4OGDVqFL755htZ4vL398crr7yinbazs9NbnpeXh+3bt2POnDnYtm2bLDEBwL///W9UVlZi/vz5eOGFF3Du3DntMmser3oXLlzA5cuXERwcrDdf7uPVp08fpKam6r2/l5cXAMDHxwdnzpzRW7+p81DKuDZt2qQdybSurg5dunTRW7+p71vKuC5evIgvvvgCc+fOxapVq1BeXq63vrWOV73U1FQ8//zz6NGjh958OY6XobJBzvOrwyWB8vJyODs7a6ft7OxQW1urXVZfrQMAJyenRiejVJycnODs7Izy8nIsWbIEEREResufe+45JCQk4P3330dubi5OnDghS1xdu3bFSy+9hHfeeQerV6/GsmXL2sTxqrdt2zaEh4c3mi/38fLz84O9/X9aT4UQUCgUAO4fl3v37umt39R5KGVc9YXYt99+ix07duDFF1/UW7+p71vKuJ544glERUVh586d6N27N95880299a11vADgzp07yM7OxsyZMxutL8fxMlQ2yHl+dbgk4OzsjIqKCu20RqPRfukNl1VUVOgVclK7ceMGXnjhBUybNg0BAQHa+UIIzJs3D66urnBwcMCECRNw6dIlWWLq378/AgMDoVAo0L9/f7i4uODWrVsArH+8ysrKkJ+fjzFjxujNt+bxqqfbPltRUaHtq6jX1HkotU8++QTx8fHYvn27ts+kXlPft5R8fX3x+OOPa183/L6sebyOHj2KqVOnNqqdA/Idr4Zlg5znV4dLAp6ensjKygIAnDt3Dh4eHtplv/3tb3H16lWUlpaipqYG33zzDUaOHClLXLdv38b8+fOxfPlyBAUF6S0rLy/H1KlTUVFRASEEcnJytH8wUtu7dy9ef/11AEBRURHKy8vxyCOPALDu8QKAs2fP4sknn2w035rHq97QoUORk5MDAMjKysLo0aP1ljd1HkrpwIED2LFjBzIyMtC7d+9Gy5v6vqX00ksv4fvvvwcAZGdn47//+7/1llvreNXH4+PjY3CZHMfLUNkg5/nV4e4O8vX1xenTpxESEgIhBJKSknDo0CGoVCoEBwcjOjoaL730EoQQmDVrFnr27ClLXOnp6SgrK0NaWhrS0tIAALNnz0ZlZSWCg4OxdOlSvPDCC3BwcMDYsWMxYcIEWeIKCgrCypUrMWfOHCgUCiQlJeHTTz+1+vECgJ9//hlubm7aad3v0VrHq96KFSsQGxuLTZs2wd3dHX5+fgCAqKgoREREGDwPpVZXV4d169ahV69e+Otf/woA+N3vfoclS5Zo4zL0fctxxZ2QkIDExER07twZDz/8MBITEwFY93jV+/nnnxslTDmPl6Gy4bXXXsPatWtlOb/4xDARkQ3rcM1BRETUfEwCREQ2jEmAiMiGMQkQEdkwJgEiIhvGJEBEZMOYBIiIbBiTAFEr/fDDD5g4cSJ27dpl7VCIzMYkQNRKgwYNwqZNm3DgwAFrh0JkNiYBIgt46KGH8NNPP1k7DCKzMQkQWcDGjRtRU1ODgoICa4dCZBYmAaJWysrKQmVlJZ566ilcvnzZ2uEQmYVJgKgVqqurkZKSgvj4eHh4eODHH3+0dkhEZmESIGqFtLQ0TJ8+HW5ubvDw8GC/ALU7TAJELZSfn48zZ85g3rx5AMAkQO0Sf0+AiMiGsSZARGTDmASIiGwYkwARkQ1jEiAismFMAkRENoxJgIjIhjEJEBHZMCYBIiIb9n89kuoNLukecgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mean_opt_penalty, lowest_error, mean_opt_beta, opt_lambda_per_fold = cross_validation_across_each_fold()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimal lambda for fold 1 is 0.20202020202020202\n",
      "Optimal lambda for fold 2 is 4.444444444444445\n",
      "Optimal lambda for fold 3 is 0.0\n",
      "Optimal lambda for fold 4 is 19.393939393939394\n",
      "Optimal lambda for fold 5 is 8.686868686868687\n",
      "The mean of all the optimal penalty values is  6.545454545454545\n"
     ]
    }
   ],
   "source": [
    "[print(f\"Optimal lambda for fold {fold + 1} is {l}\"\"\") for fold, l in enumerate(opt_lambda_per_fold)]\n",
    "print(\"The mean of all the optimal penalty values is \", mean_opt_penalty)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Exploring the performance of the fold averaged cross validation\n",
    "In the fold averaged cross validation, different lambdas do well on different training datasets, as shown by the plots of penalty terms against errors in different folds. Taking an average of the parameter that gives the lowest mse at each fold would not give an accurate representation of the optimal penalty parameter, since what we want to obtain is a parameter that does consistently well with unseen data by consistently generating models that gives an accurate predicted y value which results in a low mse. Hence, a more accurate estimate of the optimal parameter would be to take the sum of how each penalty parameter performs in each fold and take the average of these errors. In subesequent cross validations, we would be taking the average error or accuracy of each hyperparameter combination across all 5 folds, and taking the hyperparameter combination that gives us the lowest value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We take the penalty parameter that performs the best by having the lowest average MSE across all 5 folds. We call this the average MSE based cross validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_validation_ridge():\n",
    "    validation_score = defaultdict(float)\n",
    "    num_of_folds = len(folds)\n",
    "    penalty_params = np.linspace(0, 20, num=100)\n",
    "    \n",
    "    # perform cross validation across k folds\n",
    "    for i in range(num_of_folds):\n",
    "        train_x, train_y, test_x, test_y = generate_test_train_sets(folds, i)\n",
    "        optimal_lambda = None\n",
    "        min_mse = sys.maxsize\n",
    "        for j, penalty in enumerate(penalty_params):\n",
    "            #train model using function\n",
    "            beta_ridge = ridge_regression_estimate(train_x, train_y, penalty)\n",
    "            #evaluate model\n",
    "            predicted_y = predict_with_estimate(test_x, beta_ridge)\n",
    "            # calculate mse of model and add the mse to the dictionary\n",
    "            error_validation = mse(test_y, predicted_y)\n",
    "            validation_score[penalty] += error_validation\n",
    "    \n",
    "    #get average MSE for each hyperparameter lambda across all 5 folds\n",
    "    for param, error in validation_score.items():\n",
    "        validation_score[param] = error / num_of_folds\n",
    "    \n",
    "    #plot graph for the average performance of each hyperparameter\n",
    "    plot_penalty_param(validation_score, \"Average error across all 5 folds\")\n",
    "    \n",
    "    # find optimal lambda for our model\n",
    "    optimal_penalty = None\n",
    "    lowest_error = sys.maxsize\n",
    "    for param, error in validation_score.items():\n",
    "        if error < lowest_error:\n",
    "            optimal_penalty = param\n",
    "            lowest_error = error\n",
    "    \n",
    "    # retrain the ridge regression model on the entire training dataset with the optimal penalty observed\n",
    "    optimal_beta_ridge = ridge_regression_estimate(X_train, y_train, optimal_penalty)\n",
    "    return validation_score, optimal_penalty, lowest_error, optimal_beta_ridge\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAESCAYAAAAbq2nJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxXklEQVR4nO3de1hU1d4H8O8IogdQkU52NDXRxCxLi0TxWl6STFQUQUnA6CbxHINMVBoRhcyDpvWShPpkFzUveewNtTiVShSiRynMC6WlmQ0oIiANAwww6/3Dl90MzDAMzAWY7+d5ety3tfnNnmn99lprX2RCCAEiIrJLHWwdABER2Q6TABGRHWMSICKyY0wCRER2jEmAiMiOMQkQEdkxJgGSVFdXY8yYMXjuuedsHQqZ2f79+/Hiiy8CAEJCQpCent5gmz///BMPPvggZsyYIf13/PjxBtvl5eVh0qRJmDVrFv744w+Df9PQ3zlz5gwmTJjQgk9D5uRo6wCo9fjqq69w33334ezZs/j1118xYMAAW4dEVpSbm4vhw4dj27ZtjW53+PBhjBgxAq+//rqVIiNLYhIgya5duzB16lT07dsXH374IVauXIkJEyZg06ZNGDJkCAAgKioK3t7eCA4Oxrvvvosvv/wSGo0Gd999N1auXIm77roLISEh6NatGy5duoR58+bhwQcfxLp166BWq3Hjxg2MGjUKa9asAXD7DHXLli3o3LkzRo4ciY8++gjnz58HAIP7r++TTz7Brl27oNFo4ObmhhUrVmDAgAFYtmwZSktLcfXqVTz22GO4efOmzvzChQuxatUq/PTTT5DJZBg7dixeeeUVODo6YsiQIZg4cSJ++uknrF+/Hg8++KD09y5fvozVq1ejvLwcN27cwH333Ye33noLnTp1wunTp5GYmIiKigp07NgRMTEx8PHxabC/qqoqJCUlSdtFRUVh3LhxuHHjBpYuXYqSkhIAwPjx4xEVFWVweX379u3Dnj17UF1djVu3buH5559HcHBwk77/H374AaWlpQgMDIRarUZgYGCDsmlpadi1axdqa2tRWVmJN998E5s2bcKhQ4fg4OAADw8PrFixAnfeeadOuY8//hgffvghXF1d4enpKS3/9ddf8dprr0GtVkMIgYCAADz99NNNipfMRBAJIS5evCgeeOABUVxcLE6fPi0eeughUVxcLN5++22xatUqIYQQpaWlwtvbW5SVlYlPP/1UREVFierqaiGEELt37xbPPfecEEKI+fPni+XLl0v7jo6OFsePHxdCCKFUKsWIESPEmTNnxMWLF4WPj48oKCgQQgiRnJwsPD09hRCi0f1rO3HihAgODhYqlUoIIcS3334rfH19hRBCLF26VISFhUnb1p+PiYkRCQkJQqPRiKqqKhEeHi42b94shBDC09NTfPrpp3qP1dq1a8X//u//CiGEUKvVYtq0aSI9PV2o1WoxevRocfToUSGEEGfOnBHTpk0TtbW1OvsrLi4WPj4+Ijc3VwghxIULF4S3t7f4/fffxTvvvCNWrFghhBCivLxcREVFibKyMoPLtSmVShEYGCiKi4uFEEL88MMPYtiwYUIIIf7973+LF154Qfp+vvjiiwaf65133hHJycmiqqpKXLt2TTzxxBPiq6++arDd//zP/0i/iX379omgoCBRXl4urQsPD9f5O+fPnxc+Pj6isLBQCCHEihUrxOOPPy6EEGL58uXSMS8sLBRRUVGitrZW73Eny2BLgADcbgU8/vjj6N69O7p3747evXtj7969mD17NgICArBs2TIcPHgQEyZMQJcuXXD06FGcOXMGs2fPBgBoNBpUVFRI+3v00Uel6bVr1yIzMxOpqam4dOkSqqqqoFKpcOrUKYwePRr/+Mc/AADz589HcnIyABjdf52MjAxcuXIFc+fOlZaVlZWhtLQUAODl5aWzvfZ8ZmYmdu3aBZlMBicnJ8ydOxcffvghXnjhhQafQduSJUuQlZWFrVu34rfffkNhYSFUKhUuXLiADh064LHHHgMADBkyBAcOHGhwTH788Uf07dsXQ4cOBQAMHDgQjzzyCP773/9i7NixeOGFF1BQUIBRo0Zh8eLF6NKli8Hl2lxcXJCamopvvvkGv/32G3766SeoVCq9n0GfyMhIafquu+5CUFAQvvrqK0yaNMlgmczMTMyaNQvOzs4AgNDQUKSmpkKtVkvbZGdnY/To0VLrICgoCN999x0AYPLkyVi6dCl+/PFH+Pj4QC6Xo0MHDlVaE5MAQaVS4bPPPoOTk5M0YKdUKrFjxw6Eh4fj/vvvR0ZGBvbv34/Y2FgAtyvl5557TuouUKvVuHXrlrTPukoBuF25Dxo0CGPHjsWTTz6J06dPQwgBBwcHCK1HVzk4OEjTxvavvd2MGTOwZMkSab6wsBDdunVrEEf9eY1GA5lMpjNfU1Ojd1ttr7zyCmpra/Hkk0/iscceQ0FBgfR5tPcHABcuXED//v119ldbW9tgOyEEampq8NBDD+Hw4cPIzs7G8ePHMWfOHGzdutXg8rpuOgC4du0agoKCEBgYCC8vL/j6+uLo0aN6P4M+27dvx8SJE9GrVy8pJkfHxqsIY8dQ+/PV0f6eH3/8cfznP//BsWPHkJ2djU2bNmH//v3SiQFZHlMu4cCBA3Bzc8O3336LI0eO4MiRI/j666+hUqmQnp6OwMBAbN26FRUVFdKZ9JgxY7Bv3z4olUoAwNtvv42YmJgG+y4rK8OZM2fw6quv4oknnsC1a9fw+++/Q6PRYMyYMcjOzsb169cB3O7br9PU/Y8ZMwaHDh1CYWEhgNstmrCwsCZ97jFjxmDHjh0QQkCtVmPv3r0YNWqU0XLfffcdIiMjMXXqVADA6dOnUVtbi/79+0MmkyErKwsAcO7cOYSFhUGj0eiUHzZsGC5duoQff/wRAHDx4kWcPHkS3t7eWL9+PVJSUjBp0iS89tpruPfee3Hx4kWDy7WdPXsW7u7ueOmllzBmzBgpAdTW1jbpeOTk5OC9994DAJSWlmLfvn3SZzRk7Nix+Pe//y21OLZv347hw4fDyclJ2mb06NHIysrCtWvXAACffvqptG7x4sX4/PPP8dRTT2HlypVwdXXF77//3qR4yTzYEiDs2rULzzzzjM4ZWteuXRESEoIPPvgAu3fvxqpVq/D8889L6+fMmYPr168jMDAQMpkMPXv2xNq1axvsu2vXrnjhhRfg7+8PZ2dn3HXXXXjkkUdw5coV+Pj4YPny5Xj22Wfh5OSEwYMH429/+5tJ+x8zZgyef/55hIeHQyaTwdXVFe+8806DM2195HI5EhMT4efnh+rqaowdOxYLFy40Wi46OhqRkZFwdnaGq6srhg8fjt9//x1OTk5ITk7GmjVrkJSUhI4dOyI5OVmnQgQAd3d3vP3220hISEBlZSVkMhneeOMNeHh4ICwsDMuWLcO0adPg5OSEQYMG4amnnsKtW7f0Ltc2evRo7Nu3D76+vpDJZPD29oa7uzuuXLli9DMBQFxcHOLi4vDUU0+hpqYGTz/9NEaPHt1omYCAABQUFGDOnDnQaDS45557sH79ep1tBg0ahCVLliAsLAwuLi546KGHpHUvvfQSXnvtNezZswcODg6YNGkShg8f3qR4yTxkQvBR0mQbV69exWeffYaXXnoJHTp0wJdffomtW7fqtAiIyLLYEiCb+cc//oHCwkL4+fnBwcEBXbp0kS4dJSLrYEuAiMiOcWCYiMiOMQkQEdkxJgEiIjvWpgaGc3JybB0CEVGbVP/u+TptKgkAhj9IU+Tl5WHw4MFmjMY8GJdpGJdpGJdp2mNcjZ1AszuIiMiOMQkQEdkxJgEiIjvGJEBEZMeYBIiI7FibuzqIiMjeBG3OhkqlwgELXLVk9iRQXV2N2NhYKBQKqNVqREREoFevXli5ciUcHBzQr18/vP766zpvD9JXZuLEieYOjYiI6jF7EkhLS4ObmxvWrVuHkpIS+Pv744EHHkBkZCTGjx+PxYsXIyMjQ3qDlaEyTAJEZO+CNmcDAE5cLtaZ3/Oij9n+htmTgK+vL6ZMmSLNOzg4YPDgwSgtLYUQAuXl5Q1eWaevDBERWZ7FHiWtVCoREREhvRlq9erVcHd3R5cuXbBjxw506tSp0TJ+fn4N1ufk5Bh872tTVFZWonPnzs0ubymMyzSMyzSMyzStJa6Y9HwAQJJvL8Sk50Oj0WD91N7N2pdKpTL8tAVhAfn5+cLf31988sknQgghRo4cKS5cuCCEEGLHjh0iPj7eaBl9Tp061aK4zp8/36LylsK4TMO4TMO4TNNa4gpMPSYCU49J09M2fN3sfTVWd5q9O6ioqAjh4eGIi4uDj8/tfqtu3brB1dUVANCjRw98//33RssQEdkjQ+MASb69LPL3zJ4EUlNTUVZWhpSUFKSkpAAAEhMTER0dDUdHR3Ts2BEJCQkAgJiYGERFRWHbtm0NymzdurVVNMmIiNozsycBuVwOuVzeYPnu3bsbLEtKSmq0DBGRPdG++qf+lUB5eXkW+Zu8Y5iIyI7xjmEiIhuzxv0AhrAlQERkx9gSICKyocbGAayBLQEiIjvGlgARkQ3YchxAG1sCRER2jC0BIiIrs/U4gDa2BIiI7BhbAkREVtJaxgG0sSVARGTH2BIgIrKSujP+1tACqMOWABGRFQRtzpYq/9aELQEiIitrDS2AOkwCREQW1BoHg7WxO4iIyI6xJUBEZAH1z/hbWwugjtmTQHV1NWJjY6FQKKBWqxEREYFevXph5cqVcHBwQL9+/fD666+jQ4e/GiEajQbx8fH4+eef4eTkhMTERNxzzz3mDo2IiOoxexJIS0uDm5sb1q1bh5KSEvj7++OBBx5AZGQkxo8fj8WLFyMjIwMTJkyQynz99ddQq9XYs2cPcnNzsXbtWrz77rvmDo2IyOJa+xhAfWZPAr6+vpgyZYo07+DggMGDB6O0tBRCCJSXl8PRUffP5uTkYOzYsQCAYcOG4ezZs+YOi4iI9DB7EnBxcQEAKJVKLFq0CFFRUZDJZFi9ejXeffdddOnSBSNGjNApo1Qq4erqKs07ODigpqamQbIAWvay5crKSou9rLklGJdpGJdpGJdpWhJXTHo+ACDJtxdiVCoAQPw4NwAtf1G8pY6XRQaGCwoKEBkZieDgYPj5+cHHxwc7d+7EwIEDsXPnTqxduxYrV66Utnd1dUV5ebk0r9Fo9CYAABg8eHCz48rLy2tReUthXKZhXKZhXKZpSVzOmaUAbtdT2tO2jisnJ8fgOrMngaKiIoSHhyMuLg4+Prf7wLp16yad6ffo0QPff/+9TplHHnkER48exdSpU5GbmwtPT09zh0VEZDFtbRxAm9mTQGpqKsrKypCSkoKUlBQAQGJiIqKjo+Ho6IiOHTsiISEBABATE4OoqChMnjwZWVlZmDt3LoQQWLNmjbnDIiKyiKDN2ThfUIb7e3a1dSjNYvYkIJfLIZfLGyzfvXt3g2VJSUnS9OrVq80dChGRVdzfs2ureEFMc/BmMSKiZtDXBdQWWwR8bAQRkZnUtQjaErYEiIhM1JreEdxSbAkQEdkxtgSIiJqoLV8KaghbAkREdowtASKiJmhP4wDa2BIgIrJjbAkQETWiPY4DaGNLgIjIjrElQERkQHsdB9DGlgARkR1jS4CIqJ72Pg6gjS0BIiI7xpYAEZGWmPR8ODs7t+txAG1sCRAR2TG2BIiI8Fe//5nrlQAq7aIVALAlQERk18zeEqiurkZsbCwUCgXUajUiIiJw8OBBFBUVAQAUCgWGDh2KjRs36pRZtmwZFAoFOnTogISEBAwYMMDcoRER6aV91u+38bA0JmAPzJ4E0tLS4ObmhnXr1qGkpAT+/v7IyMgAANy6dQuhoaFYvny5TplvvvkGNTU12L17N7KysvDWW28hOTnZ3KEREVE9Zk8Cvr6+mDJlijTv4OAgTScnJ2P+/Pno0aOHThkPDw/U1tZCo9FAqVTC0ZFDFURkefZ0P4AhZq9tXVxcAABKpRKLFi1CVFQUAODmzZvIzs5u0AoAAGdnZygUCjz55JMoKSlBamqqwf3n5eU1O7bKysoWlbcUxmUaxmUaxqVfTHo+LhVXob97J2mZSqWCRqOxr+MlLCA/P1/4+/uLTz75RFq2Y8cOkZKSonf7NWvWiPXr10tlJ0+eLCorKxtsd+rUqRbFdf78+RaVtxTGZRrGZRrGpV9g6jERmHqswbSt4zKkJXE1VneavSVQVFSE8PBwxMXFwcfnryZVdnY2IiIi9Jbp2rUrOnbsCADo1q0bampqUFtba+7QiIj0dgGdLyjD/T272jIsmzH7JaKpqakoKytDSkoKQkJCEBISgsrKSly+fBl9+vTR2TYmJgb5+flYsGABzp07h+DgYISFhSE6OhrOzs7mDo2ISK/7e3a1q3EAbWZvCcjlcsjl8gbLDx061GBZUlKSNP3222+bOxQiIh328GhoUxltCRw/ftwacRARkQ0YbQkkJydj5MiR1oiFiMgieCmoYUaTgEwmQ2RkJDw8PNChw+2GwyuvvGLxwIiIyPKMJoHZs2dbIw4iIovgOEDjjI4J+Pn5QaVS4ccff0RZWRmeeuopa8RFRERWYDQJxMXF4erVqxg9ejQUCoXeK3+IiFqboM3ZCNqcjROXi3HicjFbAQYY7Q66cuUKdu7cCQCYNGkS5s6da/GgiIhawt5vADOF0SRQVVWFiooK/O1vf0NlZSXv5CWiNqHuBjC2ABpnNAmEhYVhxowZGDhwIH755RcsWrTIGnEREZmMj4QwndEkcOedd2Lv3r24evUqevfuje7du1sjLiIis7DnR0I0RZNuFtu5cyfc3NysEA4RUfPwUtDm4c1iRER2zGgSmDlzps7bwYiIWhM+EqJljCaBzz//HNu2bbNGLERETVZX2VPLGE0CXbp0weHDh9GvXz+pO8jDw8PigRERNUXdGT9bAM1jNAkUFxfjgw8+kOZlMhk++ugjS8ZERGSQoe4fah6jSWD79u0681VVVRYLhoioudgCaB6Dzw6KioqSprXHBJ5//nmLBkREZIh2l88ID3eM8HDHnhd9mABawGBL4ObNm9J0RkYGwsPDAQBCiEZ3WF1djdjYWCgUCqjVakRERODgwYMoKioCACgUCgwdOhQbN27UKbd582YcOXIE1dXVmDdvHubMmdPsD0VERE3TpHcMa1f8Mpms0W3T0tLg5uaGdevWoaSkBP7+/sjIyAAA3Lp1C6GhoVi+fLlOmRMnTuCHH37Arl27UFFRwauRiEgHLwO1HINJQLuyN1bxa/P19cWUKVOkee17DJKTkzF//nz06NFDp8x3330HT09PREZGQqlUIiYmpsl/j4jaNz7/x7JkwkD/zqhRo+Dj4wMhBI4fPy5NnzhxAllZWUZ3rFQqERERgcDAQPj5+eHmzZsIDQ1FWlpag5vP5HI58vPzkZqaij/++AMRERFIT09vkHxycnLg7Ozc7A9bWVmJzp07N7u8pTAu0zAu07T1uGLS8wEASb69dKZtHZe1tSQulUoFLy8vvesMtgTeeustaVr7HQJNeZ9AQUEBIiMjERwcDD8/PwBAeno6pk2bpvfuYzc3N/Tv3x9OTk7o378/OnXqhOLiYtxxxx0Nth08eLDRv29IXl5ei8pbCuMyDeMyTVuNq67L58z1SgBAfGYpfrtVg/t7drXo52mrx6sxOTk5BtcZTALe3t7N+mNFRUUIDw9HXFwcfHz+6q/Lzs5GRESE3jJeXl746KOP8Mwzz6CwsBAVFRV8YB2RHTPUBcQngppfkwaGTZGamoqysjKkpKQgJSUFALB161ZcvnwZffr00dk2JiYGUVFRePzxx3Hy5EkEBARACIG4uDg+r4jIzvGlMNZh9iQgl8v1vof40KFDDZYlJSVJ0xwMJiK+FMb6jCaB69evS5d7TpkyBYMGDcLQoUOtERsREbuALMzgHcN1VqxYgdmzZ0OtVuPRRx/F66+/bo24iMjO8G5g2zCaBKqqquDj4wOZTCZduUNERO2D0e4gJycnfPvtt9BoNMjNzYWTk5M14iIiO8G7gW3LaEsgISEB+/fvR0lJCbZt24b4+HgrhEVE9qBu4Jdsx2hL4P3332/wsDciInPhpaC2ZTQJ/PrrrygrK0PXrrxEi4jMI2hzNlQqlXQ3MC8FtZ0mJYERI0bA3d1depbPd999Z/HAiKh9qqvw+3XTrX54KahtGE0CR48etUYcRGRH7u/ZFfHj3BCfWQqAXUC2ZDQJ5ObmYv/+/aiurgYAFBYW4r333rN4YETUvtS/CihGpZIeCEe2Y/TqoMTERHh7e0OpVKJXr158sBsRmczQVUDsArI9o0mga9eumDZtGlxdXfHPf/4T169ft0ZcRNTO1FX4dXcDJ/n2YgJoBYx2B8lkMly8eBEVFRW4dOkSbty4YY24iKgd4APhWj+jLYFly5bh4sWLCAkJwauvvop58+ZZIy4iaqfYBdS6GG0JDBw4EAMHDgQA7N+/3+IBEVH7oH3zF28Ea72MJoExY8ZI06WlpejTpw+++OILiwZFRETWYTQJaN8YplAo8M4771g0ICJq2/hAuLbFpDeL3X333bh06VKj21RXVyM2NhYKhQJqtRoRERE4ePAgioqKANxOJEOHDtX7PKKbN29i1qxZ2LZtGwYMGGBKaETUCnDgt+0xmgReeeUV6XERhYWFuOOOOxrdPi0tDW5ubtLbyPz9/ZGRkQEAuHXrFkJDQ7F8+fIG5aqrqxEXF4fOnTs342MQUWvBB8K1LUaTwNy5c6XpTp06YciQIY1u7+vriylTpkjz2i+MT05Oxvz589GjR48G5f71r39h7ty52LJlS5MCJ6LWg5eCtl1Gk0B+fr7O/OXLl6XpmTNnNtjexcUFAKBUKrFo0SJERUUBuN3Vk52drbcVsH//fri7u2Ps2LFMAkRtjKEKn5eCtg0yIYRobINFixZBpVLB29sbZ8+eRVFREby8vCCTyfDKK6/oLVNQUIDIyEgEBwcjICAAALBz506UlZUhIiKiwfZPP/00ZDIZZDIZ8vLy0K9fP7z77ru48847dbbLycmBs7Nzcz8rKisrW2V3E+MyDeMyjaXjikm/faKY5NtLZ9rWcTVXe4xLpVLBy8tL/0phxDPPPCM0Go3OfGNu3LghfH19xbFjx3SWR0ZGirNnzxr7c2L+/Pnil19+0bvu1KlTRss35vz58y0qbymMyzSMyzSWiisw9ZgITD0m7ll6UNyz9KAITD0mhqxMF4Gpx4wXtmBcLdUe42qs7jR6x3BxcTH+/PNPaVqpVDa6fWpqKsrKypCSkoKQkBCEhISgsrISly9fRp8+fXS2jYmJadDdREStHx8I134YHRNYuHAhZsyYATc3N6hUKqxatarR7eVyOeRyeYPlhw4darAsKSmpwbLt27cbC4mIWgFeBdQ+GE0Cvr6+mDRpEm7dumX08lAiat94FVD7Y7A76OrVq4iMjERNTQ1yc3Mxffp0TJ48Gbm5uVYMj4haC3YBtU8GWwJr1qxBQEAAHB0d8cYbbyApKQn33nsvXn31VXbZENkpdgG1PwaTgFqtxsSJE1FSUoJr165h9OjRAACNRmO14IjI9tgF1L4ZvTooOzsbI0eOBHA7AdRdKURE7R+7gNo/gy2BgQMHYvHixTh79iwSEhJQWFiIDRs2SAmBiOwDu4DaN4NJYOnSpcjMzMSLL74IT09P/Pzzz7jvvvsQEhJizfiIyIrqKvo67AJq/wwmAZlMhvHjx0vzgwYNwqBBg6wSFBG1PuwCap9Mep8AEbVP9Qd/R3i46/zLyr/9MjowTETtm6HBX7IPRlsCSqUSmZmZUKvV0jJ9j5AmoraLg7/2y2gSeOmll9CjRw/07NkTAKS3jBFR28br/wloQhIQQmD9+vXWiIWIrIQvgqE6RpPAoEGDcPr0aQwePFha5uTkZNGgiMjy2AVEQBOSwH//+18cOXJEmpfJZDh8+LBFgyIiy2AXENVnNAmkpaVZIw4isjB2AZE+RpPA4cOH8fHHH6O6uhpCCJSWluLAgQPWiI2IzIxdQFSf0SSwadMmrFixArt378aIESOQlZVljbiIyAyCNmfj7B8lGNK7lF1ApJfRJNC9e3c8/PDD2L17N2bNmoX9+/c3un11dTViY2OhUCigVqsRERGBgwcPoqioCACgUCgwdOhQbNy4sdEyEydObOFHIyJD2AVEdYwmgY4dO+LkyZOoqanBt99+ixs3bjS6fVpaGtzc3LBu3TqUlJTA398fGRkZAIBbt24hNDQUy5cvN1qGSYCo+eoPAANAl86OrPypAaOPjVi1ahVqamoQERGBvXv3YtGiRY1u7+vri5dfflmad3BwkKaTk5Mxf/589OjRo8lliMg0fAwEmUImhBDGNsrOzsbVq1fx0EMPwcPDA506dTK6Y6VSiYiICAQGBsLPzw83b95EaGgo0tLSDFby9cvUl5OTA2dn5yZ8LP0qKyvRuXPnZpe3FMZlGsbVuJj0fABAkm8vxKTnQ6PRYP3U3jaOqqHWcrzqa49xqVQqeHl56V1ntDtow4YNuHbtGn799Vd07NgRW7ZswYYNGxotU1BQgMjISAQHB0uVeXp6OqZNm2YwAegro4/2TWumysvLa1F5S2FcpmFc+tV1AZ25XgkAiM8sxW+3atCvmyOPlwnaY1w5OTkG1xntDsrJyUFSUhKcnZ3h7++PP/74o9Hti4qKEB4ejiVLliAgIEBanp2djXHjxplUhoiaprHXQCb59rJBRNRWGE0CtbW1qKqqgkwmQ21tLTp0aLxIamoqysrKkJKSgpCQEISEhKCyshKXL19Gnz59dLaNiYlBfn6+wTJE1HR1g74jPNwxwsMde1704SAwGWW0OygsLAyzZs1CcXEx5syZgwULFjS6vVwuh1wub7D80KFDDZYlJSU1WoaIDNO+3p/3AFBzGU0CTz75JEaNGoUrV66gd+/ecHd3t0ZcRNRMvAyUTGEwCdS/ll/bG2+8YZFgiMg43gNA5mQwCZw9exaVlZWYPn06Hn74YTThSlIisjB295C5GUwCBw4cwIULF5CWloYtW7Zg+PDhmD59Ou655x5rxkdE9fAhcGROjY4JeHp64tVXXwUAnDx5Em+++SauXbuGvXv3WiU4IuIAMFlWk140/9VXX+HgwYOoqKjA9OnTrREXERnBMQAyB4NJ4IsvvsChQ4eQn5+PJ554AqtWrULv3q3v1nOi9kjf2T/AAWAyP4NJIDo6Gv3798d9992HCxcu6Dz6+c0337RKcEREZFkGk8BHH31kzTiICLz8k6zPYBLw9va2ZhxEdo+DvWQLRgeGich6ePknWRuTAJEN8fJPsjWjTxElIuvjGABZC1sCRFZU181ThwPAZGtsCRAR2TG2BIisoP6lnyM83HX+5dk/2QpbAkQWZujVj0StAVsCRFbASz+ptTJ7EqiurkZsbCwUCgXUajUiIiJw8OBBFBUVAQAUCgWGDh2q8xgKjUaD+Ph4/Pzzz3ByckJiYiIfWU1tGi/9pLbC7EkgLS0Nbm5uWLduHUpKSuDv74+MjAwAwK1btxAaGtrgrWVff/011Go19uzZg9zcXKxduxbvvvuuuUMjsjle/UOtjdmTgK+vL6ZMmSLNOzg4SNPJycmYP38+evTooVMmJycHY8eOBQAMGzYMZ8+eNXdYRBYXk56P3/b8zid/Upti9iTg4uIC4PZ7CBYtWoSoqCgAwM2bN5Gdna333cVKpRKurq7SvIODA2pqauDo2DC8vLy8ZsdWWVnZovKWwrhM01rj0mg00NTWQqVSSctUKpW0zFYxt9bjxbhMY6m4LDIwXFBQgMjISAQHB8PPzw8AkJ6ejmnTpum0DOq4urqivLxcmtdoNHoTAAAMHjy42XHl5eW1qLylMC7TtKa4tPv5z91QAwCcnZ3RpXNNqzn7b03HSxvjMk1L4srJyTG4zuyXiBYVFSE8PBxLlixBQECAtDw7Oxvjxo3TW+aRRx5BZmYmACA3Nxeenp7mDouIiPQwe0sgNTUVZWVlSElJQUpKCgBg69atuHz5Mvr06aOzbUxMDKKiojB58mRkZWVh7ty5EEJgzZo15g6LyGwMvfXLpaMMQ3p3bxVn/0RNZfYkIJfLIZfLGyw/dOhQg2VJSUnS9OrVq80dChERGcGbxYiaoCnv/G2tfclEjeFjI4iI7BhbAkQGNOXsn6itY0uAiMiOsSVApIVn/2Rv2BIguxe0ObvBG7+I7AVbAkQAzheUIWhzNs/+ye4wCZBd0tftwxe/kD1iEiD6f/f37ColBp79k71gEiC70ZRBX44NkL1hEqB2zdTXObIFQPaGSYDaPQ76EhnGJEDtDgd9iZqOSYDsAgd9ifSzmyQQtDkbKpUKB/iUx3an/mAuB32Jms5ukgC1L6YO+Jq6LZG9aPdJoK6yqDs7bE7lQa1D3XcXP84NQMMB3xEe7jr/8jsmMs7sSaC6uhqxsbFQKBRQq9WIiIjAsGHDIJfLUVZWhtraWiQlJaFv3746ZZYtWwaFQoEOHTogISEBAwYMMHdo1A6cLyhDTLoKZ65XSvNE1HxmTwJpaWlwc3PDunXrUFJSAn9/f4wcORJ+fn6YOnUqjh8/jkuXLukkgW+++QY1NTXYvXs3srKy8NZbbyE5Odks8dSdDdaNCfDssG3Rd6XPpdpaaT0HfIlaxuxPEfX19cXLL78szTs4OOD777/H9evXsWDBAhw4cADe3t46ZTw8PFBbWwuNRgOlUglHx3bfS0WNMPZUz/7undClsyNGeLhjz4s+uL9nVytGR9S+mD0JuLi4wNXVFUqlEosWLUJUVBQUCgW6du2KDz74AD179sTWrVt1yjg7O0OhUODJJ5/EihUrEBISYu6wsOdFHyT59jL7fsk86lf82v39f1bWAIBU8Sf59tKp+Pe86MNWAFEzyYQQwtw7LSgoQGRkJIKDgxEQEIDRo0fj4MGD6N69O86fP4+NGzfqJII33ngDTk5OWLx4MQoKChAWFoYDBw6gU6dOOvvNycmBs7Nzs+OqrKxE586dm13eUhgXEJOej0vFVejv3knq73fpKEN59e2f54N3dZbWr37M3e6PlykYl2naY1wqlQpeXl5615m936WoqAjh4eGIi4uDj8/tszMvLy988803mDlzJk6ePIl7771Xp0zXrl3RsWNHAEC3bt1QU1ODWq1+X22DW3Cdf15eXovKW4q9xqXd319X8f92q0ZaP6R3d739/fZ6vJqLcZmmPcaVk5NjcJ3Zu4NSU1NRVlaGlJQUhISEICQkBEuXLsVnn32GuXPn4ttvv8XChQsBADExMcjPz8eCBQtw7tw5BAcHIywsDNHR0S0646fWy1h///09u7K/n8iKzN4SkMvlkMvlDZa///77DZYlJSVJ02+//ba5Q6FWov69GcYe6KadJNjXT2RZvAyHLMJYxd/Y9f2s+Imsh0mAzMaUip/X9xO1DmYfEyD7YuzSzvoVP/v7iVoXtgTIZC0542d/P1HrwiRAemk/elvfQ/dY8RO1D0wCJDH2hNX6T2RlxU/U9tllEtCu7Ozx0dKNff76Z/gPxv9H57EN2ljxE7V9dpkEDGnPCaGxz9bUyzfrD+Sy4idq++wqCcSk58M5s1TvWa72Iwzq5oGGZ8utMVE0NdbGzvL1de306+aIA9ETG+yHFT9R+8FLRP/f+YIy/FlZgxOXi6WEYIz25ZEtmTZH+cY+l6FLNrWZcvkmn9pJ1H7YVUsgybcXBte72kXfS0u0E4L22XJjLQdbaOzM3thZfv1+fH1dO3l5edI0EbVPdpUEDNGuEOsnBEOakiiaOq2prf3rkckmlm9Ki6WxAVxtrOyJ7I9dJgHtyq5+xdfYGbKpicIamnJm39gALit+Ivtml0nAkKZWiE2tcJs6nZeXh/jM0hbvyxyfjYjsC5OAAYbOlltjZdqWYiWi1oVJoAWaUvk2tYJuaXkioubgJaJERHaMSYCIyI6ZvTuouroasbGxUCgUUKvViIiIwLBhwyCXy1FWVoba2lokJSWhb9++OuU2b96MI0eOoLq6GvPmzcOcOXPMHRoREdVj9iSQlpYGNzc3rFu3DiUlJfD398fIkSPh5+eHqVOn4vjx47h06ZJOEjhx4gR++OEH7Nq1CxUVFdi2bZu5wyIiIj3MngR8fX0xZcoUad7BwQHff/89Bg0ahAULFuDuu+/Ga6+9plPmu+++g6enJyIjI6FUKhETE2PusIiISA+ZEEJYYsdKpRIREREIDAzEsmXLsHr1asyePRvvvPMOamtr8fLLL0vbyuVy5OfnIzU1FX/88QciIiKQnp4OmUyms8+cnBw4Ozs3O6bKykp07ty52eUthXGZhnGZhnGZpj3GpVKp4OXlpXedRS4RLSgoQGRkJIKDg+Hn54e1a9diwoQJAIAJEyZg48aNOtu7ubmhf//+cHJyQv/+/dGpUycUFxfjjjvuaLBvlUrVothaWt5SGJdpGJdpGJdp7CkusyeBoqIihIeHIy4uDj4+t69l9/LywjfffIOZM2fi5MmTuPfee3XKeHl54aOPPsIzzzyDwsJCVFRUwM3NrcG+DWUyIiJqHrN3ByUmJuKLL75A//79pWVr166FXC5HRUUFXF1d8eabb6Jbt26IiYlBVFQUevXqhaSkJJw4cQJCCERHR2Ps2LHmDIuIiPSw2JgAERG1fu3usREajQbx8fH4+eef4eTkhMTERNxzzz3S+iNHjmDTpk1wdHTE7NmzERgYaJW49N0/MXHiRGn9+++/j3379sHd3R0AsGrVKp3WlCXNnDkTXbp0AQD07t0bb7zxhrTOVsdr//79+PTTTwEAVVVVyMvLQ1ZWFrp2vf3+Blscr9OnT2P9+vXYvn07rly5gmXLlkEmk2HgwIFYuXIlOnT4695LY79DS8WVl5eHhIQEODg4wMnJCf/617/w97//XWf7xr5vS8V17tw5LFy4EP369QMAzJs3D1OnTpW2tdXxio6ORlFREQBAoVBg6NChDcYsLX289NUN9957r/V+X6Kd+c9//iOWLl0qhBDihx9+EAsXLpTWqdVqMWnSJFFaWiqqqqrErFmzRGFhoVXi2rdvn0hMTBRCCFFcXCzGjx+vs37x4sXizJkzVolFW2VlpZgxY4bedbY8Xtri4+PF7t27dZZZ+3ht2bJFTJs2TcyZM0cIIcSLL74ojh8/LoQQYsWKFeLLL7/U2b6x36El43r66afF+fPnhRBC7Nq1S6xZs0Zn+8a+b0vGtXfvXvHee+8Z3N5Wx6tOaWmpmD59urh+/brOcmscL311gzV/X+3usRE5OTnSeMKwYcNw9uxZad2vv/6Kvn37olu3bnBycoKXlxdOnTpllbh8fX11Lot1cHDQWX/u3Dls2bIF8+bNw+bNm60SEwD89NNPqKioQHh4OEJDQ5Gbmyuts+XxqnPmzBn88ssvCAoK0llu7ePVt29fJCcn6/x9b29vAMC4ceNw7Ngxne0b+x1aMq4NGzZg8ODBAIDa2lp06tRJZ/vGvm9LxnX27FlkZGTg6aefRmxsLJRKpc72tjpedZKTkzF//nz06NFDZ7k1jpe+usGav692lwSUSiVcXV2leQcHB9TU1Ejr6pp1AODi4tLgx2gpLi4ucHV1hVKpxKJFixAVFaWz/qmnnkJ8fDw+/PBD5OTk4OjRo1aJq3Pnznj22Wfx3nvvYdWqVXj11VdbxfGqs3nzZkRGRjZYbu3jNWXKFDg6/tV7KoSQ7mNxcXHBn3/+qbN9Y79DS8ZVV4l9//332LFjBxYsWKCzfWPftyXjeuihhxATE4OdO3eiT58+2LRpk872tjpeAHDz5k1kZ2dj1qxZDba3xvHSVzdY8/fV7pKAq6srysvLpXmNRiN96fXXlZeX61RyllZQUIDQ0FDMmDEDfn5+0nIhBMLCwuDu7g4nJyeMHz8e58+ft0pMHh4emD59OmQyGTw8PODm5oYbN24AsP3xKisrw6VLlzBy5Eid5bY8XnW0+2fLy8ulsYo6jf0OLe3zzz/HypUrsWXLFmnMpE5j37clTZ48GUOGDJGm639ftjxe6enpmDZtWoPWOWC941W/brDm76vdJYFHHnkEmZmZAIDc3Fx4enpK6wYMGIArV66gtLQUarUap06dwsMPP2yVuOrun1iyZAkCAgJ01imVSkybNg3l5eUQQuDEiRPS/zCWtm/fPqxduxYAcP36dSiVStx5550AbHu8AODkyZMYNWpUg+W2PF517r//fpw4cQIAkJmZiUcffVRnfWO/Q0v67LPPsGPHDmzfvh19+vRpsL6x79uSnn32Wfz4448AgOzsbDzwwAM66211vOriGTdunN511jhe+uoGa/6+2t3VQZMnT0ZWVhbmzp0LIQTWrFmDAwcOQKVSISgoCMuWLcOzzz4LIQRmz56Nu+66yypxpaamoqysDCkpKUhJSQEAzJkzBxUVFQgKCkJ0dDRCQ0Ph5OQEHx8fjB8/3ipxBQQEYPny5Zg3bx5kMhnWrFmDL774wubHCwAuX76M3r17S/Pa36OtjledpUuXYsWKFdiwYQP69+8vPS+r7t4Xfb9DS6utrcXrr7+Onj174p///CcAYPjw4Vi0aJEUl77v2xpn3PHx8UhISEDHjh3x97//HQkJCQBse7zqXL58uUHCtObx0lc3vPbaa0hMTLTK74v3CRAR2bF21x1ERERNxyRARGTHmASIiOwYkwARkR1jEiAismNMAkREdoxJgIjIjjEJELXQzz//jMcffxwff/yxrUMhMhmTAFELDRo0CBs2bMBnn31m61CITMYkQGQGd9xxBy5evGjrMIhMxiRAZAbr16+HWq2GQqGwdShEJmESIGqhzMxMVFRU4LHHHsMvv/xi63CITMIkQNQCVVVVWLduHVauXAlPT09cuHDB1iERmYRJgKgFUlJSMHPmTPTu3Ruenp4cF6A2h0mAqJkuXbqEY8eOISwsDACYBKhN4vsEiIjsGFsCRER2jEmAiMiOMQkQEdkxJgEiIjvGJEBEZMeYBIiI7BiTABGRHWMSICKyY/8HgNl6bKj1wmYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The optimial penalty parameter across all folds is  2.4242424242424243\n",
      "The average mse is  26.668487647243467\n"
     ]
    }
   ],
   "source": [
    "validation_score, param, e, beta_ridge_opt = cross_validation_ridge()\n",
    "# print(validation_score)\n",
    "print(\"The optimial penalty parameter across all folds is \", param)\n",
    "print(\"The average mse is \", e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we evaluate the performance of our ridge regression model by taking the mean square error of our predicted values and the actual values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_ridge_model(X_test, y_test, beta_ridge):\n",
    "    y_predicted = predict_with_estimate(X_test, beta_ridge)\n",
    "    error = mse(y_test, y_predicted)\n",
    "    return error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Average based cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The in sample MSE is 24.574589021414514\n"
     ]
    }
   ],
   "source": [
    "error = evaluate_ridge_model(X_train, y_train, beta_ridge_opt)\n",
    "print(f\"The in sample MSE is {error}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The out of sample MSE is 19.353333946743277\n"
     ]
    }
   ],
   "source": [
    "e = evaluate_ridge_model(X_test, y_test, beta_ridge_opt)\n",
    "print(f\"The out of sample MSE is {e}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fold based cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The in sample MSE is 24.699783342933188\n"
     ]
    }
   ],
   "source": [
    "e = evaluate_ridge_model(X_train, y_train, mean_opt_beta)\n",
    "print(f\"The in sample MSE is {e}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The in sample MSE is 19.431430488065356\n"
     ]
    }
   ],
   "source": [
    "e = evaluate_ridge_model(X_test, y_test, mean_opt_beta)\n",
    "print(f\"The in sample MSE is {e}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although there is not much difference between using either cross validation methods for the MSE, we will use the averaged based cross validation in future cross validation examples since it performs slightly better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2.2\n",
    "The average in-sample MSE is 24.57 while average out-of-sample MSE is 19.35. Again, out-of-sample MSE is lower than in-sample MSE. Ridge regression has a lower out of sample MSE, but higher in sample MSE compared to linear regression. This higher in sample MSE can be explained by the fact that the ridge regression is optimised with least squares with an additional penalty term, while linear regression is optimised with least squares. The penalty term allows the regression model to have less variance and be more robust to noise and outliers in the training set, hence resulting in a lower out of sample MSE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1.3: k nearest neighbours\n",
    "\n",
    "The kNN algorithm implemented calculates the euclidean distance between the given unseen data point x, to all other points in the training data set. The k closest points to this data point, x, are chosen. Since the distance metric used to select the k closest neighbours is unweighted, the average of the y values for these k neighbours is calculated, forming the predicted value of the data point x. The average of the k nearest points are taken since this is a regression task.\n",
    "\n",
    "We standardise the datas in both the test and train datasets prevent one feature with large values from overshadowing other features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate euclidean distance between two data points\n",
    "def euclidean_dist(m, n):\n",
    "    return np.sqrt(np.sum(np.square(m - n), axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_k_nearest_neighbours(x_test, X_train, k):\n",
    "    # compute distance from x_test to X_train\n",
    "    dist = euclidean_dist(x_test, X_train)\n",
    "    enumerated_row = enumerate(dist)\n",
    "    nearest_neighbours = sorted(enumerated_row, key=lambda x: x[1])[:k]\n",
    "    \n",
    "    # return indices of k_nearest_neighbours and euclidean distance sorted by distance\n",
    "    indices = [neighbour[0] for neighbour in nearest_neighbours]\n",
    "    dist = [neighbour[1] for neighbour in nearest_neighbours]\n",
    "    return indices, dist\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_k_validation(params_to_errors_map, title):\n",
    "    plt.plot(params_to_errors_map.keys(), params_to_errors_map.values(), marker='+')\n",
    "    plt.title(title)\n",
    "    plt.xlabel(\"$k$\")\n",
    "    plt.ylabel(\"Mean Square Error\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_validation_knn():\n",
    "    validation_score = defaultdict(int)\n",
    "    num_of_folds = len(folds)\n",
    "    \n",
    "    errors_for_last_fold = {}\n",
    "    \n",
    "    # perform cross validation across k folds\n",
    "    for i in range(num_of_folds):\n",
    "        # drop one fold as test\n",
    "        train_x, train_y, test_x, test_y = generate_test_train_sets(folds, i)\n",
    "        params = list(range(1, 25))\n",
    "        \n",
    "        for param in params:\n",
    "            # find k closest neighbours for each data point in test set\n",
    "            k_nearest_neighbours_of_test_data = [find_k_nearest_neighbours(test, train_x, param) for test in test_x]\n",
    "            \n",
    "            # find the average of the values of k closest neighbours for each data point\n",
    "            predicted_y = np.array([np.average(train_y[neighbours]) for (neighbours, _) in k_nearest_neighbours_of_test_data])\n",
    "            error_validation = mse(test_y, predicted_y)\n",
    "            validation_score[param] += error_validation\n",
    "            \n",
    "            if i == num_of_folds - 1:\n",
    "                errors_for_last_fold[param] = error_validation\n",
    "    \n",
    "    \n",
    "    # plot error distribution for last fold\n",
    "    plot_k_validation(errors_for_last_fold, \"MSE of Fold 5 across different k values\")\n",
    "    min_error_fold_5 = min(errors_for_last_fold, key=errors_for_last_fold.get)\n",
    "    print(\"Optimal k for fold 5 is\", min_error_fold_5)\n",
    "    #get average MSE for each hyperparam    \n",
    "    for param, error in validation_score.items():\n",
    "        validation_score[param] = error / num_of_folds\n",
    "        \n",
    "    # find optimal k value across all 5 folds\n",
    "    optimal_k = None\n",
    "    lowest_error = sys.maxsize\n",
    "    for param, error in validation_score.items():\n",
    "        if error < lowest_error:\n",
    "            optimal_k = param\n",
    "            lowest_error = error\n",
    "            \n",
    "    return validation_score, optimal_k, lowest_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAESCAYAAAD+GW7gAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA9mklEQVR4nO3dd3zT1f7H8Ve6QumktAWZLdCWsgSBggIiIAKyFFSWiIJcB1cZgiKCqAxRES6X5ZUf6JWliHgFAaUWkL2hCKS0ZRTKaAvdO02+vz9qIxVKOrLafJ6PBw+TNPl+Pz2GNycn53uOSlEUBSGEEFWeg7ULEEIIYRkS+EIIYSck8IUQwk5I4AshhJ2QwBdCCDshgS+EEHZCAt9GxMfHExISwvPPP3/Xz6ZOnUpISAjJyckAnDp1ipEjR9K/f3/69evHyy+/TExMjOH5ISEh9O/fn4EDBxb7Ex8fX6aaZs6cSffu3Vm4cGGxxw8fPkyrVq3uOv769evve7ypU6eycuXKux5PTk4mJCTknq/ZuXMnYWFhxc6TmZlZpt/DlqxcuZKpU6cC8N5773HgwAEAli9fzmOPPca7777Lpk2beOyxxxgzZozZ67l69SpvvPHGPX82cuRIfvnlF5Of01zHFcY5WbsA8Re1Ws2lS5e4du0adevWBSA7O5sTJ04YnpOfn88rr7zCqlWraN68OQA//fQTY8eOJSIiAkdHRwD++9//4uPjU6F6vvvuO3bv3k3t2rXv+lmDBg346aefKnT80jh58iSjR4/m1VdfNfu5LG3OnDmG2xs3bmT+/Pm0a9eOF154gYkTJzJw4ECz13D9+nUuXbpk9vMI2yCBb0McHR3p06cPW7ZsMQTcjh076NGjB6tWrQIgJyeHjIwMsrOzDa8bMGAA7u7u6HQ6Q+CXVkxMDB999BGpqamoVCpGjx7NU089xfDhw1EUhbFjxzJz5kzatWtX6mN+9913rF69GgcHB3x9fZkxYwaBgYHFnrNjxw4WLlyIq6srLVq0KPFYJ0+exMnJiW3btuHu7s7EiRNp3779Xc/buHEj3333HVqtlrS0NMaOHcvw4cMB+M9//sOPP/6Ik5MTDRs2ZN68eYSHh7Nx40ZycnJwd3dn9erVLF26lK1bt+Lo6EhgYCAzZszAz8+PHTt2sHz5clQqFY6Ojrz99tu0b9++xMfvpNVqmT17NgcOHKBmzZrUrFkTDw8PoLCnO2LECH755RcSEhJ477330Gq1pKSkEB8fT0pKCsOHD2f+/PkcPXoUnU5Hs2bNmD59Ou7u7nTv3p1WrVpx/vx5Jk2aRKtWrfjoo4+4ceMGWq2Wvn378uqrrxIfH8+LL75I165diYyMJD09nSlTptC9e3emT59OQkICY8aMueenL4CCggLeeustnJyc+OSTT3By+is2hg4dyksvvUSvXr0A+OyzzwAYN24cH3zwAXFxcaSmpuLm5sb8+fNp1KiR4bXx8fH079+fkydP3vP+999/z/r169Hr9Xh7ezNjxgwaN27MsWPHmDdvHnq9HoBXXnnFcH5hhCJswtWrV5XWrVsrf/zxh9K7d2/D46NGjVLOnz+vBAcHK7dv31YURVFWrVqltGrVSunevbsyefJk5fvvv1eys7MNrwkODlb69eunDBgwwPDn9ddfv+ucWq1W6dGjh/Lrr78qiqIoN2/eVLp06aKcOHHCcJyic97p0KFDSsuWLYsd/5VXXlEURVEOHDigPP7444bX/fDDD0qfPn0UvV6vvPPOO8r//d//KUlJSUrbtm2VmJgYRVEU5YsvvlCCg4Pv2S7jxo1Ttm/fruj1euXo0aNKWFiYcuPGjWLPyczMVJ577jklOTlZURRFOXnypNK6dWtFURTlt99+U5544gklNTVVURRFmTt3rrJs2TLlhx9+UNq3b69kZGQoiqIoGzduVIYMGaJkZWUpiqIo//73v5XRo0criqIoPXr0UE6ePKkoiqLs3btXWbx48X0fv9PXX3+tvPDCC0peXp6SlZWlPP3008o777yjKIqiPP/888r27dsVRVGUbt26KadPn77r8cWLFyvz5s1T9Hq9oiiK8vnnnyszZ840vGbJkiWGc40cOVKJiIhQFEVRcnNzlZEjRypbt25Vrl69qgQHBys7d+5UFEVRfvnlF+Wxxx4z/L/s27fvPdv++eefVzZv3qy8/vrryocffmio4U4bN25U/vGPfyiKoigFBQVK586dlUuXLinbt29XZs2aZXjejBkzlI8++qjY71f0ni9y5/3Dhw8rw4cPN7yv9+7da/h78cILLyg///yzoiiKotFolA8++OCe9Yu7SQ/fxrRo0QJHR0fOnDlDzZo1ycrKIjg4uNhzXnrpJZ599lmOHj3K0aNHWbFiBStWrGDjxo2G3mNphnQuX75MXl4eTzzxBAC1atXiiSeeYO/evbRp0+a+ry1pSGfv3r08+eSThnMPGjSIOXPmFPv+4Pjx4wQHB9OkSRMAhgwZwoIFC+55niVLlhhut2vXjjZt2rB//34GDx5seNzNzY0vvviC33//ncuXLxMVFWX4BHTw4EF69+6Nl5cXAO+++y4AmzZtIiQkBHd3dwD27NnDoEGDqF69OgAvvPACX3zxBfn5+fTt25d//vOfdO3alU6dOjF27FiAEh+/08GDB+nXrx8uLi64uLjQv39/zp8/f9+2vdPu3bvJyMgwjPVrtVpq1qxZrE2gcOjv6NGjpKWlsWjRIsNjUVFRtGrVCmdnZ7p27QpAs2bNSE1NLdX5P/nkE7KysggPD0elUt318yeffJJPP/2UpKQkzp07R0BAgOFP/fr1Wb16NXFxcRw5csToe+rvv3dcXBxDhw41PJaenk5qaip9+vTho48+YufOnTzyyCNMmjSp1Me1dxL4NmjAgAFs3rwZHx+fu8Zxjx8/zsmTJ3n55Zfp1q0b3bp1Y9KkSfTr14/9+/fTu3fvUp9Hp9Pd9ZdYURQKCgrKXXvRx2xjx1TuWMLpziGCO6Wnp7Nu3TpeeeUVQ52Kotz1/Js3bzJkyBCee+452rZtS+/evdm1axdQOEx25++Ynp5Oeno6gCHci+q+83l6vd5Q88SJExk8eDD79+9n06ZNrFq1io0bN5b4+P2UdchNr9czbdo0Q1hnZWWRl5dn+HnR76DX61EUhW+//RZXV1eg8MtwtVpNSkoKzs7OODgUztG4V3CXZMCAASiKwvTp0/niiy/u+rmrqyu9evXi559/5uTJkzz77LMArFu3jg0bNjBixAj69++Pt7f3XZMGVCpVsfeBVqst9nsPHDiQKVOmGO4nJibi5eXF0KFD6datG/v372fv3r0sWbKEX375BbVaXerfy17JLB0bNHDgQH755Re2bdtGv379iv3Mx8eH5cuXc+zYMcNjSUlJZGZm3vVJwJhGjRrh5OTEjh07AEhISODXX3/lkUceKXftXbp0Ydu2bYYZRT/88APe3t40bNjQ8Jz27dsTGxtLVFQUUNjbvhc3NzfWrl1rqO/cuXOcPn2aLl26FHvemTNn8PHx4fXXX6dz586GsNfpdDzyyCOEh4cbZvYsXryYr7/++p51//DDD4ZPBqtXr6Z9+/Y4ODjQvXt3cnJyGDZsGDNnzuT8+fPk5+eX+Pjfj/u///2PvLw88vLy2LZtW5nas3Pnzqxdu5b8/Hz0ej0zZsy456chd3d3WrduzVdffQUU/sM2bNgwIiIi7nt8R0fHYkH7d61atWLChAlcuXKFDRs23PM5zz33HD/++CMnTpwwjKXv27ePp59+mmeffZbAwEB27tyJTqcr9jpPT0+0Wi2xsbEAbN26tdjvvXXrVhITEwFYv349o0aNAgq/N9BoNAwaNIhZs2aRnp5OUlLSfX9PUUh6+DaoVq1aNG7cGA8PD7y9vYv9LDAwkKVLl7Jw4UJu3ryJWq3Gw8ODuXPnFvtCbNSoUYYeXZFJkyYZeooAzs7OLFu2jNmzZ7N48WJ0Oh3jxo2jY8eO5a69U6dOvPjii4waNQq9Xo+Pjw//+c9/itXi4+PD/PnzmTx5Ms7Ozvf8EhYKw+jO+hwdHVm4cOFdQ1WdOnVi48aN9O7dG5VKRVhYGD4+PsTFxdG1a1diY2MZNmwYAE2aNGHWrFmGf0SKPPPMM9y4cYNnn30WvV5Pw4YNmT9/Pk5OTkybNo3Jkyfj5OSESqVi7ty5uLi4lPj4nYYOHcqVK1fo16/fXf/wlcbrr7/OJ598wtNPP41OpyM0NNQwrfPv5s+fz6xZs+jfvz/5+fn069ePAQMG3Hc6bpMmTVCr1TzzzDN8//339+z9q9Vq5s2bx+jRo+nYsSMNGjQo9vOiYcjevXsbetmjR4/m/fffN3ziad26NdHR0cVe5+HhwZQpUxg7diw+Pj7FPp127tyZsWPHMnr0aFQqFe7u7ixZsgSVSsXkyZOZO3cu//rXv1CpVPzzn/+kXr16pWtQO6dSFFkeWQgh7IEM6QghhJ2QwBdCCDshgS+EEHZCAl8IIeyEBL4QQtgJm5qWefz4cWuXIIQQlVLbtm2NPsemAh8Ki9ZoNISGhlq7FKuTdigk7VBI2uEv0haFitqhtJ1lGdIRQgg7IYEvhBB2QgJfCCHshAS+EELYCQl8IYSwExL4QghhAxaGRxt/UgVJ4AshhAmVJ7gVRWFRRIwZqinO5ubhCyFEZbYoIoaJPQs3I9LrFW5n5ZOQnktiRi4J6XncTPvrdkJ6LgnpuWh1llmlXgJfCCFMIK9Ax5bIGwA8tXQ/iem5JGbkUaAvHuYqFdR0U1PLU01WXgG3Mv/aJS1gauGuX+N7BBn+0TAlCXwhhKiA25l5jFt3gkMXkw2PnbqaCkC7hjUY0LoO/h7VqOWpppZnNfw81Dg73j2aHjB1K5fn9TVrrRL4QghRDtEJGazad4kfT14jr0DPYyF+jOkcyMiVR8we3OUlgS+EEKWkKAq/Ryexct8l9sbcopqzA4Pb1mN0pwCa+HtU6NjjewSZqMqSSeALIUQJFoZHM7FnMLlaHZtOXGPV/kvEJmbi76FmSq8Qhoc1oIZb8Y3ryxvc5hiz/zsJfCGEKMGiiBh0eoW1h+NIydbSvI4nC4c8SN+WdXBxuvesdksEd3lJ4AshxJ/Sc7VEXk3l5JVUTl5JAWDp7lgeD63FmM6BdAj0QaVSWbnK8pPAF0LYhaLhmSIFOj3RCZmcvJrCqSupnLyaSmxi5l2vUxQIP5dAswc86diopiVLNjmTB75Wq2XatGlcu3aN/Px8XnvtNZo0acLUqVNRqVQEBQUxc+ZMHBzkIl8hRPmsOZXMnDLuf7IoIoZmdTwNvfc/rqWRna8DwMfNhTb1vRn4YB3aNKhBq/peeFZztshUSUsyeeBv3rwZb29vPvvsM1JSUnj66adp2rQpEyZMoEOHDrz//vtERETQs2dPU59aCGEHFEVhbWQqr/XOJjVbW/gnJ//P24X/TcnWkpaTT8qfj6VkawF4ZfVxnB1VNKvjxXPt6tOmgTdt6tegvo9rpR6qKS2TB37v3r3p1auX4b6joyNnz54lLCwMgEcffZT9+/dL4AshSi0tW8vO8wnsOJvAnugkADp/suuez63u4kiN6i54uTqTnqslPiWn2M+1OoXHgv1K9eWqJaZKWpLJA9/NzQ2AzMxM3nzzTSZMmMAnn3xi+NfTzc2NjIyMEl+v0WjIzc1Fo9GYurRKR9qhkLRDIXtrh6SsAg5dzeLglWxO38yhpOVmejZ2Z3BzbzzUDrirHXFxvHdPvc9/L7J9VKM/7+lK1Za962HTbV7W94RZvrS9ceMG48aNY/jw4fTv35/PPvvM8LOsrCw8PT1LfG1oaKhsUPwnaYdC0g6FqmI73PlFqqIoXEjK5NezCew4e5PI+DQAGvu58Y+ujenVvDat6nrh4KAq59j6xSrXfmXdxNzkgX/r1i1Gjx7N+++/z8MPPwxAs2bNOHz4MB06dGDPnj107NjR1KcVQlRCiyJi6Brix44/Q/7irSwAWtf35u3eITzRrDZN/N1Ncq6qNjxTHiYP/C+++IL09HSWLVvGsmXLAHjvvfeYPXs2CxYsoFGjRsXG+IUQ9qdAp+ebg3EADFp2ACcHFQ83rslLnQPpGVqL2l7V7vv6EQ96l/mctnxBlKWYPPCnT5/O9OnT73p8zZo1pj6VEKISmvx9JBuPxxd7rECv8FCDGozs2LBUx3i+tY85Sqvy5MIrIYRFJGbkMm9bFJtOXqOutysz+jXj1TXHq9Q8d1sngS+EMKsCnZ7Vh+JYsCOa3AId47o1Zly3JlR3kfixNGlxIYTZHLuczPT/nSHqZgZdgnz5cEBzGvn99SWsfJFqWRL4QgiTS8rIY972KH44Ec8DXtVYPuIhereofdfVrPJFqmVJ4AshTGJheDRvdG/C2sNXmL/jPLlaHa891pg3usvwja2Q/wtCCJNYFBFD+LkEzt1Ip3MTXz4c2JzGfqaZQy9MQwJfCFEhN9Jy+OzX8wCkZOezbMRD9LnH8I2wPgl8IUS5ZOcX8NJXRzl8Kdnw2I20XF5fe4LxPYJkfN4GSeALIcpEr1f48eQ1Pvv1PDfTc+nb8gGm9mlKl093yZx6GyeBL4QotSOXkpm99Ryn49NoVc+LxcPb0D5ArnqtLCTwhRBGXbmdzcfbNWw/c5MHvKqxcMiDDHywLg4Of43Ty5x62yeBL4S4S9Gyxem5WpbujOWr/ZdxdFAxqWcwY7s0wtXF8a7XyJi97ZPAF0LcZVFEDH4eahaGR5Ocnc/gh+oxpVcItTzvv4qlsG0S+EKIYo78Oetm+v/O0CHQhxn9mtGirpeVqxKmIIEvhAAKh3EWRcQUe+zwpWTCzyVI4FcREvhCCKBwDN7ZUcX8HdEAMsWyCnKwdgFCCNuQnV/Ayn2X6N7U39qlCDORHr4QAoD1R66Skq1lXLfGtJQhnCpJAl8IQV6Bji/3XKBjIx/aNiz8I6oeGdIRQrDpxDUS0vMY162JtUsRZiSBL4SdK9DpWb77Ag/W86JzE19rlyPMSAJfCDv38+kbXEnO5vVuTWRJ4ypOAl8IO6bXKyzbHUtwLXd6htaydjnCzCTwhbBj4ZoEohMyGdetSbGF0ETVJIEvhJ1SFIVlu2Jp4FOdvi0fsHY5wgIk8IWwU/tibxEZn8ZrjzXGyVGiwB7I/2Uh7NSSnbHU9qzGoIfqWrsUYSES+ELYoWOXkzl8KZmxjzZC7XT32vaiapLAF8IOLd0Vi4+bC8PC6lu7FGFBZgv8yMhIRo4cCYBGo+G5555j2LBhvPvuu+j1enOdVghhxJlraew6n8ToTgFUd5HVVeyJWQJ/xYoVTJ8+nby8PACWLFnCuHHjWL9+Pfn5+ezevdscpxVClMLy3RfwUDsx8uEAa5ciLMwsgd+gQQMWL15suB8aGkpqaiqKopCVlYWTk/QqhLCG2MRMtp25wciHG+Ll6mztcoSFmSXwe/XqVSzUAwICmDNnDn369OH27dt06NDBHKcVQhjxxe8XUDs5MLpzoLVLEVZgka72nDlzWLt2LUFBQaxdu5Z58+Yxc+bMez5Xo9GQm5uLRqOxRGk2TdqhkLRDoYq2Q0Kmlh9PxNOvqSdJVy+SZMLaLE3eE4XK2g4WCXwvLy/c3d0B8Pf358SJEyU+NzQ0FI1GQ2hoqCVKs2nSDoWkHQpVtB3W/e8MDg4qpj7Vjge8XE1YmeXJe6JQUTscP368VM+3SODPnj2biRMn4uTkhLOzM7NmzTL5ORaGRzOxZ7DJjytEVZCYkct3x64y+KF6lT7sRfmZLfDr1avHhg0bAGjXrh3ffvutuU6FoigsioiRwBeiBCv3XqJAp+fVro2tXYqwoipx4dXy3y9YuwQhbFZqdj5rDsXRr1UdAnzdrF2OsKJKPT9yYXg0iyJiDPcDpm4FYHyPIOntC/Gnrw9cJitfx+vdpHdv7yp14E/sGczEnsGsPniZGT+d5ch7PfD3qGbtsoSwGZl5BSzffYHHQ2vRtLantcsRVlYlhnT8PNQA3MrIt3IlQtiWdYfjyCvQM05694JSBP6hQ4csUUeF+LoXBn5SZp6VKxHCNuj0Cr+cucmXey4C0KZBDStXJGyB0SGdxYsX07FjR0vUUm5FPfykDAl8Yd8ycrV8fyyef/0WTXpugeFx+X5LQCkCX6VSMW7cOAIDA3FwKPxAMGnSJLMXVhZFPfxb0sMXdupqcjZfH7jMhqNXycgroF3DGozpHMgTzWvTeNo2Ls/ra+0ShQ0wGviDBw+2RB0V4qZ2orqLo/TwhV1RFIXjcSms3HeJX8/exEGlom+rBxjdKZAH63tbuzxhg4wGfv/+/fnuu++IjY0lICCAYcOGWaKuMvPzUEsPX1Rpa04lMycUtDo92/64wap9l4iMT8PL1ZlXujbmhYcb3vMq2vE9gqxQrbBFRgP//fffx9PTk06dOnHkyBGmT5/Op59+aonaysTXXS09fFGlrY1Mpe4DsXxzII6b6bk08nVj1lMtGPxQ3ftuZCJj9qKI0cCPi4tj7dq1ADz++OMMHTrU7EWVh5+7mou3Mq1dhhBm8cuZmwB8+st5OjWpydxBLXgs2B8HB5WVKxOVidHAz8vLIycnB1dXV3Jzc9HpdJaoq8x8PVw4fEl6+KJq+fvV5AD7Y2/TrqEP3ZvWslJVorIyGvijRo1i4MCBBAUFERsby5tvvmmJusrMz70aKdlatDo9zo5V4noyIWhd3xtnRxXN6ngReTVVZtuICjEa+H5+fmzYsIGrV69Sr149atSwzQs4fD1cALidmU9tL1leQVR+e6KTeGXNcUJqe/DN6DAe/HCHtUsSlZzRrvDixYvx9vamZcuWNhv2UDiGD3LxlagaDsTeYuw3x2js586aMR3wcnVmxIPe1i5LVHJV4sIrAF8PufhKVA2HL95mzH+PEVDTjbUvd8C7euGn1+db+1i5MlHZGQ38p556CkdHR0vUUiHSwxdVwbHLybz09VHqeFdjzcsd8HFzsXZJogoxGvjbtm1j1apVlqilQgzr6UgPX1RSJ66k8OJXR6ntWY31Yzsa3tNCmIrRwPfw8CAiIoKAgADDkE5gYKDZCyuras6OeKidpIcvKqXT8amMWnWEmu4urBvbEX9PmXggTM9o4CcnJ/P1118b7qtUKr755htz1lRusryCqIzOXEtj5MojeLk6s25sR5llJszGaOCvXr262P28PNsNVFleQVQ2UTfTGbnyMG4ujqwf25G63nevhSOEqZQ4LXPChAmG23eO4Y8dO9asBVWE9PBFZbEwPJqYhAxGrDiM2smR9f/oSH2f6tYuS1RxJQb+7du3Dbd3795tuK0oilkLqghfdxfp4YtKYVFEDMNWHMbBQcW6sR1oWNPN2iUJO1CqNQjuDHmVynYXa/LzUJOeW0Cu1jbX+xEC4NKtrD9vKawf24FGfu5WrUfYjxID/85gt+WQv1PRzle3s2Qzc2F7FoZHEzB1K93m7wbgVmY+jy/Yw8LwaOsWJuxGiV/axsbG8tZbb6EoSrHbFy5csGR9ZXLn3rby5ZewNYMfqseGY1fJ1epIydbKQmjC4koM/H/961+G23eugW+r6+HDX4F/S8bxhY25mpzNsBWHyNHqWPtyB/r+e5+1SxJ2qMTADwsLs2QdJlE0pCNX2wpbcjU5m6FfHiIzr4C1L3egeR0v2XZQWEWVWji+pnvhuiPSwxe24lpqDsNWHCIjV8uaMR1oUdcLkG0HhXUYvfCqMlE7OeLl6iw9fGETrqfmMPTLg6TlaFn7cgda1vOydknCzhnt4SckJDB58mTGjBnDhg0biIyMLNWBIyMjGTlyJFA4p/+1115jxIgRDB06lCtXrlSs6vuQi6+ELbiRlsPQLw+RmlXYs29Vz9vaJQlhPPBnzJjB4MGDyc/Pp127dsyZM8foQVesWMH06dMNyzB89tln9O/fn7Vr1zJhwgQuXrxY8cpLIBdfCWu7mZbLsC8PkZyVzzdjwniwvre1SxICKEXg5+Xl8fDDD6NSqWjUqBFqtfElWxs0aMDixYsN90+cOEFCQgIvvvgiW7ZsMesXwn4e1STwhdUkpOcybMUhbmXm89/RYbRpYLu7xAn7Y3QM38XFhb1796LX6zl16hQuLsY3ZOjVqxfx8fGG+9euXcPT05Ovv/6aJUuWsGLFCsaPH3/P12o0GnJzc9FoNGX4Nf7imJ9JQnpOuV9vSyrSDlVJZWmH5OwC3vn1BrezC5jd8wGqZ99Eo7lpsuNXlnawBGmLQmVtB6OBP2vWLD755BNSUlJYtWoVH3zwQZmL8vb2pnv37gB0796dhQsXlvjc0NBQNBoNoaGhZT4PQHBCLP/TpBPQOBhXF9vfqet+KtIOVUllaIfEjFz++eUhknP1fPNyR9oHmH47wsrQDpYibVGoqB2OHz9equcbDfyvvvrqvgFdGm3btuX333/nqaee4ujRozRp0qRCx7ufoq0Ob2XmyeqDwiLmbD3H7vNJ3EjL5euXwswS9kKYgtEx/AsXLpCenl6hk7zzzjv89NNPDB06lL179/Lqq69W6Hj3U7SZeaKM4wsLuJWZx4q9l4hPyWHVi+0JC5SwF7bLaA//woULdOjQAR8fH8Miavv2Gb8svF69emzYsAGAunXr8tVXX1Ww1NK5s4cvhLlcTc5m/ZErbDh2FYCVL7ajY6OaVq5KiPszGvi7du2yRB0mc+cCakKYUoFOz86oRNYevsLv0UnFfjZ8xWEAxvcIkqtohc0yGvinTp1i06ZNaLVaABITE1m5cqXZCysvHzcXVCrp4QvTuZmWy7dHr/DtkavcTM+llqea8T2CGNK+PnW8XQmYulVWvhSVgtHAnz17Ni+++CK//vorwcHB5Ofb9lrzzo4O1KguF1+J8lkYHs3EnsHo9Qp7Y2+x9lAcEVGJ6PQKjwb78eHA5vRo6o+TY5VahkrYCaOB7+npSb9+/di/fz9vvPEGzz//vCXqqhA/2cxclNOiiBiqOTuy7kgcV5NzqOnmwtgujRgWVr/EbQhl5UtRWRgNfJVKRUxMDDk5OVy8eJGkpCRjL7E6WU9HlJVOr/Dej38A8MkvUXRs5MOUXk3p1bwWaqf7X88hY/aisjAa+FOnTiUmJoaRI0cyefJkhg0bZom6KsTX3YXjV7KMP1EICodxFkXEFHvs0MVkOgTWRP1gHStVJYTpGQ38oKAggoIKP7Ju2rTJ7AWZgp+HmlsZ+SiKUmn24xXW81KnAL45eJmgWh4cuZQsX8CKKsto4Hfu3NlwOzU1lfr167N9+3azFlVRvu5qcrQ6svJ1uKur1JL/wgwWhEeTlqPlg/7NefLfe61djhBmYzQN77zI6tq1ayxZssSsBZnCnXvbSuCL+zl3PZ01h+J4vmNDmtXxlC9gRZVWprlldevWNeta9qYie9uK0lAUhQ82n8XL1ZlJf37xKl/AiqrMaPd30qRJhnHwxMREata0/cvH5WpbURqbI69z5HIyc59uiXd148t+C1HZGQ38oUOHGm6r1WpatGhh1oJMwVfW0xFGZOUVMHebhhZ1PRnSvr61yxHCIowG/vXr14vdv3TpkuH2U089ZfKCTMHHzQUHlfTwRcmW7IolIT2PZSPa4uggM7mEfTAa+Dt37iQ7O5uwsDDOnDnDrVu3aNu2rU1Pd3R0UFHTXS6+Evd26VYW/7f3IoMeqkvbhrIFobAfRgM/MzOTlStXGgJ+9OjRvPXWW2YvrKJ8ZXkFUYJZP59D7eTI1D5NrV2KEBZldJZOcnIyGRkZhtuZmZlmL8oU/DzUJGXa9kJvonTWnEo22bF2RiWwMyqR8T2C8PeoZrLjClEZGO3hv/rqqwwcOBBvb2+ys7P58MMPLVFXhfm6u3AhsXL84yTub21kKnNMsKJHXoGOj7aco7GfG6MeCaj4AYWoZIwGfu/evXn88cdJS0urFFMyixT28PNkeYVKTFEUPtxyDoDtf9ygT8sHKnS8/9t7icu3s1k9JgwXJ1neWNifEt/1V69eZdy4cRQUFHDq1CkGDBhAz549OXXqlAXLKz8/dzX5BXrScwusXYooh4Xh5wl8dxtfH7gMwGtrTxAwdSuf/RpVruPdSMthyc5YejWvRZcgPxNWKkTlUWLgz507l0GDBuHk5MTHH3/Mp59+ypo1a/j8888tWV+5ycVXlZuiFP531MMNARjbJRCAXVFJXEwq+1Dd3G1R6BWF6X2bmaxGISqbEgM/Pz+fHj16kJKSws2bN+nUqRO1atVCr9dbsr5yk83MK6+lu2L5985YhrSrz8z+zQF4r28zVr3YjhtpOfRbvI//nbxW6uMdunibLZHXebVrY+r7VDdX2ULYPKMDmQcPHqRjx44A6PV6w4wdW+crPfxKaeW+S3z263kGtq7D3EEtcXBQMeJBbwC6N63FtvFdaFHHiwnfneLtjZFk599/yK5Ap+eDzWep6+3Ka481tsBvIITtKjHwg4KCeOutt1i0aBFDhgwhMTGRadOmGcLf1kkPv/JZeziOWT+fo0+L2nz+7IOGK2Cfb+1jeM4DXq6sG9uBN7o34fvj8Qxcsp/zN0vuhKw9fIWomxnM6BdKNef771wlRFVXYuC/8847DBgwgMWLFxMWFkZKSgpNmzblnXfesWR95ebl6oyTg0p6+JXED8fjmf6/M3QL8WPR0Db33STcydGBt54IYc2YDqRkaxmwZB/fHrmCUjTw/6fbmXl8vuM8nZv40qt5bXP/CkLYvBKnZapUKrp27Wq4HxISQkhIiEWKMgUHBxW+srxCpfDz6etM2RjJI41rsvz5tqWeMtmpiS/bx3dh4nenmLrpD/ZfuM3cp1vgUc0ZgPk7zpOdr+ODAc1kaq4QlGIefmXm6+EiPXwbF34ugQnfnqJtwxqseKFdmYdd/DzUfDM6jOW/X2BBeDSn41NZOvwh9IrC+iNXeblzIE38PcxUvRCVS5UOfD93NbdkeQWb9Xt0EuPWnqB5HU9Wvdie6i7lezs6OKgY160JYYE+vLn+JIOWHaCWV+F3OOMflx2shChSqsXT9uzZQ37+X8Fpq8si/52vuxrNjcoxq8jeHLxwm398c4zG/u78d3SYYRimItoH+LDtzS5M/j6SiKhEAJMcV4iqwmjgv/766/j7+/PAA4WXtVemsVA/j8IxfL1ewUHWPLcJC8OjeTTYjzH/PUp9n+qsGRNm0t2mvj5w2RD2AAFTtwIwvkeQbF8o7J7RwFcUhfnz51uiFpPz81BToFdIy9FSw022sLMFiyJiWLXvEv4eata93IGaf06fNZWJPYMNwR4wdSuX5/U16fGFqMyMTocICQkhMjKS/Px8w5/SiIyMZOTIkcUe27JlC0OGDClfpeUgm5nbFs2NdAA8XZ1ZO7Yj/p6yPLEQlmS0h3/kyBF27txpuK9SqYiIiLjva1asWMHmzZtxdXU1PKbRaNi4ceNdc6XNqWg9nVsZeQTXkpka1vL5jvMs3hlruH8tNYdO83aafZhlfA/5wlaIOxkN/M2bN5f5oA0aNGDx4sW8/fbbAKSkpDB//nymTZvGjBkzyl5lOUkP3/pOXU0l/FwCAL2a1+LXswkWG2aRMXshijMa+BEREaxbtw6tVouiKKSmprJly5b7vqZXr17Ex8cDoNPpeO+995g2bRpqtfHxWo1GQ25uLhqNppS/Qsky83UAnI29QrA6vcLHszRTtYM1ZGv1fHMymc2adHxcHZn+WC06NXTj17OU+XeqzO1gStIOf5G2KFTWdjAa+EuXLmXGjBl8++23dOjQgf3795epoLNnzxIXF8cHH3xAXl4esbGxzJkzh/fee++ezw8NDUWj0RAaGlqm89yLoii4bLiKys3bJMezNFO1g6XtjEpg+tYz3EjP5fmODZnSOwTPP6dHju/hSGho2XrelbUdTE3a4S/SFoWK2uH48eOler7RwK9RowZt2rTh22+/ZdCgQWzatKlMBbVq1YqtWwunxsXHxzNp0qQSw97UVCpV4dTMDLn4yhISM3L5cMs5tp6+QXAtdzYOf5i2DX2KPUeGWYSwHqOB7+zszNGjRykoKGDv3r0kJSVZoi6T8XV3kTF8M9PrFTYcu8rcbRpytXre6hnMK10byzaCQtgYo4H/4YcfcvHiRV577TUWLVrEm2++WaoD16tXjw0bNhh9zNz8PNRcS8216DntxcLwaAa0rsO7m/7gyKVkOgT6MHdQSxr7uVu7NCHEPRgN/Fq1anHx4kVOnDjBuHHjCAwMtERdJuPnoSYyPs3aZVQ5+QV6FkXEsHz3Bao5O/DJ4JY827a+XNEshA0zGvgLFizg5s2bXLhwAWdnZ7788ksWLFhgidpMwtddze3MPHR6xbChhqiYtBwtw1ccAuCJ5rV4v38z/D3kIiohbJ3RQdbjx4/z6aefUr16dZ5++mnDdMvKws9DjV6BlGz54tYU5v96ngc/3MHZ64XTXH8+fYOwOREsDI+2cmVCCGOM9vB1Oh15eXmoVCp0Oh0ODpXrizjDxVcZeYbbonwURSEhvfD7kPnPPsjk7yNlrRohKhGj6T1q1CgGDRpETEwMzz77LMOHD7dEXSZjWF5BZupU2NJdsXx/PJ43ewTxTNt61i5HCFFGRnv4ffr04ZFHHiEuLo569erh4+Nj7CU25c4evii/n05dY/6OaJ5uU5eJf24qImvVCFG5lBj47777bokv+vjjj81SjDkU9fAl8Mvv6OVkpnx/mrBAH+YNbmnYE0EuohKicikx8M+cOUNubi4DBgygTZs2Fl3l0pTcXBxxdXaUIZ1yunQri7HfHKNeDVe+HNkWtVPZ9pwVQtiOEsfwt2zZwtKlS8nLy+PLL7/k1KlTNGjQgC5duliyvgpTqVSymXk5JWfl89JXR3BQqfjqpfYm3ZlKCGF59x3DDw4OZvLkyQAcPXqUzz//nJs3b1r8atmKks3Myy5Xq+Mf3xzjelou68d2oGFNN2uXJISooFJtYh4eHs7PP/9MTk4OAwYMsERdJuXrribudra1y6g09HqFKRtPcywuhaXDH7prATQhROVUYuBv376drVu3cv36dZ544gk+/PBD6tWrnFPx/DzUHI9LsXYZlcbn4efZEnmdd3o3pW+rB6xdjhDCREoM/IkTJ9KoUSOaNm1KdHQ0CxcuNPzs888/t0hxpuLrriY5O58CnR4nx8p14ZilbTh6laW7LjAsrD6vdm1k7XKEECZUYuB/8803lqzDrPw81ChK4ZeQsnF2yfbF3GLaj3/QJciXjwa2MEy/FEJUDSUGflhYmCXrMKuii68SM/Ik8Esw/X9/8NPJ6zT2c2fpiIdwlk9CQlQ5dvG32nDxlczFv6fEjFzWHLpCNRdHVr3U3rAdoRCiarGLwPcvWk9H5uLfJVer4+X/HgNg1aj21PV2tXJFQghzsYvAN6ynY8Uevi0uH7wwPJqmM37h9J8bxPRfso+AqVttslYhRMUZnYdfFbi6OOKudrLaZuZFu0PZ2toz/+zehB9OxOPvoebElVRZ6liIKs4uevhgvc3MFUVhwJJ9Fj9vaWyJvE58Sg6vP9bE2qUIISzALnr4UPjFraXH8BeGR7MoIsZwP2DqVqBwWWFr9/b1eoVluy/QtLYH3Zv6y1LHQtgBuwl8X3c1MYmZFj3nxJ7BODmo+PzPMfGf3+hMi7peFq2hJOGaBGITM1k0tDUODiqr/wMkhDA/uxnS8fNQW2WJ5N80CTTyLVx4bPf5RIuf/14URWHZrlga+FSnb0tZOkEIe2E3ge/rriY1W0tegc5i50xIzyUyPo3Bbevh76Fm1/kki537fvbH3iYyPo1XuzaWpSaEsCN287e96OKr2xZcJvk3TQIAPZvVYlhYA05eSSEly/rLNC/bHYu/h5rBbetauxQhhAXZT+C7W34z89/OJdDApzpB/u50a+qPXoE9Mdbt5Z+4ksKBC7cZ26WR7F4lhJ2xm8D3tfDetll5Bey/cJuezWqhUqloVdeLmm4u7Iyy7jj+sl0X8HJ1ZniHBlatQwhheXYT+EVDOpbq4e+NSSK/QM/jobUAcHBQ0TXEj9+jk9DprbM/8PmbGfymSeClTgG4qe1mgpYQ4k92E/g13Qr3Y7VUDz/8XCJers60C6hheKxbiD+p2VpOXbXOZizLd8dS3cWRFx8JsMr5hRDWZTeBX83ZEc9qThbZ21anV9gZlUC3EL9iyww/GuSHo4OKXVGWH8e/cjubLadvMKJDA9mMXAg7ZbbAj4yMZOTIkQBoNBqGDx/OyJEjGTNmDLdu3TLXae/L10NtkR7+iSsppGRr6dmsdrHHvao707ZBDauM4/9nzwUcVSpe7iK7WAlhr8wS+CtWrGD69Onk5RWG65w5c5gxYwarV6+mZ8+erFixwhynNcrPXW2R9XTCzyXg7Kji0WDfu37Wrak/526kczMt1+x1FElMz+X7Y/EMbluPWrIBjBB2yyyB36BBAxYvXmy4v2DBAkJDQwHQ6XSo1WpznNYoXwutp/PbuQQ6NqqJxz02EunW1A+w7FW3K/ddokCvlz1qhbBzZpmq0atXL+Lj4w33/f39AThx4gRr1qxh7dq1Jb5Wo9GQm5uLRqMxeV1O2ixupmWb5dhF4tPyuXgri96Nq93zPIqi4OfmyOZjF3jQI+u+xzJFO2Tk6fjmwBUeDXAnO/EKGttY3aFMzPV+qGykHf4ibVGorO1gsbl527ZtY/ny5Xz55Zf4+PiU+LzQ0FA0Go3hE4EpBd+M5SdNOoFNgqnmbJ6Ljvb8fgGA57s9SJ0Sdo/qeV7HTyev0TgoBBenkj9kmaIdFv0WQ06BwtSBbWha27NCx7IWc70fKhtph79IWxQqaofjx4+X6vkWmaXz008/sWbNGlavXk39+vUtccp7Krra1pxf3IafS6B5Hc8Swx4Kp2dm5es4ejnZbHVA4cVfXx24xOOh/pU27IUQpmP2wNfpdMyZM4esrCzeeOMNRo4cyb///W9zn/aezH3x1e3MPI5fSTFcbFWSTk1q4uLowC4zz9ZZf+QKqdlaXu8mG5wIIcw4pFOvXj02bNgAwJEjR8x1mjLxNXMPf2dUIopSuFja/VR3caJDIx92nk9ker9mZqklr0DHir0X6djIh4ca1DD+AiFElWc3F17BnT1881x89ZsmgQe8qtG8jvHhk+5N/bmYlEXc7ft/cVteP564RkJ6HuOkdy+E+JNdBX5Nd/Mtr5Cr1bEn+haPhxYulmZMt5DCmUvmGNYp0OlZ/vsFWtb1onOTu68FEELYJ7sKfGdHB2pUdyYp0/QXPR24cIscrY7HjQznFAnwdaORr5tZNkXZduYmcbezGdetcan+8RFC2Ae7Cnwo2szc9EM64ecScVc70bFRyVNO/+6xEH8OXrxNdn6Byeoo2r6wsZ8bT/xtaQchhH2zu8D3NcPyCnq9QoQmga7BfmXaVKR7U3/yC/QcvHDbZLWM//YkUTczeO2xJjg4SO9eCPEXuwt8c2xmfvpaGokZeTzezL9Mr2sfWIPqLo4mW0xNURQ2R96grrcrA1vXMckxhRBVh90Fvq+76VfM/O1cAo4OKsMXsaWldnKkUxNfdp9PQlEqvinKwYuFnxRe6dqo2LLMQggBdhj4fh5qsvN1ZOWZbtz8N00C7RrWKNc6892b+nMtNYeYxMxyn39heDQBU7cyfMVhAN7/6SwBU7eyMDy63McUQlQ9drfPne8dm5mbYpu/q8nZRN3MYHrf8q3r8VhI4eqZO6MSCa7lUa5jTOwZTF6Bni/+XMfn8ry+5TqOEKJqs8sePphueYXwcwmA8atrS/KAlyuhD3hWaD5+5NVUvtxzgSHtrLdOkRDC9tlhD9+0F1/9pkkgyN+dhjXdyn2MbiF+/GfPRdJytHi53r2G/v3kFeh4e+Np/DzUTOsbSm0v2eBECHFvdtvDN0Xgp2VrOXwpudQXW5Wke1N/dHqFfTFl3/px6a4LnE/I4ONBLfFydWZiz+AK1SKEqLrsLvBruqlxUEGSCdbT2R2diE6vlHs4p0jr+t54uTqXeXrm2etpLNsVy6A2denetGI1CCGqPrsLfEcHFT5uLibp4YefS8DX3YXW9bwrdBwnRwceDfbj9+hE9PrSTc/U6vS8vfE03tVdeL+/eVbcFEJULXYX+FA4U6eiX9rmF+j5/XwSPZrWMskVrd2b+nErM58/rqWV6vn/+f0CZ6+nM/upFuWaDiqEsD92Gfh+HhW/+OrIpWQy8goqPH5f5NEgP1Qq2FWKzc2jEzL4d0QsfVs9QO8Wsl6OEKJ07DPwTdDD/02TQDVnB5MtP1zTXU3r+t5Gp2cW6PRM+T4S92pOfDSguUnOLYSwD3YZ+L5/9vDLu5yBoiiEn0ugcxM/XF1Mtxl6txB/IuPT7vvpY+W+S0TGp/HBgObU/PMiMiGEKA27DHw/dzV5BXoyy7m8guZGBtdSc+hZxsXSjOnetPB4e6LvvUb+haRMPg+P5olmtejf6gGTnlsIUfXZZeD7elTs4qvfNAmoVJh8KmSzBzzx81Cz8x7j+Dq9wjsbT+Pq7Mjsp1rIxiZCiDKzy8D3cy+8GrUigd+6vrfhIi5TcXBQ0S3Ejz3RSRTo9MV+9s3ByxyLS+H9fs3w95SraYUQZWefgV+BzcxvpuVyOj6twhdblaRbiD8ZuQUcj0sxPBZ3O4tPfznPYyF+DHqorlnOK4So+uwy8P9aT6fse9v+pvlzsbRQ8wR+pyBfnBxUhr1u9XqFqT/8gZODio8HtZShHCFEudll4Neo7oKjg6pcPfyiwG/i727qsgDwrOZM+wAfw/TMdUeucPDibab1DeUBL1eznFMIYR/sbrVMKBwrr1nG5RWy8gr436lrHIgt3FXKnD3tbk39mLstigX7FA5czaFzE1+Gtpelj4UQFWOXgQ+l39s2JiGDNYfiWHfkClrdX/P2A6ZuBWB8jyCTr1DZvak/c7dFEX4hk+oujjKUI4QwCbsNfF93NUklBL5Wp2fH2QRWH7rMoYvJuDg60LflA4x8uCEPNahB4LvbzLqrVGM/d+rVcCU+JYepfZpS36e62c4lhLAfdhv4fh5qjsclF3vsRloO6w9fYf3RqyRl5FGvhivv9G7Kc+3qWeyq1oXh0SyKiDHcf/+ns7z/01mzfJIQQtgXuw78zDwder3CgQu3WX3oMr9pEtErCo8F+zHy4YZ0DfbH8R4rYY7vEWS2uib2DDYEe8DUrbI/rRDCZMwW+JGRkcyfP5/Vq1cTFxfH1KlTUalUBAUFMXPmTBwcrDtBqGgz826f7ybudjY+bi6M7dKIER0aGB1CkZ62EKIyMkvqrlixgunTp5OXVzhG/vHHHzNhwgTWrVuHoihERESY47SlsjA8moCpW5n18zkA4m5nAzAsrL7NjZePeNDb2iUIIaoQswR+gwYNWLx4seH+2bNnCQsLA+DRRx/lwIED5jhtqUzsGczleX259PGTAFye15fL8/oypVdTq9VUkudb+1i7BCFEFWKWwO/VqxdOTn+NFimKYphW6ObmRkZGhjlOWyYyzVEIYW8s8qXtneP1WVlZeHp6lvhcjUZDbm4uGo3G7HWNeNDbIucpL0u1g62Tdigk7fAXaYtCZW0HiwR+s2bNOHz4MB06dGDPnj107NixxOeGhoai0WgIDQ01e11zzH+KCrFUO9g6aYdC0g5/kbYoVNQOx48fL9XzLTJV5p133mHx4sUMGTIErVZLr169LHFaIYQQdzBbD79evXps2LABgMDAQNasWWOuUwkhhCgFu1wtUwgh7JEEvhBC2AkJfCGEsBMqRVEU40+zjNJ+0yyEEKK4tm3bGn2OTQW+EEII85EhHSGEsBMS+EIIYSdsZj18vV7PBx98wPnz53FxcWH27Nk0bNjQ2mVZzVNPPYWHhwdQeE3Dxx9/bOWKLMvWl9e2lDvb4ezZs7z66qsEBAQAMGzYMJ588knrFmhmWq2WadOmce3aNfLz83nttddo0qSJ3b0f7tUOtWvXLvP7wWYC/7fffiM/P5/vvvuOU6dOMW/ePJYvX27tsqyiaFnp1atXW7kS61ixYgWbN2/G1dUV+Gt57Q4dOvD+++8TERFBz549rVyl+f29Hc6dO8dLL73E6NGjrVyZ5WzevBlvb28+++wzUlJSePrpp2natKndvR/u1Q7jxo0r8/vBZv5ZPH78OF26dAGgdevWnDlzxsoVWU9UVBQ5OTmMHj2aF154gVOnTlm7JIuy5eW1Lenv7XDmzBl2797NiBEjmDZtGpmZmVaszjJ69+7N+PHjDfcdHR3t8v1wr3Yoz/vBZgI/MzMTd3d3w31HR0cKCgqsWJH1VKtWjTFjxrBy5Uo+/PBDJk+ebFdtURmW17aEv7dDq1atePvtt1m7di3169dn6dKlVqzOMtzc3HB3dyczM5M333yTCRMm2OX74V7tUJ73g80Evru7O1lZWYb7er2+2JvdngQGBjJgwABUKhWBgYF4e3uTlJRk7bKspizLa1dlPXv2pEWLFobb586ds3JFlnHjxg1eeOEFBg4cSP/+/e32/fD3dijP+8FmAv+hhx5iz549AJw6dYrgYPvdN3bjxo3MmzcPgISEBDIzM/Hz87NyVdZTtLw2wJ49e2jXrp2VK7KOMWPGcPr0aQAOHjxI8+bNrVyR+d26dYvRo0czZcoUnnnmGcA+3w/3aofyvB9s5sKrolk60dHRKIrC3Llzady4sbXLsor8/Hzeffddrl+/jkqlYvLkyTz00EPWLsui4uPjmTRpEhs2bODSpUvMmDEDrVZLo0aNmD17No6OjtYu0SLubIezZ88ya9YsnJ2d8fX1ZdasWcWGQaui2bNns337dho1amR47L333mP27Nl29X64VztMmDCBzz77rEzvB5sJfCGEEOZlM0M6QgghzEsCXwgh7IQEvhBC2AkJfCGEsBMS+EIIYSck8IUQwk5I4AshhJ2QwBfiPg4cOMCUKVOsXYYQJiGBL8R9REVF0axZM2uXIYRJSOALcR9RUVGEhoaSn5/P1KlTWbBgAXJxuqis7HM5SiFKKSoqipo1azJmzBieeeYZBg4caO2ShCg3WUtHiBJotVo6duxInTp1+Oijj2jTpo21SxKiQmRIR4gSXLhwgZYtW+Lk5FTlV2MU9kECX4gSREVF0aZNGxYsWMC7777LrVu3rF2SEBUigS9ECaKioggKCiIwMJDJkyczYcIEtFqttcsSotxkDF8IIeyE9PCFEMJOSOALIYSdkMAXQgg7IYEvhBB2QgJfCCHshAS+EELYCQl8IYSwExL4QghhJ/4fMaZZ3sbsSigAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimal k for fold 5 is 2\n",
      "The optimal value for k is 2\n"
     ]
    }
   ],
   "source": [
    "validation_score, optimal_k, lowest_error = cross_validation_knn()\n",
    "print(f\"The optimal value for k is {optimal_k}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.1\n",
    "In general, the MSE increases as the number of neighbours increases past 5. In addition, the MSE is generally higher at extreme ends (1 and 25). When the number of neighbours considered is 1, the MSE is high because the model is not robust to noise or outliers and is unable to take into account other data points in the vicinity. On the other hand, when the values of k neighbours that are considered becomes very high, we might be considering data points that are irrelevant to the current data point. This is exacerbated by the fact that the distance metric is unweighted and all neighbours are considered equally regardless of their distance to the data point in consideration. Hence, an appropriate number of neighbours, such as 2, 3 or 5 gives a considerably low MSE, with 2 neighbours giving the optimal MSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predict the value of y giveen the training dataset\n",
    "def knn_predict(X_test,dataset, labels, optimal_k):\n",
    "    test_neighbours = [find_k_nearest_neighbours(x_test, dataset, optimal_k) for x_test in X_test]\n",
    "    predicted_y = np.array([np.average(labels[neighbours]) for (neighbours, _) in test_neighbours])\n",
    "    return predicted_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test acc:  13.76362745098039\n",
      "train acc:  3.970810643564357\n"
     ]
    }
   ],
   "source": [
    "test_set_pred = knn_predict(X_test, X_train, y_train, optimal_k)\n",
    "test_set_accuracy = mse(test_set_pred, y_test)\n",
    "train_set_pred = knn_predict(X_train, X_train, y_train, optimal_k)\n",
    "train_set_accuracy = mse(train_set_pred, y_train)\n",
    "print(\"test acc: \", test_set_accuracy)\n",
    "print(\"train acc: \", train_set_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.2\n",
    "A summary table of the out of sample MSE and the in sample MSE for the different models is shown below: \n",
    "<br>\n",
    "<br>\n",
    "\n",
    "|Table    |In Sample MSE | Out of Sample MSE|\n",
    "|:-----:|:-----:|:-----: |\n",
    "|**Linear Regression** |24.37 |19.54 |\n",
    "|**Ridge Regression**|24.69 |19.54 |\n",
    "|**K Nearest Neighbours**|3.97|13.76|\n",
    "\n",
    "The in sample MSE for kNN is much lower than both linear and ridge regression. This is probably due to the fact that the data point itself is considered to be one of the nearest neighbours as the distance to itself is 0. Hence, this causes the predicted value for the data points to be much closer to the actual value.\n",
    "\n",
    "The kNN model also performs better than both linear and ridge regression for out of sample test data, as shown by its lower MSE. This might be because the kNN algorithm is a local model and is able to give a better prediction when the data itself is not entirely linear separable. The model generalises well to data points that do not \"fit\" nicely on a linear line, and hence would also be able to better predict the outliers in the dataset that do not follow the pattern of a line. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2: Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to Task 1, we import the classifiation train and test data, standardise them, and also split the training dataset into 5 different folds for cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(SEED_VALUE)\n",
    "classification_train_data = np.loadtxt(\"./classification_train.csv\", delimiter=\",\")\n",
    "classification_test_data = np.loadtxt(\"./classification_test.csv\", delimiter=\",\")\n",
    "train_pd = pd.read_csv(\"./classification_train.csv\")\n",
    "np.random.shuffle(classification_train_data)\n",
    "np.random.shuffle(classification_test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "classification_train_X = classification_train_data[:, :-1]\n",
    "classification_train_y = classification_train_data[:, -1]\n",
    "classification_test_X = classification_test_data[:, :-1]\n",
    "classification_test_y = classification_test_data[:, -1]\n",
    "\n",
    "# again, we standardise the data set to allow each feature to have values over similar ranges. This is a good practice.\n",
    "classification_train_X, mean, sigma = standardize(classification_train_X)\n",
    "classification_test_X, _, _ = standardize(classification_test_X, mean, sigma)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "classification_folds = split_data(classification_train_data, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task 2.1: Logistic Regression\n",
    "\n",
    "For Logistic Regression, we conduct linear regression on our dataset and pass the predicted values from this linear model through the logistic function. This gives us the probability, p of the data point belonging to a certain class. If p is greater than the decision threshold, we predict the data point to be in class 1. Otherwise, the data point will be in class 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model learns parameters $\\boldsymbol \\beta$ and $\\boldsymbol \\beta_0$ and pass the output of this function through the logistic function. The output of this operation is $\\hat{\\boldsymbol y}_{\\text{log}}$:\n",
    "\n",
    "$$\n",
    "\\hat{\\boldsymbol y}_{\\text{log}} = f(\\boldsymbol X \\boldsymbol\\beta+ \\beta_0)\n",
    "$$\n",
    "where $\\boldsymbol X = [X^{(1)}, X^{(2)}, \\dots, X^{(n)}]$, and $X^{(i)} \\in \\mathbb R^d$.\n",
    "\n",
    "$\\hat{\\boldsymbol y}_{\\text{log}}$ gives the probability of the given data point belonging to one of the two classes. A value of 1 is assigned to this data point if the probability is > $\\tau$, a decision threshold. Otherwise, this value is 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logistic_fn(x):\n",
    "    return 1 / (1 + np.exp(-x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predicts value of ylog given a test dataset, X, and optimal values of beta and beta_0\n",
    "def predict_log_reg(X, beta_optimal, beta_0_optimal):\n",
    "    \"\"\"\n",
    "        Obtain the predicted value using beta optimals and pass \n",
    "        this outcome through the logistic function.\n",
    "    \"\"\"\n",
    "    y_log = logistic_fn(X @ beta_optimal + beta_0_optimal)\n",
    "    return y_log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the lectures, the cost function for logistic regression is \n",
    "$$\n",
    "\\mathcal L = - \\frac{1}{n} \\sum_{i=1}^n y^{(i)} \\log(\\hat{y}_{\\text{log}}^{(i)}) + (1-y^{(i)}) \\log (1-\\hat{y}_{\\text{log}}^{(i)}) \\, .\n",
    "$$\n",
    "\n",
    "By using gradient descent to optimise the cost function, we take the derivatives with respect to $\\boldsymbol \\beta$ and $\\boldsymbol \\beta_0$, giving:\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathcal L}{\\partial \\boldsymbol \\beta^{(i)}} = \\frac{1}{n} \\sum_{i=1}^n (  \\hat{y}_{\\text{log}}^{(i)} - y^{(i)} ) X^{(i)}\n",
    "$$\n",
    " \n",
    "$$\n",
    "\\frac{\\partial \\mathcal L}{\\partial \\beta_0} = \\frac{1}{n} \\sum_{i=1}^n ( \\hat{y}_{\\text{log}}^{(i)} - y^{(i)})\n",
    "$$\n",
    "\n",
    "$\\boldsymbol \\beta$ and $\\boldsymbol \\beta_0$ are updated using the formulas: \n",
    "\n",
    "$$\n",
    "{\\boldsymbol \\beta^{(i)}} = {\\boldsymbol \\beta^{(i-1)}} - \\alpha * \\frac{\\partial \\mathcal L}{\\partial \\boldsymbol \\beta}\n",
    "$$\n",
    "\n",
    "$$\n",
    "{\\boldsymbol \\beta_0^{(i)}} = {\\boldsymbol \\beta_0^{(i-1)}} - \\alpha * \\frac{\\partial \\mathcal L}{\\partial \\boldsymbol \\beta_0}\n",
    "$$\n",
    "\n",
    "where $\\boldsymbol \\alpha$ is the learning rate.\n",
    "\n",
    "$\\boldsymbol \\beta$ and $\\boldsymbol \\beta_0$ are initialised with zeros. Both $\\boldsymbol \\beta$ and $\\boldsymbol \\beta_0$ are continuously updated for every iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# given number of features, d, initialise a 1D beta vector with shape (d,) with zeros as values. \n",
    "# initialise beta_0 to 0 as well\n",
    "def initialise(d):\n",
    "    beta = np.zeros(d, dtype=np.float64)\n",
    "    beta_0 = 0\n",
    "    return beta, beta_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "#performs a forward pass on the data by predicting y_log. Returns the loss value, and the derivatives of the loss with respect to  beta and beta_0\n",
    "def forward(X, y, beta, beta_0):\n",
    "    n = X.shape[0]\n",
    "    # predict y_log\n",
    "    y_predicted = predict_log_reg(X, beta, beta_0)\n",
    "\n",
    "    # calculate the difference between y_predicted and y_actual\n",
    "    y_diff = y_predicted - y\n",
    "    \n",
    "    # calculate gradients\n",
    "    grad_beta = (y_diff @ X) / n\n",
    "    grad_beta_0 = np.sum(y_diff) / n\n",
    "\n",
    "    # cost function\n",
    "    cost = (-1) * np.mean(y * np.log(y_predicted) + (1 - y) * np.log(1 - y_predicted), axis=0)\n",
    "    \n",
    "    grads = {}\n",
    "    grads[\"beta\"] = grad_beta\n",
    "    grads[\"beta_0\"] = grad_beta_0\n",
    "    return cost, grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The actual optimisation is conducted by updating $\\beta$ and $\\beta_0$ with a learning rate ot $\\alpha$ using gradient descent for 5000 iterations. The optimal learning rate is obtained through cross validation. The default value is set to 0.01."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def optimise_beta(X, y, beta, beta_0,learning_rate=0.01,num_of_iterations=5000, print_cost=False):\n",
    "    costs = []\n",
    "    \n",
    "    for i in range(num_of_iterations):\n",
    "        # calculate cost and gradients\n",
    "        cost, grads = forward(X, y, beta, beta_0)\n",
    "        \n",
    "        # update beta\n",
    "        beta = beta - learning_rate * grads[\"beta\"]\n",
    "        beta_0 = beta_0 - learning_rate * grads[\"beta_0\"]\n",
    "        \n",
    "        #record the cost \n",
    "        if i % 500 == 0:\n",
    "            costs.append(cost)\n",
    "            if print_cost:\n",
    "                print(\"cost after iteration {}: {} \".format(i, cost))\n",
    "\n",
    "    params = {\"beta\":beta, \"beta_0\":beta_0}\n",
    "    return params, costs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The labels for the test set is predicted after the $\\beta$ parameters are calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_class(X, beta, beta_0, decision_threshold=0.5):\n",
    "    #compute vector y_log that predicts possibilities of data point belonging to class 1\n",
    "    y_log = predict_log_reg(X, beta, beta_0)\n",
    "    \n",
    "    # if y_log > decision threshold, classify data point as 1, otherwise 0\n",
    "    y_log[y_log > decision_threshold] = 1\n",
    "    y_log[y_log <= decision_threshold] = 0\n",
    "    \n",
    "    return y_log\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the task is classification, we can use accuracy as a metric to determine the performance of our model. Accuracy is calculated using (TP + TN) / (Total number of data points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_accuracy(y_actual, y_predicted):\n",
    "    return 1 - np.mean(np.abs(y_predicted - y_actual))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform cross_validation on given learning_rate and threshold and returns \n",
    "# the average accuracy across all folds\n",
    "def log_cross_validation(learning_rate, threshold):\n",
    "    accuracy = 0\n",
    "    n = len(classification_folds)\n",
    "    # perform cross validaton across 5 folds\n",
    "    for i in range(n):\n",
    "        # initialise beta\n",
    "        beta, beta_0 = initialise(classification_train_X.shape[1])\n",
    "        \n",
    "        train_X, train_y, test_X, test_y = generate_test_train_sets(classification_folds, i)\n",
    "        # optimise beta using sgd\n",
    "        params, costs = optimise_beta(train_X, train_y, beta, beta_0, learning_rate)\n",
    "        \n",
    "        # predict y_values and calculate accuracy of our model\n",
    "        y_test_predicted= predict_class(test_X, params[\"beta\"], params[\"beta_0\"], threshold)\n",
    "        accuracy += calc_accuracy(test_y, y_test_predicted)\n",
    "    \n",
    "    average_acc = accuracy / n\n",
    "    return average_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "#performs grid search across the various hyperparameter combinations and choose the one giving highest accuracy\n",
    "def grid_search(grid):\n",
    "    opt_combi = None\n",
    "    max_acc = 0\n",
    "    for combi, acc in grid.items():\n",
    "        if acc >= max_acc:\n",
    "            opt_combi = combi\n",
    "            max_acc = acc\n",
    "    return opt_combi, max_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "#finds optimal hyperparameters for logistic regression model\n",
    "def tune_hyperparameters(learning_rates, thresholds):\n",
    "    # perform grid search on hyperparameters\n",
    "    grid = {}\n",
    "    \n",
    "    # loop through both parameters to obtain average accuracy for each combination\n",
    "    for rate in learning_rates:\n",
    "        for threshold in thresholds:\n",
    "            # perform cross validation for each learning rate, threshold pair and take the average accurae\n",
    "            average_accuracy = log_cross_validation(rate, threshold)\n",
    "            #store resultant accuracy in grid\n",
    "            grid[(rate, threshold)] = average_accuracy\n",
    "    \n",
    "    opt_combi, max_acc = grid_search(grid)\n",
    "    return opt_combi, grid      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We run the model to obtain the best hyperparameters for our dataset. \n",
    "The learning rates are defined as "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def model(X_train, y_train, X_test, y_test, num_of_iterations=5000):\n",
    "    beta, beta_0 = initialise(X_train.shape[1])\n",
    "    learning_rate = [0.1, 0.01, 0.02, 0.05, 0.001, 0.002, 0.0001]\n",
    "    decision_threshold = np.linspace(0.1, 0.9, num=9)\n",
    "    \n",
    "    # perform cross validation to obtain the optimal learning rate and decision threshold\n",
    "    (optimal_learning_rate, optimal_decision_threshold), grid = tune_hyperparameters(learning_rate, decision_threshold)\n",
    "    print(\"best learning_rate\", optimal_learning_rate)    \n",
    "    print(\"best threshold\", optimal_decision_threshold)\n",
    "    \n",
    "    # train the final classification model over the entire training set using the optimal hyperparameters \n",
    "    params, costs = optimise_beta(X_train, y_train, beta, beta_0, optimal_learning_rate,num_of_iterations, print_cost=True)\n",
    "    \n",
    "    # predict y_log for the test set and training set\n",
    "    y_pred_test = predict_class(X_test, params[\"beta\"], params[\"beta_0\"], optimal_decision_threshold)\n",
    "    y_pred_train = predict_class(X_train, params[\"beta\"], params[\"beta_0\"], optimal_decision_threshold)\n",
    "    \n",
    "    # calculate mean accuracies for both the test and train set to determine the performance of our model\n",
    "    accuracy_test = calc_accuracy(y_test, y_pred_test)\n",
    "    accuracy_train = calc_accuracy(y_train, y_pred_train)\n",
    "    print(\"accuracy of model on test set\", accuracy_test)\n",
    "    print(\"accuracy of model on train set\", accuracy_train)\n",
    "    return {\"costs\":costs, \"accuracy_grid\": grid, \n",
    "            \"learning_rate\":learning_rate, \"decision_threshold\": decision_threshold}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best learning_rate 0.1\n",
      "best threshold 0.5\n",
      "cost after iteration 0: 0.6931471805599452 \n",
      "cost after iteration 500: 0.5303239866632354 \n",
      "cost after iteration 1000: 0.5303210871508574 \n",
      "cost after iteration 1500: 0.530321071766102 \n",
      "cost after iteration 2000: 0.5303210716655677 \n",
      "cost after iteration 2500: 0.5303210716649102 \n",
      "cost after iteration 3000: 0.5303210716649058 \n",
      "cost after iteration 3500: 0.5303210716649057 \n",
      "cost after iteration 4000: 0.5303210716649057 \n",
      "cost after iteration 4500: 0.5303210716649058 \n",
      "accuracy of model on test set 0.74\n",
      "accuracy of model on train set 0.745\n"
     ]
    }
   ],
   "source": [
    "d = model(classification_train_X, classification_train_y, classification_test_X, classification_test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.2\n",
    "Compare the performance of your optimal model on the training data and on the test data by\n",
    "their mean accuracies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The in sample mean accuracy is 74.5, while the out of sample mean accuracy is 74. The model performs slightly better on the train set than the test set. This is to be expected since the model is fitted on the training set. However, in question 2.2.2, we explore the reasons why accuracy is not the best metric to measure the performance of this model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2.2 Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random forest is an ensembling method that aggregates and randomisies a list of decision trees to reduce overfitting of the model and to allow our model to become more robust to noise. We create B decision trees to constitute the forest by sampling B samples of size N from our training dataset, with replacement. To build each decision tree, we split the dataset on each feature descriptor and find the feature descriptor that gives us the maximum information gain when we split on that feature. This information gain can be calculated by cross entropy. We continue splitting on other features until all features have been used, or when the maximal depth of the tree has been reached. Each leaf in the tree gives us a probability pair that makes up the probability of that leaf predicting a certain class. We then use the various trees in the forest to predict a value for the test data point and sum up the probabilities across all these trees. The higher probability would dictate the class that the data point is assigned to. There are several hyperparameters that we have to define before we train our model, namely number of trees, depth of tree and the number of features to be chosen as a potential splitting point at each split. The optimal hyperparameters will be chosen again by 5-fold cross validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross Entropy loss\n",
    "\n",
    "The cost function that we want to optimise for random forests is cross entropy. Entropy is defined as the average amount of information of the current state of the dataset. Cross entropy has the formula:\n",
    "\n",
    "$$\n",
    "CE[\\pi(R_\\alpha)] = \\sum_{i=1}^n  \\pi_n(R_\\alpha)log(1 - \\pi_n(R_\\alpha))\n",
    "$$\n",
    "given region $R_\\alpha$, across all the classes n\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "# return value of cross entropy using equation above\n",
    "def cross_entropy(y):\n",
    "    \"\"\" \n",
    "    Calculate the cross entropy for labels.\n",
    "    Arguments:\n",
    "        y: vector of training labels, of shape (N,).\n",
    "    Returns:\n",
    "        (float): the cross entropy for y.\n",
    "    \"\"\"\n",
    "    \n",
    "    count = Counter(y)\n",
    "    N = len(y)\n",
    "    probability = np.zeros(len(count.keys()) + 1)\n",
    "    \n",
    "    for label in count.keys():\n",
    "        probability[int(label)] = count[label] / N\n",
    "    \n",
    "    # if the calculated probability is 1, this number is set to a number slightly smaller than 1 to overcome the np.log error\n",
    "    altered_arr = np.where(probability == 1, 0.9999, probability)\n",
    "    cross_ent = np.sum(probability * np.log(1 - altered_arr))\n",
    "    return cross_ent\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_dataset(X, y, column, value):\n",
    "    \"\"\"\n",
    "    Return the split of data whose column-th feature equals value.\n",
    "    Arguments:\n",
    "      X: training features, of shape (N, D).\n",
    "      y: vector of training labels, of shape (N,).\n",
    "      column: the column of the feature for splitting.\n",
    "      value: the value of the column-th feature for splitting.\n",
    "      sample_weights: weights for each samples, of shape (N,).\n",
    "    Returns:\n",
    "      (np.array): the subset of X whose column-th feature equals value.\n",
    "      (np.array): the subset of y whose column-th feature equals value.\n",
    "    \"\"\" \n",
    "    ret = []\n",
    "    featVec = X[:, column]\n",
    "    X = X[:,[i for i in range(X.shape[1]) if i!=column]]\n",
    "    \n",
    "    # get the indices of rows with column having value equal to value passed in as parameter\n",
    "    for i in range(len(featVec)):\n",
    "        if featVec[i]==value:\n",
    "            ret.append(i)\n",
    "    \n",
    "    # returns a subset of X whose column-th feature equals value\n",
    "    sub_X = X[ret,:]\n",
    "    sub_y = y[ret]\n",
    "\n",
    "    return sub_X, sub_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Information gain is defined as the difference between the initial entropy and the weighted average entropy of the produced subsets. We want to choose the feature that gives us the maximum information gain after the split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the information gain by splitting on this feature column\n",
    "def info_gain(X, y, column):\n",
    "    \"\"\"\n",
    "    Calculate the resulted information gain given a vector of features.\n",
    "    Arguments:\n",
    "      X: training features, of shape (N, D).\n",
    "      y: vector of training labels, of shape (N,).\n",
    "      column: the column of the feature for calculating. 0 <= column < D\n",
    "    Returns:\n",
    "      (float): the resulted info gain after splitting by this feature.\n",
    "    \"\"\"\n",
    "\n",
    "    n = X.shape[0]   \n",
    "    old_entropy = cross_entropy(y)\n",
    "    \n",
    "    unique_vals = np.unique(X[:, column])\n",
    "    new_entropy = 0.0\n",
    "    \n",
    "    #split the values of i-th feature and calculate the cost \n",
    "    for value in unique_vals:\n",
    "        sub_X, sub_y = split_dataset(X, y, column, value) \n",
    "        prob = sub_X.shape[0] / n\n",
    "        new_entropy += prob * cross_entropy(sub_y) \n",
    "\n",
    "    ig = old_entropy - new_entropy # information gain\n",
    "    return ig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def choose_best_feature(X, y, max_no_of_features):\n",
    "    best_feature_idx = 0\n",
    "    n_features = X.shape[1]\n",
    "    \n",
    "    #select a random subset of features to split at as part of feature bagging\n",
    "    feature_subset = np.random.permutation(range(n_features))[:max_no_of_features]\n",
    "\n",
    "    # use C4.5 algorirhm to select best feature to split on from feature subset\n",
    "    best_gain_cost = 0.0\n",
    "    for feature in feature_subset:\n",
    "        info_gain_cost = info_gain(X, y, feature)          \n",
    "        if info_gain_cost > best_gain_cost:\n",
    "            best_gain_cost = info_gain_cost\n",
    "            best_feature_idx = feature              \n",
    "\n",
    "    return best_feature_idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_tree(X, y, features, depth, max_depth=10, max_no_of_features=4, min_samples_leaf=2):\n",
    "    mytree = dict()\n",
    "    # if (i) no feature, (ii) all lables are the same, (iii) depth exceed or (iv) X is too small\n",
    "    if len(features)==0 or len(np.unique(y))==1 or depth>=max_depth or len(X) <= min_samples_leaf: \n",
    "        return probability_label(y)\n",
    "    else:\n",
    "        best_feature_index = choose_best_feature(X, y, max_no_of_features)\n",
    "        features = features[:]\n",
    "        np.delete(features, best_feature_index)\n",
    "        \n",
    "        mytree = {best_feature_index:{}}\n",
    "        unique_vals = np.unique(X[:, best_feature_index])\n",
    "        # split on each unique value for the column of the best feature chosen and create a dictionary of decision trees recursively\n",
    "        for value in unique_vals:\n",
    "            sub_X, sub_y = split_dataset(X, y, best_feature_index, value)\n",
    "            mytree[best_feature_index][value] = build_tree(sub_X, sub_y, features, depth+1, max_depth, max_no_of_features)\n",
    "\n",
    "        return mytree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# returns the label that appears the most in y\n",
    "labels = np.unique(classification_train_y)\n",
    "def probability_label(y):\n",
    "    # returns the number of occurences of each label in y\n",
    "    count = Counter(y)\n",
    "    probability_of_returning_label = [count[label] / len(y) for label in labels]\n",
    "    return probability_of_returning_label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have all the functions we need to build a decision tree. We can construct a random forest using all the functions from above. Bootstrapping is used to create multiple decision trees. We first produce B random samples from our training dataset. Each sample has size of 800, which is the size of the training dataset. I chose this value since our training dataset is on the lower end of a medium sized dataset and it would be better to have all the data represented when building the decision trees. In additiion, each sample is also stated to have the same size as the original dataset in \"Applied Predictive Modeling\" by Max Khun and Kjell Johnson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# produce num_of_trees random samples from S, each of size dataset\n",
    "def generate_random_samples(X, y, num_of_trees):\n",
    "    N, _ = X.shape\n",
    "    forest_dataset = []\n",
    "    \n",
    "    # generate N random samples from the dataset with replacement\n",
    "    for i in range(num_of_trees):\n",
    "        indices = np.random.choice(N, N)\n",
    "        sample_X = X[indices]\n",
    "        sample_y = y[indices]\n",
    "        forest_dataset.append((sample_X, sample_y))\n",
    "    return forest_dataset\n",
    "\n",
    "# generate a random forest with num_of_trees\n",
    "def random_forest(num_of_trees, max_depth, max_num_of_features, X, y):\n",
    "    forest_dataset = generate_random_samples(X, y, num_of_trees)\n",
    "    forest = []\n",
    "    feature_names = list(range(classification_train_X.shape[1]))\n",
    "    for tree in forest_dataset:\n",
    "        X, y = tree\n",
    "        root = build_tree(X, y, feature_names, 0, max_depth, max_num_of_features)\n",
    "        forest.append(root)\n",
    "    return forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# classify a data point x using the given treee\n",
    "def classify(tree, x):\n",
    "    #first element\n",
    "    feature_name = list(tree.keys())[0]\n",
    "    second_dit = tree[feature_name]            \n",
    "    key = x[feature_name]\n",
    "    if key not in second_dict:\n",
    "        key = np.random.choice(list(second_dict.keys()))\n",
    "    value_of_key = second_dict[key]\n",
    "    if isinstance(value_of_key, dict):\n",
    "        label = classify(value_of_key, x)\n",
    "    else:\n",
    "        label=value_of_key\n",
    "    return label\n",
    "\n",
    "# predict the class of a data point x given forest\n",
    "def rf_predict(forest, x):\n",
    "    # values at each index represent sum or probabilities from each tree that gave a predicted value of that index\n",
    "    res = [0, 0]\n",
    "    \n",
    "    # sum up the probabilities of the data point being classified as 0 or 1 obtained from each tree\n",
    "    for tree in forest:\n",
    "        outcome = classify(tree, x)\n",
    "        res[0] += outcome[0]\n",
    "        res[1] += outcome[1]\n",
    "    \n",
    "    # return label with max probability\n",
    "    if res[0] >= res[1]:\n",
    "        return 0\n",
    "    return 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate the confusion matrix to evaluate the performance of our model\n",
    "def rf_confusion_matrix(y_pred, y_actual):\n",
    "    data = {'y_predicted': y_pred, 'y_actual':y_actual}\n",
    "    df = pd.DataFrame(data, columns=['y_predicted', 'y_actual'])\n",
    "    confusion_matrix = pd.crosstab(df['y_actual'], df['y_predicted'], rownames=['Actual'], colnames=['Predicted'])\n",
    "    return confusion_matrix\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rf_accuracy(y_pred, y_actual):\n",
    "    n = len(y_actual)\n",
    "    return 1 - np.mean(np.abs(y_pred - y_actual))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def rf_cross_validation(num_trees, max_depth, max_num_of_features):\n",
    "    total_acc = 0\n",
    "    n = len(classification_folds)\n",
    "\n",
    "    for i in range(n):\n",
    "        train_X, train_y, test_X, test_y = generate_test_train_sets(classification_folds, i)\n",
    "        \n",
    "        #train a forest based on the hyperparameters given \n",
    "        forest = random_forest(num_trees, max_depth, max_num_of_features, train_X, train_y)\n",
    "        # predict the class for the data points in the y se\n",
    "        y_pred_test = np.array([rf_predict(forest, x) for x in test_X])\n",
    "        # calculate the accuracy of the model\n",
    "        acc = rf_accuracy(y_pred_test, test_y)\n",
    "        total_acc += acc\n",
    "    \n",
    "    return total_acc / n\n",
    "\n",
    "# tune hyperparameters of random forest\n",
    "def rf_tune_hyperparameters(number_of_trees, max_depth, maximum_number_of_features):\n",
    "    i = len(number_of_trees)\n",
    "    j = len(max_depth)\n",
    "    k = len(maximum_number_of_features)\n",
    "    print(\"i, {}, j, {}, k, {}\".format(i, j, k))\n",
    "    print(\"i\", number_of_trees)\n",
    "    print(\"j\", max_depth)\n",
    "    print(\"k\", maximum_number_of_features)\n",
    "    grid = {}\n",
    "    \n",
    "    # loop through each hyperparameter\n",
    "    for num_trees in number_of_trees:\n",
    "        for depth in max_depth:\n",
    "            for num_of_features in maximum_number_of_features: \n",
    "                print(\"num_trees: {}, depth: {}, features: {}\".format(num_trees, depth, num_of_features))\n",
    "                avg_accuracy = rf_cross_validation(num_trees, depth, num_of_features)\n",
    "                grid[(num_trees, depth, num_of_features)] = avg_accuracy\n",
    "    \n",
    "\n",
    "    chosen_combi, max_acc = grid_search(grid)\n",
    "    \n",
    "    t, d, f = chosen_combi\n",
    "    print(\"max_accuracy: \", max_acc)\n",
    "    print(\"num_trees: {}, depth: {}, features: {}\".format(t, d, f))\n",
    "    \n",
    "    return chosen_combi, grid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 275,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i, 11, j, 8, k, 5\n",
      "i [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]\n",
      "j [3, 4, 5, 6, 7, 8, 9, 10]\n",
      "k [2, 3, 4, 5, 6]\n",
      "num_trees: 5, depth: 3, features: 2\n",
      "num_trees: 5, depth: 3, features: 3\n",
      "num_trees: 5, depth: 3, features: 4\n",
      "num_trees: 5, depth: 3, features: 5\n",
      "num_trees: 5, depth: 3, features: 6\n",
      "num_trees: 5, depth: 4, features: 2\n",
      "num_trees: 5, depth: 4, features: 3\n",
      "num_trees: 5, depth: 4, features: 4\n",
      "num_trees: 5, depth: 4, features: 5\n",
      "num_trees: 5, depth: 4, features: 6\n",
      "num_trees: 5, depth: 5, features: 2\n",
      "num_trees: 5, depth: 5, features: 3\n",
      "num_trees: 5, depth: 5, features: 4\n",
      "num_trees: 5, depth: 5, features: 5\n",
      "num_trees: 5, depth: 5, features: 6\n",
      "num_trees: 5, depth: 6, features: 2\n",
      "num_trees: 5, depth: 6, features: 3\n",
      "num_trees: 5, depth: 6, features: 4\n",
      "num_trees: 5, depth: 6, features: 5\n",
      "num_trees: 5, depth: 6, features: 6\n",
      "num_trees: 5, depth: 7, features: 2\n",
      "num_trees: 5, depth: 7, features: 3\n",
      "num_trees: 5, depth: 7, features: 4\n",
      "num_trees: 5, depth: 7, features: 5\n",
      "num_trees: 5, depth: 7, features: 6\n",
      "num_trees: 5, depth: 8, features: 2\n",
      "num_trees: 5, depth: 8, features: 3\n",
      "num_trees: 5, depth: 8, features: 4\n",
      "num_trees: 5, depth: 8, features: 5\n",
      "num_trees: 5, depth: 8, features: 6\n",
      "num_trees: 5, depth: 9, features: 2\n",
      "num_trees: 5, depth: 9, features: 3\n",
      "num_trees: 5, depth: 9, features: 4\n",
      "num_trees: 5, depth: 9, features: 5\n",
      "num_trees: 5, depth: 9, features: 6\n",
      "num_trees: 5, depth: 10, features: 2\n",
      "num_trees: 5, depth: 10, features: 3\n",
      "num_trees: 5, depth: 10, features: 4\n",
      "num_trees: 5, depth: 10, features: 5\n",
      "num_trees: 5, depth: 10, features: 6\n",
      "num_trees: 6, depth: 3, features: 2\n",
      "num_trees: 6, depth: 3, features: 3\n",
      "num_trees: 6, depth: 3, features: 4\n",
      "num_trees: 6, depth: 3, features: 5\n",
      "num_trees: 6, depth: 3, features: 6\n",
      "num_trees: 6, depth: 4, features: 2\n",
      "num_trees: 6, depth: 4, features: 3\n",
      "num_trees: 6, depth: 4, features: 4\n",
      "num_trees: 6, depth: 4, features: 5\n",
      "num_trees: 6, depth: 4, features: 6\n",
      "num_trees: 6, depth: 5, features: 2\n",
      "num_trees: 6, depth: 5, features: 3\n",
      "num_trees: 6, depth: 5, features: 4\n",
      "num_trees: 6, depth: 5, features: 5\n",
      "num_trees: 6, depth: 5, features: 6\n",
      "num_trees: 6, depth: 6, features: 2\n",
      "num_trees: 6, depth: 6, features: 3\n",
      "num_trees: 6, depth: 6, features: 4\n",
      "num_trees: 6, depth: 6, features: 5\n",
      "num_trees: 6, depth: 6, features: 6\n",
      "num_trees: 6, depth: 7, features: 2\n",
      "num_trees: 6, depth: 7, features: 3\n",
      "num_trees: 6, depth: 7, features: 4\n",
      "num_trees: 6, depth: 7, features: 5\n",
      "num_trees: 6, depth: 7, features: 6\n",
      "num_trees: 6, depth: 8, features: 2\n",
      "num_trees: 6, depth: 8, features: 3\n",
      "num_trees: 6, depth: 8, features: 4\n",
      "num_trees: 6, depth: 8, features: 5\n",
      "num_trees: 6, depth: 8, features: 6\n",
      "num_trees: 6, depth: 9, features: 2\n",
      "num_trees: 6, depth: 9, features: 3\n",
      "num_trees: 6, depth: 9, features: 4\n",
      "num_trees: 6, depth: 9, features: 5\n",
      "num_trees: 6, depth: 9, features: 6\n",
      "num_trees: 6, depth: 10, features: 2\n",
      "num_trees: 6, depth: 10, features: 3\n",
      "num_trees: 6, depth: 10, features: 4\n",
      "num_trees: 6, depth: 10, features: 5\n",
      "num_trees: 6, depth: 10, features: 6\n",
      "num_trees: 7, depth: 3, features: 2\n",
      "num_trees: 7, depth: 3, features: 3\n",
      "num_trees: 7, depth: 3, features: 4\n",
      "num_trees: 7, depth: 3, features: 5\n",
      "num_trees: 7, depth: 3, features: 6\n",
      "num_trees: 7, depth: 4, features: 2\n",
      "num_trees: 7, depth: 4, features: 3\n",
      "num_trees: 7, depth: 4, features: 4\n",
      "num_trees: 7, depth: 4, features: 5\n",
      "num_trees: 7, depth: 4, features: 6\n",
      "num_trees: 7, depth: 5, features: 2\n",
      "num_trees: 7, depth: 5, features: 3\n",
      "num_trees: 7, depth: 5, features: 4\n",
      "num_trees: 7, depth: 5, features: 5\n",
      "num_trees: 7, depth: 5, features: 6\n",
      "num_trees: 7, depth: 6, features: 2\n",
      "num_trees: 7, depth: 6, features: 3\n",
      "num_trees: 7, depth: 6, features: 4\n",
      "num_trees: 7, depth: 6, features: 5\n",
      "num_trees: 7, depth: 6, features: 6\n",
      "num_trees: 7, depth: 7, features: 2\n",
      "num_trees: 7, depth: 7, features: 3\n",
      "num_trees: 7, depth: 7, features: 4\n",
      "num_trees: 7, depth: 7, features: 5\n",
      "num_trees: 7, depth: 7, features: 6\n",
      "num_trees: 7, depth: 8, features: 2\n",
      "num_trees: 7, depth: 8, features: 3\n",
      "num_trees: 7, depth: 8, features: 4\n",
      "num_trees: 7, depth: 8, features: 5\n",
      "num_trees: 7, depth: 8, features: 6\n",
      "num_trees: 7, depth: 9, features: 2\n",
      "num_trees: 7, depth: 9, features: 3\n",
      "num_trees: 7, depth: 9, features: 4\n",
      "num_trees: 7, depth: 9, features: 5\n",
      "num_trees: 7, depth: 9, features: 6\n",
      "num_trees: 7, depth: 10, features: 2\n",
      "num_trees: 7, depth: 10, features: 3\n",
      "num_trees: 7, depth: 10, features: 4\n",
      "num_trees: 7, depth: 10, features: 5\n",
      "num_trees: 7, depth: 10, features: 6\n",
      "num_trees: 8, depth: 3, features: 2\n",
      "num_trees: 8, depth: 3, features: 3\n",
      "num_trees: 8, depth: 3, features: 4\n",
      "num_trees: 8, depth: 3, features: 5\n",
      "num_trees: 8, depth: 3, features: 6\n",
      "num_trees: 8, depth: 4, features: 2\n",
      "num_trees: 8, depth: 4, features: 3\n",
      "num_trees: 8, depth: 4, features: 4\n",
      "num_trees: 8, depth: 4, features: 5\n",
      "num_trees: 8, depth: 4, features: 6\n",
      "num_trees: 8, depth: 5, features: 2\n",
      "num_trees: 8, depth: 5, features: 3\n",
      "num_trees: 8, depth: 5, features: 4\n",
      "num_trees: 8, depth: 5, features: 5\n",
      "num_trees: 8, depth: 5, features: 6\n",
      "num_trees: 8, depth: 6, features: 2\n",
      "num_trees: 8, depth: 6, features: 3\n",
      "num_trees: 8, depth: 6, features: 4\n",
      "num_trees: 8, depth: 6, features: 5\n",
      "num_trees: 8, depth: 6, features: 6\n",
      "num_trees: 8, depth: 7, features: 2\n",
      "num_trees: 8, depth: 7, features: 3\n",
      "num_trees: 8, depth: 7, features: 4\n",
      "num_trees: 8, depth: 7, features: 5\n",
      "num_trees: 8, depth: 7, features: 6\n",
      "num_trees: 8, depth: 8, features: 2\n",
      "num_trees: 8, depth: 8, features: 3\n",
      "num_trees: 8, depth: 8, features: 4\n",
      "num_trees: 8, depth: 8, features: 5\n",
      "num_trees: 8, depth: 8, features: 6\n",
      "num_trees: 8, depth: 9, features: 2\n",
      "num_trees: 8, depth: 9, features: 3\n",
      "num_trees: 8, depth: 9, features: 4\n",
      "num_trees: 8, depth: 9, features: 5\n",
      "num_trees: 8, depth: 9, features: 6\n",
      "num_trees: 8, depth: 10, features: 2\n",
      "num_trees: 8, depth: 10, features: 3\n",
      "num_trees: 8, depth: 10, features: 4\n",
      "num_trees: 8, depth: 10, features: 5\n",
      "num_trees: 8, depth: 10, features: 6\n",
      "num_trees: 9, depth: 3, features: 2\n",
      "num_trees: 9, depth: 3, features: 3\n",
      "num_trees: 9, depth: 3, features: 4\n",
      "num_trees: 9, depth: 3, features: 5\n",
      "num_trees: 9, depth: 3, features: 6\n",
      "num_trees: 9, depth: 4, features: 2\n",
      "num_trees: 9, depth: 4, features: 3\n",
      "num_trees: 9, depth: 4, features: 4\n",
      "num_trees: 9, depth: 4, features: 5\n",
      "num_trees: 9, depth: 4, features: 6\n",
      "num_trees: 9, depth: 5, features: 2\n",
      "num_trees: 9, depth: 5, features: 3\n",
      "num_trees: 9, depth: 5, features: 4\n",
      "num_trees: 9, depth: 5, features: 5\n",
      "num_trees: 9, depth: 5, features: 6\n",
      "num_trees: 9, depth: 6, features: 2\n",
      "num_trees: 9, depth: 6, features: 3\n",
      "num_trees: 9, depth: 6, features: 4\n",
      "num_trees: 9, depth: 6, features: 5\n",
      "num_trees: 9, depth: 6, features: 6\n",
      "num_trees: 9, depth: 7, features: 2\n",
      "num_trees: 9, depth: 7, features: 3\n",
      "num_trees: 9, depth: 7, features: 4\n",
      "num_trees: 9, depth: 7, features: 5\n",
      "num_trees: 9, depth: 7, features: 6\n",
      "num_trees: 9, depth: 8, features: 2\n",
      "num_trees: 9, depth: 8, features: 3\n",
      "num_trees: 9, depth: 8, features: 4\n",
      "num_trees: 9, depth: 8, features: 5\n",
      "num_trees: 9, depth: 8, features: 6\n",
      "num_trees: 9, depth: 9, features: 2\n",
      "num_trees: 9, depth: 9, features: 3\n",
      "num_trees: 9, depth: 9, features: 4\n",
      "num_trees: 9, depth: 9, features: 5\n",
      "num_trees: 9, depth: 9, features: 6\n",
      "num_trees: 9, depth: 10, features: 2\n",
      "num_trees: 9, depth: 10, features: 3\n",
      "num_trees: 9, depth: 10, features: 4\n",
      "num_trees: 9, depth: 10, features: 5\n",
      "num_trees: 9, depth: 10, features: 6\n",
      "num_trees: 10, depth: 3, features: 2\n",
      "num_trees: 10, depth: 3, features: 3\n",
      "num_trees: 10, depth: 3, features: 4\n",
      "num_trees: 10, depth: 3, features: 5\n",
      "num_trees: 10, depth: 3, features: 6\n",
      "num_trees: 10, depth: 4, features: 2\n",
      "num_trees: 10, depth: 4, features: 3\n",
      "num_trees: 10, depth: 4, features: 4\n",
      "num_trees: 10, depth: 4, features: 5\n",
      "num_trees: 10, depth: 4, features: 6\n",
      "num_trees: 10, depth: 5, features: 2\n",
      "num_trees: 10, depth: 5, features: 3\n",
      "num_trees: 10, depth: 5, features: 4\n",
      "num_trees: 10, depth: 5, features: 5\n",
      "num_trees: 10, depth: 5, features: 6\n",
      "num_trees: 10, depth: 6, features: 2\n",
      "num_trees: 10, depth: 6, features: 3\n",
      "num_trees: 10, depth: 6, features: 4\n",
      "num_trees: 10, depth: 6, features: 5\n",
      "num_trees: 10, depth: 6, features: 6\n",
      "num_trees: 10, depth: 7, features: 2\n",
      "num_trees: 10, depth: 7, features: 3\n",
      "num_trees: 10, depth: 7, features: 4\n",
      "num_trees: 10, depth: 7, features: 5\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_trees: 10, depth: 7, features: 6\n",
      "num_trees: 10, depth: 8, features: 2\n",
      "num_trees: 10, depth: 8, features: 3\n",
      "num_trees: 10, depth: 8, features: 4\n",
      "num_trees: 10, depth: 8, features: 5\n",
      "num_trees: 10, depth: 8, features: 6\n",
      "num_trees: 10, depth: 9, features: 2\n",
      "num_trees: 10, depth: 9, features: 3\n",
      "num_trees: 10, depth: 9, features: 4\n",
      "num_trees: 10, depth: 9, features: 5\n",
      "num_trees: 10, depth: 9, features: 6\n",
      "num_trees: 10, depth: 10, features: 2\n",
      "num_trees: 10, depth: 10, features: 3\n",
      "num_trees: 10, depth: 10, features: 4\n",
      "num_trees: 10, depth: 10, features: 5\n",
      "num_trees: 10, depth: 10, features: 6\n",
      "num_trees: 11, depth: 3, features: 2\n",
      "num_trees: 11, depth: 3, features: 3\n",
      "num_trees: 11, depth: 3, features: 4\n",
      "num_trees: 11, depth: 3, features: 5\n",
      "num_trees: 11, depth: 3, features: 6\n",
      "num_trees: 11, depth: 4, features: 2\n",
      "num_trees: 11, depth: 4, features: 3\n",
      "num_trees: 11, depth: 4, features: 4\n",
      "num_trees: 11, depth: 4, features: 5\n",
      "num_trees: 11, depth: 4, features: 6\n",
      "num_trees: 11, depth: 5, features: 2\n",
      "num_trees: 11, depth: 5, features: 3\n",
      "num_trees: 11, depth: 5, features: 4\n",
      "num_trees: 11, depth: 5, features: 5\n",
      "num_trees: 11, depth: 5, features: 6\n",
      "num_trees: 11, depth: 6, features: 2\n",
      "num_trees: 11, depth: 6, features: 3\n",
      "num_trees: 11, depth: 6, features: 4\n",
      "num_trees: 11, depth: 6, features: 5\n",
      "num_trees: 11, depth: 6, features: 6\n",
      "num_trees: 11, depth: 7, features: 2\n",
      "num_trees: 11, depth: 7, features: 3\n",
      "num_trees: 11, depth: 7, features: 4\n",
      "num_trees: 11, depth: 7, features: 5\n",
      "num_trees: 11, depth: 7, features: 6\n",
      "num_trees: 11, depth: 8, features: 2\n",
      "num_trees: 11, depth: 8, features: 3\n",
      "num_trees: 11, depth: 8, features: 4\n",
      "num_trees: 11, depth: 8, features: 5\n",
      "num_trees: 11, depth: 8, features: 6\n",
      "num_trees: 11, depth: 9, features: 2\n",
      "num_trees: 11, depth: 9, features: 3\n",
      "num_trees: 11, depth: 9, features: 4\n",
      "num_trees: 11, depth: 9, features: 5\n",
      "num_trees: 11, depth: 9, features: 6\n",
      "num_trees: 11, depth: 10, features: 2\n",
      "num_trees: 11, depth: 10, features: 3\n",
      "num_trees: 11, depth: 10, features: 4\n",
      "num_trees: 11, depth: 10, features: 5\n",
      "num_trees: 11, depth: 10, features: 6\n",
      "num_trees: 12, depth: 3, features: 2\n",
      "num_trees: 12, depth: 3, features: 3\n",
      "num_trees: 12, depth: 3, features: 4\n",
      "num_trees: 12, depth: 3, features: 5\n",
      "num_trees: 12, depth: 3, features: 6\n",
      "num_trees: 12, depth: 4, features: 2\n",
      "num_trees: 12, depth: 4, features: 3\n",
      "num_trees: 12, depth: 4, features: 4\n",
      "num_trees: 12, depth: 4, features: 5\n",
      "num_trees: 12, depth: 4, features: 6\n",
      "num_trees: 12, depth: 5, features: 2\n",
      "num_trees: 12, depth: 5, features: 3\n",
      "num_trees: 12, depth: 5, features: 4\n",
      "num_trees: 12, depth: 5, features: 5\n",
      "num_trees: 12, depth: 5, features: 6\n",
      "num_trees: 12, depth: 6, features: 2\n",
      "num_trees: 12, depth: 6, features: 3\n",
      "num_trees: 12, depth: 6, features: 4\n",
      "num_trees: 12, depth: 6, features: 5\n",
      "num_trees: 12, depth: 6, features: 6\n",
      "num_trees: 12, depth: 7, features: 2\n",
      "num_trees: 12, depth: 7, features: 3\n",
      "num_trees: 12, depth: 7, features: 4\n",
      "num_trees: 12, depth: 7, features: 5\n",
      "num_trees: 12, depth: 7, features: 6\n",
      "num_trees: 12, depth: 8, features: 2\n",
      "num_trees: 12, depth: 8, features: 3\n",
      "num_trees: 12, depth: 8, features: 4\n",
      "num_trees: 12, depth: 8, features: 5\n",
      "num_trees: 12, depth: 8, features: 6\n",
      "num_trees: 12, depth: 9, features: 2\n",
      "num_trees: 12, depth: 9, features: 3\n",
      "num_trees: 12, depth: 9, features: 4\n",
      "num_trees: 12, depth: 9, features: 5\n",
      "num_trees: 12, depth: 9, features: 6\n",
      "num_trees: 12, depth: 10, features: 2\n",
      "num_trees: 12, depth: 10, features: 3\n",
      "num_trees: 12, depth: 10, features: 4\n",
      "num_trees: 12, depth: 10, features: 5\n",
      "num_trees: 12, depth: 10, features: 6\n",
      "num_trees: 13, depth: 3, features: 2\n",
      "num_trees: 13, depth: 3, features: 3\n",
      "num_trees: 13, depth: 3, features: 4\n",
      "num_trees: 13, depth: 3, features: 5\n",
      "num_trees: 13, depth: 3, features: 6\n",
      "num_trees: 13, depth: 4, features: 2\n",
      "num_trees: 13, depth: 4, features: 3\n",
      "num_trees: 13, depth: 4, features: 4\n",
      "num_trees: 13, depth: 4, features: 5\n",
      "num_trees: 13, depth: 4, features: 6\n",
      "num_trees: 13, depth: 5, features: 2\n",
      "num_trees: 13, depth: 5, features: 3\n",
      "num_trees: 13, depth: 5, features: 4\n",
      "num_trees: 13, depth: 5, features: 5\n",
      "num_trees: 13, depth: 5, features: 6\n",
      "num_trees: 13, depth: 6, features: 2\n",
      "num_trees: 13, depth: 6, features: 3\n",
      "num_trees: 13, depth: 6, features: 4\n",
      "num_trees: 13, depth: 6, features: 5\n",
      "num_trees: 13, depth: 6, features: 6\n",
      "num_trees: 13, depth: 7, features: 2\n",
      "num_trees: 13, depth: 7, features: 3\n",
      "num_trees: 13, depth: 7, features: 4\n",
      "num_trees: 13, depth: 7, features: 5\n",
      "num_trees: 13, depth: 7, features: 6\n",
      "num_trees: 13, depth: 8, features: 2\n",
      "num_trees: 13, depth: 8, features: 3\n",
      "num_trees: 13, depth: 8, features: 4\n",
      "num_trees: 13, depth: 8, features: 5\n",
      "num_trees: 13, depth: 8, features: 6\n",
      "num_trees: 13, depth: 9, features: 2\n",
      "num_trees: 13, depth: 9, features: 3\n",
      "num_trees: 13, depth: 9, features: 4\n",
      "num_trees: 13, depth: 9, features: 5\n",
      "num_trees: 13, depth: 9, features: 6\n",
      "num_trees: 13, depth: 10, features: 2\n",
      "num_trees: 13, depth: 10, features: 3\n",
      "num_trees: 13, depth: 10, features: 4\n",
      "num_trees: 13, depth: 10, features: 5\n",
      "num_trees: 13, depth: 10, features: 6\n",
      "num_trees: 14, depth: 3, features: 2\n",
      "num_trees: 14, depth: 3, features: 3\n",
      "num_trees: 14, depth: 3, features: 4\n",
      "num_trees: 14, depth: 3, features: 5\n",
      "num_trees: 14, depth: 3, features: 6\n",
      "num_trees: 14, depth: 4, features: 2\n",
      "num_trees: 14, depth: 4, features: 3\n",
      "num_trees: 14, depth: 4, features: 4\n",
      "num_trees: 14, depth: 4, features: 5\n",
      "num_trees: 14, depth: 4, features: 6\n",
      "num_trees: 14, depth: 5, features: 2\n",
      "num_trees: 14, depth: 5, features: 3\n",
      "num_trees: 14, depth: 5, features: 4\n",
      "num_trees: 14, depth: 5, features: 5\n",
      "num_trees: 14, depth: 5, features: 6\n",
      "num_trees: 14, depth: 6, features: 2\n",
      "num_trees: 14, depth: 6, features: 3\n",
      "num_trees: 14, depth: 6, features: 4\n",
      "num_trees: 14, depth: 6, features: 5\n",
      "num_trees: 14, depth: 6, features: 6\n",
      "num_trees: 14, depth: 7, features: 2\n",
      "num_trees: 14, depth: 7, features: 3\n",
      "num_trees: 14, depth: 7, features: 4\n",
      "num_trees: 14, depth: 7, features: 5\n",
      "num_trees: 14, depth: 7, features: 6\n",
      "num_trees: 14, depth: 8, features: 2\n",
      "num_trees: 14, depth: 8, features: 3\n",
      "num_trees: 14, depth: 8, features: 4\n",
      "num_trees: 14, depth: 8, features: 5\n",
      "num_trees: 14, depth: 8, features: 6\n",
      "num_trees: 14, depth: 9, features: 2\n",
      "num_trees: 14, depth: 9, features: 3\n",
      "num_trees: 14, depth: 9, features: 4\n",
      "num_trees: 14, depth: 9, features: 5\n",
      "num_trees: 14, depth: 9, features: 6\n",
      "num_trees: 14, depth: 10, features: 2\n",
      "num_trees: 14, depth: 10, features: 3\n",
      "num_trees: 14, depth: 10, features: 4\n",
      "num_trees: 14, depth: 10, features: 5\n",
      "num_trees: 14, depth: 10, features: 6\n",
      "num_trees: 15, depth: 3, features: 2\n",
      "num_trees: 15, depth: 3, features: 3\n",
      "num_trees: 15, depth: 3, features: 4\n",
      "num_trees: 15, depth: 3, features: 5\n",
      "num_trees: 15, depth: 3, features: 6\n",
      "num_trees: 15, depth: 4, features: 2\n",
      "num_trees: 15, depth: 4, features: 3\n",
      "num_trees: 15, depth: 4, features: 4\n",
      "num_trees: 15, depth: 4, features: 5\n",
      "num_trees: 15, depth: 4, features: 6\n",
      "num_trees: 15, depth: 5, features: 2\n",
      "num_trees: 15, depth: 5, features: 3\n",
      "num_trees: 15, depth: 5, features: 4\n",
      "num_trees: 15, depth: 5, features: 5\n",
      "num_trees: 15, depth: 5, features: 6\n",
      "num_trees: 15, depth: 6, features: 2\n",
      "num_trees: 15, depth: 6, features: 3\n",
      "num_trees: 15, depth: 6, features: 4\n",
      "num_trees: 15, depth: 6, features: 5\n",
      "num_trees: 15, depth: 6, features: 6\n",
      "num_trees: 15, depth: 7, features: 2\n",
      "num_trees: 15, depth: 7, features: 3\n",
      "num_trees: 15, depth: 7, features: 4\n",
      "num_trees: 15, depth: 7, features: 5\n",
      "num_trees: 15, depth: 7, features: 6\n",
      "num_trees: 15, depth: 8, features: 2\n",
      "num_trees: 15, depth: 8, features: 3\n",
      "num_trees: 15, depth: 8, features: 4\n",
      "num_trees: 15, depth: 8, features: 5\n",
      "num_trees: 15, depth: 8, features: 6\n",
      "num_trees: 15, depth: 9, features: 2\n",
      "num_trees: 15, depth: 9, features: 3\n",
      "num_trees: 15, depth: 9, features: 4\n",
      "num_trees: 15, depth: 9, features: 5\n",
      "num_trees: 15, depth: 9, features: 6\n",
      "num_trees: 15, depth: 10, features: 2\n",
      "num_trees: 15, depth: 10, features: 3\n",
      "num_trees: 15, depth: 10, features: 4\n",
      "num_trees: 15, depth: 10, features: 5\n",
      "num_trees: 15, depth: 10, features: 6\n",
      "max_accuracy:  0.70625\n",
      "num_trees: 14, depth: 3, features: 2\n"
     ]
    }
   ],
   "source": [
    "# train hyperparamters of random forest\n",
    "trees = list(range(5, 16))\n",
    "depths = list(range(3, 11))\n",
    "features = list(range(2, 7))\n",
    "(opt_num_trees, opt_depth, opt_num_features), grid = rf_tune_hyperparameters(trees, depths, features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rf_model(X, y, X_test, y_test, opt_num_trees, opt_depth, opt_num_features):\n",
    "    \n",
    "    print(\"generating forest\")\n",
    "    forest = random_forest(opt_num_trees, opt_depth, opt_num_features, X, y)\n",
    "\n",
    "    y_pred_test = np.array([rf_predict(forest, x) for x in X_test])\n",
    "\n",
    "    confusion_matrix_test = rf_confusion_matrix(y_pred_test, y_test)\n",
    "    \n",
    "    X_train = X[:, :-1]\n",
    "    y_pred_train = np.array([rf_predict(forest, x) for x in X_train])\n",
    "    \n",
    "    confusion_matrix_train = rf_confusion_matrix(y_pred_train, y)\n",
    "    \n",
    "    plt.figure()\n",
    "    plt.title(\"Confusion matrix for training dataset\")\n",
    "    sn.heatmap(confusion_matrix_train, annot=True)\n",
    "    \n",
    "    plt.figure()\n",
    "    plt.title(\"Confusion matrix for test dataset\")\n",
    "    sn.heatmap(confusion_matrix_test, annot=True)\n",
    "    \n",
    "    return confusion_matrix_test, confusion_matrix_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 321,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generating forest\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEPCAYAAACX9KXAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAo7UlEQVR4nO3deVxU5f4H8M/M4LDNIFKp3RSSFLMUIc0lcEFTLCWU1IAcNXfLDMwNFyDJJXG3Wy55LUlCEbe0rCRvXFHJMNekkgo3RAV/ODMIAzPP7w9vczVZZWCOzefda14vOPOc5/meCb98ec5zzpEJIQSIiMiq5NYOgIiImIyJiCSByZiISAKYjImIJIDJmIhIApiMiYgkgMm4EkajERs3bkRISAiCg4Px4osvIj4+HgaDoVZ9Tpw4EYGBgfj0009rvP+pU6cwefLk+x7f0rRaLYYPH17h+8HBwbh582a1+9u+fTt69uyJ0aNH33dMJ0+eRHR0dI33W7lyJXbu3Flpm9TUVLz77rv3GVnlTp06hV69elXZLjk5GZs3b66TGObMmYPTp0/XSd9UOTtrByBlsbGxKCwsxCeffAK1Wo2ioiJMnToVs2fPRnx8/H31mZeXh4MHD+L48eNQKBQ13r9du3ZYtWrVfY1dFwoLC3Hq1KkK39+1a1eN+tu5cyciIyMRHBx83zGdO3cOeXl5Nd7vrbfeqrJN79690bt37/sJy2IyMzPRqlWrOun70KFDeOWVV+qkb6ock3EFLl68iM8//xwHDx6ESqUCADg5OeGdd97BsWPHANyuCt955x1kZWVBJpOhW7dumDJlCuzs7NCuXTuMGzcO6enpuHr1KsaMGYOXXnoJY8aMQVlZGUJCQrB69Wr06dMHhw8fhpubGwCgdevWOHz4MOzt7REVFYWcnBzI5XI8/fTTmDdvHo4ePYq4uDjs2bOnxuOHh4ffc5zt2rXDa6+9hkOHDqGoqAiTJk3Cvn378Msvv6Bx48ZYs2YNnJycsG3bNmzZsgWlpaUoLCzE2LFjER4ejqioKBQXFyM4OBjbt29H+/bt0bt3b2RlZWHJkiUYPHgwDh8+jMTERBw8eBCbN29GQUEBBg0ahCVLlqBLly7mWBYsWIBTp07h4sWLuHHjBl5++eUKj69t27Z3jdOuXTsAQG5uLlatWgWtVouoqCgMHDgQ8+fPh5OTE/R6PVJSUrB48WKcOHECer0eQgi8++676NChA2bOnIlWrVph9OjRFX5+27dvx1dffYW1a9dCo9HAx8cHx44dQ25uLrp27Yq4uDjI5XJs374d69atg4ODA7p06YJNmzbhp59+uufzT0xMxCeffAKVSgUvLy/z9uvXryM6Ohr5+fm4du0aHnvsMaxYsQLHjh3Dt99+i/T0dDg4OCAwMLDcdg899BASExORlJSEBg0awN7eHvPmzUPLli2Rl5eHefPmITc3F6Wlpejfvz8mTJiA5cuX4+rVq5g6dSoWL16M9u3bW+4fFFVNULn27dsnXn755UrbTJ8+XcTFxQmTySRKSkrEqFGjxNq1a4UQQnh5eYmEhAQhhBCnTp0Sbdu2FcXFxeLChQvCx8fH3IeXl5fIz8+/5/sdO3aIUaNGCSGEKCsrE7NnzxZ//PGHOHLkiOjfv/99j/9XXl5e4pNPPhFCCLF27Vrh6+srrly5IoxGoxg0aJDYvXu30Ol0YujQoaKgoEAIIcSPP/5oPobyjmfHjh33HE9ZWZl49dVXxdq1a8XIkSPFhx9+WO5nOmzYMPHll19W6/juHOdOKSkpYty4cUIIIY4cOSKefPJJcfHiRSGEEMeOHRNvvvmmMBqN5mMeP368EEKIGTNmiI8++qjSz+/OvocNGyYmT54sjEaj0Gq1wt/fXxw+fFj8+uuvomvXriI3N1cIIcTq1auFl5fXPXH+9NNPomvXruLq1atCCCHmzp0rAgIChBBCfPzxx+ZjNZlMYsyYMWLDhg33xFlRu7KyMvH000+LvLw8IYQQO3bsEElJSUIIITQajUhNTRVCCFFcXCw0Go3Yu3evEEKIgIAAcfLkyXI/V6pbnDOugFwuh8lkqrRNWloahg0bBplMBqVSidDQUKSlpZnf//PP2aeffhoGgwFFRUXVHr9Dhw44d+4cNBoN1q1bhxEjRsDDw6NOxg8MDAQAuLu7w8vLC02aNIFcLkezZs1QWFgIZ2dnrFmzBt999x1WrFiBNWvWVHosHTt2vGebQqHAkiVLsH79egghMH78+Co/g6qOr7xxyvPoo4/iscceAwD4+voiIiICSUlJeO+997Bv3z7o9fpy96vO5xcQEAC5XA6VSgUPDw8UFhbi4MGD8PPzQ9OmTQEAw4YNK7f/w4cPw8/PD4888ggA3DU9MGLECDzzzDPYuHEjYmNj8euvv5Y7fkXtFAoF+vXrh9DQUMybNw8uLi4YPHgwioqKcPToUaxcuRLBwcEYOnQocnNzkZWVVa3PkuoOk3EFvL298dtvv0Gn0921PS8vD+PGjUNxcTFMJhNkMpn5PZPJhLKyMvP39vb2AGBuI6q4DcidJwabN2+Ob775BuPGjYNOp8Nrr72Gb7/99q72lhq/QYMG5X79pytXrmDgwIG4dOkSOnTogIiIiEqPw8nJqdztly5dgr29Pc6fP4/CwsJK+wCqPr6Kxqksnn//+9/mXwS9e/dGWFhYhftV5/NzcHAwfy2TySCEgEKhuKttZecGKmoXHx+PlStXolGjRnjllVfg5+dX7viVtVuyZAnWrFkDd3d3rFu3DlOmTIHJZIIQAklJSdi1axd27dqFLVu2VOuXI9UtJuMKNGnSBEFBQZg1a5Y5Iet0OsTGxsLV1RUODg7w9/fHp59+CiEEDAYDtm7diueee65G47i5uZlPgO3Zs8e8PTExEVFRUfD398e0adPg7+9/z5yjJcavjtOnT8PNzQ2vv/46/P39ceDAAQC3V4bY2dnBaDRW+Yvm5s2bmDZtGhYtWoQBAwZg9uzZVY57v8enUCjuStp3Sk9PR0BAAMLDw9G2bVvs378fRqOxyj5rwt/fH4cPHzafRExOTi63nZ+fH9LT03HlyhUAwI4dO8zvHTx4ECNGjMDAgQPx0EMP4dChQ+Y47zy+itoVFBSgR48ecHV1xciRIxEREYFTp05BpVLBx8cHGzduBHD7/0tYWBhSU1Pv6ZvqF5NxJWJiYtCyZUuEhoYiODgYQ4YMQcuWLc1Lm+bMmYOCggIEBQUhKCgILVq0wIQJE2o0xpw5czBv3jwMGjQI2dnZ5j9ZBw4cCKPRiBdffBEhISHQarXQaDT37Fvb8avDz88PTZo0Qb9+/fDCCy8gNzcXbm5uyMnJwSOPPAJvb2/0798fN27cqPQ4e/bsCX9/f0yaNAkXLlyocnnW/R6fj48PLly4gEmTJt3zXmhoKL7//nsEBQVh0KBBaN68OS5evFjllFRNtGjRAlFRURg9ejRCQkKQnZ0NR0fHe9q1bt0a06ZNw4gRIxASEoKSkhLze2+88QYWL16MoKAgTJw4Ec888wzOnz8PAOjevTuSkpKwdu3aCtu5ublh4sSJGDlyJEJCQrB06VLzz+2SJUtw4sQJBAUFYciQIRgwYABeeuklAECfPn0wbdo0HDx40GKfB1WPTFRV0hBRjVy4cAG7du3C66+/Drlcjq+//hrr16+vsEImAri0jcjimjZtiqtXryIoKAgKhQJqtRoLFiywdlgkcayMiYgkgHPGREQSwGRMRCQBTMZERBIgiRN4fZv3s3YIJDGN5A5VNyKbtCVnZ632L73+W7XbNnjYs1Zj1YQkkjERUb0xWfYiH0thMiYi2yIsd4GPJTEZE5FtseDVlpbEZExENkWwMiYikgCjNG+ExGRMRLaFJ/CIiCSA0xRERBLAE3hERNbHE3hERFLAypiISAKMpdaOoFxMxkRkWzhNQUQkAZymICKSAFbGREQSwMqYiMj6hIkn8IiIrI+VMRGRBHDOmIhIAmp5o6CBAwdCrVYDAJo1a4YJEyZg5syZkMlkaNWqFWJiYiCXy7F161YkJSXBzs4OEydOREBAQKX9MhkTkW2pRWVcUlICAEhISDBvmzBhAiIiItC5c2dER0cjNTUVPj4+SEhIQEpKCkpKShAeHg4/Pz8olcoK+2YyJiLbUos546ysLNy6dQujRo1CWVkZpkyZgjNnzqBTp04AgO7duyM9PR1yuRy+vr5QKpVQKpVwd3dHVlYWvL29K+ybyZiIbEstbi7v4OCA0aNHY8iQIfjjjz8wduxYCCEgk8kAAM7OztBqtdDpdOapjD+363S6SvtmMiYi21KLyrhFixbw8PCATCZDixYt4OrqijNnzpjf1+v1cHFxgUqlgl6vv2v7ncm5PPL7joqI6AEkhLHar7/atm0bFi1aBADIy8uDTqeDn58fMjIyAABpaWno2LEjvL29kZmZiZKSEmi1WmRnZ8PLy6vSuFgZE5FtqUVlPHjwYERFRSEsLAwymQwLFixAo0aNMHfuXCxbtgyenp4IDAyEQqGARqNBeHg4hBCIjIyEvb19pX3LhBDiviOzkL7N+1k7BJKYRnIHa4dAErUlZ2et9r914KNqt3UMGFOrsWqClTER2RZegUdEJAG1WE1Rl5iMici28HJoIiIJ4DQFEZEEMBkTEUkApymIiCSAJ/CIiCSA0xRERBLAaQoiIglgZUxEJAFMxkREEmD92/GUi8mYiGxLGVdTEBFZH0/gERFJAOeMiYgkgHPGREQSwMqYiEgCmIyJiKxPGO990KgUMBkTkW1hZUxEJAFc2kZEJAEmrqYgIrI+iU5TyK0dwN/Jkz6tEb91cbnv2TvYY/n2pWj+RLP76rvL852xes8qrNi5HC+E9QMAKOwUmL5iGpamLMGqz1eiS58u9x071Z2WPq0QnfTuPds7vdAVC3bHY/6uxegV+vx99f1M72exYHc84nYsQq/QPgBu/1y8sTwCsckLMH/XYnR4/tlaxf+3YzRW/1WPWBlbyJAJg/H8y71RXFR8z3utvFvhrYVv4uGmD99X3wo7BcbHjMebAyajuKgYy3csw5H9GXg2oCNu3riJxRHxULuq8eG+f+LIN0dqeyhkQS+NH4RuIT1R8pefC5lcjvAZGkQFTUWxvhjL9q/G0a8yoL2hrXbfCjsFRkSPwqygqSi+VYK4lIXITD0K357PQHdDi39GroDKVY33vliGzP1HLX1oDy5Wxn9vuTm5eGdsXLnvNVA2wDtj5uFC9kXzNoWdAlPiI7F0WzyWpSyFdxfvu/ZJykw0f+3e0h2X/7gMXaEOZaVlOHP0NNp2aou0Pf/BJ0s2mdsZy6S5ZMeWXTl/BUvHL7pnuzCZMKX3JNzSFkHdSA3IgOKi4tu/eBdPQuzW+Xhn2wI81aXtXfutPbrR/PVjLZvhyh+50N/Uw1hahqyjZ9Hm2adweO8hbFm62dzOKNGlXFZjEtV/1aM6qYzz8/Pxww8/QKvVwsXFBT4+PmjcuHFdDCUZB79MR5NmTcp976cffrpn2wth/VBYUIhl05ZD7arG0m1LMO758Zi/KQ5KB3uoXdWI37oY+Veu4/OEvdBr9eZ9i3S34Kx2Nlfhjs6OmLt2Dj5e8kndHBzdt++/PIxHmpX/s28ymtCpXxeMihuHH7/NRFmpEc+H94W24CbWTn8fKlc1YpPnY2qfyZj5yVwo7e2hclUhOuld3MjLx9cJ+1CkLTL3d0t/C04uTuYq3MHZAVPWTMeWJYnljm+zbGU1RXJyMrZs2YIOHTrA2dkZv/76K9asWYMhQ4YgLCzM0sM9sFo8+TjadmqLJ32fBHC7Ula7qjF7+FwAtyvjaUOn/7dtCzg5O5r3dVI5Qn9TBwB45NGHEfNRND7ftAcHdv67fg+Cau37fUdw9KsMvL50Mnq83BPuT3rgyWefQksfLwCAQqGAylWNRSNu/9W19uhGzAudAwBwf9IDjqr//Vw4OjtCf/P2L+2HHn0Yb6+bia8TvkT6rrR6PiqJs5XVFCkpKfjss8/QoEED8zaDwYCwsDAm4ztcOHcR13KvI+n9LVA6KBH+Zhh0hbpy254/dx6PtXgMalcVbumL0a5TOySvTYHrw65YuHkB3p/7AY6nH6/fA6BacVQ5YvqG2ZiviUWZoQzFRcUwmQQunbuI/Nx87PznNjSwVyLkzcHQV/BzcencRTR9/FE4N1ShuKgYbTo/jc/X7UTDhxti9qcx+Ff0epxOP1nPRyZ9QqJzxhZPxmVlZSgpKbkrGRcXF0Mmk1l6KEkLGNgTjk6O+CLxy3Lf37v5C0S89xaWJC+Gk8oJn2/aA3HH3aRCO4SbvzaWGbF23jos+HQB5DIZ9m39GvlX8jExdgJUDVV49a1wvPrW7fazh8+BodhQtwdH980vuDscnByQ+tnXOLgzDbFbF8BYVobzWTn4z47voLCTY/yiNxCz5V04qp3w9aYv7/q5GP/sa+avjWVGbIrbiNkJMZDJ5TiwdT9u5BVgRMxoOLuoEPLmUIS8ORQAsHDEPJSW8OcCQL2vkqgumRCWvZ/ct99+i0WLFsHDwwNqtRo6nQ45OTmIiopCz549y92nb/N+lgyB/gYayR2sHQJJ1JacnbXaXz/v1Wq3dY7eXHUjC7F4ZdyrVy90794d2dnZ0Ol0UKlUeOKJJ2Bnx1V0RCQBEp2mqJOlbXZ2dmjdujU6dOiA1q1bw87ODsnJyXUxFBFRzVhgaVt+fj569OiB7Oxs5OTkICwsDOHh4YiJiYHpv8l+69atCAkJwdChQ3HgwIEqw6rTdcamO34DOTo6VtKSiKieCFP1X+UoLS1FdHQ0HBxuT6UtXLgQERERSExMhBACqampuHbtGhISEpCUlIQNGzZg2bJlMBgqn7O3+NzBhQsXsHDhQpw+fRp2dnYwmUzw8vJCVFSUpYciIqq5Wi5te++99xAaGop169YBAM6cOYNOnToBALp374709HTI5XL4+vpCqVRCqVTC3d0dWVlZ8Pb2rrBfiyfj2bNn4+2330b79u3N244fP46oqCgkJSVZejgiohoRtbhSdfv27XBzc0O3bt3MyVgIYV4t5uzsDK1WC51OB7Vabd7P2dkZOl35SxT/ZPFkbDAY7krEAODj42PpYYiI7k8tKuOUlBTIZDIcPnwYZ8+exYwZM1BQUGB+X6/Xw8XFBSqVCnq9/q7tdybn8lg8Gbdu3RpRUVHo1q0b1Go19Ho9vvvuO7Ru3drSQxER1VwtLofevPl/S900Gg1iY2MRHx+PjIwMdO7cGWlpaejSpQu8vb2xYsUKlJSUwGAwIDs7G15eXpX2bfFkHBsbi/379yMzM9O8tC0gIAB9+vSx9FBERDVn4cuhZ8yYgblz52LZsmXw9PREYGAgFAoFNBoNwsPDIYRAZGQk7O3tK+3H4hd93A9e9EF/xYs+qCK1vehDGxFU7bbqFZ/Xaqya4JUYRGRbJHqrWSZjIrIttnLXNiIiSWMyJiKyPgmcJisXkzER2RZWxkREEsBkTERkfaJMmrfQZDImItsizVzMZExEtkVwmoKISAKYjImIJIDTFERE1sdpCiIiCRBlTMZERNbHaQoiIuurxb3l6xSTMRHZFiZjIiLrY2VMRCQBoszaEZSPyZiIbAorYyIiCWAyJiKSAiGzdgTlYjImIpvCypiISAKEiZUxEZHVmYxMxkREVsdpCiIiCeA0BRGRBAhp3rSNyZiIbAsrYyIiCXjgTuAdPHiwwp38/f3rJBgiorr2wFXGe/furXAnJmMielCJB+0KvIULF5a7/erVq3UWDBFRXavN0jaj0Yg5c+bg999/h0KhwMKFCyGEwMyZMyGTydCqVSvExMRALpdj69atSEpKgp2dHSZOnIiAgIBK+65yznjVqlVITExEaWkpiouL8fjjj1daNRMRSZmpFpXxgQMHAABJSUnIyMgwJ+OIiAh07twZ0dHRSE1NhY+PDxISEpCSkoKSkhKEh4fDz88PSqWywr7lVQ2elpaGtLQ0BAUF4YsvvkCTJk3u+0CIiKxNCFm1X3/1/PPPIy4uDgBw+fJlPPzwwzhz5gw6deoEAOjevTsOHTqEkydPwtfXF0qlEmq1Gu7u7sjKyqo0riqTsaurK5RKJfR6PTw8PHDr1q37OX4iIkkwGWXVfpXHzs4OM2bMQFxcHAIDAyGEgEx2u62zszO0Wi10Oh3UarV5H2dnZ+h0ukrjqnKaomnTpti2bRscHR2xdOnSKjskIpIyS6ymeO+99zB16lQMHToUJSUl5u16vR4uLi5QqVTQ6/V3bb8zOZenysp43rx56Nq1K6ZPn47GjRtj+fLltTgEIiLrMglZtV9/tXPnTqxduxYA4OjoCJlMhrZt2yIjIwPA7Wndjh07wtvbG5mZmSgpKYFWq0V2dja8vLwqjavKynj37t3mr9VqNU6fPo2WLVvW6OCJiKSiNkvb+vbti6ioKLz66qsoKyvDrFmz8MQTT2Du3LlYtmwZPD09ERgYCIVCAY1Gg/DwcAghEBkZCXt7+0r7lglR+ZXaS5cu/e8BCJw9exaurq7mbZbSt3k/i/ZHD75Gcgdrh0AStSVnZ632P/l4ULXbev/xea3GqokqK+O3337b/LUQAuPHj6/TgIiI6lJtlrbVpSqTscFgMH997do1XLx4sU4DIiKqS6YH7XLoP/Xr1w8ymQxCCDg4OGDMmDEWD+LbvFMW75MebLcu/8faIdDf1ANbGa9YsQLe3t7m77///vs6DYiIqC49cPem+OGHH3Du3Dl8/PHHeO211wAAJpMJmzdvxp49e+otQCIiS3rgKmMXFxdcv34dBoMB165dAwDIZDJMmzat3oIjIrI0iT7oo+Jk7OXlBS8vLwwZMgQFBQVo06YN9u/fj+eee64+4yMisiijqcpr3ayiyqjmz5+P48ePAwB+//13zJw5s65jIiKqM6YavOpTlck4Ly8PYWFhAICxY8fyfsZE9EATkFX7VZ+qVa///vvvAICcnByYTPX9+4KIyHJMovqv+lTl0rZZs2YhIiIC+fn5cHBwwKBBg+ojLiKiOmGq54q3uqqsjNu3b4+4uDg899xzuHXrFvLz8+sjLiKiOiHVaYoKK2ODwYC9e/di8+bNUCqV0Ol0SE1NhYMDb+BCRA8u44NWGffq1Qs///wzlixZgsTERDRu3JiJmIgeeFJdTVFhZTx8+HDs2bMHly5dwuDBg1HFnTaJiB4IUl2CUGFlPG7cOOzevRsajQZ79uzB6dOnER8fj19++aU+4yMisiipzhlXeQKvU6dOiI+PxzfffIOmTZti+vTp9REXEVGdMMmq/6pPVT7poz7YKR+zdggkMbyFJlWkwcOetdp/V9PwarcNvpJYq7Fqosp1xkREfydGawdQASZjIrIpJpk0l7YxGRORTbH6vGwFmIyJyKZIdWkbkzER2RSJPo+UyZiIbItUL4dmMiYim8LKmIhIAjhnTEQkAVxNQUQkAZymICKSAE5TEBFJgJGVMRGR9bEyJiKSACZjIiIJqM1qitLSUsyaNQuXLl2CwWDAxIkT0bJlS8ycORMymQytWrVCTEwM5HI5tm7diqSkJNjZ2WHixIkICAiotG8mYyKyKbVZTbF79264uroiPj4eN27cwKBBg/Dkk08iIiICnTt3RnR0NFJTU+Hj44OEhASkpKSgpKQE4eHh8PPzg1KprLBvJmMisim1mabo168fAgMDzd8rFAqcOXMGnTp1AgB0794d6enpkMvl8PX1hVKphFKphLu7O7KysuDt7V1h31U+domI6O/EWIPXXzk7O0OlUkGn02Hy5MmIiIiAEAKy/94j2dnZGVqtFjqdDmq1+q79dDpdpXExGRORTantM/Byc3MxfPhwBAcHIygoCHL5/9KoXq+Hi4sLVCoV9Hr9XdvvTM7lYTImIptiqsHrr65fv45Ro0Zh2rRpGDx4MADgqaeeQkZGBgAgLS0NHTt2hLe3NzIzM1FSUgKtVovs7Gx4eXlVGhfnjInIptRmNcWaNWtw8+ZNfPDBB/jggw8AALNnz8a7776LZcuWwdPTE4GBgVAoFNBoNAgPD4cQApGRkbC3t6+0bz4dmiSJT4emitT26dDzPV6tdtvZOZtrNVZNsDImIpvCp0MTEUkAr8AjIpIA3kKTiEgCTBK9vTyTMRHZFGmmYiZjIrIxnDMmIpIAo0RrYyZjIrIprIyJiCSAJ/CIiCRAmqmYyZiIbAynKYiIJIAn8IiIJIBzxlSpGdMnIWhAXzRQNsCaNZ9g48dJ1g6JLGTwyDegUjkDAJo92hTvzp5ifu/U2Z8Rv2o9BAQedmuERdHTYW9f8XPSyvPvg0fw4cZE2CkUGDSgLwa/9AJKy8owd8FyXM7Ng6G0FONHhCGgWxeLHteDSpqpmMlYEnp074quXTuiW49gODk54u0pE6wdEllISYkBAPDx+4vveU8IgdhFq7B8/my4N/sHtu3eh8tXrqKFR7Nq919aVob3Vq1D0kcr4eTogGET3kZPv874z5Ef4OqixqLoafi/wpsY/NokJuP/YmVMFerbtwdOn85CyrYNcFGrMWNmnLVDIgv5+dxvKC4uwdiIWTAaTXhr/Ai0b9sGAPDH+UtwbahGwpad+PW339H9uU5o4dEMpWVlmBe/GucvXIZJmPDm2BHo9Mz/HmTZIygc332eCAD47Y8LcG/2DzR0uf1In2e8n0bmiTMIDOiGvj39zfvYKRT1eNTSxhN4VKGHHnKDh3szvDRwBFq0cMeO7RvxdNvu1g6LLMDBwR4jw1/Gy0H9kHPhEia8PRd7PvsIdnYK3CgsxPFTZzErciLcmz+GN6bF4KnWLfHHhUto1NAFcVGR+L/Cmxjx+jTs2rwWE96ei+KSEhTe1GLkpOlo8vBDeGVQf6icnc3jOTs5QqvTw8nJEQCg1xchcvZ8vDl2uLU+AskRtlQZ5+fn44cffoBWq4WLiwt8fHzQuHHjuhjqb6Gg4AZ+/jkbpaWl+OWXbBQXl+CRRx7CtWv51g6Naunx5o/Bvdk/IJPJ8Lh7M7g2dMG1/AI82uQRuDZ0gXuzf+CJFh4AAP/OHfHTz+dwKTcPx06cwcmffgYAGI1G/F/hTaxZevsvph5B4eZpj5/P/Y6ioiLzePqiW3BR307OuXnX8FZUHEJD+qN/34D6PGxJk+pqCos/kDQ5ORnjx4/HsWPHcPnyZWRmZmLChAn47LPPLD3U30Z6+lEE9u0JAHj00SZwdnJEfv4N6wZFFrF979eIX70eAHD1Wj70+iI88pAbAKD5P5qi6NYtnL94GQCQeeI0nmjhgRYezfHC8z3w8fuLsWZpHPr26gYXtarc/j0fb46ci5dReFOL0tJSZJ44jfZt2+B6wQ2Mi5yNKa+/hpABgfVzsA+I2jyQtC5Z/Bl4oaGhSEhIQIMGDczbDAYDwsLCkJKSUu4+fAYesGjhbPTo8Rzkcjnmzl2Er7/5ztohWdXf5Rl4paWlmD1/GXLzrkIGGSJfH4XLuXkounULQ4JfREbmcSz/cCOEEPBp9xSiIibAYDAg5r1VyL2SB52+CKEhAzD4pRcqHOPP1RRCCAzq3xdhLwdh4Yo12JeadtfJwDVL4+BQxUMxHwS1fQaexiOk2m0TcrbXaqyasHgyHjx4MD7++GOoVP/7TX7z5k2MGjUK27ZtK3cfJmP6q79LMibLq20yHlaDZPxpPSZji88Zv/766wgJCYGHhwfUajV0Oh1ycnIQFRVl6aGIiGrMZpa29erVC927d0d2djZ0Oh1UKhWeeOIJ2Nlx4QYRWZ9Nraaws7ND69at79qWnJyMIUOG1MVwRETVVmZLybg8jo6O9TUUEVGFbKoyLs+AAQPqaygiogrZzBV4Go0GpaWld20TQkAmkyEpiTe/ISLrsvACMouxeDKeOnUq5syZg3/+859Q8Hp4IpIYm1lN0b59ewQHB+Pnn39Gnz59LN09EVGtSPVy6DqZMx4zZkxddEtEVGs2UxkTEUmZVOeMLX6jICIiKbPEjYJOnDgBjUYDAMjJyUFYWBjCw8MRExMDk+n2nlu3bkVISAiGDh2KAwcOVBkXkzER2RRRg//Ks379esyZMwclJSUAgIULFyIiIgKJibdv1pSamopr164hISEBSUlJ2LBhA5YtWwaDwVBpXEzGRGRTTBDVfpXH3d0dq1evNn9/5swZdOrUCQDQvXt3HDp0CCdPnoSvry+USiXUajXc3d2RlZVVaVxMxkRkU4zCVO1XeQIDA++6186f11EAgLOzM7RaLXQ6HdRqtbmNs7MzdDpdpXHxBB4R2RRLXw4tl/+vptXr9XBxcYFKpYJer79r+53Judx+LBoVEZHEmYSo9qs6nnrqKWRkZAAA0tLS0LFjR3h7eyMzMxMlJSXQarXIzs6Gl5dXpf2wMiYim2LphW0zZszA3LlzsWzZMnh6eiIwMBAKhQIajQbh4eEQQiAyMhL2VTxlxeJP+rgffNIH/RWf9EEVqe2TPvwe61XttumXvq3VWDXBypiIbAqvwCMikoCKVklYG5MxEdkUm7+5PBGRFEjgNFm5mIyJyKZwzpiISAJYGRMRSYBRok/BYzImIptS3Svr6huTMRHZFK6mICKSAFbGREQSwMqYiEgCWBkTEUkAL4cmIpIATlMQEUmAYGVMRGR9vByaiEgCeDk0EZEEsDImIpIAo4lzxkREVsfVFEREEsA5YyIiCeCcMRGRBLAyJiKSAJ7AIyKSAE5TEBFJAKcpiIgkgLfQJCKSAK4zJiKSAFbGREQSYOItNImIrI8n8IiIJECqyVgmpBoZEZENkVs7ACIiYjImIpIEJmMiIglgMpYAk8mE6OhovPLKK9BoNMjJybF2SCQhJ06cgEajsXYYVMe4mkIC9u/fD4PBgC1btuD48eNYtGgRPvzwQ2uHRRKwfv167N69G46OjtYOheoYK2MJyMzMRLdu3QAAPj4+OH36tJUjIqlwd3fH6tWrrR0G1QMmYwnQ6XRQqVTm7xUKBcrKyqwYEUlFYGAg7Oz4B6wtYDKWAJVKBb1eb/7eZDLxHyCRjWEyloBnnnkGaWlpAIDjx4/Dy8vLyhERUX1j+SUBffr0QXp6OkJDQyGEwIIFC6wdEhHVM14OTUQkAZymICKSACZjIiIJYDImIpIAJmMiIglgMiYikgAmY7KIjIwMdO3aFRqNBhqNBkOHDkVCQkKN+1myZAm2b9+Os2fP4v3336+w3TfffIO8vLxq9ZmWloaZM2fWOBai+sR1xmQxXbp0wfLlywEABoMB/fr1Q3BwMFxcXGrcV5s2bdCmTZsK39+0aRNiY2PRpEmT+46XSEqYjKlO6HQ6yOVyjBw5Es2aNcPNmzexbt06xMbGIicnByaTCREREejcuTO++uorfPjhh3Bzc0NpaSk8PT2RkZGBpKQkLF++HMnJyfjss89gMpnQu3dvtGvXDmfPnsWMGTOQmJiILVu2YM+ePZDJZHjxxRcxfPhwZGdnY9asWXB0dISjoyMaNmxo7Y+EqFJMxmQxR44cgUajgUwmQ4MGDTB37lx89NFHCAoKQp8+fZCYmIhGjRphwYIFuHHjBoYNG4a9e/ciPj4eycnJcHV1xbhx4+7qMz8/33wbSaVSiUWLFuHZZ59FmzZtEBsbi/Pnz+OLL75AYmIiZDIZRo4cCX9/f6xcuRKTJ0+Gn58f1q1bh99++81KnwpR9TAZk8XcOU3xp48++ggtWrQAAPzyyy/IzMzEyZMnAQBlZWW4fv06VCoVGjVqBADw9fW9a/8LFy6gVatWcHBwAADMmjXrrvd/+eUXXL58GSNHjgQAFBYW4vz58/j111/h7e0N4Pa9P5iMSep4Ao/qnEwmAwB4enqif//+SEhIwPr169GvXz+4uLhAq9WioKAAAHDq1Km79nV3d8dvv/0Gg8EAAJg8eTLy8vIgk8kghICnpydatmyJTZs2ISEhASEhIfDy8oKnpyd+/PFHAOD9oemBwMqY6k1oaCjmzJmDYcOGQafTITw8HEqlEgsXLsTo0aPRsGHDe24d6ubmhrFjx2LYsGGQyWQICAhAkyZN4Ovri+nTp+Nf//oXunbtirCwMBgMBnh7e6NJkyaIiYlBZGQkNmzYADc3N9jb21vpqImqhzcKIiKSAE5TEBFJAJMxEZEEMBkTEUkAkzERkQQwGRMRSQCTMRGRBDAZExFJAJMxEZEE/D+L8oxbTHKyhQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEPCAYAAACX9KXAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjuElEQVR4nO3de1xUdf4/8NfAcJOLyjdl20xXQsxLiGmigiSaggKCpCLkqKl5K1lc84oKpablXVsv2cUCwSQyTV1bI1dWTUrNG3lJNCQvmMrCzKjMMPP5/eHPKRQQ5sIcnNdzH+fxmDlzzufzPlTv+ez7fM5nZEIIASIisio7awdARERMxkREksBkTEQkAUzGREQSwGRMRCQBTMZERBLAZGwGOp0On3zyCWJiYhAVFYX+/ftj8eLF0Gg0JrU5YcIEhIaGIi0trdbnnzx5EgkJCUb3b25KpRLDhw+v8vOoqCiUlpbWuL0vv/wSPXv2xOjRo42O6cSJE5g7d67R5xcWFmLSpEk1OjYiIgK5ublma6+2/vOf/2DlypUWaZvMQ27tAB4HKSkpKCkpwaeffgp3d3fcvn0bb775JpKSkrB48WKj2iwqKsL+/ftx7Ngx2Nvb1/r85557DqtWrTKqb0soKSnByZMnq/x827ZttWrvq6++wuTJkxEVFWV0TOfPn0dRUZHR51+5cgUXL140+nxLt/dnJ0+eRElJiUXaJjMRZJLCwkLRoUMHoVQqK+y/fv262L17txBCiNLSUjFlyhQRHh4uIiIixLvvviu0Wq0QQoj27duLVatWidjYWBESEiI2bdoklEql6Nevn3j22WfFgAEDREFBgfD19RU3b940tH//vUqlEpMmTRIDBgwQ0dHRIikpSeh0OnHo0CERHh5uVP+Vad++vVi6dKl4+eWXRb9+/cTOnTvFpEmTRGhoqFAoFEKtVgshhMjMzBSDBg0SUVFRomfPnob2hg0bZrie8vJy0a5dO5GQkCD69u0rTpw4Ybie1atXi9jYWFFeXi6uX78uAgMDxffff18hlgULFgh/f38REhIiPvnkk2qv78F+7rty5Yp48cUXxfPPPy9mzJghhBAiOzvbEHtsbKw4evSoEEKI8+fPi9jYWDFw4EARHR0t0tLSRHl5uXjppZfEc889J0aNGvXQ3+uXX34RgwcPFhERESIhIUGEhISIQ4cOCSGEWLt2rRg0aJCIiIgQvXv3Fv/+978rba+y46qK5741a9aI6OhoMWDAADFhwgRx7do1cezYMdG9e3cREBAgli1bVuk/X7I+JmMT7d69W7z88svVHjNt2jQxb948odfrRVlZmRg1apRYv369EOJeUk1NTRVCCHHy5EnRvn17cffuXVFYWCj8/f0NbVSVjLdu3Wr4j7e8vFwkJSWJX3/9tUIyNqb/B/n6+opPP/1UCCHE+vXrRceOHcW1a9eETqcTAwcOFNu3bxcqlUoMGTJE3Lp1SwghxE8//WS4hsquZ+vWrQ9dT3l5uXjllVfE+vXrxciRI8XatWsr/ZsOGzZM/Otf/6rR9f25nz/LysoSY8eOFUIIcfHiRREREWGI/dy5cyIwMFCo1Woxc+ZMQ3vXr18XiYmJD33hPSgqKkps2bJFCCHE4cOHRevWrcWhQ4fEb7/9JhQKhbhz544QQogdO3aIiIgIIYSo0F51x1UVz9atW0ViYqLhi2jz5s1izJgxQgghVq1aJd56661KYyVpYJnCRHZ2dtDr9dUek5OTg4yMDMhkMjg6OmLo0KH49NNPMXbsWABA7969AQDt2rWDRqPB7du3a9x/p06dsHz5cigUCnTv3h0jRoxAixYtcO3aNZP6d3Jyeqiv0NBQAEDz5s3h6+sLLy8vAECzZs1QUlICV1dXrFu3Dvv27cOvv/6KM2fOVHstnTt3fmifvb09lixZgsjISLRr1w7jxo175N/gUddXWT8POnDgAK5fv46RI0ca9slkMly6dAl9+vTB9OnTceLECXTr1g2zZ8+GnV3Vt1uKi4tx9uxZREdHA7j3z6hVq1YAgKeeegrvvfcevv76axQUFOD48eNQq9UPtVHdcVXFs3fvXpw8eRIvv/wyAECv1+POnTuPvHaSBt7AM5Gfnx8uXLgAlUpVYX9RURHGjh2Lu3fvQq/XQyaTGT7T6/UoLy83vL+f+O4fIx6xXMifbww+/fTT2LNnD8aOHQuVSoVXX30V3333XYXjzdW/g4NDpa/vu3btGqKjo3H58mV06tQJiYmJ1V5HgwYNKt1/+fJlODk54dKlSzWqcz7q+qrq58E2unXrhm3bthm2LVu2oFWrVggJCcE333yDfv364fTp04iMjKzwZVeVP/8d5fJ74568vDzExsZCpVIhMDAQY8aMqfTc6o6rKh69Xo8xY8YY4s/KykJGRsYj4yRpYDI2kZeXFyIjIzFr1ixDQlapVEhJSUGjRo3g7OyMoKAgpKWlQQgBjUaDLVu2oHv37rXqx9PT03ADbMeOHYb96enpmDlzJoKCgjB16lQEBQXh559/rnCuOfqviVOnTsHT0xMTJ05EUFAQ9u7dC+DezBC5XA6dTvfIL5rS0lJMnToVixYtQkREBJKSkh7Zr7HXZ29vb0ja3bp1w4EDB5Cfnw8A2LdvHwYMGIC7d+9iypQp2LVrF8LDw5GcnAw3NzdcunQJ9vb20Gq1D7XbuHFjtGvXDpmZmQDuJdZz584BAH788Ue0b98er776Krp06YLs7GzodDpDPPfbq+64quIJCgrCF198Yfj3cOXKlZg2bdpD10rSxGRsBsnJyfDx8cHQoUMRFRWFwYMHw8fHB/PnzwcAzJ49G7du3UJkZCQiIyPRsmVLjB8/vlZ9zJ49G2+//TYGDhyI/Px8NGnSBAAQHR0NnU6H/v37IyYmBkqlEgqF4qFzTe2/JgIDA+Hl5YWwsDD069cPV69ehaenJwoKCtCkSRP4+fkhPDwcxcXF1V5nz549ERQUhDfeeAOFhYXYtGlTtf0ae33+/v4oLCzEG2+8AR8fH7z99tv4xz/+gQEDBmDlypVYu3YtXF1dMXHiRHz99dcYMGAAhgwZgpdeegkvvPACfHx84OTkhEGDBj30JbNs2TLs2rULkZGRWLNmDby9vQHcm+JWXFyMfv36oX///mjQoAFKSkqgUqkqtFfdcVXFM3jwYPTs2RNDhgxBeHg4zp49i0WLFgEAunbtiv3792PevHmP/LuQdcjEo4YqRERkcRwZExFJAJMxEZEEMBkTEUkAkzERkQQwGRMRSYAknsCTOz5l7RBIYnY17mHtEEii+hZtNul87Y0LNT7W4Qlvk/qqDUkkYyKiOqPXWTuCSjEZE5FtEdWvJWMtTMZEZFsesbCXtTAZE5FNERwZExFJgE6aCyYxGRORbeENPCIiCWCZgohIAngDj4jI+ngDj4hICiQ6MubaFERkW3Tamm9VOH78uOEXdU6fPo34+HgoFAqMHj0aN27cAABs2bIFMTExGDJkiOEnyKrDkTER2RYTyxQbNmzA9u3b4eLiAgBYsGAB5syZgzZt2mDz5s3YsGEDxowZg9TUVGRlZaGsrAzx8fEIDAyEo6Njle1yZExEtkWvr/lWiebNm2P16tWG98uWLUObNm0A3PvxXScnJ5w4cQIdO3aEo6Mj3N3d0bx5c5w5c6basJiMici2CH3Nt0qEhoZCLv+jqNC0aVMAwNGjR5GWloaRI0dCpVLB3d3dcIyrq6vhV7urwjIFEdkWC9zA27VrF9auXYsPPvgAnp6ecHNzg1qtNnyuVqsrJOfKcGRMRDZF6LU13mpi27ZtSEtLQ2pqKp5++mkAgJ+fH44cOYKysjIolUrk5+fD19e32nY4MiYi22LGkbFOp8OCBQvw5JNPYtKkSQCAF154AQkJCVAoFIiPj4cQApMnT4aTk1O1bcmEEMJskRmJv/RBD+IvfVBVTP2lj7tHvqrxsc6dok3qqzY4MiYi28KFgoiIJICPQxMRSYBEH4dmMiYi28LF5YmIJIAjYyIi6xOCN/CIiKyPI2MiIgngbAoiIgngyJiISAI4m4KISAJYpiAikgCWKYiIJIDJmIhIAlimICKSAN7AIyKSAJYpiIgkgGUKIiIJ4MiYiEgCmIyJiCTA+j/7WSkmYyKyLeWcTUFEZH28gUdEJAGsGRMRSQBrxkREEsCRMRGRBEg0GdtZOwAiorokdLoab1U5fvw4FAoFAKCgoABxcXGIj49HcnIy9P8/2W/ZsgUxMTEYMmQI9u7d+8i4ODImItti4sh4w4YN2L59O1xcXAAACxcuRGJiIgICAjB37lxkZ2fD398fqampyMrKQllZGeLj4xEYGAhHR8cq2+XImIhsi9DXfKtE8+bNsXr1asP7vLw8dOnSBQAQHByMgwcP4sSJE+jYsSMcHR3h7u6O5s2b48yZM9WGxWRMRLZFL2q+VSI0NBRy+R9FBSEEZDIZAMDV1RVKpRIqlQru7u6GY1xdXaFSqaoNi2UKIrItZr6BZ2f3x5hWrVbDw8MDbm5uUKvVFfb/OTlXhslYAuzs7LB+3WK09n0GOp0Oo1/7By5cKLB2WGQlLROi0CS0E+wc5CjcuAe3Dv6M9qsmAAJQnSnE6RkfS3aubL1QzY05Y7Rt2xa5ubkICAhATk4OunbtCj8/P6xYsQJlZWXQaDTIz8+Hr69vte2wTCEBERF9AADBPaOR8tYSLFmcbOWIyFoad2+LRi/44oeIZPwY/Rac//p/aP22AucXbcGPUSmADGga1tnaYdZven3NtxqYPn06Vq9ejdjYWGi1WoSGhqJJkyZQKBSIj4/HiBEjMHnyZDg5OVXbjkwI63/Fyh2fsnYIVmdvbw+dTgeFYjC6d+uMCROnWzskq9rVuIe1Q7AKn6ShgADcWjeD3N0F597aBP9P30SO/0QAQJOwTvi/F/1wZuYnVo7UevoWbTbp/NtLxtT42AZvfmhSX7VhkTLFzZs3cfjwYSiVSnh4eMDf3x9Nmza1RFePDZ1Oh48/WoHoqDDEDh1r7XDIShw93eHcrAl+GvYuXJo3RcfPpkJmJzN8rlPdhdyjgRUjfAxIdKEgs5cpMjMzMW7cOBw9ehRXrlzBkSNHMH78eGRkZJi7q8fOqNGJaNOuB9atXYwGDVysHQ5ZgbZYhZt7j0NodbidfxX6Mi3k7n8kX3s3Z5SX3LZihI8BE2dTWIrZR8ZZWVnIyMiAg4ODYZ9Go0FcXBzi4uLM3d1j4ZVXXkazp57Eu++9j9u370Cv10Onk+a3N1lWce5ZtHgtDAXrdsLJqzHsGzjh1v5TaNy9LYoP/ownevnj1oGfrR1mvSYk+ji02ZNxeXk5ysrKKiTju3fvGubh0cO2bt2Fjz5cjr3ZWXBwcMA/3kxGWVmZtcMiK7ix5ygad30WAbsXQGYnw+kZH+POpd/RdulrsHOUQ/3LZRR9fcjaYdZvZp5NYS5mT8YTJ05ETEwMWrRoAXd3d6hUKhQUFGDmzJnm7uqxcfv2HcTFj7d2GCQRv8xLf2jf4YFvWyGSx1Qdlx9qyuzJuFevXggODkZ+fj5UKhXc3NzwzDPPVHhihYjIamylTAEAcrkcrVu3rrAvMzMTgwcPtkR3REQ1Zysj4z/T6/WGRwXvr3BERGRVEp3aZvZkXFhYiIULF+LUqVOQy+XQ6/Xw9fVlzZiIpMFWRsZJSUmYMmUKOnToYNh37NgxzJw5E5s3m/bkDBGRqUS5jcym0Gg0FRIxAPj7+5u7GyIi49jKyLh169aYOXMmevToAXd3d6jVauzbt++hG3pERFZhKzXjlJQUfPvttzhy5IhhaltISAj69Olj7q6IiGrPVkbGMpkMffr0YfIlIkkStpKMiYgkzVZu4BERSRpHxkREEsBkTERkfRL4caNKMRkTkW3hyJiISAKYjImIrE+U28hDH0REkibNXMxkTES2hQ99EBFJAZMxEZEEsExBRGR9LFMQEUmAKGcyJiKyPhPKFFqtFjNmzMDly5dhZ2eHefPmQS6XY8aMGZDJZGjVqhWSk5MNv/1ZG0zGRGRTTFlbft++fSgvL8fmzZtx4MABrFixAlqtFomJiQgICMDcuXORnZ1t1BLCtU/fRET1mb4W2wNatmwJnU4HvV4PlUoFuVyOvLw8dOnSBQAQHByMgwcPGhUWR8ZEZFNMGRk3aNAAly9fRr9+/VBcXIx169bhxx9/hEwmAwC4urpCqVQa1TaTMRHZFFFu/LkbN25EUFAQpkyZgqtXr2LEiBHQarWGz9VqNTw8PIxqm2UKIrIpQl/z7UEeHh5wd3cHADRs2BDl5eVo27YtcnNzAQA5OTno3LmzUXHJhAQW95Q7PmXtEEhidjXuYe0QSKL6Fm026fyikBdrfKzX3n0V3qvVasyaNQu///47tFothg8fjvbt22POnDnQarXw9vbG/PnzYW9vX+u4WKYgItsiZEaf6urqipUrVz60Py0tzZSIADAZE5GNMeUGniUxGRORTRF640fGlsRkTEQ2Ra9jMiYisjqWKYiIJIBlCiIiCbD+ZN7KMRkTkU3hyJiISALq3Q28/fv3V3lSUFCQRYIhIrK0ejcy3rlzZ5UnMRkTUX0lTHgCz5KqTMYLFy6sdP/169ctFgwRkaXV26ltq1atQnp6OrRaLe7evYu//e1v1Y6aiYikTC/RkfEjl9DMyclBTk4OIiMjsWvXLnh5edVFXEREFiGErMZbXXrkyLhRo0ZwdHSEWq1GixYtcOfOnbqIi4jIIurdbIr7/vKXv+CLL76Ai4sLli5dCpVKVRdxERFZRL2bTXHf22+/jatXryIsLAxbt27F8uXL6yIuIiKLkGrN+JHJePv27YbX7u7uOHXqFHx8fCwaFBGRpdS7qW335efnAwCEEDh9+jQaNWqE6OhoS8dFRGQR9XZtiilTphheCyEwbtw4iwZERGRJ9bZModFoDK9///13/PbbbxYNiIjIkvT19QZeWFgYZDIZhBBwdnbGmDFjzB5EI2dXs7dJ9VtI3jvWDoEeU/V2ZLxixQr4+fkZ3v/www8WDYiIyJLq3Q28w4cP4/z589i4cSNeffVVAIBer8emTZuwY8eOOguQiMic6t3I2MPDAzdu3IBGo8Hvv/8OAJDJZJg6dWqdBUdEZG4SnUxRdTL29fWFr68vBg8ejFu3bqFNmzb49ttv0b1797qMj4jIrHT6Ry7JYxWPjGrBggU4duwYAODixYuYMWOGpWMiIrIYfS22uvTIZFxUVIS4uDgAwGuvvcb1jImoXhOQ1XirSzX6DbyLFy+iZcuWKCgogF4v0ZWZiYhqQG9i0Xj9+vX47rvvoNVqERcXhy5dumDGjBmQyWRo1aoVkpOTYWdX+1LII8+YNWsWEhMTERQUhNGjR7NmTET1mh6yGm8Pys3NxU8//YSMjAykpqbi2rVrWLhwIRITE5Geng4hBLKzs42K65HJuEOHDpg3bx66d++OO3fu4ObNm0Z1REQkBaaUKfbv3w9fX1+8/vrrGD9+PHr27Im8vDx06dIFABAcHIyDBw8aFVeVZQqNRoOdO3di06ZNcHR0hEqlQnZ2NpydnY3qiIhICnQm1IKLi4tx5coVrFu3Dr/99hsmTJgAIQRksntturq6QqlUGtV2lSPjXr164ezZs1iyZAnS09PRtGlTJmIiqvdMmU3RqFEjBAUFwdHREd7e3nBycqqQfNVqNTw8PIyKq8pkPHz4cBw8eBBLly7Fvn37IKS67hwRUS2Ykow7deqE//73vxBCoKioCHfu3EG3bt2Qm5sL4N5vhnbu3NmouGTiEVn2hx9+QGZmJnJycjBo0CBERUXB19fXqM6q8oSHeduj+u/qhd3WDoEkyuEJb5PO3+kVV+Njw4syHtr33nvvITc3F0IITJ48Gc2aNcOcOXOg1Wrh7e2N+fPnw97evtZxPTIZ31daWopt27YhKysLX331Va07qg6TMT2IyZiqYmoy/vovNU/GkdceTsaWUuPJcB4eHlAoFGZPxEREdcmUqW2WVKOHPoiIHhc6awdQBSZjIrIpelk9W0KTiOhxJNV5YUzGRGRTpLq6DpMxEdkUif4eKZMxEdkWUx6HtiQmYyKyKRwZExFJAGvGREQSwNkUREQSwDIFEZEEsExBRCQBOo6MiYisjyNjIiIJYDImIpIAzqYgIpIAzqYgIpIAlimIiCSAi8sTEUkAyxRERBLAMgURkQRwNgURkQToJZqOmYyJyKbwBh4RkQSwZkxEJAGcTUFEJAFSrRnbWTsAIqK6JGqxVeXmzZt48cUXkZ+fj4KCAsTFxSE+Ph7JycnQ640rhDAZE5FN0ddiq4xWq8XcuXPh7OwMAFi4cCESExORnp4OIQSys7ONiovJmIhsig6ixltl3n33XQwdOhRNmzYFAOTl5aFLly4AgODgYBw8eNCouJiMicimmDIy/vLLL+Hp6YkePXoY9gkhIJPduyvo6uoKpVJpVFy8gUdENsWUG3hZWVmQyWT4/vvvcfr0aUyfPh23bt0yfK5Wq+Hh4WFU20zGRGRTTJlLsWnTJsNrhUKBlJQULF68GLm5uQgICEBOTg66du1qVNssUxCRTTH1Bt6Dpk+fjtWrVyM2NhZarRahoaFGxcWRMRHZlKpuzNVWamqq4XVaWprJ7TEZE5FNkepDH0zGEuDo6IDVaxehxd+ehlKpwvQpb+FCfoG1wyIzOZF3BsvWfoyN779XYf+evfvxYdoWyGQyDBrQD4MGhNW67f/sP4S1n6RDbm+PgRF9MWhAP2jLyzHnneW4crUIGq0W40bEIaSHcXXMx5E0UzGTsSQoRsZCrbqNsN5D4OPTEouWzMWQgaOtHRaZwcebMvH17u/g4uxUYb9Op8PydZ/g849WoYGLM6JeGYfewd3QuFHDGretLS/Hu6s+wOYPV6KBizOGjZ+CnoEB+O+hw2jk4Y5Fc6fifyWlGPTqG0zGfyLVkTFv4ElA62efwbd7cgAA589fhK/vM1aOiMzl6b8+iRXvzH5ov729PbZv+gDubq74X6kSAkADF5d7o9qFyzFi4lQoJkzBD0dPVDjvxch4w+sLvxaiebO/oqGHOxwcHPC8XzscOZ6H0JAemPTacMNxcnt7i11ffWTuG3jmwpGxBJw6cQZ9w3pi14496PRCBzz5Vy/Y2dkZ/Yw7SUefkCBcvlpU6WdyuT32/OcAFiz7J4K7dYFcbo/M7f9C44YemDdzMv5XUooRE6di26b1GD9lDu6WlaGkVImRb0yD1xP/h9iB4XBzdTW059rABUqVGg0auAAA1OrbmJy0oEJiJkBIdGRskWR88+ZNHD58GEqlEh4eHvD39zc8OkgP25T6BXxbe2PbzlTkHjqK48fymIhtRJ+egegd3A1JC5Zh++5s/JL/K44ez8OJn88CuFfO+F9JKdYtnQfg3sj4fu357PmLuH37tqEt9e078HC/l5yvFv2Ov8+ch6Ex4QjvG1LHVyVt5ppNYW5mL1NkZmZi3LhxOHr0KK5cuYIjR45g/PjxyMjIMHdXj42OnZ7Doe+PICpcgV079qDg10Jrh0QWplKrMfL1qdBoNLCzs4OLsxPsZDK0bPE0+r30Ija+/x7WLZ2Hvr16wMPdrdI2vP/2NAp+u4KSUiW0Wi2OHD+FDu3b4MatYoydnIR/THwVMRHGzXl9nNlMmSIrKwsZGRlwcHAw7NNoNIiLi0NcXJy5u3ssXDhfgJlJiXg9YTRKSpRIfH2WtUMiC9n57724fecOBkf1R3jfEIx4fRrkcnv4PtMSEaG9oNPpkPzuKox8fSpU6tsYGhMBO7s/xkz7vk43vHaQyzFt0msYOzkJQggMDO8LryZPYOGKdShVqrBuYwbWbbw3CFq3dB6cnZweiscW6YU0R8YyIcwb2aBBg7Bx40a4uf3xbV5aWopRo0bhiy++qPScJzx8zRkCPQauXtht7RBIohye8Dbp/GEtYmp8bFrBlyb1VRtmHxlPnDgRMTExaNGiBdzd3aFSqVBQUICZM2eauysiolqT6tQ2syfjXr16ITg4GPn5+VCpVHBzc8MzzzwDuZwTN4jI+mxqNoVcLkfr1q0r7MvMzMTgwYMt0R0RUY2V21IyroyLi0tddUVEVCWbGhlXJiIioq66IiKqklRn8Js9GSsUCmi12gr77v8syebNm83dHRFRrZh5ApnZmD0Zv/nmm5g9ezb++c9/wp7PxBORxNjMbIoOHTogKioKZ8+eRZ8+fczdPBGRSaT6OLRFasZjxoyxRLNERCazmZExEZGU2UzNmIhIymxmNgURkZTZ/DxjIiIpYM2YiEgCdEKahQomYyKyKSxTEBFJgFQXl2cyJiKbIs1UzGRMRDaGN/CIiCTAlGSs1Woxa9YsXL58GRqNBhMmTICPjw9mzJgBmUyGVq1aITk5ucLvFtYUkzER2RRTZlNs374djRo1wuLFi1FcXIyBAwfi2WefRWJiIgICAjB37lxkZ2cbtS5P7dM3EVE9JmrxvweFhYXh73//u+G9vb098vLy0KVLFwBAcHAwDh48aFRcTMZEZFOEEDXeHuTq6go3NzeoVCokJCQgMTHRsF77/c+VSqVRcTEZE5FN0UPUeKvM1atXMXz4cERFRSEyMrJCfVitVsPDw8OouJiMicimmDIyvnHjBkaNGoWpU6di0KBBAIC2bdsiNzcXAJCTk4POnTsbFRdv4BGRTdGZsG7bunXrUFpaijVr1mDNmjUAgKSkJMyfPx/Lli2Dt7c3QkNDjWpbJiSwuOcTHr7WDoEk5uqF3dYOgSTK4Qlvk85v79W1xseeKjpkUl+1wZExEdkUrk1BRCQBXJuCiEgCODImIpIAjoyJiCSAi8sTEUkAyxRERBIgODImIrI+rmdMRCQBEnjOrVJMxkRkUzgyJiKSAJ2eNWMiIqvjbAoiIglgzZiISAJYMyYikgCOjImIJIA38IiIJIBlCiIiCWCZgohIAriEJhGRBHCeMRGRBHBkTEQkAXouoUlEZH28gUdEJAFSTcYyIdXIiIhsiJ21AyAiIiZjIiJJYDImIpIAJmMJ0Ov1mDt3LmJjY6FQKFBQUGDtkEhCjh8/DoVCYe0wyMI4m0ICvv32W2g0Gnz++ec4duwYFi1ahLVr11o7LJKADRs2YPv27XBxcbF2KGRhHBlLwJEjR9CjRw8AgL+/P06dOmXliEgqmjdvjtWrV1s7DKoDTMYSoFKp4ObmZnhvb2+P8vJyK0ZEUhEaGgq5nP8H1hYwGUuAm5sb1Gq14b1er+d/gEQ2hslYAp5//nnk5OQAAI4dOwZfX18rR0REdY3DLwno06cPDhw4gKFDh0IIgXfeecfaIRFRHePj0EREEsAyBRGRBDAZExFJAJMxEZEEMBkTEUkAkzERkQQwGZNZ5Obmolu3blAoFFAoFBgyZAhSU1Nr3c6SJUvw5Zdf4vTp03j//ferPG7Pnj0oKiqqUZs5OTmYMWNGrWMhqkucZ0xm07VrVyxfvhwAoNFoEBYWhqioKHh4eNS6rTZt2qBNmzZVfv7ZZ58hJSUFXl5eRsdLJCVMxmQRKpUKdnZ2GDlyJJo1a4bS0lJ88MEHSElJQUFBAfR6PRITExEQEIBvvvkGa9euhaenJ7RaLby9vZGbm4vNmzdj+fLlyMzMREZGBvR6PXr37o3nnnsOp0+fxvTp05Geno7PP/8cO3bsgEwmQ//+/TF8+HDk5+dj1qxZcHFxgYuLCxo2bGjtPwlRtZiMyWwOHToEhUIBmUwGBwcHzJkzBx9++CEiIyPRp08fpKeno3HjxnjnnXdQXFyMYcOGYefOnVi8eDEyMzPRqFEjjB07tkKbN2/eNCwj6ejoiEWLFuGFF15AmzZtkJKSgkuXLmHXrl1IT0+HTCbDyJEjERQUhJUrVyIhIQGBgYH44IMPcOHCBSv9VYhqhsmYzObPZYr7PvzwQ7Rs2RIAcO7cORw5cgQnTpwAAJSXl+PGjRtwc3ND48aNAQAdO3ascH5hYSFatWoFZ2dnAMCsWbMqfH7u3DlcuXIFI0eOBACUlJTg0qVL+OWXX+Dn5wfg3tofTMYkdbyBRxYnk8kAAN7e3ggPD0dqaio2bNiAsLAweHh4QKlU4tatWwCAkydPVji3efPmuHDhAjQaDQAgISEBRUVFkMlkEELA29sbPj4++Oyzz5CamoqYmBj4+vrC29sbP/30EwBwfWiqFzgypjozdOhQzJ49G8OGDYNKpUJ8fDwcHR2xcOFCjB49Gg0bNnxo6VBPT0+89tprGDZsGGQyGUJCQuDl5YWOHTti2rRp+Pjjj9GtWzfExcVBo9HAz88PXl5eSE5OxuTJk/HRRx/B09MTTk5OVrpqoprhQkFERBLAMgURkQQwGRMRSQCTMRGRBDAZExFJAJMxEZEEMBkTEUkAkzERkQQwGRMRScD/Az3rJsVaXkqKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cm_test, cm_train = rf_model(classification_train_X, classification_train_y, classification_test_X, classification_test_y, opt_num_trees, opt_depth, opt_num_features )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculates f1 score of the class\n",
    "def calc_f1(precision, recall):\n",
    "    if recall == 0 or precision == 0:\n",
    "        return 0\n",
    "    return 2 * precision * recall / (precision + recall)\n",
    "# generate metrics for our model given confusion matrix\n",
    "def rf_score(confusion_matrix):\n",
    "    \n",
    "    unstacked_mat = confusion_matrix.unstack().reorder_levels(('Actual', 'Predicted'))\n",
    "    #taking 0 to be TP, 1 to be TN\n",
    "    TP = unstacked_mat.loc[0, 0]\n",
    "    TN = unstacked_mat.loc[1, 1]\n",
    "    FP = unstacked_mat.loc[0, 1]\n",
    "    FN = unstacked_mat.loc[1, 0]\n",
    "    \n",
    "    results = {}\n",
    "    # calculate the precision , recall and f1 score for both classes, 0 and 1\n",
    "    results[\"precision_0\"] = TP / (TP + FN)\n",
    "    results[\"recall_0\"] = TP / (TP + FP)\n",
    "    results[\"f1_0\"] = calc_f1(results[\"precision_0\"], results[\"recall_0\"])\n",
    "    results[\"precision_1\"] = TN / (TN + FP)\n",
    "    results[\"recall_1\"] = TN / (TN + FN)\n",
    "    results[\"f1_1\"] = calc_f1(results[\"precision_1\"], results[\"recall_1\"])\n",
    "\n",
    "    # calculate the macro average \n",
    "    results[\"macro_avg_f1\"] = (results[\"f1_0\"] + results[\"f1_1\"]) / 2\n",
    "    results[\"macro_avg_precision\"] = (results[\"precision_0\"] + results[\"precision_1\"]) / 2\n",
    "    results[\"macro_avg_recall\"] = (results[\"recall_0\"] + results[\"recall_1\"]) / 2\n",
    "    results[\"accuracy\"] = (TP + TN) / (TP + TN + FP + FN)\n",
    "    \n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 322,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance metrics for train dataset\n",
      "precision_0 : 0.9469026548672567\n",
      "recall_0 : 0.45147679324894513\n",
      "f1_0 : 0.6114285714285714\n",
      "precision_1 : 0.8107714701601164\n",
      "recall_1 : 0.9893428063943162\n",
      "f1_1 : 0.8912\n",
      "macro_avg_f1 : 0.7513142857142857\n",
      "macro_avg_precision : 0.8788370625136865\n",
      "macro_avg_recall : 0.7204097998216307\n",
      "accuracy : 0.83\n",
      "**************************\n",
      "Performance metrics for test dataset\n",
      "precision_0 : 0.25\n",
      "recall_0 : 0.047619047619047616\n",
      "f1_0 : 0.08\n",
      "precision_1 : 0.6808510638297872\n",
      "recall_1 : 0.9343065693430657\n",
      "f1_1 : 0.7876923076923078\n",
      "macro_avg_f1 : 0.4338461538461539\n",
      "macro_avg_precision : 0.4654255319148936\n",
      "macro_avg_recall : 0.49096280848105667\n",
      "accuracy : 0.655\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None, None, None, None, None, None, None, None, None, None]"
      ]
     },
     "execution_count": 322,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_metric = rf_score(cm_test)\n",
    "train_metric = rf_score(cm_train)\n",
    "print(\"Performance metrics for train dataset\")\n",
    "[print(f\"{key} : {value}\") for key, value in train_metric.items()]\n",
    "print(\"**************************\")\n",
    "print(\"Performance metrics for test dataset\")\n",
    "[print(f\"{key} : {value}\") for key, value in test_metric.items()]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 323,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of occurences of each class in the training dataset\n",
      "Class 1.0: 563 values\n",
      "Class 0.0: 237 values\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None, None]"
      ]
     },
     "execution_count": 323,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_set_values = Counter(classification_train_y)\n",
    "print(\"Number of occurences of each class in the training dataset\")\n",
    "[print(f\"Class {i}: {value} values\") for i, value in training_set_values.items()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2.2\n",
    "The accuracy of our model for our training data is 83%, which is much higher than that for our test dataset, which is 65.5%. \n",
    "<br>\n",
    "Upon inspecting the values in our training dataset, we observe that there are more than double  the number of samples from class 1 (563) than there are class 0 (237). This implies that our dataset is lopsided and hence accuracy is not an accurate representation of the performance of our model since the performance of the class with a higher number of samples will skew the accuracy value. Hence, we look at other metrics, such as precision, recall and f1. These metrics are defined for each class. \n",
    "<br>\n",
    "\n",
    "For Class 1, the summary table of the metrics for both the test and train dataset are as given:\n",
    "<br>\n",
    "\n",
    "Class 1 | Train dataset | Test Dataset\n",
    ":--- | :---: | :---:|\n",
    "**precision** | 0.81 | 0.68\n",
    "**recall** | 0.98 | 0.93\n",
    "**f1** | 0.89 | 0.79\n",
    "\n",
    "<br>\n",
    "\n",
    "The model has a high recall value for both test and train datasets. This means that the model is able to correctly recognise data points that belong in class 1. However, the precision of the model for the test dataset is much lower than that of the train dataset, indicating that the model is not very good at differentiating between class 1 and 0, since the proportion of data points labeled as class 1 is indeed class 1 is low in the test set. This is most likely due to the fact that our data is lopsided and we have trained our model to have a bias towards class 1. During bootstrapping, there is a chance that B would contain entirely of data points with labels of 1s, thus affecting the performance of our model. In hindsight, when choosing B samples to construct the random forest, we could restrict our model to choose an equal amount of data points belonging to both classes, thus giving undersampling the majority class to allow our model to perform better. Since the f1 score considers both precision and recall, it is a more accurate metric than accuracy to represent the performance of our model.\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "For Class 0, the summary table of the metrics for both the test and train dataset are as given:\n",
    "\n",
    "Class 0 | Train dataset | Test Dataset\n",
    ":--- | :---: | :---:|\n",
    "**precision** | 0.94 | 0.25\n",
    "**recall** | 0.45 | 0.04\n",
    "**f1** | 0.61 | 0.08\n",
    "\n",
    "<br>\n",
    "Th model has a low recall value in both train and test datasets, indicating that class 0 is not correctly recognised by our model. In fact, in the test dataset, the recall is close to zero, indicating that our model is not able to recognise data points belonging to class 0 when given unseen data points.  The precision for our training dataset is very high, at 0.94, which indicates that an example labeled as zero is indeed zero. However, this value falls to 0.25 in the test dataset. This is because our model falsely identifies a few data points (9) as 1, and due to the small amount of zeros (3) being correctly classified, the precision is adversely affected. \n",
    "\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "The macro averaged metrics of both our training and test datasets are as follows:\n",
    "<br>\n",
    "\n",
    "Macro Averaged Metrics | Train dataset | Test Dataset\n",
    ":--- | :---: | :---:|\n",
    "**precision** | 0.75 | 0.43\n",
    "**recall** | 0.88 | 0.47\n",
    "**f1** | 0.72 | 0.49\n",
    "\n",
    "<br>\n",
    "The macro averaged statistics takes an average of precision, recall and f1 of both classes. \n",
    "Overall, our model performs relatively well on the training dataset, with a f1 score of 0.72. This is to be expected since our model is fitteed to the training dataset. However, due to the model's inability to recognise and classify data points belonging to class 0, our model has a rather mediocre f1 score on the test dataset. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2.3 SVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (i) Linear SVM\n",
    "\n",
    "The cost function, hinge loss for Linear SVM is defined as:\n",
    "$$\n",
    "\\mathcal L (\\boldsymbol w) = \\frac{1}{2} \\| \\boldsymbol w \\|^2 + \\frac{\\lambda}{n} \\sum_{i=1}^n \\max \\bigg( 0, 1-y_i (\\boldsymbol w \\cdot x_i + b) \\bigg) \\, \n",
    "$$\n",
    "\n",
    "where $\\boldsymbol w$ is the vector of weights, $\\lambda$ the regularisation parameter, and $b$ the intercept which is included in our `X` as an additional column of $1$'s.\n",
    "To implement a hard margin SVM, we want a very large $\\lambda$ so that the loss function heavily penalises data points that are on the wrong side of the hyper plane. This allows us to implement a hard margin SVM that tolerates no mistakes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add extra column of 1s to both train and test dataset!!! to cater to b\n",
    "def compute_cost(W, X, y, reg_param=1e100):\n",
    "    n = X.shape[0]\n",
    "    \n",
    "    distances = 1 - y * (np.dot(X, W))\n",
    "    distances[distances < 0] = 0\n",
    "    hinge_loss = reg_param * (np.sum(distances) / n)\n",
    "    \n",
    "    cost = 1/2 * np.dot(W, W) + hinge_loss\n",
    "    return cost\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform gradient descent to optimise the weights vector, we need to calculate the gradient of the loss function with respect to the weight vector. The formula is:\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathcal L}{\\partial \\boldsymbol w} = \n",
    "\\frac{1}{N} \\sum_{i=1}^{n}\n",
    "\\left\\{\\begin{array}{ll}\n",
    "\\boldsymbol{w} & \\text { if } \\max \\left(0,1-y_{i} *\\left(\\boldsymbol{w} \\cdot x_{i}\\right)\\right)=0 \\\\\n",
    "\\boldsymbol{w}-\\lambda y_{i} x_{i} & \\text { otherwise }\n",
    "\\end{array}\\right.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_cost_gradient(W, X_batch, y_batch, reg_param=1e100):\n",
    "    # convert singular data points into a list\n",
    "    if type(y_batch) == np.float64:\n",
    "        y_batch = np.asarray([y_batch])\n",
    "        X_batch = np.asarray([X_batch])\n",
    "    \n",
    "    distance = 1 - (y_batch * np.dot(X_batch, W))\n",
    "    dw = np.zeros(W.shape[0])\n",
    "    \n",
    "    for i, d in enumerate(distance):\n",
    "        if max(0, d) == 0:\n",
    "            di = W\n",
    "        else:\n",
    "            di = W - (reg_param * y_batch[i]* X_batch[i])\n",
    "        dw += di\n",
    "    \n",
    "    #compute average\n",
    "    dw = dw / len(y_batch)\n",
    "    return dw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We add an extra column to both train_X and test_X datasets, as well as the weight vector so that the parameter b will be optimised when optimising the weight vector. We perform gradient descent to optimise the weights, with equation:\n",
    "$$\n",
    "\\boldsymbol w = \\boldsymbol w - \\alpha \\frac{\\partial \\mathcal L}{\\partial \\boldsymbol w}\n",
    "$$\n",
    "where $\\frac{\\partial \\mathcal L}{\\partial \\boldsymbol w}$ is the gradient of the hinge loss as we have computed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sgd(X, y_train, max_epochs=2000, stop_criterion=0.01, learning_rate=1e-5, reg_param=1e100, print_outcome=False):\n",
    "    \n",
    "    #add extra column to X\n",
    "    X = np.hstack((X, np.ones((X.shape[0], 1))))\n",
    "    \n",
    "    # change labels from 0 to -1\n",
    "    y = np.array(y_train)\n",
    "    y[y == 0] = -1\n",
    "    #initialise zero weights\n",
    "    weights = np.random.rand(X.shape[1]) * 10\n",
    "    nth = 0\n",
    "    prev_cost = np.inf\n",
    "    \n",
    "    #perform single value stochastic gradient descent\n",
    "    for epoch in range(max_epochs):\n",
    "        #shuffle to prevent repeating update cycles\n",
    "        np.random.shuffle([X, y])\n",
    "        \n",
    "        for ind, x in enumerate(X):\n",
    "            ascent = compute_cost_gradient(weights, x, y[ind], reg_param)\n",
    "            weights = weights - (learning_rate * ascent)\n",
    "                                 \n",
    "        #convergence check on 2^n ith iteration\n",
    "        if epoch == 2 ** nth or epoch == (max_epochs - 1):\n",
    "            cost = compute_cost(weights, X, y, reg_param)\n",
    "            if print_outcome:\n",
    "                print(\"ascent: \", ascent)\n",
    "                print(\"Current epoch is: {}, Cost is: {}\".format(epoch, cost))\n",
    "            if abs(prev_cost - cost) < stop_criterion * prev_cost:\n",
    "                return weights\n",
    "            prev_cost = cost\n",
    "            nth += 1\n",
    "    return weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "W = sgd(classification_train_X, classification_train_y, max_epochs=2000, stop_criterion=0.001, learning_rate=1e-3, reg_param=1e100,print_outcome=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-6.80111508e+97, -4.16182645e+96,  6.70469630e+96,  1.79322009e+97,\n",
       "        3.13880965e+97, -1.36515054e+97,  1.10865353e+97, -2.23396587e+97,\n",
       "        5.04989568e+97,  2.56909206e+97,  4.77607399e+96,  4.63752371e+97])"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "def svm_f1_score(y_pred, y_actual, macro=False):\n",
    "    confusion_matrix = np.zeros(4).reshape((2, 2))\n",
    "    for i in range(len(y_pred)):\n",
    "        pred = y_pred[i]\n",
    "        actual = y_actual[i]\n",
    "        if pred == -1 and actual == -1:\n",
    "            # TP\n",
    "            confusion_matrix[0, 0] += 1\n",
    "        elif pred == 1 and actual == -1:\n",
    "            # FN\n",
    "            confusion_matrix[0, 1] += 1\n",
    "        elif pred == -1 and actual == 1:\n",
    "            # FP\n",
    "            confusion_matrix[1, 0] += 1\n",
    "        else:\n",
    "            # TN\n",
    "            confusion_matrix[1, 1] += 1\n",
    "    \n",
    "    TP = confusion_matrix[0, 0]\n",
    "    TN = confusion_matrix[1, 1]\n",
    "    FP = confusion_matrix[1, 0]\n",
    "    FN = confusion_matrix[0, 1]\n",
    "    \n",
    "    precision_0 = TP / (TP + FN)\n",
    "    recall_0 = 0\n",
    "    if (TP + FP) != 0:\n",
    "        recall_0 = TP / (TP + FP)\n",
    "    f1_0 = calc_f1(precision_0, recall_0)\n",
    "    specificity_0 = TN / (TN + FP)\n",
    "    \n",
    "    precision_1 = TN / (TN + FP)\n",
    "    recall_1 = 0\n",
    "    if (TN + FN) != 0: \n",
    "        recall_1 = TN / (TN + FN)\n",
    "    specificity_1 = TP / (TP + FN)\n",
    "    f1_1 = calc_f1(precision_1, recall_1)\n",
    "    \n",
    "    macro_avg_f1 = (f1_0 + f1_1) / 2\n",
    "    macro_avg_precision = (precision_0 + precision_1) / 2\n",
    "    macro_avg_recall = (recall_0 + recall_1) / 2\n",
    "    accuracy = (TP + TN) / (TP + TN + FP + FN)\n",
    "    macro_specificity = (specificity_1 + specificity_0) / 2\n",
    "    \n",
    "    if macro:\n",
    "        return macro_avg_f1, macro_avg_precision, macro_avg_recall, accuracy, macro_specificity\n",
    "    return f1_1, precision_1, recall_1, accuracy, specificity_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score(W, X, y, title, plot_matrix=False, macro=False):\n",
    "    y_preds = np.array([])\n",
    "    # change the labels from 0 t0 -1\n",
    "    y[y == 0] = -1\n",
    "    X = np.hstack((X, np.ones((X.shape[0], 1))))\n",
    "    for i in range(X.shape[0]):\n",
    "        y_pred = np.sign(np.dot(X[i], W))\n",
    "        y_preds = np.append(y_preds, y_pred)\n",
    "    \n",
    "    confusion_matrix = rf_confusion_matrix(y_preds, y)\n",
    "    if plot_matrix:\n",
    "        plt.figure()\n",
    "        plt.title(title)\n",
    "        sn.heatmap(confusion_matrix, annot=True)\n",
    "    f1_score, precision, recall, accuracy, _ = svm_f1_score(y_preds, y, macro)\n",
    "    \n",
    "    metrics = {\"f1_score\": f1_score, \"precision\": precision, \"recall\": recall, \"accuracy\":accuracy}\n",
    "    return metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance metrics for train dataset\n",
      "f1_score : 0.6280867699921664\n",
      "precision : 0.6314949299638015\n",
      "recall : 0.625728228389621\n",
      "accuracy : 0.6825\n",
      "**************************\n",
      "Performance metrics for test dataset\n",
      "f1_score : 0.6067036482639195\n",
      "precision : 0.605433901054339\n",
      "recall : 0.6083333333333334\n",
      "accuracy : 0.665\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None, None, None, None]"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEPCAYAAACX9KXAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAApPklEQVR4nO3deXhU1f3H8fckwyQhk4VIwbYaasAISiORzRhAARFaQGQLIWGURVAUMZQ9oGCjIkVRkR2lSFgSIIAIWAUEUlbL8hNRQQ0UEJB9yQyQ9f7+oEYDCQSyXZzPy+c+z3CXc86F4zffnHvuGYthGAYiIlKuPMq7ASIiomAsImIKCsYiIiagYCwiYgIKxiIiJqBgLCJiAgrGvxE//vgj99xzD926dbvq2LBhw7jnnns4ffr0Ncs4dOgQL7zwQoHHjh07RnR0dIm0tTh++OEH2rVrR7t27XjkkUeoW7du3p9nzZpV5HLWrFnDq6++WnoN/Z+JEyeyevXqUq9Hbn3W8m6AlBwvLy/279/P4cOH+eMf/wjAhQsX2LFjR5GuP3LkCPv37y/wWNWqVUlKSiqxtt6sGjVq8NFHHwGwePFiPv30U6ZNm3bD5TRv3pzmzZuXdPOusnXrVmrUqFHq9citT8H4N8TT05O//OUvfPzxxzz77LMAfPbZZzRv3pyZM2fmnff5558zZcoUsrKy8Pb2ZujQoYSFhTFy5EiOHTtGr169eOWVV4iNjaV69eocPnyYN954g549e7Jz506ys7MZN24c69atw9PTk/DwcEaNGsWhQ4cYMWIEmZmZGIZBp06diI2Nvaqdq1evZuLEieTm5uLr68vw4cMJCwvjvffe4/Dhw5w4cYLDhw9TtWpVxo0bR5UqVYr8d1C7dm2aN2/Onj17ePPNN9m7dy/JyclkZWVx7tw5evfuTUxMTL5A7nA4qFOnDjt27ODo0aNERESQkJCAh0f+Xxw/++wzpkyZgsViwdPTkyFDhlC/fn3S09N57bXX+O6778jKyiIiIoIhQ4aQnJzM7t27+cc//oGnpyctWrS4yX9ZcQuG/CYcOnTIqFOnjvHVV18ZrVq1ytv/1FNPGXv37jVCQ0ONU6dOGfv37zfatGljnD592jAMw/juu++MyMhIw+VyGVu2bDFat26dV15oaKjxn//8J1/5hmEYH374oREbG2tcvHjRyMnJMV588UVjyZIlxvDhw41p06YZhmEYx48fN+Li4oycnJx87fzhhx+Mhx56yDh48KBhGIaxadMmIzIy0khPTzcmTJhgNG/e3EhPTzcMwzCeeeYZ49133y30nlNSUow+ffrk2xcaGmosWbLEMAzDcDqdRlRUVN697ty5M+8efn1tt27djP79+xs5OTlGenq60ahRI2Pz5s1X1de8eXNj586dhmEYxr///W/jvffeMwzDMIYNG2bMnj3bMAzDyM7ONgYNGmRMnz49r+xPPvmk0HsQ+Zky49+Y2rVr4+npye7du7nttttwuVyEhobmHd+4cSPHjx+ne/fuefssFgsHDx68qiyr1UqdOnWu2r9p0ybatWuHt7c3AO+88w4AAQEBDB06lF27dhEREcHIkSOvyi63bNnCgw8+yJ133glAREQEQUFB7N69G4AGDRpgt9sBuPfeezl37twN/x3Uq1cPAF9fX6ZOncr69ev573//y549e7hw4UKB1zRt2hQPDw/sdjvVqlUrsN7WrVvTr18/Hn74YSIjI+nduzcA69at46uvvmLRokUAXLp06YbbLKJg/Bv0+OOPs2zZMoKCgmjXrl2+Y7m5uUREROQFUICjR49SpUoVtm3blu9cm82G1Xp1F7ly38mTJ8nNzaVp06Z8+umnbNq0ic2bNzNp0iQWL17M7bffnq9+i8WS73rDMMjOzgbIC/Bw+YeEcRNLp1SsWBGAn376iS5duhAVFUXdunVp1aoVa9euLfCaotQ7YMAAOnbsyMaNG1m8eDEzZ85k0aJF5Obm8u6771K9enUAzp8/f9U9ilyPZlP8BrVr145//etfrFy5kjZt2uQ7FhERwcaNG0lLSwNg/fr1PP7441y6dAlPT0+ysrKuW35ERATLly8nMzOT3NxcRo8ezYoVKxg4cCArV66kdevWjBo1CrvdflXGHRERwYYNGzh06BAAmzdv5ujRo9x///0ldPe/2L17N0FBQTz33HM0atQoLxDn5OTccFnZ2dk0a9aMixcv0rVrV0aNGsXevXvJzMykUaNGzJo1C8MwyMzMpG/fvsyZMwe4PI7/8w8akWtRZvwbVLVqVapXr46fnx+BgYH5jtWoUYO///3v/O1vf8MwDKxWK1OmTMHX15caNWrg5eVFp06dePvttwstPzo6msOHD9OhQwcMw6BBgwY4HA4OHDjAiBEjSE5OxtPTk0cffZT69etfVf+oUaPo168fOTk5eHt7M3XqVPz8/Er87yEyMpJFixbRqlUrLBYLDRo0ICgoiAMHDtxwWVarlfj4eAYNGoTVasVisfD6669js9kYMWIEr732Gm3btiUrK4uHHnqIp59+GoBmzZoxfvx4srKyaN++fUnfovyGWIyb+T1QRERKlIYpRERMQMFYRMQEFIxFRExAwVhExAQUjEVETMAUU9tsXneUdxPEZHI1yUcKkZ15uFjXZ53cV+RzK1QOKVZdN8IUwVhEpMzk3vhLP2VBwVhE3IuRW94tKJCCsYi4l1wFYxGRcmcoMxYRMYEccy7cpGAsIu5FD/BERExAwxQiIiagB3giIuVPD/BERMxAmbGIiAnkXP+rxcqDgrGIuBcNU4iImICGKURETMCkmbHWMxYR95KbW/StEKdOneLhhx8mLS2NAwcO0LVrV2JiYhg1ahS5/7tuwYIFdOjQgaioKNauXXvdZikYi4hbMXKzirwVJCsri5dffhlvb28AxowZQ1xcHPPmzcMwDNasWcOJEydITEwkKSmJDz74gPHjx5OZmXnNdikYi4h7KWZmPHbsWKKjo6lSpQoAX3/9NQ0aNACgSZMmbNq0iV27dhEeHo7NZsPPz4/g4GD27NlzzWYpGIuIezFyi75dYfHixQQFBdG4ceNfijMMLBYLAL6+vqSnp+N0OvHz88s7x9fXF6fTec1m6QGeiLiXYiwUlJKSgsViYfPmzXz77bcMHTqU06dP5x13uVz4+/tjt9txuVz59v86OBdEmbGIuJdiZMZz585lzpw5JCYmUqtWLcaOHUuTJk3YunUrAKmpqdSrV4+wsDC2b99ORkYG6enppKWlERoaes1mKTMWEfdSwvOMhw4dyksvvcT48eMJCQmhZcuWeHp64nA4iImJwTAMBgwYgJeX1zXLsRhG+X8Nr74dWq6kb4eWwhT326EvbZxb5HO9I2OLVdeNUGYsIu5Fb+CJiJQ/w9A3fYiIlD9lxiIiJmDStSkUjEXEvSgzFhExgZzs8m5BgRSMRcS9aJhCRMQENEwhImICCsYiIiagYQoRERPQAzwRERPQMIWIiAlomEJExASUGYuImICCsYiICZh0rWwFYxFxL9maTSEiUv70AE9ExAQ0ZiwiYgIaMxYRMQFlxiIiJqBgLCJS/owcfSGpiEj5U2YsImICmtomImICuZpNISJS/kw6TOFR3g34LalfP5xVny28an+XqHZs+PfHrF+3lIkTx2CxWG647NatH2XTxuWkrv+Inj1jALBarfxz5rt8viaFjRuW06ZNi2Lfg5S8BvXDWbOqgH7RpR2bNnzMv9d/xKSJb9xUv2jTugWbN61gQ+oyev2qX8z65wTWfb6YzRvVL66Sk1P07apLcxg+fDjR0dHExsZy8OBBvv76axo3bozD4cDhcLBy5UoAFixYQIcOHYiKimLt2rXXbZYy4xIycGBfYmM64nJdyLff29ub0a8M5oEHHuXixUskzp5I69aPsnz5qiKXbbVaGTduNA891BqX6wLr1y1hxYpVtGzZlFOnz9Cj54sEBQXyxdZPb6hcKX2DBvYlNrYjF1wX8+339vbm76OHUOeB5ly8eIk5iZNuql+8OW4UD/6vX6SuX8ryFato1bIpp06doXuP/gQFVWLbF+oX+RQjM/45qCYlJbF161bGjBlDs2bN6NGjBz179sw778SJEyQmJpKSkkJGRgYxMTFERkZis9kKLVuZcQnZl3aAqC69r9qfkZHBww8/wcWLl4DL/wNdupSB1Wpl2tQ3WbN6EWs/X0yTJhH5rjt4YEfe51o17yYt7b+cPXuOrKwsNm76D40iG5CSspzRo8flnZdt0q+TcWdp+w7QOargftH44Xb5+kXG//rF9GlvsnZNCuvXLuHhK/rFjwd35n2uVSt/v9i08T80atSQRSnLGTX6H3nnZZt0YZxyk2sUfbvCo48+SkJCAgBHjhyhcuXK7N69m3Xr1hEbG0t8fDxOp5Ndu3YRHh6OzWbDz8+P4OBg9uzZc81mlXhmnJGRwfz589myZQvp6en4+flRr149unXrhre3d0lXZxpLlq6kWrU7rtpvGAbHj58E4LnneuBr92X16lT69HFw6tRpnnl2EEFBgXy+JoU64c1ZtiwRH29vgoICWfXZQo4c+Ylp02dz/tz5vDKd6S78A/zzsnC73Zek+dMZPWrcVfVL+Vqy5Pr94vnnemD3rciq1ak80+dJTp06TZ9nBhEUVIm1n6dwf51mLF+WiI/P5X6xZtVCDh/5iWnTZnPufHpemelOJwH+fvn6xYKk6bz8q8AsFHs2hdVqZejQoaxatYoJEyZw7NgxOnfuTO3atZkyZQqTJk2iZs2a+Pn55V3j6+uL0+m8drnFalUBhg8fTs2aNYmLi8PX1xeXy0VqaioDBw5k0qRJJV3dLcFisTBmzAhC7w6hy/+y59q1a9IosiH164cDl/+Bg4ICefxxB3A5M27xWGcA/ly7FnY/e155dj9fzp29HJzvuOP3LFzwPlOnzSYpeWkZ3pUUl8ViYeyYkdwdGkLnX/eLRg1okK9fVKLN//rFjwd30rzF//rFn2vhZ/fNK8/PbufsuZ/7xR9YtPB9pk79kKSkpWV4V7eAEphNMXbsWAYNGkRUVBRJSUlUrVoVgBYtWpCQkEC9evVwuVx557tcrnzBuSAlPkxx/Phx+vTpQ82aNbnzzjupWbMmffr04cyZMyVd1S1j8qSxeHt707FTr7xfS/fuTSM5eSktHutM28cdpKQs58yZcwVe/+2e76lR4y4qVQqkQoUKNG7UkC1bt1OlSmVWrphH/IjX+fDD5LK8JSkBUyaPxcvbiw4de/6qX/xAcvJHNG/RmdZtf+4XZwu8/ttvv6dGjZC8ftGocUO2bLncLz5ZOY/4+NeZpX5xFSM3t8jblZYuXcq0adMA8PHxwWKx0K9fP3bt2gXA5s2bue+++wgLC2P79u1kZGSQnp5OWloaoaGh12xXiWfGXl5eLF26lMaNG+Pn54fT6SQ1NZWKFSuWdFWmFt3lCXztvmzf/iU9ekSzYcMXfPbpAgAmTvyAGTPmMHXKP1i9ahH+/namTpuN8avVpIKrPZD3OTs7myFDXmHF8jl4eHgw68Nkjhz5ibfeeoXAwADih8cRPzwOgLaPO7h06VKZ3qsUXXT0E9h9L/eLnj26smHDVlZ/drlfTJj4AdNnzGHa1HF8vnoRfv5+TJ36Yb5+cUdweN7n7OxsBg95hZUr5l7uF7OSOHLkJ8a/9QqVAgMYEf8iI+JfBKB1W/WLPMV4Hfqxxx5j+PDhxMbGkp2dTXx8PL///e9JSEigQoUKVK5cmYSEBOx2Ow6Hg5iYGAzDYMCAAXh5eV2zbIthlOx6cmfOnGHSpEns2LEDl8uFr68vDzzwAH379uW2224r8Bqb19VjauLeck26zKGUv+zMw8W63vX32CKf6/vy3GLVdSNKPDOuVKkSI0eOLOliRURKhru/9NG/f/+yqkpEpHDFmNpWmsrspY+f5+aJiJQrd18oKCAgoKyqEhEpnLssFLR///5Cj911110lXZ2IyA0xst1kcfn4+HgOHTpESEhIvik5FouF2bNnl3R1IiI3xl0y45kzZ9KtWzfGjRuX91aKiIhpmHTMuMRnU/j4+PDKK69w5MgRAEp4GrOISPGYdDZFqUxtq127NuHhl98Ueuqpp0qjChGRm2LkGkXeylKpz6ZQZiwipuIuD/CuVLdu3dKuQkSk6NzlAd6V4uLiSrsKEZGic9dgLCJiJmYdOlUwFhH3osxYRMQEFIxFRMqfkW3Olz4UjEXEvZgzFisYi4h7KeuXOYpKwVhE3IuCsYiICWiYQkSk/GmYQkTEBIxsBWMRkfKnYQoRkfJn0rXlFYxFxM0oGIuIlD9lxiIiJmBkl3cLCqZgLCJupTiZcU5ODiNHjmT//v14enoyZswYDMNg2LBhWCwW7r77bkaNGoWHhwcLFiwgKSkJq9VK3759adq06TXLVjAWEbdSnGC8du1aAJKSkti6dWteMI6Li6Nhw4a8/PLLrFmzhjp16pCYmEhKSgoZGRnExMQQGRmJzWYrtGwFYxFxL4blpi999NFHeeSRRwA4cuQIlStXZt26dTRo0ACAJk2asHHjRjw8PAgPD8dms2Gz2QgODmbPnj2EhYUVWnapfDu0iIhZGblF3wpitVoZOnQoCQkJtGzZEsMwsFguB3hfX1/S09NxOp34+fnlXePr64vT6bxmu5QZi4hbMXJvPjP+2dixYxk0aBBRUVFkZGTk7Xe5XPj7+2O323G5XPn2/zo4F0SZsYi4ldwcS5G3Ky1dupRp06YB4OPjg8VioXbt2mzduhWA1NRU6tWrR1hYGNu3bycjI4P09HTS0tIIDQ29Zrsshgm+nc/mdUd5N0FMJrf8u6WYVHbm4WJd/2PDZkU+946tn+f784ULFxg+fDgnT54kOzub3r17U716dV566SWysrIICQnh1VdfxdPTkwULFpCcnIxhGDzzzDO0bNnymnUpGIspKRhLYYobjA/Vb17kc+/8z5pi1XUjNGYsIm7FrD/nFYxFxK2UxAO80qBgLCJupaAHc2ZQaDDesGFDoRc1atSoVBojIlLabrnMeMWKFYVepGAsIrcqoxhv4JWmQoPxmDFjCtx//PjxUmuMiEhpu2WX0JwwYQLz5s0jKyuLS5cu8ac//emaWbOIiJnlmjQzvu4beKmpqaSmptK2bVtWrlxJ1apVy6JdIiKlwjAsRd7K0nUz48DAQGw2Gy6Xi2rVqnHx4sWyaJeISKm45WZT/Oz2229n0aJF+Pj48NZbb1135SERETMz62yK674OnZuby9GjRwkICGDJkiVERERQo0aNEm2EXoeWK+l1aClMcV+H3h3Spsjn1t63vFh13YjrZsbLli3L++zn58fu3btLPBiLiJSVW25q28/S0tIAMAyDb7/9lsDAQJ544onSbpeISKkw6y9d1w3GAwcOzPv881JwIiK3KrNObbtuMM7MzMz7fOLECX788cdSbZCISGnKNekDvOsG41atWmGxWDAMA29vb55++ukSb8RtPv4lXqbc2g7+UHYPTsS93LKZ8TvvvJPvG02/+OKLUm2QiEhpuuUe4G3bto0ffviBWbNm0aNHD+DyNLe5c+eyfLmyFhG5Nd1ymbG/vz8nT54kMzOTEydOAGCxWBg8eHCZNU5EpKSZdDJF4cE4NDSU0NBQOnfuzOnTp6lVqxarV6/moYceKsv2iYiUqJzc6y7JUy6u26rXXnuN//u//wNg//79DBs2rLTbJCJSanJvYCtL1w3Gx44do2vXrgD07t1b6xmLyC3NwFLkrSwVKV/fv38/AAcOHCA316QrM4uIFEGuUfStLF13alt8fDxxcXGcOnUKb29v2rdvXxbtEhEpFbllnPEW1XUz4/vvv5+EhAQeeughLl68yKlTp8qiXSIipcKswxSFZsaZmZmsWLGCuXPnYrPZcDqdrFmzBm9v77Jsn4hIicq51TLjZs2asXfvXt58803mzZtHlSpVFIhF5JZn1tkUhWbGTz75JMuXL+fw4cN06tSJ66xBLyJySzDrFIRCg3GfPn3o06cPX3zxBQsXLmT37t2MGzeOdu3aERoaWpZtFBEpMcUZC87KyiI+Pp7Dhw+TmZlJ3759uf3223n22Wf505/+BEDXrl3561//yoIFC0hKSsJqtdK3b1+aNm16zbKvO5uiQYMGNGjQgPPnz/PRRx8xZMgQli5detM3IyJSnoqzguayZcsIDAxk3LhxnDlzhvbt2/P888/To0cPevbsmXfeiRMnSExMJCUlhYyMDGJiYoiMjMRmsxVa9nWD8c/8/f1xOBw4HI6bvxMRkXJWnKltrVq1omXLlnl/9vT0ZPfu3ezfv581a9ZQrVo14uPj2bVrF+Hh4dhsNmw2G8HBwezZsyffCphXKnIwFhH5LcgpxrW+vr4AOJ1O+vfvT1xcHJmZmXTu3JnatWszZcoUJk2aRM2aNfHz88t3ndPpvGbZ5lwxQ0SklORaLEXeCnL06FGefPJJ2rVrR9u2bWnRogW1a9cGoEWLFnzzzTfY7XZcLlfeNS6XK19wLoiCsYi4FeMGtiudPHmSnj17MnjwYDp16gRAr1692LVrFwCbN2/mvvvuIywsjO3bt5ORkUF6ejppaWnXnfigYQoRcSvFmdo2depUzp8/z+TJk5k8eTIAw4YN4/XXX6dChQpUrlyZhIQE7HY7DoeDmJgYDMNgwIABeHl5XbNsi2GCCcS/D7y3vJsgJqPvwJPCVKgcUqzr5/8htsjndj0yt1h13QhlxiLiVsz6OrSCsYi4leLMMy5NCsYi4lZuudehRUR+i8r9IVkhFIxFxK1omEJExAQ0TCEiYgI5yoxFRMqfMmMRERNQMBYRMQHNphARMQHNphARMQENU4iImEBxFpcvTQrGIuJWNEwhImICGqYQETEBzaYQETGBXJOGYwVjEXEreoAnImICGjMWETEBzaYQETEBjRmLiJiAOUOxgrGIuBmNGYuImECOSXNjBWMRcSvKjEVETEAP8ERETMCcoVjBWETcTHGGKbKysoiPj+fw4cNkZmbSt29fatSowbBhw7BYLNx9992MGjUKDw8PFixYQFJSElarlb59+9K0adNrlq1gLCJupTgP8JYtW0ZgYCDjxo3jzJkztG/fnpo1axIXF0fDhg15+eWXWbNmDXXq1CExMZGUlBQyMjKIiYkhMjISm81WaNkKxiLiVoozZtyqVStatmyZ92dPT0++/vprGjRoAECTJk3YuHEjHh4ehIeHY7PZsNlsBAcHs2fPHsLCwgot2+OmWyVXCa8bRsryWYUeH/fOaOJHDbipslu0eoRPPk/m48/mEftkJwCsVivvTXuDpSsTWbkmicf+cu1fg6R8nDpzlubtHew7cCjf/lVrN9ClV3+in36RRcv+dVNlr9uwhS69+hPbZwCLln0CQFZ2NsP+Po4n+w4i+ukXWfvvLcW+h98S4wa2K/n6+mK323E6nfTv35+4uDgMw8BiseQdT09Px+l04ufnl+86p9N5zXYpMy4hz/XvSacuj3PhwsUCjzu6R1Hz3lA2b/zPDZdttVp55fVh/KVpFBcuXGTZp3P47F/raNaiMWdOn+WFZ4ZRqVIAq1IX89kna4t7K1KCsrKzeeUfE/D28sq3Pycnh7en/pPkDyZQ0cebdrHP0LxJBJUCA26o7LETppP0/rtU9PGm27MDeSSyIf/eso1Afz/eeHkwZ8+dp1OPfjRt/GBJ39otq7izKY4ePcrzzz9PTEwMbdu2Zdy4cXnHXC4X/v7+2O12XC5Xvv2/Ds4FUWZcQg789xC9HC8WeKxu/ft5oP79JM5akLfParXy1nsJLFk5m48+SSSiUf1813y5NzXv8933hPDffQc4d+48WVlZfLFlBw0j6vLx0k8Z+9qEvPOyc7JL+K6kuN6c+D5dnmjN7yoH5dvv6enJsrnT8bP7cvZ8OgZQ0ceHrOxsXhrzNk89NxhH34F8sWNXvusebhuT93nffw8RfMcfCPD3o0KFCjwQdh/bv/yalk0b80LvJ/POs3p6luo93mpyb2C70smTJ+nZsyeDBw+mU6fLv6Hee++9bN26FYDU1FTq1atHWFgY27dvJyMjg/T0dNLS0ggNDb1mu5QZl5AVy1ZxR/AfrtpfpWplBg17np7d+tO2fau8/TFPduT0qTMMfOElKlUKYMnKRB6JeJy5C6fh7eNFYKUAUpbP4qcjx/lwZhLnz//yK47T6cLf384F1wUAfO0VmTH7Hca+OuGq+qX8LF2xiqDAACIb1mVGYvJVx61WT1at28hr4yfRJKIBVqsnC5d9QqUAfxKGD+DsufM89dxgPpo7jWcHvsSljAzOnU+ne78hVK18G13at8bu65tXnm9FH9KdLipW9AHA5brAgBGv5QvMAkYxMuOpU6dy/vx5Jk+ezOTJkwEYMWIEr776KuPHjyckJISWLVvi6emJw+EgJiYGwzAYMGAAXlf8dnSlUgnGp06dYtu2baSnp+Pv70+dOnWoUqVKaVRlem2faEXQbZWYs3AqVapUxsfHhx++30+te0NpGFGXB+pdHtD3tHpSqVIAsZ2fAS5nxh3bdAeg1n2h2O2//E9nt/ty7lw6AH/44+3MnDOBWR8ksWTRirK9ObmmJSs+w2KBzdt2svf7fcQnvMnEsaOofNsvWXKLRyJp3iSCEa+NZ9m/1vB92n/Z8eXX7PpmL3B5OOPsufNMfSsBuJwZz5r4DwD2/rCfCxcu5JXlunARf7/L/eTosRO8ODyB6A6taf2YniX8WnFmU4wcOZKRI0detX/OnDlX7YuKiiIqKqrIZZd4MF64cCHJycnUrVsXX19fvv/+e6ZOnUrnzp3p2rVrSVdneh9Mm8MH0y7/Q0XFPEGNu+9iwbyl+PnZOXrkGBPGT8fb24sXBz7D2bPnCyzj+737uKt6NQIDA3C5LvDgQ/WY8t4/qfy720haPIP4wa+xIVUPaczmw8m/jCV27zeElwe/kBeInS4X/YaMZvrbr2Gz2fDx9sLDYuGuandS9XeV6fNUNJcyMpj+YRL+fvYCyw/5050c+PEI586nU9HHm+1f7qZ7TEdOnj5DnwEjGPG3vjxYL7xM7vVW4javQ6ekpDB//nwqVKiQty8zM5OuXbu6VTBu36k1vr4VmfPhwgKPJ/4zmTcn/J3FKz7E7mfnw/fnYxi//MS+/54meZ+zs7MZPWIs8xdPx8PDg/lzFvPT0eMkvDGcgMAABgx5lgFDngUgttMzXLqUUbo3JzdtxWdruXDxIp3b/ZXWjzXlqeeHYLV6Elr9Ltq0bEZOTg6jxk6g+/ODcbouEN2hDR4evzzaWf/xvLzPFaxWhrzQmz4DRmAYBu1bP0bV31VmzDtTOZ/uZOqs+UydNR+AqW8lXPUQ0V3lGuZ8B89iGCXbsk6dOjFr1izs9l9+mp8/f56ePXuyaNGiAq/5feC9JdkE+Q04+MPy8m6CmFSFyiHFur5btQ5FPnfOgcXFqutGlHhm/Nxzz9GhQweqVauGn58fTqeTAwcOMHz48JKuSkTkhrnNQkHNmjWjSZMmpKWl4XQ6sdvtVK9eHatVEzdEpPwVZzZFaSqVCGm1Wrnnnnvy7Vu4cCGdO3cujepERIos252CcUF8fHzKqioRkUK5VWZckDZt2pRVVSIihXKbqW0Oh4OsrKx8+35eSCMpKamkqxMRuSElPIGsxJR4MB40aBAjR45k0qRJeOqdeBExGbeZTXH//ffTrl079u7dS4sWLUq6eBGRYnGrb4d++umnS6NYEZFic5vMWETEzNxmzFhExMzcZjaFiIiZuf08YxERM9CYsYiICeQY5hyoUDAWEbeiYQoRERMw6+LyCsYi4lbMGYoVjEXEzegBnoiICSgYi4iYgGZTiIiYgGZTiIiYgNamEBExAY0Zi4iYgFkzY4/yboCISFnKIbfIW2G+/PJLHA4HAF9//TWNGzfG4XDgcDhYuXIlAAsWLKBDhw5ERUWxdu3a67ZLmbGIuJXivoE3Y8YMli1blveN99988w09evSgZ8+eeeecOHGCxMREUlJSyMjIICYmhsjISGw2W6HlKjMWEbdi3MB/BQkODua9997L+/Pu3btZt24dsbGxxMfH43Q62bVrF+Hh4dhsNvz8/AgODmbPnj3XbJeCsYi4lVzDKPJWkJYtW2K1/jKoEBYWxpAhQ5g7dy533nknkyZNwul04ufnl3eOr68vTqfzmu1SMBYRt1LczPhKLVq0oHbt2nmfv/nmG+x2Oy6XK+8cl8uVLzgXRMFYRNxKcTPjK/Xq1Ytdu3YBsHnzZu677z7CwsLYvn07GRkZpKenk5aWRmho6DXL0QM8EXErJf069OjRo0lISKBChQpUrlyZhIQE7HY7DoeDmJgYDMNgwIABeHl5XbMci2GCSXe/D7y3vJsgJnPwh+Xl3QQxqQqVQ4p1fUjl8CKfu+/kzmLVdSOUGYuIWzG0UJCISPnT69AiIiZggpHZAikYi4hbUWYsImICObkaMxYRKXdaXF5ExAQ0ZiwiYgIaMxYRMQFlxiIiJqAHeCIiJqBhChERE9AwhYiICRT3a5dKi4KxiLgVzTMWETEBZcYiIiaQqyU0RUTKnx7giYiYgFmDsSm+dklExN3p26FFRExAwVhExAQUjEVETEDBuBxdvHiR6Oho0tLSrjp2+vRpevbsSUxMDHFxcVy8eLEcWijl4csvv8ThcFy1//PPP6djx4506dKFBQsWlEPLpDQpGJeTr776itjYWA4dOlTg8cmTJ9OmTRvmzZvHvffeS3Jychm3UMrDjBkzGDlyJBkZGfn2Z2VlMWbMGGbOnEliYiLJycmcOHGinFoppUHBuJxkZmYyadIkQkJCCjy+fft2GjduDECTJk3YtGlTWTZPyklwcDDvvffeVfvT0tIIDg4mICAAm81G3bp12bZtWzm0UEqL5hmXk7p1617zuNPpxM/PDwBfX1/S09PLollSzlq2bMmPP/541f5f9we43CecTmdZNk1KmYJxGXr77bfZsWMHALNmzcLT07PQc+12Oy6XC29vb1wuF/7+/mXVTDGhn/vDz1wuV77gLLc+BeMyNGDAgCKf+8ADD7B+/Xo6dOhAamrqdTNp+W2rXr06Bw4c4OzZs1SsWJFt27bRq1ev8m6WlCCNGZvI2bNn6devHwB9+/ZlxYoVREdHs3PnTrp161bOrZPy8PHHH5OcnEyFChUYNmwYvXr1Ijo6mo4dO1K1atXybp6UIL0OLSJiAsqMRURMQMFYRMQEFIxFRExAwVhExAQUjEVETEDBWErE1q1biYiIwOFw4HA4iIqKIjEx8YbLefPNN1m8eDHffvstEydOLPS8VatWcezYsSKVmZqayrBhw264LSJlSS99SIl58MEHefvtt4HLa2+0atWKdu3a3dTbg7Vq1aJWrVqFHp89ezajR4/WXFv5zVAwllLhdDrx8PCge/fu3HHHHZw/f57p06czevRoDhw4QG5uLnFxcTRs2JBPP/2UKVOmEBQURFZWFiEhIWzdupWkpCTefvttFi5cyPz588nNzaV58+b8+c9/5ttvv2Xo0KHMmzeP5ORkli9fjsVi4a9//StPPvkkaWlpxMfH4+Pjg4+PDwEBAeX9VyJyTQrGUmK2bNmCw+HAYrFQoUIFXnrpJd5//33atm1LixYtmDdvHpUqVeL111/nzJkzdOvWjRUrVjBu3DgWLlxIYGAgffr0yVfmqVOnmDFjBsuWLcNms/HGG29Qv359atWqxejRozl48CArV65k3rx5WCwWunfvTqNGjXj33Xfp378/kZGRTJ8+nX379pXT34pI0SgYS4n59TDFz95//33uuusuAL777ju2b9/Orl27AMjOzubkyZPY7XYqVaoEQHh4eL7rDx06xN133423tzcA8fHx+Y5/9913HDlyhO7duwNw7tw5Dh48yPfff09YWBhweZ0PBWMxOz3Ak1JnsVgACAkJoXXr1iQmJjJjxgxatWqFv78/6enpnD59Gri86P6vBQcHs2/fPjIzMwHo378/x44dw2KxYBgGISEh1KhRg9mzZ5OYmEiHDh0IDQ0lJCSEnTt3ArB79+4yvFuRm6PMWMpMdHQ0I0eOpFu3bjidTmJiYrDZbIwZM4ZevXoREBCA1Zq/SwYFBdG7d2+6deuGxWKhadOmVK1alfDwcIYMGcLMmTOJiIiga9euZGZmEhYWRtWqVRk1ahQDBgzggw8+ICgoCC8vr3K6a5Gi0UJBIiImoGEKERETUDAWETEBBWMRERNQMBYRMQEFYxERE1AwFhExAQVjERETUDAWETGB/we4DwrqI0iifwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEPCAYAAACX9KXAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAk0klEQVR4nO3de1yUZf7/8dcAIspBpVb77npY8VC6rmUaihplaViaB9QCEzVdTcstzEo8AJYllqXmoSzLr4WSplm/0rYTulGSrMc8fM0KXcVDiGcYlcPM/fvDjQ0FHZiBuWnezx734+Hcc891f8Z4vLm87uu+bothGAYiIuJWXu4uQEREFMYiIqagMBYRMQGFsYiICSiMRURMQGEsImICCuPfqcOHD3PjjTcyZMiQK96Li4vjxhtv5NSpU1dtIysri7///e+lvpednU1UVJRLanXW1ep0RG5uLkOHDnVZPatWrWL58uUua088g8L4d6xmzZocOHCAI0eOFO87f/4827Ztc+jzR48e5cCBA6W+16BBA1asWOGSOp11tTodcfbsWXbt2uWyerZu3crFixdd1p54BoXx75i3tzf33nsvn3zySfG+L774grvvvrvEcevXr2fQoEH069ePqKgotm/fjs1mY+rUqRw6dIiRI0dy+PBh7rjjDkaMGEFERATbt2+nXbt2ABQVFZGUlERERAT33XcfU6ZMoaCggMzMTKKiooiMjKR///5l9ha/+uor+vXrR58+fYiOjmbnzp0AzJ8/n7i4OEaOHEnPnj0ZNmwYx48fL/HZy+sE2LZtG4MHD6Z///4MGDCADRs2AJCTk8OIESPo378//fv3Z+7cuQBMmjSJixcv0rdvX2w2W4n2t2zZwsCBA4mMjCQyMpLPP/8cgIKCAmbMmEH//v3p06cPcXFx5OXl8eWXX7J+/XqWLl2q3rGUjyG/S1lZWcYtt9xi7Nq1y+jZs2fx/mHDhhn79u0zWrZsaZw8edI4cOCA0bt3b+PUqVOGYRjGjz/+aHTp0sWwWq3Gpk2bjF69ehW317JlS2Pz5s0l2jcMw3jnnXeMhx56yLhw4YJhs9mMJ554wvjwww+NSZMmGW+88YZhGIZx/PhxIzY21rDZbCXq/Pnnn43OnTsbhw4dMgzDMNLT040uXboYubm5xrx584y7777byM3NNQzDMB555BHj1VdfveK7/rbOM2fOGPfcc4+RlZVlGIZh/PLLL0Z4eLhx5MgRY8GCBUZ8fLxhGIZhtVqN2NhY49y5cyW+y+WGDh1qrF271jAMw9i7d68xbdo0wzAMY/78+cbMmTMNu91uGIZhvPLKK0ZiYqJhGIYxceJE46233nLg/5LIf/m4+5eBVK42bdrg7e3N7t27ue6667BarbRs2bL4/Y0bN3L8+HGGDx9evM9isXDo0KEr2vLx8eGWW265Yn96ejp9+/bFz88PoLjHWadOHSZOnMjOnTsJCwtj6tSpeHmV/MfYpk2b6NSpE40aNQIgLCyM4OBgdu/eDUBoaCgBAQEAtG7dmrNnz171++7YsYOcnBwee+yxEt9n37593H777YwePZpjx47RuXNnJkyYQGBg4FXbvPfee3nuuedYv349nTt35sknnwTgn//8J7m5uaSnpwNQWFjIddddd9XaRK5GYewB+vTpw8cff0xwcDB9+/Yt8Z7dbicsLKw4QAGOHTtG/fr12bJlS4ljfX198fG58kfm8n0nTpzAbrfTrVs3Pv/8c9LT0/nuu+9YuHAha9as4YYbbihxfovFUuLzhmFQVFQEUBzwcClUjWsspWKz2WjWrBmrVq0q3pednU1wcDA1atQgNTWV7777jk2bNjFo0CAWL15M3bp1y2wvKiqKbt26sXHjRr755hsWLFjAZ599ht1uZ/Lkydxxxx0AWK1W8vPzr1qbyNVozNgD9O3bl88++4xPP/2U3r17l3gvLCyMjRs3kpmZCcDXX39Nnz59uHjxIt7e3hQWFl6z/bCwMNauXUtBQQF2u51p06axbt06JkyYwKeffkqvXr1ITEwkICDgih53WFgY3377LVlZWQB89913HDt2jJtvvtnh7/fbOm+55RYOHjzI5s2bAdi7dy8RERFkZ2fz8ssv89prr9G9e3emTJlC8+bN+emnn/Dx8cFms5Ua9FFRUezdu5fIyEimT5/OuXPnyMnJoWvXrixfvrz4O8fHxzN79uzien79ZSLiKPWMPUCDBg1o1qwZgYGBV/QCmzdvznPPPceTTz6JYRj4+Pjw+uuv4+/vT/PmzalZsyYDBw5kzpw5ZbYfFRXFkSNHiIyMxDAMQkNDiYmJ4eDBg0yZMoWVK1fi7e1N9+7due222644f2JiIuPGjcNms+Hn58eiRYsIDAx0+Pv9ts5Vq1Yxb948XnrpJfLz8zEMg5deeomGDRsybNgw4uLi6N27N76+vtx444306tULb29v2rZtS69evVi+fDn16tUrbvupp55ixowZzJ07F4vFwrhx42jYsCGPPvooL774Iv3798dms9GqVSvi4uIACA8PZ+bMmQA88sgjDn8P8WwW41r/7hMRkUqnYQoRERNQGIuImIDCWETEBBTGIiImoDAWETEBU0xt8/H9k7tLEJNpEtTA3SWISWWecGyhq7IUntjv8LE1rg9x6lzlYYowFhGpMnbbtY9xA4WxiHgWw+7uCkqlMBYRz2JXGIuIuJ2hnrGIiAnYzLmIk8JYRDyLLuCJiJiAhilERExAF/BERNxPF/BERMzApD1jrU0hIp7FVuj4Vobvv/+emJgYAA4ePEh0dDSDBw8mMTER+3/C/v333ycyMpIHHniADRs2XLMshbGIeBbD7vhWisWLFzN16tTiB9AmJSURGxtLSkoKhmGQmppKTk4OycnJrFixgrfffpvZs2dTUFBw1bIUxiLiWex2x7dSNG7cmPnz5xe/3rNnD6GhocCl5x+mp6ezc+dO2rVrh6+vL4GBgTRu3JgffvjhqmUpjEXEszjZM46IiMDH57+X2wzDwGKxAODv709ubi55eXklHqrr7+9PXl7eVcvSBTwR8SwuvoDn5fXfPq3VaiUoKIiAgACsVmuJ/dd64rl6xiLiUQx7ocObI1q3bk1GRgYAaWlpdOjQgbZt27J161by8/PJzc0lMzOTli1bXrUd9YxFxLO4uGc8ceJE4uPjmT17NiEhIURERODt7U1MTAyDBw/GMAzGjx9PzZo1r9qOxTAMw6WVVYCe9CGX05M+pCzOPunj4taPHD7Wr30/p85VHuoZi4hn0UJBIiImoNuhRURMwKS3QyuMRcSzaHF5ERETUM9YRMT9DEMX8ERE3E89YxERE9BsChERE1DPWETEBDSbQkTEBDRMISJiAhqmEBExAYWxiIgJaJhCRMQEdAFPRMQENEwhImICGqYQETEB9YxFRExAYSwiYgLuf+xnqRTGIuJZiio+m6KgoIBJkyaRlZVFQEAACQkJWCwW4uLisFgstGjRgsTERLy8vMrdtsJYRDyLExfw3n//fWrXrs3777/P/v37mT59OjVq1CA2NpaOHTuSkJBAamoqPXr0KHfb5Y9vEZHqzG53fLvMzz//THh4OAAhISFkZmayZ88eQkNDAQgPDyc9Pb1CZSmMRcSzGIbj22VatWrFhg0bMAyDHTt2kJ2djWEYWCwWAPz9/cnNza1QWQpjEfEsTvSMBwwYQEBAAEOHDmXDhg385S9/KTE+bLVaCQoKqlBZCmMR8SxOhPGuXbto3749ycnJdO/enUaNGtG6dWsyMjIASEtLo0OHDhUqSxfwRMSjGLaKP5C0SZMmvPrqqyxZsoTAwEBeeOEFzp8/T3x8PLNnzyYkJISIiIgKta0wFhHP4sRNH8HBwSxduvSK/cuWLXOioEsUxiLiWbQ2hYiICdh1B56IiPtpbQr5LR8fH95aPJs/N2lIzZq+vJD0KllZR3ltwUyKior48af9jH7kKQyT3kcvlcPLy4sZc+IJad4Em93OxL9PIzAogDeXz+Xf+w8BkPK/q1n30RdurrQac+ICXmVSGLvJQ4MjOXnyNMMffpzg4Hps+dfnbNu+k+dfmMM/PlvPu+/Mp9d93Vm77kt3lypV6O6IS3d3PdBrBB27tGfy9CdZ/3kaS15fxtuvOX+RSFDPWEpa/cFaPlizrvh1UVERO3bspl5wXQACAwMoLCx0U3XiLl/+45+s/+IbAP7U8H84mXOKNje3IqT5n+l+7538O/MQz099GWveeTdXWo2ZdMzYYrj438H5+fm89957bNq0idzcXAIDA+nQoQNDhgzBz8+v1M/4+P7JlSVUKwEB/ny0ZilvLVmOYRjMf/UFjuec5NzZc3S7eyD5+fnuLtEtmgQ1cHcJbjVrwbP06NWNcQ8/Q4P/qc++//uJ3d/v5dHxI6lTN5CkxLnuLtFtMk9sc+rz52eNcPjY2k8vcepc5eHyO/AmTZpEQUEBsbGxzJw5k9jYWOx2OxMmTHD1qaq9hg3/yFdfrmLZ8tWsWPERc155jjvviqTNX+8gedlqZr2U4O4SxU2eHpdI9479mTEnnm83fMfu7/cC8MW69bT+601urq6asxuOb1XI5WF8/PhxRo8ezU033USjRo246aabGD16NKdPn3b1qaq1+vWv5x+fpjB58gyWvrMSgFOnz3DuXB4AR49lU69eHXeWKG7Qb1AvxjzxMAAXz1/Ebrfz2jsv07bdXwDoHB5aHMxSMYbd7vBWlVw+ZlyzZk0++ugjbr/9dgIDA8nLyyMtLY3atWu7+lTVWtzEv1Ovbh2mTH6CKZOfAGDMmKdJWfYaRUVFFBQU8sjYp91cpVS1z9el8uK8abz3yVv4+Pjw/NSXOXYkm2kzJ1JQWMiJ4yeZMv55d5dZvZl0NoXLx4xPnz7NwoUL2bZtG1arFX9/f2699VbGjh3LddddV+pnPHnMWErn6WPGUjZnx4ytzz3k8LH+CcudOld5uLxnXK9ePaZOnerqZkVEXMOkU9uqbAnNxx9/vKpOJSJSNpNewKuyecbTp0+vqlOJiJTN0xcKqlNHMwNExARMetOHy8P4wIEDZb7XtGlTV59ORKRcjCJzzqZweRhPnjyZrKwsQkJCSixyY7FYePfdd119OhGR8vGUnvGSJUsYMmQIs2bNokEDTU8SEZMx6Zixy2dT1KpVi2effZajR48CaAlIETEXk86mqJSpbW3atKFdu3YADBs2rDJOISJSIYbdcHirSpU+m0I9YxExFScu4BUWFhIXF8eRI0fw8vJi+vTp+Pj4EBcXh8VioUWLFiQmJuLlVf5+bqWHcfv27Sv7FCIijnOix/v1119TVFTEihUr2LhxI3PnzqWwsJDY2Fg6duxIQkICqamp9OjRo9xtV/odeLGxsZV9ChERxzkxZty0aVNsNht2u528vDx8fHzYs2cPoaGhAISHh5Oenl6hsvSkDxHxKM4MndauXZsjR45w7733cvr0aRYtWsTmzZuxWCwA+Pv7k5ubW6G2FcYi4lmcGKZYunQpXbt2ZcKECRw7doxhw4aVeDya1WolKCioQm1X2UJBIiKm4MQwRVBQEIGBgcClJR6Kiopo3bo1GRkZAKSlpdGhQ4cKleXy9YwrQusZy+W0nrGUxdn1jM8Ou9vhY+u8k1ritdVqZfLkyeTk5FBYWMjQoUNp06YN8fHxFBYWEhISwvPPP4+3t3e561IYiykpjKUsTodxTDnCODn12ge5iMaMRcSjVPXNHI5SGIuIZ1EYi4iYgDnXCVIYi4hn0TCFiIgJGEUKYxER99MwhYiI+5l0bXmFsYh4GIWxiIj7qWcsImICRpG7KyidwlhEPIp6xiIiJqAwFhExA8Pi7gpKpTAWEY+inrGIiAkYdvWMRUTczm5TGIuIuJ2GKURETEDDFCIiJuD+B82VTmEsIh5FPWMREROodhfwvv322zI/1LVr10opRkSksjnTM16zZg0ffvghAPn5+ezdu5eUlBRmzJiBxWKhRYsWJCYm4uXlVe62ywzjdevWlfkhhbGIVFeGE3fgRUZGEhkZCcCzzz7LgAEDWLhwIbGxsXTs2JGEhARSU1Pp0aNHudsuM4yTkpJK3X/8+PFyn0RExCxcMbVt165d/PzzzyQmJrJgwQJCQ0MBCA8PZ+PGja4N41/NmzePlJQUCgsLuXjxIn/+85+v2msWETEzuwvWpnjjjTd47LHHADAMA4vlUpv+/v7k5uZWqM1rDmykpaWRlpbG/fffz6effkqDBg0qdCIRETMwDIvDW2nOnTvH/v376dSpE0CJ8WGr1UpQUFCF6rpmGNetWxdfX1+sVitNmjThwoULFTqRiIgZ2G0Wh7fSbN68mc6dOxe/bt26NRkZGcClzmuHDh0qVNc1w/iGG25g9erV1KpVi1deeYW8vLwKnUhExAwMu8XhrTQHDhygYcOGxa8nTpzI/PnzefDBByksLCQiIqJCdVkM4+r3o9jtdo4dO0adOnX48MMPCQsLo3nz5hU6WVl8fP/k0vak+msSpOEwKV3miW1OfX53SG+Hj22zf61T5yqPa17A+/jjj4v/HBgYyO7du10exiIiVcWZqW2V6ZphnJmZCVy6Yrh3717q1q1Lv379KrsuEZFKUW3XppgwYULxnw3D4JFHHqnUgkREKpMrprZVhmuGcUFBQfGfc3JyOHz4cKUWJCJSmezVdaGgnj17YrFYMAwDPz8//va3v7m8iHq1AlzeplRvP/yw2t0lyO9Ute0Zz507l7Zt2xa//te//lWpBYmIVKZqdwFvy5Yt/PzzzyxdupSHH34YuDTNbfny5axdW3XTPUREXKna9YyDgoI4ceIEBQUF5OTkAGCxWHj66aerrDgREVcz6WSKssO4ZcuWtGzZkkGDBnHq1ClatWrFV199VeI2QBGR6sZmL/9aw1XhmlW98MIL7NixA7h0G2BcXFxl1yQiUmns5diq0jXDODs7m+joaABGjRql9YxFpFozsDi8VSWH+usHDhwA4ODBg9jtVf37QkTEdeyG41tVuubUtsmTJxMbG8vJkyfx8/Ojf//+VVGXiEilsFdxj9dR1+wZ33zzzUyfPp3OnTtz4cIFTp48WRV1iYhUCrMOU5TZMy4oKGDdunUsX74cX19f8vLySE1Nxc/PryrrExFxKVt16xnfdddd7Nu3j5dffpmUlBTq16+vIBaRas+ssynK7BkPHTqUtWvXcuTIEQYOHMg11qAXEakWzDoFocye8ejRo/n444+JiYlh7dq17N69m1mzZvHjjz9WZX0iIi5l1jHja17ACw0NZdasWXz55ZfccMMNPPPMM1VRl4hIpbBbHN+qksP3BQYFBRETE8NHH31UieWIiFQuOxaHt6p0zXnGIiK/JzYnP//GG2+wfv16CgsLiY6OJjQ0lLi4OCwWCy1atCAxMREvr/Kvf2HOFTNERCqJ3WJxeLtcRkYG27dv57333iM5OZlffvmFpKQkYmNjSUlJwTAMUlNTK1SXwlhEPIpRju1y3377LS1btuSxxx5jzJgx3HnnnezZs4fQ0FAAwsPDSU9Pr1BdGqYQEY/izNS206dPc/ToURYtWsThw4cZO3YshmFg+U8v2t/fn9zc3Aq1rTAWEY/izCyJunXrEhISgq+vLyEhIdSsWZNffvml+H2r1UpQUFCF2tYwhYh4FBsWh7fLtW/fnm+++QbDMMjOzubChQuEhYWRkZEBQFpaGh06dKhQXeoZi4hHcaZn3K1bNzZv3lx8V3JCQgINGzYkPj6e2bNnExISQkRERIXaVhiLiEdx9nbo0m58W7ZsmZOtKoxFxMOYdZUdhbGIeJSqvs3ZUQpjEfEoZl21TWEsIh7Fpp6xiIj7qWcsImICCmMRERPQbAoRERPQbAoRERPQMIWIiAk4u7h8ZVEYi4hH0TCFiIgJaJhCRMQENJtCRMQE7CaNY4WxiHgUXcATETEBjRmLiJiAZlOIiJiAxoxFREzAnFGsMBYRD6MxYxERE7A52Tfu168fgYGBADRs2JAxY8YQFxeHxWKhRYsWJCYm4uXlVe52FcYi4lGc6Rnn5+cDkJycXLxvzJgxxMbG0rFjRxISEkhNTaVHjx7lbrv88S0iUo3ZMRzeLvfDDz9w4cIFRowYwdChQ9mxYwd79uwhNDQUgPDwcNLT0ytUl3rGIuJRnBmk8PPzY+TIkQwaNIh///vfjBo1CsMwsFguzZfz9/cnNze3Qm0rjEXEozgzTNG0aVOaNGmCxWKhadOm1K1blz179hS/b7VaCQoKqlDbGqYQEY9iw3B4u9zq1auZOXMmANnZ2eTl5dGlSxcyMjIASEtLo0OHDhWqSz1jEfEoztz0MXDgQCZNmkR0dDQWi4UZM2ZQr1494uPjmT17NiEhIURERFSobYWxm3h5eTFn3vM0b9EUm83G449NIiDAn6SX4rHZbBQUFPDYIxPJyTnp7lLFSTv3/MDs15ewdMFLJfb/89tNvP6/Kfh4e9O/9z0M7HNvudsurY3CoiLiZ8zh6LFsCgoLeWRYNN1u7+Sqr1PtOTNm7OvryyuvvHLF/mXLljnR6iUKYzeJuLcbAL0iouncNZTnXphEnTqBTHpmOrt3/cDQhx/k77GjSJgy082VijOWLF/FJ5+tp5ZfzRL7C4uKeHHem6x461Vq1/JjyJgJ3NmlI9dfF+xw22W18c2mLdQNCmRmwtOcOXuOgQ+PUxj/hllvh9aYsZv8Y10qTz4RD0CjRn8kJ+cEo0c8ye5dPwDg4+1dPKdRqq9Gf/wf5s6YesX+/f/OonHDP1InKJAaNWpwa9u/sPX7PZd6tUlzGPbo08SMncC/tu0s8bk77h98zTYiut3O30cNLT7Ox9u78r5gNWQvx1aV1DN2I5vNxoLXZ3Jf7x6MGPY42dk5ANwW2o6Ro4fQ596H3FyhOKtHt64cOZZ9xX6r1UqAv3/xa//atcjNs/LBJ59Rr04Q0yeN58zZcwx79Gn+3/I3GDMhnov5+Zw9l8vwcc/Q4PrreLB/r1LbqF271n/OcZ7xU14oEcwChkl7xpUSxidPnmTLli3k5uYSFBTELbfcQv369SvjVNXeuLFx1E98mc9S36drx17c0/NOYieMZfADozl58rS7y5NK4u/vz/nz54tfW89fICjQn4yt37Pt+z3s/L99wKVf2GfOnmPRK9OBSz3jX8ee9/18oNQ2AI5l5/DEpOlERfai1z3dquprVQvO3g5dWVwexqtWrWLlypW0b98ef39/fvrpJxYtWsSgQYOIjo529emqrUEP9uWPf2rAq7Pf5PyFC9gNg/t692DYww/Sr3cMZ06fdXeJUolC/tyIg4ePcvZcLrVr+bH1+90MHzyA4ydO0eAP1zN6WBQX8/N5850VBAUGlKuNE6dOM3r8FKY8OZZOHdpV8TczP7MuFGQxDMOlvyaioqJITk6mRo0axfsKCgqIjo7mgw8+KPUzf6hzoytLqBZq167FvNeSqF//enxq+DBvzmLmvTaDI1nHOHv2HADpGzfzUtJ8N1fqHkcz/+HuElzmyLFsnk5IImXxXNZ9sYHzFy4wqO99xTMhDMOgf697iB5wPwUFBSS+OI9jv2STZz1PVGTvq86yKK2NpLmL+Cw1jaZNGhYft+iV6fjVrFlmO9VJjetDnPp8TJNIh49NPrjGqXOVh8vDeODAgSxdupSAgP/+Nj937hwjRoxg9erVpX7GE8NYru73FMbiWs6G8ZByhPGyKgxjlw9TPProo0RGRtKkSRMCAwPJy8vj4MGDTJo0ydWnEhEpN7NObXN5GN91112Eh4eTmZlJXl4eAQEBNGvWDB8fTdwQEffzqNkUPj4+3HhjyaGHVatWMWjQoMo4nYiIw4o8KYxLU6tWrao6lYhImTyqZ1ya3r17V9WpRETKZNapbS4P45iYGAoLC0vs+3Xx5RUrVrj6dCIi5eLiCWQu4/Iwfuqpp5g6dSoLFy7EW/fEi4jJeMxsiptvvpm+ffuyb9++Cj2UT0SkMnnM7dAAf/vb3yqjWRERp3lMz1hExMw8ZsxYRMTMPGY2hYiImXn8PGMRETMw65ixHrskIh7FZtgd3spy8uRJ7rjjDjIzMzl48CDR0dEMHjyYxMRE7PaKDYQojEXEoxjl+K80hYWFJCQk4OfnB0BSUhKxsbGkpFxaVzo1NbVCdSmMRcSj2A3D4a00L774IlFRUcWPktuzZw+hoaEAhIeHk56eXqG6FMYi4lGMcmyXW7NmDcHBwdx+++3/be8/yz3ApWcb5ubmVqguXcATEY/izAW8Dz74AIvFwnfffcfevXuZOHEip06dKn7farUSFBRUobYVxiLiUZwJ4+XLlxf/OSYmhmnTpjFr1iwyMjLo2LEjaWlpdOrUqUJta5hCRDyKK2ZT/NbEiROZP38+Dz74IIWFhURERFSoLpc/kLQi9EBSuZweSCplcfaBpLf9MdzhYzcfTXPqXOWhYQoR8Sgm6H+WSmEsIh7FrHfgKYxFxKOoZywiYgI2k67bpjAWEY9S1p117qYwFhGPoiU0RURMQD1jERETUM9YRMQE1DMWETEBR29zrmoKYxHxKBqmEBExAUM9YxER99Pt0CIiJqDboUVETEA9YxERE7DZNWYsIuJ2mk0hImICGjMWETEBjRmLiJiAesYiIibgzAU8m83G1KlTOXDgAN7e3iQlJWEYBnFxcVgsFlq0aEFiYiJeXl7lblthLCIexZlhig0bNgCwYsUKMjIyisM4NjaWjh07kpCQQGpqKj169Ch32+WPbxGRaswwDIe3y3Xv3p3p06cDcPToUa6//nr27NlDaGgoAOHh4aSnp1eoLoWxiHgUu2E4vJXGx8eHiRMnMn36dCIiIjAMA4vFAoC/vz+5ubkVqkvDFCLiUVwxz/jFF1/kqaee4oEHHiA/P794v9VqJSgoqEJtqmcsIh7FmZ7xRx99xBtvvAFArVq1sFgstGnThoyMDADS0tLo0KFDheqyGCaY5/GHOje6uwQxmaOZ/3B3CWJSNa4PcerzNf0aOXxs/sWsEq/Pnz/PpEmTOHHiBEVFRYwaNYpmzZoRHx9PYWEhISEhPP/883h7e5e7LoWxmJLCWMribBj71mzo8LEF+YedOld5aMxYRDyKCfqfpTJFz1hExNPpAp6IiAkojEVETEBhLCJiAgpjN7pw4QJRUVFkZmZe8d6pU6cYMWIEgwcPJjY2lgsXLrihQnGH77//npiYmCv2r1+/ngEDBvDggw/y/vvvu6EyqUwKYzfZtWsXDz30EFlZWaW+/9prr9G7d29SUlJo3bo1K1eurOIKxR0WL17M1KlTS9zVBVBYWEhSUhJLliwhOTmZlStXkpOT46YqpTIojN2koKCAhQsXEhJS+pzJrVu3cvvttwPOLT4i1Uvjxo2ZP3/+FfszMzNp3LgxderUwdfXl/bt27NlyxY3VCiVRfOM3aR9+/ZXfT8vL4/AwEDAucVHpHqJiIjg8OErbzT47c8DXPqZyMvLq8rSpJIpjKvQnDlz2LZtGwBLly696i2TAQEBWK1W/Pz8nFp8RH4ffv15+JXVai0RzlL9KYyr0Pjx4x0+9tZbb+Xrr78mMjKStLS0a/ak5fetWbNmHDx4kDNnzlC7dm22bNnCyJEj3V2WuJDGjE3kzJkzjBs3DoCxY8eybt06oqKi2L59O0OGDHFzdeIOn3zyCStXrqRGjRrExcUxcuRIoqKiGDBgAA0aNHB3eeJCuh1aRMQE1DMWETEBhbGIiAkojEVETEBhLCJiAgpjERETUBiLS2RkZBAWFkZMTAwxMTE88MADJCcnl7udl19+mTVr1rB3714WLFhQ5nFffvkl2dnZDrWZlpZGXFxcuWsRqUq66UNcplOnTsyZMwe4tPZGz5496du3b4XuHmzVqhWtWrUq8/13332XadOmaa6t/G4ojKVS5OXl4eXlxfDhw2nYsCHnzp3jzTffZNq0aRw8eBC73U5sbCwdO3bk888/5/XXXyc4OLj4CbsZGRmsWLGCOXPmsGrVKt577z3sdjt33303f/3rX9m7dy8TJ04kJSWFlStXsnbtWiwWC/fddx9Dhw4lMzOTyZMnU6tWLWrVqkWdOnXc/VciclUKY3GZTZs2ERMTg8VioUaNGsTHx/PWW29x//3306NHD1JSUqhXrx4zZszg9OnTDBkyhHXr1jFr1ixWrVpF3bp1GT16dIk2T548yeLFi/n444/x9fVl5syZ3HbbbbRq1Ypp06Zx6NAhPv30U1JSUrBYLAwfPpyuXbvy6quv8vjjj9OlSxfefPNN9u/f76a/FRHHKIzFZX47TPGrt956i6ZNmwLw448/snXrVnbu3AlAUVERJ06cICAggHr16gHQrl27Ep/PysqiRYsW+Pn5ATB58uQS7//4448cPXqU4cOHA3D27FkOHTrETz/9RNu2bYFL63wojMXsdAFPKp3FYgEgJCSEXr16kZyczOLFi+nZsydBQUHk5uZy6tQp4NKi+7/VuHFj9u/fT0FBAQCPP/442dnZWCwWDMMgJCSE5s2b8+6775KcnExkZCQtW7YkJCSE7du3A7B79+4q/LYiFaOesVSZqKgopk6dypAhQ8jLy2Pw4MH4+vqSlJTEyJEjqVOnDj4+JX8kg4ODGTVqFEOGDMFisdCtWzcaNGhAu3bteOaZZ1iyZAlhYWFER0dTUFBA27ZtadCgAYmJiYwfP563336b4OBgatas6aZvLeIYLRQkImICGqYQETEBhbGIiAkojEVETEBhLCJiAgpjERETUBiLiJiAwlhExAQUxiIiJvD/ATKFkMQuwKMUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "metrics_train = score(W, classification_train_X, classification_train_y, plot_matrix=True, title=\"Metrics on Train set\", macro=True)\n",
    "metrics_test = score(W, classification_test_X, classification_test_y, title=\"Metrics on test set\", plot_matrix=True, macro=True)\n",
    "print(\"Performance metrics for train dataset\")\n",
    "[print(f\"{key} : {value}\") for key, value in metrics_train.items()]\n",
    "print(\"**************************\")\n",
    "print(\"Performance metrics for test dataset\")\n",
    "[print(f\"{key} : {value}\") for key, value in metrics_test.items()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (ii) Kernel SVM\n",
    "\n",
    "The cost function, hinge loss for Linear SVM is defined as:\n",
    "$$\n",
    "\\mathcal L (\\boldsymbol w) = \\frac{1}{2} k(w, w) + \\frac{\\lambda}{n} \\sum_{i=1}^n \\max \\bigg( 0, 1-y_i (k(\\boldsymbol w, x_i)) \\bigg) \\, \n",
    "$$\n",
    "where k is the kerrnel function.\n",
    "\n",
    "In this case, we us the RBF kernel, given by the equation:\n",
    "$$\n",
    "k(x_i, x_j) = e ^ {- \\frac{\\|{x_i - x_j}\\|^2}{2 \\sigma^2}}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 363,
   "metadata": {},
   "outputs": [],
   "source": [
    "def euclidean_dist(m, n):\n",
    "    return np.sqrt(np.sum(np.square(m - n)))\n",
    "\n",
    "def gaussian(w, X, sigma):\n",
    "    euclidean_distance = euclidean_dist(X, w)\n",
    "    exponent = - euclidean_distance / 2 * sigma ** 2\n",
    "    return np.exp(exponent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 364,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we compute the hinge loss for the rbf svm by replacing the dot products in linear svm with the kernel function\n",
    "def compute_cost_rbf(W, X, y, reg_param=1e100, sigma=0.01):\n",
    "    n = X.shape[0]\n",
    "    \n",
    "    distances = 1 - y * (gaussian(W, X, sigma))\n",
    "    distances[distances < 0] = 0\n",
    "    hinge_loss = reg_param * np.sum(distances) / n\n",
    "    \n",
    "    cost = 1/2 * gaussian(W, W, sigma) + hinge_loss\n",
    "    return cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform gradient descent to optimise the weights vector, we need to calculate the gradient of the loss function with respect to the weight vector. The formula is:\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathcal L}{\\partial \\boldsymbol w} = \n",
    "\\frac{1}{N} \\sum_{i=1}^{n}\n",
    "\\left\\{\\begin{array}{ll}\n",
    "\\boldsymbol{w} & \\text { if } \\max \\left(0,1-y_{i} *\\left(k (\\boldsymbol{w},x_{i})\\right) + b\\right)=0 \\\\\n",
    "\\boldsymbol{w}-\\lambda y_{i} x_{i} & \\text { otherwise }\n",
    "\\end{array}\\right.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 365,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_cost_gradient_rbf(W, X_batch, y_batch, reg_param=1e100, sigma=0.01):\n",
    "    # convert singular data points into a list\n",
    "    if type(y_batch) == np.float64:\n",
    "        y_batch = np.asarray([y_batch])\n",
    "        X_batch = np.asarray([X_batch])\n",
    "    \n",
    "    distance = 1 - y_batch * (gaussian(W, X_batch, sigma))\n",
    "    dw = np.zeros(W.shape[0])\n",
    "    \n",
    "    for i, d in enumerate(distance):\n",
    "        if max(0, d) == 0:\n",
    "            di = W\n",
    "        else:\n",
    "            di = W - (reg_param * y_batch[i]* X_batch[i])\n",
    "        dw += di\n",
    "    \n",
    "    #compute average\n",
    "    dw = dw / len(y_batch)\n",
    "    \n",
    "    return dw\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 366,
   "metadata": {},
   "outputs": [],
   "source": [
    "def kernel_sgd(X, y, sigma, max_epochs=2000, stop_criterion=0.01, learning_rate=1e-5, reg_param=1e100, print_outcome=False):\n",
    "    \n",
    "    # add extra column of 1s to X to predict b\n",
    "    X = np.hstack((X, np.ones((X.shape[0], 1))))\n",
    "    \n",
    "    # change labels from 0 to -1\n",
    "    y = np.array(y)\n",
    "    y[y == 0] = -1\n",
    "    # initialise random weights\n",
    "    weights = np.random.rand(X.shape[1])\n",
    "    nth = 0\n",
    "    # initialise starting cost as infinity\n",
    "    prev_cost = np.inf\n",
    "\n",
    "    # stochastic gradient descent\n",
    "    for iteration in range(1, max_epochs):\n",
    "        # shuffle to prevent repeating update cycles\n",
    "        np.random.shuffle([X, y])\n",
    "        for ind, x in enumerate(X):\n",
    "            ascent = compute_cost_gradient_rbf(weights, x, y[ind], reg_param, sigma)\n",
    "            weights = weights - (learning_rate * ascent)\n",
    "\n",
    "        # convergence check on 2^n'th iteration\n",
    "        if iteration==2**nth or iteration==max_iterations-1:\n",
    "            # compute cost\n",
    "            cost = compute_cost_rbf(weights, X, y, sigma)\n",
    "            if print_outcome:\n",
    "                print(\"Iteration is: {}, Cost is: {}\".format(iteration, cost))\n",
    "            # stop criterion\n",
    "            if abs(prev_cost - cost) < stop_criterion * prev_cost:\n",
    "                return weights\n",
    "\n",
    "            prev_cost = cost\n",
    "            nth += 1\n",
    "\n",
    "    return weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 385,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score_rbf(W, X, y, sigma, title=\"\", plot_matrix=False, macro=False,):\n",
    "    y_preds = np.array([])\n",
    "    # change the labels from 0 t0 -1\n",
    "    y[y == 0] = -1\n",
    "    \n",
    "    # extract b from weights to add separately to the prediction\n",
    "    b = W[-1]\n",
    "    W = W[:-1]\n",
    "    for i in range(X.shape[0]):\n",
    "        y_pred = np.sign(gaussian(W, X[i], sigma) + b)\n",
    "        y_preds = np.append(y_preds, y_pred)\n",
    "    \n",
    "    confusion_matrix = rf_confusion_matrix(y_preds, y)\n",
    "    if plot_matrix:\n",
    "        plt.figure()\n",
    "        plt.title(title)\n",
    "        sn.heatmap(confusion_matrix, annot=True)\n",
    "    f1_score, precision, recall, accuracy, TNR = svm_f1_score(y_preds, y, macro)\n",
    "    \n",
    "    return f1_score, precision, recall, accuracy, TNR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 397,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_validation_rbf(sigma):\n",
    "    num_of_folds = len(classification_folds)\n",
    "    \n",
    "    avg_f1_score = 0\n",
    "    \n",
    "    for i in range(num_of_folds):\n",
    "        train_X, train_y, test_X, test_y = generate_test_train_sets(classification_folds, i)\n",
    "        \n",
    "        # perform sgd with weights to get the best W\n",
    "        W = kernel_sgd(train_X, train_y,max_epochs=2000,stop_criterion=0.01, learning_rate=1e-3, reg_param=1e100, sigma=sigma, print_outcome=False)\n",
    "        \n",
    "        # calculate f1 score for data set\n",
    "        f1, _, _, _, _ = score_rbf(W, test_X, test_y, sigma, macro=True)\n",
    "        avg_f1_score += f1\n",
    "    \n",
    "    return avg_f1_score / num_of_folds\n",
    "\n",
    "def tune_rbf_hyperparameters():\n",
    "    sigmas = np.linspace(0.001, 0.1, 100)\n",
    "    grid = {}\n",
    "    \n",
    "\n",
    "    for i, sigma in enumerate(sigmas):\n",
    "        f1 = cross_validation_rbf(sigma)\n",
    "        grid[sigma] = f1\n",
    "    \n",
    "    \n",
    "    opt_sigma, max_f1 = grid_search(grid)\n",
    "    \n",
    "    print(\"opt_sigma\", opt_sigma)\n",
    "    print(\"max_f1: \", max_f1)\n",
    "    \n",
    "    return opt_sigma"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 398,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "opt_sigma 0.1\n",
      "max_f1:  0.41278348777722995\n"
     ]
    }
   ],
   "source": [
    "sigma = tune_rbf_hyperparameters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 413,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rbf_train(X, y, sigma, macro=False):\n",
    "    W = kernel_sgd(X, y, sigma, max_epochs=2000, stop_criterion=0.001, learning_rate=1e-3, reg_param=1e100)\n",
    "    f1_train, precision_train, recall_train, accuracy_train, TNR = score_rbf(W, X, y, sigma, plot_matrix=True, title=\"Confusion Matrix for Train dataset\", macro=macro)\n",
    "    metrics = {\"f1_train\":f1_train, \"precision_train\":precision_train, \"accuracy_train\":accuracy_train, \"TNR\":TNR, \"recall_train\":recall_train}\n",
    "    return W, metrics\n",
    "    \n",
    "def rbf_predict(W,X_test, y_test, sigma, plot_matrix=False, macro=False):\n",
    "    f1, precision, recall, accuracy, TNR = score_rbf(W, X_test, y_test, sigma, plot_matrix=plot_matrix, title=\"Confusion Matrix for Test dataset\", macro=macro)\n",
    "    metrics = {\"f1\":f1, \"precision\":precision, \"accuracy\":accuracy, \"TNR\":TNR, \"recall\":recall}\n",
    "    return metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 414,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEPCAYAAACX9KXAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAonUlEQVR4nO3de1hU1d4H8O/AMIBcRCLxnBSUFPWVJqmOpqCJaSiCCEoBOaZ4xTyG5QXwWpSXqEyNvNTbsSaVUMwMrV5CjVMqJZ1CVOqENxQlRUNm0Blg1vuHj3NCuc8AmzPfz/Ps52n2rL3W2tPz/Pi59lpry4QQAkRE1Kas2roDRETEYExEJAkMxkREEsBgTEQkAQzGREQSwGBMRCQBDMbNVF1djX/84x8IDw9HaGgogoKCkJycDL1eb1KdsbGxCAwMxMcff9zk648fP465c+c2u/27DR8+HP3794dWq61xfvfu3ejduze+/PLLeq8vLy/HpEmT6vw+NDQUN27caHR/du/ejWHDhmHq1KmNvubPDh8+jNDQUISGhsLPzw+PP/648fP+/fsbXc+OHTuwZcuWZvXhDl9fX1y4cKHeMnl5eVi2bJlJ7dRl586d2LZtW4vUTc0jb+sOtFcrVqxAWVkZPvzwQzg5OaGiogLz58/H4sWLkZyc3Kw6S0pK8O233+Knn36CtbV1k69/6KGHsH79+ma1XZdOnTohMzMT48aNM57bs2cP3NzcGry2rKwMx48fr/P7zz77rEl92bNnD+bNm4fQ0NAmXXfH4MGDjW1u2LAB169fb1awi4qKalb7TfXbb7+hpKSkRerOzc1Fr169WqRuah5mxs1w4cIFfP7551i5ciWcnJwAAB06dMDLL7+MESNGALidFc6fPx/BwcEICQnB66+/jqqqKgC3g+aGDRsQGRmJ4cOHY/v27dBoNJg2bRqqqqoQHh6O8+fPo3fv3rh27Zqx3TuftVot5s6di9DQUISFhWHJkiUwGAzIyclBcHBws9qvy9ixY7F3717j54sXL6KiogJeXl7Gc7t27UJERATGjRuHgIAAY30JCQm4desWQkNDUV1dDR8fH7zwwgsIDAzE8ePHjffzzjvvIDIyEtXV1bhy5Qr8/f1x9OjRGv1YuXIljh8/jnXr1mHr1q313t/d7TT2/+kTTzyBmJgYBAYG4vfff8emTZsQERGBkJAQjBgxApmZmQBuB/JXXnkFwO1/PWzYsAHR0dEICAjA22+/XWv9x44dQ2hoKMaNG4elS5fCYDAAAAwGA1599VVEREQgKCgIo0ePRm5uLi5duoT169fj2LFjSEhIqLPcnbonTJiA8PBwhIeH46uvvgIA6PV6rFy5EmFhYRg7dizi4+Oh0WiQmZmJAwcOYOvWrcyOpURQk3355Zdi/Pjx9ZZZuHChSEpKEgaDQeh0OhETEyM2b94shBDC29tbqNVqIYQQx48fFz4+PuLWrVuiqKhI9O/f31iHt7e3KC0tvefzp59+KmJiYoQQQlRVVYnFixeLs2fPiqNHj4oxY8Y0u/27BQQEiNzcXDFo0CBRUlIihBAiJSVFqNVqMXHiRPHFF18IjUYjnn76aXHt2jUhhBD/+te/jPdQ2/18+umn99xPVVWVePbZZ8XmzZvF5MmTxcaNG2v9Te+02Zj7+3M7tVm/fr14+eWXjZ+LioqEt7e3+OGHH4QQQly4cEGoVCpx8+ZNIYQQGRkZIjg4+J5rAwICxOrVq4UQQly+fFk89NBD4vz58zXa0ul0YvDgweLw4cNCCCE+//xz4e3tLYqKisSPP/4o/v73v4vq6mohhBCbN28WM2fOFEIIkZ6eLmbMmCGEEPWWmzRpksjIyBBCCHHq1CmxYsUKIYQQGzZsEKtXrxYGg0EIIcSbb74pli9fLoQQYtGiReL999+v9zei1sVhimawsrIyZjZ1yc7Oxo4dOyCTyaBQKBAZGYkPP/wQM2bMAAA8+eSTAIB+/fpBr9ejoqKi0e0/+uijWLt2LVQqFQYPHoznnnsOnp6euHz5sknt29ra3tOWjY0NAgMDkZGRgZiYGHzxxRdQq9XG7MvBwQGbNm3CN998g7Nnz6KgoKDee3nsscfuOWdtbY033ngDISEh6NevH2bOnNngb9DQ/dXWTkPkcjn69+8PAHjggQfw+uuv4/PPP8e5c+fw888/3zN2fsed39Ld3R333XcfysrK0K1bN+P3v/76K+RyOQYNGgQACA4ONg6P+Pr6omPHjkhNTUVRURFycnLg4OBwTxv1lRs9ejReeeUVHDhwAIMHD8aLL74IADh06BDKy8tx+PBhAEBlZSXuu+++Jv8u1Do4TNEMSqUSp0+fhkajqXG+pKQEM2bMwK1bt2AwGCCTyYzfGQwG4z+jARgD350yooEtQv78YLBbt27IzMzEjBkzoNFoMGXKFBw4cKBGeXO2P27cOOzduxc//vgjevToARcXF+N3ly9fxrhx43Dx4kU8+uijiIuLq/c+OnToUOv5ixcvwtbWFufPn0dZWVm9ddy5n/rur6526qNQKCCX385PTpw4gWeeeQYajQZ+fn6YNm1andf9+Y+YTCar9be8+9yddg4dOmT84/Pkk0/WOR5dX7nIyEjs3bsXfn5++PbbbzF27FjodDoYDAYkJibis88+w2effYadO3di3bp1jfkpqA0wGDeDu7s7QkJCkJiYaAzIGo0GK1asgIuLC+zs7ODv74+PP/4YQgjo9XqkpaVh8ODBTWrH1dXVOOaZkZFhPL99+3YkJCTA398fCxYsgL+/P06ePFnjWnO0f8fDDz+MW7duYe3atQgLC6vxXX5+PlxdXTF79mz4+/vj4MGDAG7PDJHL5aiurm7wD82NGzewYMECrF69GsHBwVi8eHGDfTLn/dXmhx9+gI+PD6ZMmYIBAwYgKysL1dXVzaqrd+/eEELgm2++AQBkZWUZ/+B89913CAgIQHR0NHx8fPD1118b27G2tjb+gamvXGRkJE6dOoXw8HAkJSXhxo0bxrH3bdu2Qa/Xw2AwYOnSpXjrrbfuqZukgcG4mZYvX46ePXsiMjISoaGhiIiIQM+ePfHqq68CAJYsWYJr164hJCQEISEh6NGjB2bNmtWkNpYsWYJXXnkFYWFhKCwsxP333w/gdqZaXV2NoKAghIeHo7y8HCqV6p5rTW3/z0JDQ3HmzBkMGTKkxnk/Pz+4u7tj1KhRGD16NC5dugRXV1ecO3cO999/P5RKJcaMGYPr16/Xe5/Dhg2Dv78/5syZg6KiogYfLJn7/u4WHByM69evY/To0QgKCkKHDh1QVlZ2z7+GGsPGxgYpKSlYt24dQkNDkZmZaRwuiIyMxPfff4+QkBCEhYWhW7duuHDhAgwGA/r374+ioiLMmTOn3nLz58/H+vXrMW7cOKhUKsyZMwddu3bF7Nmz8cADDyAsLAxBQUEQQiA+Ph4AMHToUKSmpmLz5s1m+83INDLRUNpCREQtjpkxEZEEMBgTEUkAgzERkQQwGBMRSQCDMRGRBEhiBZ5c8UBbd4GI2okq/UWTrq+8errRZW3cvBouZCaSCMZERK3G0LzFOy2NwZiILIuof1+ZtsJgTESWpYFNvtoKgzERWRTBzJiISAKqTdsgady4ccaXSnTt2hUTJ07ErFmz0L17dwC33wQTFBSEtLQ0pKamQi6XIzY2FgEBAfXWy2BMRJbFhAd4Op0OAKBWq43ndu7ciSlTpiAmJsZ47sqVK1Cr1UhPT4dOp0N0dDT8/PygUCjqrJvBmIgsiwnDFAUFBbh58yZiYmJQVVWFF198Efn5+Thz5gyysrLg6emJxMRE5OXlwdfXFwqFAgqFAh4eHigoKIBSqayzbgZjIrIsJjzAs7Ozw9SpUxEREYGzZ89i+vTpmDFjBiIiIuDj44ONGzciJSUFffr0MQ5lALffiNPQ9qtcgUdEFkUIQ6OPu/Xo0QNjx46FTCYzvvVmyJAh8PHxAQCMHDkSJ0+ehKOjY43XdGm12hrBuTYMxkRkWQyGxh932bVrF1avXg3g9mvWNBoNnn/+eeTl5QEAjhw5gn79+kGpVCI3Nxc6nQ7l5eUoLCyEt7d3vd2SxObyXA5NRI1l6nJoXcE3jS5r2+eJGp/1ej0SEhJQXFwMmUyG+fPnw9bWFklJSbCxsYGbmxuSkpLg6OiItLQ0fPLJJxBCYObMmQgMDKy3LQZjImpXTA7Gpw42uqxt3/qno5kTH+ARkWXhCjwiIgngCjwiIglgZkxE1PaEobKtu1ArBmMisizMjImIJIBjxkREEsA3fRARSQAzYyIiCeCYMRGRBJi4uXxLYTAmIsvCzJiIqO0JwQd4RERtj5kxEZEEcDYFEZEEMDMmIpIAzqYgIpIADlMQEUkAhymIiCSAwZiISAI4TEFEJAF8gEdEJAEcpiAikgAOUxARSQAzYyIiCWAwJiKSACHauge1YjAmIstSZdpsinHjxsHJyQkA0LVrV8yaNQvx8fGQyWTo1asXli9fDisrK6SlpSE1NRVyuRyxsbEICAiot14GYyKyLCY8wNPpdAAAtVptPDdr1izExcVh4MCBWLZsGbKystC/f3+o1Wqkp6dDp9MhOjoafn5+UCgUddbNYExElsWEMeOCggLcvHkTMTExqKqqwosvvogTJ05gwIABAIChQ4fiu+++g5WVFXx9faFQKKBQKODh4YGCggIolco662YwJiLLYsKYsZ2dHaZOnYqIiAicPXsW06dPhxACMpkMAODg4IDy8nJoNBrjUMad8xqNpt66GYyJyLKYkBn36NEDnp6ekMlk6NGjB1xcXHDixAnj91qtFs7OznB0dIRWq61x/s/BuTZWze4VEVF7ZDA0/rjLrl27sHr1agBASUkJNBoN/Pz8kJOTAwDIzs7GY489BqVSidzcXOh0OpSXl6OwsBDe3t71douZMRFZFFHd/BeSTpgwAQkJCYiKioJMJsPKlSvRqVMnLF26FG+99Ra8vLwQGBgIa2trqFQqREdHQwiBefPmwdbWtt66ZUK0/aQ7ueKBtu4CEbUTVfqLJl1fsemFRpftMGudSW01BTNjIrIs3JuCiEgCDG0+GFArBmMisiwS3ZuCsynI7ORyObb+Yz0OHdiNI99lIDh4ZK3lNr67BitfS2hWG8FjRuLI4X34NnsvpsZEN6ldsnDV1Y0/WhEzYzK7Z6PDUVp6HZOnzIWraycc+/4rZGRk1igzfdpEPOTTF9n/PNLk+uVyOd5IXo7HB4+BVluB7G/2IGNfJkYFBjTYLpFUM2MGYzK7XekZSN+9z/i56q6NWR4f+CgGDnwEW977GH36PAjgdoB9N2U1evXsASsrKyxb/jq+yf5PoL5w/l/o6uELAOjbtxcKC8/ijz/KAACHv/sB/v4DG2yXCIDljBnrdDrs2LEDR48eRXl5OZycnPDYY49h4sSJsLOzM3dzJEFabQUAwNHRAWmpW7BsxevG77p06YxlS1/E+IipiJgw1nh+akw0SkuvYcbM+XB17YSDB9LxcP/hyNirhr29HVxdXZCVuRMXiy9j8+aPUHaj3HhtuUaDjs5O9bZLZGQpsykSEhLQp08fxMXFwcHBAVqtFtnZ2XjppZeQkpJi7uZIorp2/St27XwfmzZ9iNTUPcbzE8YH4z43V2TsVcO9S2d0sLdHQUEhfHz6wN9/AAb87Xb2K5fL4eraCcFjVQBuZ8ZPjowAADz0UF84OToY63RydMQfZTfqbZfIyFIy499//x1vvfVWjXN9+vRBdHS0uZsiierc2Q1f7N+OF15YggMHv63x3TspH+CdlA8AAJNUT6NPnwfxkToNzs6OuHjxElav2QA7OzskJszF9et/1Fr/qVP/Rs+eXujUyQUajRb+QwbizbWb6m2X6A4h0TFjs8+msLW1xZ49e1BaWgq9Xo9r165hz5496NChg7mbIomKX/R3dHLpiMWJLyArcyeyMnciKioM06Y+W+c1W977GL1798SBr3fhn9mf4dy5C/jz4tA748XA7bHgBQtfxv592/DtP/di69ZUFBdfrrVdDo3RPSQ6m8Lsy6GvX7+OlJQU/Pjjj9BqtXBwcMAjjzyC2NhY3HfffbVew+XQRNRYpi6H1r5Sd1JwN4dl20xqqynMPkzRqVMnLFmyxNzVEhGZh6UMU9Rl7ty5rdUUEVHdDKLxRytqtXnGSUlJrdUUEVHdLGVqW106duzYWk0REdXNUqa2nTlzps7vevToYe7miIiaRFS17iyJxjJ7ME5MTERRURG8vLxqTE2SyWT46KOPzN0cEVHTWEpm/MEHH2DixIlITk6Gu7u7uasnIjKNRMeMzT6bwt7eHi+//DKKi4sBABJ4qxMR0X9IdDZFi0xt8/Hxga/v7RVTzz33XEs0QUTULMIgGn20phafTcHMmIgkxVIe4N3t0UcfbekmiIgaz1Ie4N0tLi6upZsgImo8Sw3GRERSItWhUwZjIrIsEs2M+XZoIrIsZpjaVlpaiieeeAKFhYU4ceIEhgwZApVKBZVKhf379wMA0tLSEB4ejqeffhoHDx5ssFvMjInIoogq0xZ9VFZWYtmyZcYXF5w8eRJTpkxBTEyMscyVK1egVquRnp4OnU6H6Oho+Pn5QaFQ1FkvM2MisiyGJhy1WLNmDSIjI9G5c2cAQH5+Pg4dOoRnn30WiYmJ0Gg0yMvLg6+vLxQKBZycnODh4YGCgoJ6u8VgTEQWxZRFH7t374arqyuGDBliPKdUKrFw4UJs27YN3bp1Q0pKCjQaDZycnIxlHBwcoNFo6u0XgzERWRYTxozT09Nx+PBhqFQqnDp1CosWLcLQoUPh4+MDABg5ciROnjwJR0dHaLVa43VarbZGcK4NgzERWRYThim2bduGjz/+GGq1Gn379sWaNWswe/Zs5OXlAQCOHDmCfv36QalUIjc3FzqdDuXl5SgsLIS3t3e93eIDPCKyKObec2LFihVISkqCjY0N3NzckJSUBEdHR6hUKkRHR0MIgXnz5sHW1rbeesz+dujm4NuhiaixTH079LWwJxpd1vXTb0xqqymYGRORZZHmdsYMxkRkWSS6tzyDMRFZGAZjIqK2x8yYiEgCRFVb96B2DMZEZFGYGRMRSQCDMRGRFAhZW/egVgzGRGRRmBkTEUmAMDAzJiJqc4ZqBmMiojbHYQoiIgngMAURkQS0/T6VtWMwJiKLwsyYiEgC2t0DvG+//bbOi/z9/VukM0RELa3dZcb79u2r8yIGYyJqr0R7W4G3atWqWs///vvvLdYZIqKW1m6ntq1fvx7bt29HZWUlbt26he7du9ebNRMRSZlBopmxVUMFsrOzkZ2djZCQEOzfvx/u7u6t0S8iohYhhKzRR2tqMDN2cXGBQqGAVquFp6cnbt682Rr9IiJqEe1uNsUdXbp0wa5du2Bvb48333wTGo2mNfpFRNQipDqbQiZE/etRDAYDLl26hI4dO+LTTz/FoEGD0LNnT7N2Qq54wKz1EdF/ryr9RZOuz/cKbnRZn9MZJrXVFA1mxnv37jX+t5OTE/Lz880ejImIWku7m9p2R2FhIQBACIFTp07BxcUF48aNa+l+ERG1CHPsTVFaWorw8HB88MEHkMvliI+Ph0wmQ69evbB8+XJYWVkhLS0NqampkMvliI2NRUBAQL11NhiMX3rppT/dhMDMmTNNvxMiojZi6tS2yspKLFu2DHZ2dgBur8mIi4vDwIEDsWzZMmRlZaF///5Qq9VIT0+HTqdDdHQ0/Pz8oFAo6qy3walter3eeBQXF+PChQsm3QgRUVsyGGSNPmqzZs0aREZGonPnzgCAEydOYMCAAQCAoUOH4vDhw8jLy4Ovry8UCgWcnJzg4eGBgoKCevvVYGY8atQoyGQyCCFgZ2eHadOmNfXeG3Sz+J9mr5OIqDamZMa7d++Gq6srhgwZgi1btgC4PWIgk92u08HBAeXl5dBoNHBycjJe5+Dg0OBMtAaD8dtvvw2lUmn8/P333zfrJoiIpMCUB3jp6emQyWQ4cuQITp06hUWLFuHatWvG77VaLZydneHo6AitVlvj/J+Dc23qDMbHjh3Db7/9hq1bt2LKlCkAbk9z27ZtGzIyWm+6BxGROZmSGW/bts343yqVCitWrEBycjJycnIwcOBAZGdn4/HHH4dSqcTbb78NnU4HvV6PwsJCeHt711t3ncHY2dkZV69ehV6vx5UrVwAAMpkMCxYsaPaNEBG1NXO/6GPRokVYunQp3nrrLXh5eSEwMBDW1tZQqVSIjo6GEALz5s2Dra1tvfU0uOijpKQE165dQ9++ffH111/jiSeegI2NjVlvpvLqabPWR0T/vWzcvEy6/rsuExpd1u/yLpPaaooGZ1O89tpr+OmnnwAAZ86cQXx8fEv3iYioxRiacLSmBoNxSUkJoqKiAADTp0/nfsZE1K4JyBp9tKYGgzFwOyMGgHPnzsFgkOjOzEREjWAQjT9aU4NT2xITExEXF4fS0lLY2dkhLCysNfpFRNQiDK2c8TZWg5nxww8/jKSkJAwePBg3b95EaWlpa/SLiKhFSHWYos7MWK/XY9++fdi2bRsUCgU0Gg2ysrKM67GJiNqj6vaWGQ8fPhy//PIL3njjDWzfvh2dO3dmICaidk+qsynqzIwnTZqEjIwMXLx4ERMmTEAD05GJiNoFqU5BqDMznjFjBvbu3QuVSoWMjAzk5+cjOTkZv/76a2v2j4jIrKQ6ZtzgA7wBAwYgOTkZmZmZ6NKlCxYuXNga/SIiahEGWeOP1tTgcujWwOXQRNRYpi6H/qxLdKPLhl7eblJbTdHgPGMiov8m1W3dgTowGBORRTHIpDm1jcGYiCxKm4/L1oHBmIgsilSntjEYE5FFae1ZEo3FYExEFkWqy6EZjInIojAzJiKSAI4ZExFJAGdTEBFJAIcpiIgkgMMUREQSUM3MmIio7TEzJiKSAAZjIiIJ4GwKIiIJMGU2RXV1NZYsWYIzZ87A2toaq1atQnl5OWbNmoXu3bsDAKKiohAUFIS0tDSkpqZCLpcjNjYWAQEB9dbNYExEFsWUYYqDBw8CAFJTU5GTk4NVq1Zh+PDhmDJlCmJiYozlrly5ArVajfT0dOh0OkRHR8PPzw8KhaLOuhmMiciimLK5/IgRIzBs2DAAQHFxMdzc3JCfn48zZ84gKysLnp6eSExMRF5eHnx9faFQKKBQKODh4YGCggIolco662YwJiKLYuqiD7lcjkWLFiEzMxPr169HSUkJIiIi4OPjg40bNyIlJQV9+vSBk5OT8RoHBwdoNJp6623whaRERP9NDE046rJmzRp89dVXWLp0Kfz9/eHj4wMAGDlyJE6ePAlHR0dotVpjea1WWyM414bBmIgsimjCcbc9e/Zg8+bNAAB7e3vIZDLMmTMHeXl5AIAjR46gX79+UCqVyM3NhU6nQ3l5OQoLC+Ht7V1vvzhMQUQWxWDC5LannnoKCQkJePbZZ1FVVYXExET85S9/QVJSEmxsbODm5oakpCQ4OjpCpVIhOjoaQgjMmzcPtra29dYtE0K0+bS7yqun27oLRNRO2Lh5mXT9K57PNrrssnPbTGqrKZgZE5FF4Qo8IiIJ4BaaREQSYMqYcUtiMCYiiyLNUMxgTEQWhmPGREQSUC3R3JjBmIgsCjNjIiIJ4AM8IiIJkGYoZjAmIgvDYQoiIgngAzwiIgngmDFZlAmTn4ejowMAoOtfuuDVxS8avzt+6hckr38PAgJurp2wetlC2NrW/Tqa2hz69ig2/mM75NbWCAt+ChPGjkZlVRWWrlyL4ksl0FdWYuZzUQgY8rhZ74vaP2mGYgZjagE6nR4AsPWd1+/5TgiBFavXY+1ri+HR9a/YtfdLFF/+HT08uza6/sqqKqxZvwWp769DB3s7TJz1Eob5DcQ/jx6Di7MTVi9bgD/KbmDClDkMxnQPZsZkMX757TRu3dJhelwiqqsNeGHmc3jYpy8A4Oz5i3Dp6AT1J3vw79NnMHTwAPTw7IrKqiq8krwB54uKYRAG/H36cxjwyH/eF/ZESDS++Xw7AOD02SJ4dP0rOjrffnPCI8p+yP35BAIDhuCpYf7Ga+TW1q1419Re8AEeWQw7O1tMjh6P8SGjcK7oIma9tBQZO96HXG6N62Vl+On4KSTOi4VHtwfw/ILl+J/ePXG26CI6dXRGUsI8/FF2A8/NXoDPtm3GrJeW4pZOh7Ib5Zg8ZyHc3e7DM2Fj4OjgYGzPoYM9yjVadOhgDwDQaiswb/Fr+Pv0SW31E5CECUvKjEtLS3Hs2DGUl5fD2dkZ/fv3R+fOnVuiKZKg7t0egEfXv0Imk6G7R1e4dHTGldJr+Iv7/XDp6AyPrn/Fgz08AQD+Ax/DyV9+w8VLJfjx5xPIO/kLAKC6uhp/lN3ApjeTANzOjO8Me/zy2xlUVFQY29NW3ISz0+3gfKnkCl5ISEJk+BiMeSqgNW+b2gmpzqYw+zvwdu7ciZkzZ+LHH39EcXExcnNzMWvWLOzYscPcTZFE7d73f0je8B4A4PcrpdBqK3D/fa4AgG5/7YKKmzdx/kIxACD353w82MMTPTy7YfSIJ7D1ndex6c0kPDV8CJydHGut36t7N5y7UIyyG+WorKxE7s/5eNinL65eu44Z8xbjxdlTEB4c2Do3S+2OOV5I2hLM/tqlyMhIqNVq2NjYGM/p9XpERUUhPT291mv42qX/LpWVlVj82lu4VPI7ZJBh3uwYFF8qQcXNm4gIDUJO7k9Yu/EfEEKg/0P/g4S4WdDr9Vi+Zj0uXS6BRluByPBgTBg7us427symEEIgbMxTiBofglVvb8KXWdk1HgZuejMJdg28e4zaF1Nfu6TyDG90WfW53Sa11RRmD8YTJkzA1q1b4ej4n6zmxo0biImJwa5du2q9hsGYiBrL1GA8sQnB+ONWDMZmHzOePXs2wsPD4enpCScnJ2g0Gpw7dw4JCQnmboqIqMksZmrb8OHDMXToUBQWFkKj0cDR0REPPvgg5HJO3CCitmdRsynkcjl69+5d49zOnTsRERHREs0RETValSUF49rY29u3VlNERHWyqMy4NsHBwa3VFBFRnSxmBZ5KpUJlZWWNc0IIyGQypKammrs5IqImMWUCWXV1NZYsWYIzZ87A2toaq1atghAC8fHxkMlk6NWrF5YvXw4rKyukpaUhNTUVcrkcsbGxCAiofxGS2YPx/PnzsWTJEqSkpMCaewMQkcSYMpvi4MGDAIDU1FTk5OQYg3FcXBwGDhyIZcuWISsrC/3794darUZ6ejp0Oh2io6Ph5+cHhaLu3QnNHowffvhhhIaG4pdffsHIkSPNXT0RkUlMWQ49YsQIDBs2DABQXFwMNzc3HDp0CAMGDAAADB06FN999x2srKzg6+sLhUIBhUIBDw8PFBQUQKlU1ll3i4wZT5s2rSWqJSIymanzjOVyORYtWoTMzEysX78eBw8ehEwmAwA4ODigvLwcGo0GTk5OxmscHByg0Wjqr9ekXhERtTPmWHS8Zs0azJ8/H08//TR0Op3xvFarhbOzMxwdHaHVamuc/3Nwro3ZNwoiIpIyUzYK2rNnDzZv3gzg9nRdmUwGHx8f5OTkAACys7Px2GOPQalUIjc3FzqdDuXl5SgsLIS3t3e9/TL73hTNwb0piKixTN2b4qluoxpd9v+KvqzxuaKiAgkJCbh69Sqqqqowffp0PPjgg1i6dCkqKyvh5eWFV199FdbW1khLS8Mnn3wCIQRmzpyJwMD6dxJkMCaidsXUYDyiW+O3V/266CuT2moKjhkTkUWpFtJc9sFgTEQWxeKXQxMRSYGh7Udma8VgTEQWRZqhmMGYiCyMxWwuT0QkZQzGREQSwNkUREQSwNkUREQSIIF1brViMCYii8IxYyIiCWBmTEQkAdUSfQsegzERWRSuwCMikgDOpiAikgBmxkREEsDMmIhIApgZExFJAJdDExFJAIcpiIgkQDAzJiJqe1wOTUQkAVwOTUQkAcyMiYgkoNrAMWMiojbH2RRERBJgyphxZWUlEhMTcfHiRej1esTGxqJLly6YNWsWunfvDgCIiopCUFAQ0tLSkJqaCrlcjtjYWAQEBNRbN4MxEVkUU8aM9+7dCxcXFyQnJ+P69esICwvD888/jylTpiAmJsZY7sqVK1Cr1UhPT4dOp0N0dDT8/PygUCjqrJvBmIgsiimZ8ahRoxAYGGj8bG1tjfz8fJw5cwZZWVnw9PREYmIi8vLy4OvrC4VCAYVCAQ8PDxQUFECpVNZZN4MxEVkUUx7gOTg4AAA0Gg3mzp2LuLg46PV6REREwMfHBxs3bkRKSgr69OkDJyenGtdpNJp667Zqdq+IiNohA0Sjj9pcunQJkyZNQmhoKEJCQjBy5Ej4+PgAAEaOHImTJ0/C0dERWq3WeI1Wq60RnGvDYExEFkUI0ejjblevXkVMTAwWLFiACRMmAACmTp2KvLw8AMCRI0fQr18/KJVK5ObmQqfToby8HIWFhfD29q63XxymICKLYsoWmps2bcKNGzfw7rvv4t133wUAxMfHY+XKlbCxsYGbmxuSkpLg6OgIlUqF6OhoCCEwb9482Nra1lu3TEhgbWDl1dNt3QUiaids3LxMut6hQ/dGl9VWnDWpraZgZkxEFoWbyxMRSYCBW2gSEbU9CYzM1orBmIgsilSDsSQe4BERWTrOMyYikgAGYyIiCWAwJiKSAAZjkoSff/4ZKpXqnvMHDhzA+PHj8cwzzyAtLa0NekbUOjibgtrce++9h71798Le3r7G+crKSqxatQq7du2Cvb09oqKiEBAQgPvvv7+NekrUcpgZU5vz8PDAhg0b7jlfWFgIDw8PdOzYEQqFAo8++iiOHTvWBj0kankMxtTmAgMDIZff+480jUbT5D1hidorBmOSrObsCUvUXjEYk2Q9+OCDOHfuHP744w/o9XocO3YMvr6+bd0tohbBB3gkOZ9//jkqKirwzDPPID4+HlOnToUQAuPHj4e7u3tbd4+oRXA5NBGRBHCYgohIAhiMiYgkgMGYiEgCGIyJiCSAwZiISAIYjMkscnJyMGjQIKhUKqhUKjz99NNQq9VNrueNN97A7t27cerUKbzzzjt1lsvMzERJSUmj6szOzkZ8fHyT+0LUmjjPmMzm8ccfx9q1awEAer0eo0aNQmhoKJydnZtcV9++fdG3b986v//oo4+wYsUKzjum/xoMxtQiNBoNrKysMHnyZHTt2hU3btzAli1bsGLFCpw7dw4GgwFxcXEYOHAgvvrqK2zcuBGurq6orKyEl5cXcnJykJqairVr12Lnzp3YsWMHDAYDnnzySTz00EM4deoUFi1ahO3bt+OTTz5BRkYGZDIZgoKCMGnSJBQWFiIxMRH29vawt7dHx44d2/onIaoXgzGZzdGjR6FSqSCTyWBjY4OlS5fi/fffR0hICEaOHInt27ejU6dOWLlyJa5fv46JEydi3759SE5Oxs6dO+Hi4oIZM2bUqLO0tNS4xaZCocDq1avxt7/9DX379sWKFStw/vx57N+/H9u3b4dMJsPkyZPh7++PdevWYe7cufDz88OWLVtw+vTpNvpViBqHwZjM5s/DFHe8//776NGjBwDg119/RW5uLvLy8gAAVVVVuHr1KhwdHdGpUycAuGfviaKiIvTq1Qt2dnYAgMTExBrf//rrryguLsbkyZMBAGVlZTh//jz+/e9/Q6lUAgAeeeQRBmOSPD7AoxYnk8kAAF5eXhgzZgzUajXee+89jBo1Cs7OzigvL8e1a9cAAMePH69xrYeHB06fPg29Xg8AmDt3LkpKSiCTySCEgJeXF3r27ImPPvoIarUa4eHh8Pb2hpeXF/71r38BAPLz81vxbomah5kxtZrIyEgsWbIEEydOhEajQXR0NBQKBVatWoWpU6eiY8eO9+xr7OrqiunTp2PixImQyWQICAiAu7s7fH19sXDhQnzwwQcYNGgQoqKioNfroVQq4e7ujuXLl2PevHn43//9X7i6usLW1raN7pqocbhREBGRBHCYgohIAhiMiYgkgMGYiEgCGIyJiCSAwZiISAIYjImIJIDBmIhIAhiMiYgk4P8B0L8NVv+4rsgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEPCAYAAACX9KXAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjjUlEQVR4nO3de1xUZf4H8M8g92FQKaO2xIXwghlqeAdJLcMboZgKLKN4iawtg7zhFc287FqKoptJ67qRZHgNL9W6yk9Sk03zAqaZgGBo5IWEGYSBmef3h+usJJeBGWYOzuf9ep3XyzPnnOf5noHX1y/Pec45MiGEABERWZSNpQMgIiImYyIiSWAyJiKSACZjIiIJYDImIpIAJmMiIglgMjaSVqvFP/7xD4SGhiIkJATDhg3DypUrodFojGrz9ddfR1BQED799NMGH5+VlYVp06Y1uv/fGzRoELp16wa1Wl3t8507d6Jjx4746quv6jy+tLQU48ePr3V7SEgISkpKDI5n586dGDBgACZPnmzwMfc7duwYQkJCEBISAn9/f/Tp00e/vn///ga3N2nSJNy6dave/d59910kJiaarL2GunLlCt566y2Tt0umYWvpAJq7RYsW4fbt2/jnP/8JhUKBsrIyzJgxA/PmzcPKlSsb1WZRURGOHDmC06dPo0WLFg0+/tlnn8XatWsb1XdtWrdujQMHDmDkyJH6z3bv3o1HH3203mNv376NrKysWrd/8cUXDYpl9+7diI2NRUhISIOOu6dfv376PhMTE1FcXIyFCxc2qi0AOHr0aKOPNUd791y9ehV5eXlN0jYZj5WxEX7++Wfs2bMHy5Ytg0KhAAA4Oztj8eLFePHFFwHcrQpnzJiBESNGIDg4GH/9619RVVUF4G7STExMRFhYGAYNGoSUlBSoVCpMmTIFVVVVCA0NRUFBATp27FitUrq3rlarMW3aNISEhGDUqFGYP38+dDodMjMzMWLEiEb1X5uXX34ZaWlp+vXCwkKUlZXBy8tL/9n27dsxZswYjBw5EgMHDtS3N2fOHJSXlyMkJARarRZdunTB22+/jaCgIGRlZenPZ926dQgLC4NWq8X169cREBCA48ePV4tj2bJlyMrKwpo1a7B58+Y6z+/3/RgiJycHkyZN0v+ls337dgCo9bueM2cOAGDChAm4du1atbZUKpW+f6VSidzcXP229PR0hIWFITQ0FAMGDEBCQoL+u7q/vdr2qy0eADh06JD+5xAWFoZTp05Bq9Vi/vz5KCgoaPRfFNTEBDXaV199JUaPHl3nPrNmzRJLliwROp1OVFRUiEmTJomPPvpICCFEhw4dRHJyshBCiKysLNGlSxdRXl4urly5Irp166Zvo0OHDuLmzZsPrO/atUtMmjRJCCFEVVWVmDdvnrh8+bI4fvy4GD58eKP7/72BAweKkydPir59+4qioiIhhBDr168XycnJIjIyUnz55ZdCpVKJsWPHilu3bgkhhDh16pT+HGo6n127dj1wPlVVVeJPf/qT+Oijj0RUVJT48MMPa/xO7/VpyPnd309N1q5dKxYvXiyEEKKyslIMGzZMZGdnCyGEKCkpEUOHDhWnTp2q9buu6edzz9KlS8WsWbOETqcTN2/eFIGBgWLt2rVCp9OJyMhIkZeXJ4QQ4pdffhE+Pj76Nu61V9d+tcWTl5cnRowYof85XLx4Ufj7+wu1Wl3t94Kkh5WxEWxsbPTVSG0yMjIQGRkJmUwGe3t7hIWFISMjQ7/9hRdeAAA888wz0Gg0KCsrM7h/Pz8/XLp0CUqlEhs3bsSECRPQrl27Junfzs4OQUFB2Lt3LwDgyy+/1FffACCXy7FhwwYcPnwYCQkJ2LBhQ53n0qNHjwc+a9GiBd5//30kJSVBCIHXXnut3u+gvvOrqZ/aXL58GQUFBZg7dy5CQkIQGRmJ8vJy/PDDDwZ917/37bffYuTIkZDJZHBzc8PgwYMBADKZDBs2bMC5c+ewbt06rFixAkII3Llzp9rxde1XWzxHjx7Fr7/+iqioKISEhGDGjBmQyWQoKCgw+Hsgy2AyNoKvry9yc3OhUqmqfV5UVITo6GiUl5dDp9NBJpPpt+l0Ov2f0QDg4OAAAPp9RD2PCrn/wmDbtm1x4MABREdHQ6VSYeLEiTh06FC1/U3Z/8iRI5GWlobvv/8enp6eaNWqlX7bL7/8gpEjR6KwsBB+fn6IiYmp8zycnZ1r/LywsBAODg4oKCjA7du362zj3vnUdX619VMTrVYLhUKBL774Qr+kpqZi9OjRBn3XNbn/+7w3/l9WVoZRo0bh3Llz6Ny5M2bNmgVbW9sHvvu69qstHp1Oh759+z5wDu3btzf4eyDLYDI2gru7O4KDgzF37lx9QlapVFi0aBFatWoFR0dHBAQE4NNPP4UQAhqNBqmpqejXr1+D+nFzc9OPed6rTAEgJSUFc+bMQUBAAGbOnImAgAD88MMP1Y41Rf/3dO3aFeXl5Vi9ejVGjRpVbVt2djbc3NzwxhtvICAgAOnp6QDuJjhbW1totdp6/6MpKSnBzJkzsWLFCowYMQLz5s2rNyZTnp+npyccHR31F/euXbuGESNGIDs7u87vukWLFtX+A7inf//+2L59O3Q6HW7fvo2DBw8CAPLz86FSqRATE4NBgwYhMzMTGo1G/1fWvfbq2q+2ePr27YujR48iJycHAHD48GG8/PLLKC8vR4sWLVBZWdmo74aaHpOxkeLj4+Ht7Y2wsDCEhIRgzJgx8Pb2xnvvvQcAmD9/Pm7duoXg4GAEBwfD09MTU6dObVAf8+fPx7vvvotRo0YhJycHbdq0AXC3UtVqtRg2bBhCQ0NRWloKpVL5wLHG9n+/kJAQ5OXloX///tU+9/f3h7u7O4YMGYKhQ4fi2rVrcHNzQ35+Ptq0aQNfX18MHz4cxcXFdZ7ngAEDEBAQgDfffBNXrlzBli1b6ozHlOdnb2+Pv/3tb9i+fTuCg4MxadIkvP322/Dz86vzux4yZAiUSiUuXrxYrb233noLtra2GDp0KKZOnYoOHToAuHsBdsCAARg6dCiGDh2K9PR0eHt7Iz8/v1p7Mpms1v1qi8fb2xvvvvsu3nnnHbz88stYs2YNPvzwQ8jlcnh7e8PBwQGvvPJKvf8xkvnJBH8qREQWx8qYiEgCmIyJiCSAyZiISAKYjImIJIDJmIhIAiTxoCBb+yctHQIRNRNVmkKjjq+8kVv/Tv9l96hX/TuZiCSSMRGR2ei0lo6gRkzGRGRdRN3Pk7EUJmMisi71PNzLUpiMiciqCFbGREQSoH3woU5SwGRMRNaFF/CIiCSAwxRERBLAC3hERJbHC3hERFLAypiISAK00nz1FJMxEVkXDlMQEUkAhymIiCSAlTERkQSwMiYisjyhk+YFPL7pg4isi05n+FKLM2fOQKlUAgAuXbqE8PBwhIWFYdGiRdBq795unZqaitDQUIwdOxbp6en1hsXKmIisi5FjxklJSUhLS4OTkxMAYNWqVXjnnXfQs2dPxMXF4dChQ+jWrRuSk5OxY8cOVFRUICIiAv7+/rC3t6+1XVbGRGRddFrDlxp4eHggMTFRv56YmIiePXtCo9Hg+vXreOSRR3D27Fl0794d9vb2UCgU8PDwwIULF+oMi8mYiKyL0Bm+1CAoKAi2tv8bVGjRogUKCwsxYsQIFBcXw9PTEyqVCgqFQr+PXC6HSqWqMywmYyKyLiYYM/69J598Ev/6178QHh6OFStWwMXFBWq1Wr9drVZXS841YTImIuuirTJ8McDUqVNx+fJlAHcrYBsbG/j6+uLkyZOoqKhAaWkpcnJy0KFDhzrb4QU8IrIuJp5nHB0djbi4ONjZ2cHJyQnvvfce2rRpA6VSiYiICAghEBsbCwcHhzrbkQkhhEkjawRb+yctHQIRNRNVmkKjjr+TsdngfZ0Co4zqqyFYGRORdeEdeEREEsBnUxARSQArYyIiCTBwloS5MRkTkXXhMAURkQRwmIKISAKYjImIJIDDFEREEsALeEREEsBhCiIiCeAwBRGRBLAyJiKSACZjIiIJsPyDKmvEZExE1qWKsymIiCyPF/CIiCSAY8ZERBLAMWMiIglgZUxEJAFMxkRElie0WkuHUCMmYyKyLhKtjG0sHQARkVkJneFLLc6cOQOlUgkAOH/+PCIiIqBUKjF58mTcuHEDAJCamorQ0FCMHTsW6enp9YbFypiIrIvOuNkUSUlJSEtLg5OTEwBg6dKlWLBgAXx8fLB161YkJSVhypQpSE5Oxo4dO1BRUYGIiAj4+/vD3t6+1nZZGRORddHpDF9q4OHhgcTERP36qlWr4OPjAwDQarVwcHDA2bNn0b17d9jb20OhUMDDwwMXLlyoMywmY5KE2bPexJGMNGQe/xITo8Lg49Meh9N3IeP/diNx7TLY2PBXlUxEqzV8qUFQUBBsbf83qPDYY48BAL7//nt8+umniIqKgkqlgkKh0O8jl8uhUqnqDIu/4WRxzwf2Rd++PdD/+RAMemE02rb9A95bEof5C1YgcMBIODs7ITj4JUuHSQ8LIyvjmuzfvx/x8fHYuHEj3Nzc4OLiArVard+uVqurJeeacMyYLO6ll55HdvYF7Nj+d7gqFJgdtwTvLU2ATqeDnZ0dHndvg1+Lblg6THpYGDlm/HtffPEFPv/8cyQnJ6NVq1YAAF9fXyQkJKCiogIajQY5OTno0KFDne2YPBlXVFTgs88+w/Hjx1FaWgqFQoEePXogMjISjo6Opu6OHgKPPOKGdh5P4eWRE+Dp6YFdO/+BZ7oEwsPjSXz95ee4XVKCHy/mWDpMeliY8EFBWq0WS5cuxRNPPIG33noLANCzZ09MmzYNSqUSEREREEIgNjYWDg4OdbYlE8K0N2q/88476NSpEwIDAyGXy6FWq5GRkYEzZ85g/fr1NR5ja/+kKUOgZmbZ0jm4fv0WVid8BAA4eeIAhgwNw/XrNwEAkyaGIyCgNyZNjrFglCQVVZpCo44v+8tEg/d1nv0Po/pqCJOPGf/666+Ijo5Gp06d0LZtW3Tq1AnR0dEoLi42dVf0kDh69DsEvTQAAPDEE+6QOzshaeMH8Pb2BACUqtTQSXSiPjU/QqczeDEnkw9TODg4YPfu3ejfvz8UCgVUKhUyMjLg7Oxs6q7oIbFv/7/Rv39vfHtsH2xsbDDt7XkoLVVj08erodFUoqzsDqKnzrB0mPSwkOjt0CYfpiguLsb69evx/fffQ61WQy6X47nnnsPrr7+ORx55pMZjOExBRIYydphC/e6fDN5XvnCLUX01hMkr49atW2P+/PmmbpaIyDQkOuRltnnG06ZNM1dXRES10wnDFzMy2zzjJUuWmKsrIqLaWfs78Fq2bGmuroiIamfmitdQJk/GeXl5tW7z9PQ0dXdERA0iqqQ5m8LkyXju3Lm4cuUKvLy8cP9EDZlMhk8++cTU3RERNYy1VMabNm1CZGQkVq5cCXd3d1M3T0RkHImOGZt8NoWTkxMWL16Mq1evAgBMPI2ZiMg4Ep1N0SRT27p06YLu3bsDACZMmNAUXRARNYrQCYMXc2ry2RSsjIlIUqzlAt7v+fn5NXUXRESGs5YLeL8XExPT1F0QERnOWpMxEZGUSHXolMmYiKwLK2MiIglgMiYisjxRJc2bPpiMici6SDMXMxkTkXUx980chjLbw+WJiCTBBLdDnzlzBkqlUr9+4MABTJ8+Xb9++vRpjBkzBmFhYVi3bp1BYbEyJiLrYuQwRVJSEtLS0uDk5AQAeO+993DkyBH4+Pjo94mPj0diYiLatm2L6OhonDt3Ds8880yd7bIyJiKrYuyzKTw8PJCYmKhff+6557Bo0SL9ukqlgkajgYeHB2QyGQICAvDtt9/WGxeTMRFZFVElDF5qEhQUBFvb/w0qDBs2DDKZTL+uUqng4uKiX5fL5SgtLa03Lg5TEJF1aeLZFC4uLlCr1fp1tVoNV1fXeo9jZUxEVkXoDF8aw8XFBXZ2digoKIAQAkeOHEGPHj3qPY6VMRFZFzPMM168eDFmzJgBrVaLgIAAdO3atd5jZEICT82wtX/S0iEQUTNRpSk06vgbQ583eN9HvzxsVF8NwcqYiKyKqLJ0BDVjMiYiqyLR95EyGRORdWEyJiKSAiGrfx8LYDImIqvCypiISAKEjpUxEZHF6bRMxkREFsdhCiIiCeAwBRGRBFj+nuOaMRkTkVVhZUxEJAHN7gLekSNHaj0oICCgSYIhImpqza4y3rdvX60HMRkTUXMlmtsdeMuXL6/x819//bXJgiEiamrNdmrb2rVrkZKSgsrKSpSXl+OPf/xjnVUzEZGU6SRaGdf72qWMjAxkZGQgODgY+/fvh7u7uzniIiJqEkLIDF7Mqd7KuFWrVrC3t4darUa7du1w584dc8RFRNQkmt1sinsef/xxbN++HU5OTvjggw+gUqnMERcRUZOQ6myKet+Bp9PpcO3aNbRs2RK7du1C37594e3tbdIg+A48IjKUse/Ay/YaYfC+XXL3GtVXQ9RbGaelpen/rVAokJ2dbfJkTERkLlKd2lbvBbycnBzk5OTg0qVL2LNnD7755htzxEVE1CSEMHypzZkzZ6BUKgEA+fn5CA8PR0REBOLj46HT3Z07l5qaitDQUIwdOxbp6en1xlVvZTx9+vT7TkLgtddeq7dRIiKpMnZqW1JSEtLS0uDk5ATg7j0ZMTEx6N27NxYuXIiDBw+iW7duSE5Oxo4dO1BRUYGIiAj4+/vD3t6+1nbrrYw1Go1+uXr1Kn7++WejToSIyJJ0OpnBS008PDyQmJioXz937hx69eoFAAgMDMSxY8dw9uxZdO/eHfb29lAoFPDw8MCFCxfqjKveynjIkCGQyWQQQsDR0RFTpkxpyHkb5M5VDn0QkXkYWxkHBQVVK0qFEJDJ7rYpl8tRWloKlUoFhUKh30cul9c7E63eZJyQkABfX1/9+n/+858GB09EJBWmvoBnY/O/AQa1Wg1XV1e4uLhArVZX+/z+5FyTWpPxiRMncOnSJWzevBkTJ04EcHea25YtW7B3r/mmexARmZKpb4fu3LkzMjMz0bt3b2RkZKBPnz7w9fVFQkICKioqoNFokJOTgw4dOtTZTq3J2NXVFTdu3IBGo8H169cBADKZDDNnzjTpiRARmZOpX/Qxe/ZsLFiwAKtWrYKXlxeCgoLQokULKJVKREREQAiB2NhYODg41NlOvTd9FBUV4datW/Dx8cG///1vPP/887CzszPpyVTeyDVpe0T08LJ71Muo448+/orB+/r/st2ovhqi3tkUS5cuxenTpwEAeXl5iIuLa+qYiIiajK4BiznVm4yLiooQHh4OAHj11Vf5PGMiatYEZAYv5lRvMgbuVsTA3TtN7t1dQkTUHOmE4Ys51Tu1be7cuYiJicHNmzfh6OiIUaNGmSMuIqImoTNzxWuoeivjrl27YsmSJejXrx/u3LmDmzdvmiMuIqImIdVhilorY41Gg3379mHLli2wt7eHSqXCwYMH4ejoaM74iIhMStvcKuNBgwbhxx9/xPvvv4+UlBQ89thjTMRE1OxJdTZFrZXx+PHjsXfvXhQWFuKVV15BPdORiYiaBalOQai1Mo6OjkZaWhqUSiX27t2L7OxsrFy5EhcvXjRnfEREJiXVMeN6L+D16tULK1euxIEDB/D4449j1qxZ5oiLiKhJ6GSGL+ZU7+3Q5sDboYnIUMbeDv3F4xEG7xvyS4pRfTVEvfOMiYgeJlpLB1ALJmMisio6mTSntjEZE5FVsfi4bC2YjInIqkh1ahuTMRFZFXPPkjAUkzERWRWp3g7NZExEVoWVMRGRBHDMmIhIAjibgohIAjhMQUQkARymICKSAK0RlbFGo8GcOXNw5coVuLi4YOHChZDJZIiLi4NMJkP79u0RHx8PGxuDXi9aDZMxEVkVYyrj1NRUODs7IzU1Fbm5uViyZAns7OwQExOD3r17Y+HChTh48CAGDx7c4LYbnr6JiJoxY970cenSJQQGBgIAvLy8kJOTg3PnzqFXr14AgMDAQBw7dqxRcTEZE5FVEQ1Yfs/Hxwfp6ekQQuD06dMoKiqCEAKy/z58SC6Xo7S0tFFxMRkTkVUx5uHyo0ePhouLC8aPH4/09HQ888wz1caH1Wo1XF1dGxUXkzERWRVjhimysrLg5+eH5ORkvPjii2jbti06d+6MzMxMAEBGRgZ69OjRqLh4AY+IrIoxD5dv164d1qxZg02bNkGhUGDp0qUoKyvDggULsGrVKnh5eSEoKKhRbfO1S0TUrBj72qW/tos0eN9Z+Z8a1VdDsDImIqvCmz6IiCTA4kMBtWAyJiKropNoOmYyJiKrwrdDExFJAMeMiYgkgI/QJCKSAI4ZExFJgDRTMZMxEVkZjhkTEUmAVqK1MZMxEVkVVsZERBLAC3hERBIgzVTMZExEVobDFEREEsALeEREEiDVMWO+domaxNlzFxD15qxaty/6yxqs/nBTo9r+vyPHMW7yNPwpOhbb074EAFRWVSHu3ZUY//oMhE15G+nfHG9U2/TwM+aFpE2JlTGZ3KYt27Dnq0NwcnSocXvq7v34KecyenR/tsFtV1ZV4S9rN2Lrx2vg7OSIyKnTMcC/N745fgKtXBVYsXAmfrtdglcmvomB/fsYeyr0EGJlTFaj7R+eQMKy+TVuO519HmfPXcCYkGH6zyqrqrBg+WpMeGMmlK9Px3++P1vtmOeDI/T/zr18BR5P/QEtXRWws7PDc77P4OSZcwga2B9vvTpev59tixYmPit6WBjzQtKmxMqYTG7wwAAUXit64PPrN27hb3//FGuWL8DXB7/Rf75jz1do3dIVS+bE4rfbJZjwxkx8seUjTJ2+AOUVFbhdUoqoN2fB/dFHMG7UcLjI5fpj5c5OKFWp4ezsBABQq8sQO29ptcRMdD8h0cq4SZLxzZs3ceLECZSWlsLV1RXdunXDY4891hRdUTPydfo3+O12CV6fsRA3bxbjTkUFPD3a4qecy/j+zDmc/eFHAIBWq8Vvt0uw4YMlAO5WxpvX/RUA8OOlPJSVlenbVJfdgavibnK+VnQdb89ZgrDQ4Rj+0kAznx01F1Yzm2Lbtm34/PPP4efnB7lcjp9++gkbNmzAmDFjEB4eburuqBmJHBOCyDEhAIDd+w4gr+AKRg4fDFVZGdzbPIroCWEor6jAxn9uhavCpcY2vP7YFvk/X8XtklI4Ozni5JlsREWMxo1bxYiOnYd577yOPj26m/O0qJkxZvihsrIScXFxKCwshI2NDZYsWQJbW1vExcVBJpOhffv2iI+Ph41Nw0eATZ6Md+zYgc8++wx2dnb6zzQaDcLDw5mMrdS+f6Wj7M6dauPE9xsbMhTxf1mLqD/PhEpdhrDQEdV+mQ/vSdH/287WFrPeehXRsfMghMCo4S/Bvc2jWJ6wASWlKmzY/Bk2bP4MALDhgyVwdKj5IiJZL51ofGV8+PBhVFVVYevWrTh69CgSEhJQWVmJmJgY9O7dGwsXLsTBgwcxePDgBrdt8mRcVVWFioqKasm4vLwcMplEH69PTeLJJ9yRkpQAADUOGYwc/r9fVnt7eyxfMMPgtgcE9MGAgOozJebETMWcmKmNC5asijGDFJ6entBqtdDpdFCpVLC1tcXp06fRq1cvAEBgYCCOHj0qjWT8xhtvIDQ0FO3atYNCoYBKpUJ+fj7mzJlj6q6IiBrMmKltzs7OKCwsxNChQ1FcXIwNGzbgu+++0xebcrkcpaWljWrb5Ml40KBBCAwMRE5ODlQqFVxcXPD000/D1pYTN4jI8oyZTbF582YEBARg+vTpuHbtGiZMmIDKykr9drVaDVdX10a13STzjG1tbdGxY0f4+fmhY8eOsLW1xbZt25qiKyKiBqmCMHj5PVdXVygUCgBAy5YtUVVVhc6dOyMzMxMAkJGRgR49ejQqLrOVq05OTubqioioVsZUxlFRUZg7dy4iIiJQWVmJ2NhYdOnSBQsWLMCqVavg5eWFoKCgRrUtE8KIS4smUnkj19IhEFEzYfeol1HHh7Z72eB9d+anGdVXQ5i8MlYqldXGUABACAGZTIatW7eaujsiogaRQP1ZI5Mn4xkzZmD+/PlYv349WvD5AEQkMVJ9UJDJk3HXrl0REhKCH3/8sVFz7YiImpLV3A4NAFOmTGmKZomIjGY1lTERkZRZzZgxEZGU8YWkREQSYFXPMyYikiqOGRMRSYBWSHOggsmYiKwKhymIiCTAmIfLNyUmYyKyKtJMxUzGRGRleAGPiEgCmIyJiCSAsymIiCSAsymIiCSAz6YgIpIAjhkTEUkAK2MiIgnQSvS5bUzGRGRVeAceEZEEcDYFEZEEGFMZ79y5E7t27QIAVFRU4Pz580hJScGyZcsgk8nQvn17xMfHw8bGpsFty4QERrMrb+RaOgQiaibsHvUy6vhOj/U0eN8Lv35X67bFixejU6dOSE9Px8SJE9G7d28sXLgQ/fv3b9TLmBuevomImjGdEAYvtcnKysKlS5cwbtw4nDt3Dr169QIABAYG4tixY42Ki8MURGRVTHE79EcffYQ///nPAO5OlZPJZAAAuVyO0tLSRrXJZExEVsXYC3glJSXIzc1Fnz59AKDa+LBarYarq2uj2uUwBRFZFSF0Bi81+e6779CvXz/9eufOnZGZmQkAyMjIQI8ePRoVF5MxEVkVHYTBS03y8vLw1FNP6ddnz56NxMREjBs3DpWVlQgKCmpUXJxNQUTNirGzKTzcnjV434JbWUb11RAcMyYiq8IHBRERSYBWx2dTEBFZHG+HJiKSAAlcJqsRkzERWRWOGRMRSQArYyIiCeAFPCIiCeAwBRGRBHCYgohIAvjaJSIiCeA8YyIiCWBlTEQkAToTPFy+KTAZE5FV4QU8IiIJkGoylsTzjImIrB3f9EFEJAFMxkREEsBkTEQkAUzGJAlnzpyBUql84PNDhw5h9OjRGDduHFJTUy0QGZF5cDYFWVxSUhLS0tLg5ORU7fPKykosX74c27dvh5OTE8LDwzFw4EC0adPGQpESNR1WxmRxHh4eSExMfODznJwceHh4oGXLlrC3t4efnx9OnDhhgQiJmh6TMVlcUFAQbG0f/CNNpVJBoVDo1+VyOVQqlTlDIzIbJmOSLBcXF6jVav26Wq2ulpyJHiZMxiRZTz/9NPLz8/Hbb79Bo9HgxIkT6N69u6XDImoSvIBHkrNnzx6UlZVh3LhxiIuLw+TJkyGEwOjRo+Hu7m7p8IiaBG+HJiKSAA5TEBFJAJMxEZEEMBkTEUkAkzERkQQwGRMRSQCTMZlEZmYm+vbtC6VSCaVSibFjxyI5ObnB7bz//vvYuXMnzp8/j3Xr1tW634EDB1BUVGRQmxkZGYiLi2twLETmxHnGZDJ9+vTB6tWrAQAajQZDhgxBSEgIXF1dG9yWj48PfHx8at3+ySefYNGiRZx3TA8NJmNqEiqVCjY2NoiKisJTTz2FkpISbNy4EYsWLUJ+fj50Oh1iYmLQu3dvfP311/jwww/h5uaGyspKeHl5ITMzE1u3bsXq1auxbds2fPbZZ9DpdHjhhRfw7LPP4vz585g9ezZSUlLw+eefY+/evZDJZBg2bBjGjx+PnJwczJ07F05OTnByckLLli0t/ZUQ1YnJmEzm+PHjUCqVkMlksLOzw4IFC/Dxxx8jODgYgwcPRkpKClq3bo1ly5ahuLgYkZGR2LdvH1auXIlt27ahVatWiI6OrtbmzZs39Y/YtLe3x4oVK9CzZ0/4+Phg0aJFKCgowP79+5GSkgKZTIaoqCgEBARgzZo1mDZtGvz9/bFx40bk5uZa6FshMgyTMZnM/cMU93z88cfw9PQEAFy8eBEnT57E2bNnAQBVVVW4ceMGXFxc0Lp1awB44NkTV65cQfv27eHo6AgAmDt3brXtFy9exNWrVxEVFQUAuH37NgoKCvDTTz/B19cXAPDcc88xGZPk8QIeNTmZTAYA8PLywvDhw5GcnIykpCQMGTIErq6uKC0txa1btwAAWVlZ1Y718PBAbm4uNBoNAGDatGkoKiqCTCaDEAJeXl7w9vbGJ598guTkZISGhqJDhw7w8vLCqVOnAADZ2dlmPFuixmFlTGYTFhaG+fPnIzIyEiqVChEREbC3t8fy5csxefJktGzZ8oHnGru5ueHVV19FZGQkZDIZBg4cCHd3d3Tv3h2zZs3Cpk2b0LdvX4SHh0Oj0cDX1xfu7u6Ij49HbGws/v73v8PNzQ0ODg4WOmsiw/BBQUREEsBhCiIiCWAyJiKSACZjIiIJYDImIpIAJmMiIglgMiYikgAmYyIiCWAyJiKSgP8H7GYPNKGQtiwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "W, train_metrics = rbf_train(classification_train_X, classification_train_y, sigma, macro=True)\n",
    "test_metrics = rbf_predict(W, classification_test_X, classification_test_y, sigma, plot_matrix=True, macro=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 415,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance metrics for train dataset\n",
      "f1_train : 0.4130594277329421\n",
      "precision_train : 0.5\n",
      "accuracy_train : 0.70375\n",
      "TNR : 0.5\n",
      "recall_train : 0.351875\n",
      "**************************\n",
      "Performance metrics for test dataset\n",
      "f1 : 0.4065281899109792\n",
      "precision : 0.5\n",
      "accuracy : 0.685\n",
      "TNR : 0.5\n",
      "recall : 0.3425\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None, None, None, None, None]"
      ]
     },
     "execution_count": 415,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Performance metrics for train dataset\")\n",
    "[print(f\"{key} : {value}\") for key, value in train_metrics.items()]\n",
    "print(\"**************************\")\n",
    "print(\"Performance metrics for test dataset\")\n",
    "[print(f\"{key} : {value}\") for key, value in test_metrics.items()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2.3.1\n",
    "We take the macro averaged statistcs of the linear SVM and the optimal kernel SVM and compare them for both train and test datasets:\n",
    "\n",
    "For the training dataset:\n",
    "\n",
    "Performance Metrics | Linear SVM | Kernel SVM\n",
    ":--- | :---: | :---:|\n",
    "**precision** | 0.62 | 0.5\n",
    "**recall** | 0.62 |  0.35\n",
    "**f1** | 0.62 | 0.5\n",
    "\n",
    "<br>\n",
    "Overall, the kernel SVM performs much poorer than the linear SVM on the training set. \n",
    "The kernel SVM has a much lower precision than the linear SVM of 0.5. This might be due to the fact that the kernel SVM predicts all ones for the training data points. In fact, the kernel is unable to recognise both classes due to the low macro average recall score. \n",
    "\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "For the test dataset:\n",
    "\n",
    "Performance Metrics | Linear SVM | Kernel SVM\n",
    ":--- | :---: | :---:|\n",
    "**precision** | 0.65 | 0.5\n",
    "**recall** | 0.66 | 0.34\n",
    "**f1** | 0.65 | 0.41\n",
    "\n",
    "<br>\n",
    "\n",
    "The kernel SVM also performs much poorer than the linear SVM on the test set.\n",
    "The same observations for the training set applies here as well.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 416,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_ROC_curve(W, X_test, y_test):\n",
    "    sigmas = np.linspace(0.001, 0.1, 100)\n",
    "    results = [rbf_predict(W, X_test, y_test, sigma, plot_matrix=False) for sigma in sigmas]\n",
    "    TPR_vals = [metrics[\"recall\"] for metrics in results]\n",
    "    TNR_vals = [metrics[\"TNR\"] for metrics in results]\n",
    "    FPR = [1 - TNR for TNR in TNR_vals]\n",
    "    \n",
    "    plt.figure()\n",
    "    plt.scatter(TPR_vals, TNR_vals)\n",
    "    plt.title(\"ROC Curve\")\n",
    "    plt.xlabel(\"FPR\")\n",
    "    plt.ylabel(\"TNR\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 417,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEPCAYAAACp/QjLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAaMklEQVR4nO3dfXBU1eHG8ScvbLJkAxTUdEYNYjQ2xTKQdMQ6RGoogiiohCQEG1QiLQxUIYoEBCbFGLA2tIJCrQWk8YW3atHa1mpDjTLYIVuiBhc6xMoIMjQKjOwmYTfs/f3Ryf6IJ9EkkJtd8v38tXvOuXuerE6evTe7S5RlWZYAADhLdE8HAACEH8oBAGCgHAAABsoBAGCgHAAABsoBAGCI7ekAgB2uueYapaamKjo6WlFRUWpsbJTL5VJJSYm+973vSZIaGhq0Zs0aVVZWyuFwSJKysrI0e/ZsxcfHhx7rlVde0ebNm9XU1KRAIKCMjAwtWLBA/fr1a3Pvzq4HwkEUn3NAb3DNNddo9+7dGjhwYGhs/fr1+tvf/qYtW7aoublZd911l4YPH6558+bJ6XSqsbFR5eXl8ng82rRpk2JjY/Wb3/xGVVVVWr16tS666CIFAgGVlZXpwIEDevHFF419O7seCBdcVkKv1NzcrKNHj6p///6SpL/+9a8KBoNatGiRnE6nJMnpdOqRRx6R1+vVm2++qYaGBj3zzDMqKyvTRRddJEnq06ePHn74YU2dOlV+v7/VHh1Zv2bNGi1fvjx0zNn3CwoKNHfuXE2YMEGbNm3SyJEjQ3ucOXNGmZmZqqur06lTp1RcXKzJkydr4sSJKisrU3Nzc/c+gbjgUQ7oNe6++25NnDhRo0aN0rhx4yRJK1askCTt3btX3//+941joqKi9IMf/EBut1sff/yx4uPjdcUVV7Ra43Q6NWnSpNClqBadXd+Wfv366c9//rPuvvtuXX311aqsrJQkvfvuu7rsssuUkpKisrIyDR06VC+//LL++Mc/6sSJE9q4cWNHnxagTfzNAb3Gpk2bNHDgQO3bt08/+clPNHLkSA0aNCg0396rbb/fr5iYGEVHRysYDHZ4v86ub8vZhTVlyhS98sorGj9+vF5++WXl5uZKkv7xj3/oww8/1Pbt2yVJTU1N57QnIHHmgF5o6NChWrRokYqLi3X48GFJUnp6uqqrq41f5sFgUHv27NGIESN01VVXqbm5WZ988kmrNadPn9bMmTN17NixVuMdWR8VFaWz/+wXCARare3bt2/o9i233KL3339fdXV12rNnj8aPHx/K+OSTT2rHjh3asWOHtm3bpmXLlnXpuQFaUA7olW677TYNGzYsdFlp3LhxcjqdKisrC73ybmpq0qOPPqqEhASNHTtWDodDM2fO1COPPKLPP/9c0v/OKsrKytTY2KikpKRWe3Rk/be+9S3t27dPlmXJ6/Vq586d7WaOi4vTrbfequLiYt18882hv42MGjVKzz33nCzLkt/v1+zZs/X888+f9+cMvQuXldBrLV26VJMmTdI777yjzMxMbdiwQWvXrtXkyZMVHR2tM2fOKCsrSxs2bFCfPn0kSbNmzZLT6VRhYaGk/50FXHfddVq7dm2be3zT+pb9b775ZiUlJem6667T172BMCcnR88//7xKSkpCY4888ogee+wxTZw4UYFAQDfccIPuu+++8/EUoRfjrawAAAOXlQAABsoBAGCgHAAABsoBAGCgHAAAhgviraxut7unIwBARMrIyGhz/IIoB6n9H7AtHo9HaWlp3Zjm/Iu0zOTtfpGWOdLySpGXubN5v+6FNZeVAAAGygEAYKAcAAAGygEAYKAcAAAGygEAYKAcAAAGygEAYKAcAAAGygEAYKAcAAAGygEAYKAcAAAGygEAYKAcAAAGygEAYKAcAAAGygEAYKAcAAAGygEAYKAcAAAGygEAYKAcAAAGygEAYLCtHILBoJYtW6a8vDwVFBTo0KFDreYrKyuVnZ2tvLw8bd26tdXcF198odGjR6uurs6uuADQq9lWDm+99Zb8fr+2bNmiBx98UCtXrgzNBQIBrVixQhs2bFBFRYW2bNmi+vr60NyyZcsUHx9vV1QA6PVsKwe3263MzExJ0vDhw1VbWxuaq6urU3Jysvr37y+Hw6GMjAxVV1dLkh5//HFNnTpVl1xyiV1RAaDXs60cvF6vXC5X6H5MTIyam5tDc4mJiaG5hIQEeb1evfzyyxo4cGCoVAAA9oi1ayOXyyWfzxe6HwwGFRsb2+acz+dTYmKiKioqFBUVpd27d8vj8WjhwoVat26dLr74YuPxPR5Ph7M0NTV1an04iLTM5O1+kZY50vJKkZf5fOa1rRzS09O1c+dOTZgwQTU1NUpNTQ3NpaSk6NChQzp58qT69u2r6upqFRYWavz48aE1BQUFKikpabMYJCktLa3DWTweT6fWh4NIy0ze7hdpmSMtrxR5mTub1+12tztnWzmMHTtWu3bt0tSpU2VZlsrKyvTaa6+poaFBeXl5Ki4uVmFhoSzLUnZ2tpKSkuyKBgD4CtvKITo6WsuXL281lpKSErqdlZWlrKysdo+vqKjotmwAgNb4EBwAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMlAMAwEA5AAAMsXZtFAwGVVJSogMHDsjhcKi0tFSDBw8OzVdWVurpp59WbGyssrOzlZubq0AgoMWLF+vIkSPy+/2aPXu2xowZY1dkAOi1bCuHt956S36/X1u2bFFNTY1WrlypdevWSZICgYBWrFih7du3y+l0Kj8/XzfddJOqqqo0YMAAPfHEEzpx4oTuvPNOygEAbGBbObjdbmVmZkqShg8frtra2tBcXV2dkpOT1b9/f0lSRkaGqqurNX78eI0bNy60LiYmxq64ANCr2VYOXq9XLpcrdD8mJkbNzc2KjY2V1+tVYmJiaC4hIUFer1cJCQmhY++//37Nmzev3cf3eDwdztLU1NSp9eEg0jKTt/tFWuZIyytFXubzmde2cnC5XPL5fKH7wWBQsbGxbc75fL5QWRw9elRz5szRtGnTNHHixHYfPy0trcNZPB5Pp9aHg0jLTN7uF2mZIy2vFHmZO5vX7Xa3O2fbu5XS09NVVVUlSaqpqVFqampoLiUlRYcOHdLJkyfl9/tVXV2tESNG6PPPP9eMGTO0YMECTZkyxa6oANDr2XbmMHbsWO3atUtTp06VZVkqKyvTa6+9poaGBuXl5am4uFiFhYWyLEvZ2dlKSkpSaWmpvvzyS61du1Zr166VJD377LOKj4+3KzYA9Eq2lUN0dLSWL1/eaiwlJSV0OysrS1lZWa3mlyxZoiVLltiSDwDw//gQHADAQDkAAAyUAwDAQDkAAAyUAwDAQDkAAAyUAwDAQDkAAAyUAwDAQDkAAAyUAwDAQDkAAAyUAwDAQDkAAAyUAwDAQDkAAAyUAwDAQDkAAAyUAwDAQDkAAAyUAwDAQDkAAAyUAwDAQDkAAAyUAwDAQDkAAAydLof6+vruyAEACCPtlsO+ffs0a9YsLVy4UMePH5ckvfDCC5o8ebJt4QAAPSO2vYmlS5eqqKhIn332mX71q1+poaFB//3vf/XCCy/YmQ8A0APaPXNwOp0aNWqUcnNzVVVVpcsuu0y///3vlZycbGc+AEAPaPfMISYmJnT7kksu0fz5820JBADoee2Wg2VZCgQCsixL8fHxoduS5HA4Or1RMBhUSUmJDhw4IIfDodLSUg0ePDg0X1lZqaefflqxsbHKzs5Wbm7uNx4DAOge7ZbDkSNHNH78+FAhjBs3TpIUFRWlv//9753e6K233pLf79eWLVtUU1OjlStXat26dZKkQCCgFStWaPv27XI6ncrPz9dNN92kvXv3tnsMEO6uKH7dGPtk5a09kATovHbLobKy8rxu5Ha7lZmZKUkaPny4amtrQ3N1dXVKTk5W//79JUkZGRmqrq5WTU1Nu8cA4aytYmgZpyAQCdoth6eeeqrN8aioKM2ZM6fTG3m9XrlcrtD9mJgYNTc3KzY2Vl6vV4mJiaG5hIQEeb3erz0GANB92v0te9FFF7W639jYqGeffVaXXnppl8rB5XLJ5/OF7geDwdAv+a/O+Xw+JSYmfu0xX+XxeDqcpampqVPrw0GkZSZv+87XPjzH3S/SMp/PvO2Ww9SpU0O3q6urtXTpUt11112aNWtWlzZKT0/Xzp07NWHCBNXU1Cg1NTU0l5KSokOHDunkyZPq27evqqurVVhYqKioqHaP+aq0tLQOZ/F4PJ1aHw4iLTN5P2535nztw3Pc/SItc2fzut3udue+9vpMIBDQqlWrtHv3bpWXl+u73/1ux1N+xdixY7Vr1y5NnTpVlmWprKxMr732mhoaGpSXl6fi4mIVFhbKsixlZ2crKSmpzWMAAN2v3XL46KOPtGjRImVmZmrbtm3q06fPOW0UHR2t5cuXtxpLSUkJ3c7KylJWVtY3HgNEgk9W3sq7lRDR2i2H3NxcJSQkaM+ePSooKJD0v88+REVFafPmzbYFBCIVRYBI1m45jBkzRg8//LCdWQAAYaLdcjhx4oQuvfRSO7MAAMJEu+Xw6aefatWqVW3OFRUVdVsgAEDPa7cc4uPjNWTIEDuzAADCxNd+CO7OO++0MwsAIEy0++85XHvttXbmAACEkXbLYeHChXbmAACEkXbLAQDQe1EOAAAD5QAAMFAOAAAD5QAAMFAOAAAD5QAAMFAOAAAD5QAAMFAOAAAD5QAAMFAOAAAD5QAAMFAOAAAD5QAAMFAOAAAD5QAAMFAOAAAD5QAAMFAOAAAD5QAAMFAOAAAD5QAAMFAOAABDrF0bNTU1acGCBfriiy+UkJCgxx9/XAMHDmy1ZuvWrdq8ebNiY2M1e/Zs3XTTTTp16pQWLFggr9erQCCg4uJijRgxwq7YANAr2Xbm8NJLLyk1NVUvvvii7rjjDq1du7bVfH19vSoqKrR582atX79eq1atkt/v18aNG3X99dfr+eef14oVK7R8+XK7IgNAr2XbmYPb7dZ9990nSbrxxhuNcvjggw80YsQIORwOORwOJScna//+/brnnnvkcDgkSWfOnFFcXJxdkQGg1+qWcti2bZs2bdrUamzQoEFKTEyUJCUkJOjUqVOt5r1eb2i+ZY3X61W/fv0k/e/MYsGCBVq8eHGbe3o8ng7na2pq6tT6cBBpmcnb/SItc6TllSIv8/nM2y3lkJOTo5ycnFZjc+fOlc/nkyT5fL7QL/0WLpcrNN+ypqUsDhw4oKKiIj388MO67rrr2twzLS2tw/k8Hk+n1oeDSMtM3u4XaZkjLa8UeZk7m9ftdrc7Z9vfHNLT0/X2229LkqqqqpSRkdFqftiwYXK73Tp9+rROnTqluro6paam6uDBg3rggQdUXl6u0aNH2xUXAHo12/7mkJ+fr4ULFyo/P199+vRReXm5JGnjxo1KTk7WmDFjVFBQoGnTpsmyLM2fP19xcXEqLy+X3+/XY489Jul/Zxjr1q2zKzYA9Eq2lYPT6dTq1auN8XvvvTd0Ozc3V7m5ua3mKQIAsB8fggMAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGCgHAICBcgAAGGwrh6amJv3sZz/TtGnTNHPmTB0/ftxYs3XrVk2ePFm5ubnauXNnq7m6ujplZGTo9OnTdkUGgF7LtnJ46aWXlJqaqhdffFF33HGH1q5d22q+vr5eFRUV2rx5s9avX69Vq1bJ7/dLkrxerx5//HE5HA674gJAr2ZbObjdbmVmZkqSbrzxRu3evbvV/AcffKARI0bI4XAoMTFRycnJ2r9/vyzL0tKlS1VUVCSn02lXXADo1WK740G3bdumTZs2tRobNGiQEhMTJUkJCQk6depUq3mv1xuab1nj9Xr11FNPafTo0frOd77THVEBAG3olnLIyclRTk5Oq7G5c+fK5/NJknw+n/r169dq3uVyheZb1iQmJurVV1/Vt7/9bf3hD39QfX29ZsyYoRdeeMHY0+PxdDhfU1NTp9aHg0jLTN7uF2mZIy2vFHmZz2febimHtqSnp+vtt9/WsGHDVFVVpYyMjFbzw4YN069//WudPn1afr9fdXV1Sk1N1Ztvvhlak5WVpQ0bNrT5+GlpaR3O4vF4OrU+HERaZvJ2v0jLHGl5pcjL3Nm8bre73TnbyiE/P18LFy5Ufn6++vTpo/LycknSxo0blZycrDFjxqigoEDTpk2TZVmaP3++4uLi7IoHADiLbeXgdDq1evVqY/zee+8N3c7NzVVubm67j1FZWdkt2QAArfEhOACAgXIAABgoBwCAgXIAABgoBwCAgXIAABgoBwCAgXIAABgoBwCAgXIAABgoBwCAgXIAABgoBwCAgXIAABgoBwCAgXIAABgoBwCAgXIAABgoBwCAgXIAABgoBwCAgXIAABgoBwCAgXIAABgoBwCAIcqyLKunQ5wrt9vd0xEAICJlZGS0OX5BlAMA4PzishIAwEA5AAAMsT0d4HwLBoMqKSnRgQMH5HA4VFpaqsGDB4fmP/jgA61cuVKWZeniiy/WE088obi4ON1xxx1KTEyUJF122WVasWJFWOd95plnVFlZqUAgoPz8fOXk5NiSt6uZX3/9db3yyiuSpNOnT8vj8WjXrl3q169fWOaNjo5WcXGxjhw5oujoaD366KNKSUnp9qxdzRsVFaVFixbp008/lcvl0rJly3TFFVfYkvebMtfX16uoqCi01uPx6MEHH1ReXt7X/pzhljc/P1+S9P777+uXv/ylKioqbMl6LpmnTJmixYsX68iRI/L7/Zo9e7bGjBnTsQ2tC8wbb7xhLVy40LIsy9q7d681a9as0FwwGLQmTZpkffLJJ5ZlWdbWrVuturo6q6mpybr99tt7Im6X8r733nvWT3/6U+vMmTOW1+u1Vq9eHfaZz1ZSUmJt3rw5rPO++eab1v33329ZlmW9++671ty5c8M6b0VFhbVkyRLLsiyrrq7OmjFjhm15vynz2f71r39ZBQUFVnNzc4ePCZe8lmVZv/3tb63bbrvNysnJsS1ri65k3r59u1VaWmpZlmUdP37cGj16dIf3u+AuK7ndbmVmZkqShg8frtra2tDcf/7zHw0YMECbNm3Sj3/8Y508eVJXXnml9u/fr8bGRs2YMUPTp09XTU1NWOd99913lZqaqjlz5mjWrFn64Q9/aFvermZu8eGHH+rgwYPKy8sL67xDhgzRmTNnFAwG5fV6FRtr30l2V/IePHhQN954oyTpyiuvVF1dnW15vylzC8uy9Oijj6qkpEQxMTEdOiac8kpScnKy1qxZY1vOs3Ul8/jx4/XAAw+E5lt+jo644MrB6/XK5XKF7sfExKi5uVmSdOLECe3du1fTpk3Txo0b9d5772n37t2Kj49XYWGh1q9fr5///Od66KGHQseEY94TJ06otrZWTz75ZCivZeObzrqSucUzzzyjOXPm2Ja1q3n79u2rI0eO6JZbbtHSpUtVUFAQ1nnT0tK0c+dOWZalmpoaHTt2TGfOnAmLzC0qKyt19dVXh14sdOSYcMorSePGjbP1hcLZupI5ISFBLpdLXq9X999/v+bNm9fh/S64cnC5XPL5fKH7wWAw9B9zwIABGjx4sK666ir16dNHmZmZqq2t1ZAhQzRp0iRFRUVpyJAhGjBggOrr68M274ABAzRq1Cg5HA5deeWViouL0/Hjx23J29XMkvTll1/q448/1vXXX29b1q7mfe655zRq1Ci98cYb2rFjh4qLi3X69OmwzZudnS2Xy6Xp06dr586dGjp0aKdeJXZn5havvvqqcnNzO3VMd+lK3p7W1cxHjx7V9OnTdfvtt2vixIkd3u+CK4f09HRVVVVJkmpqapSamhqau/zyy+Xz+XTo0CFJUnV1ta6++mpt375dK1eulCQdO3ZMXq9XF198cdjmzcjI0DvvvCPLsnTs2DE1NjZqwIABtuTtamZJ2rNnj2644Qbbcp5L3n79+oXeoNC/f381Nzfb9kq8K3k//PBDZWRkqKKiQj/60Y90+eWX25K1I5lb7Nu3T+np6Z06prt0JW9P60rmzz//XDNmzNCCBQs0ZcqUTu13wX0IruUv+v/+979lWZbKysr00UcfqaGhQXl5edq9e7fKy8tlWZZGjBihJUuWyO/3a9GiRfrss88UFRWlhx56yLb/KbqSV5J+8Ytf6J///Kcsy9L8+fND1yLDOfPvfvc7xcbG6p577rEta1fz+nw+LV68WPX19QoEApo+fXqnXnXZnff48eMqKipSY2OjEhMT9dhjjykpKcmWvB3JfPz4cd17773asWPH1x5j9zvCOpO3xeHDh1VUVKStW7fakvVcMpeWluovf/lLq0tjzz77rOLj479xvwuuHAAA5+6Cu6wEADh3lAMAwEA5AAAMlAMAwEA5AAAMF9wX7wF2Onz4sCZNmqShQ4eGxkaOHKkNGzaExvx+v0aOHKmioiKtWbNGf/rTn3TJJZdIkk6ePKkJEyZo9uzZPZIfaA/lAJyjq666qtU3dB4+fFhVVVWhsWAwqPz8fO3fv1+SdM8994S+4dPv92vChAnKzc3VoEGD7A8PtIPLSkA3a2pqkt/vl9PpNOZOnDih5uZmxcXF9UAyoH2cOQDn6ODBg62+mG/evHmtxmJiYjR9+vTQd+8/99xzev3113X06FElJSWptLS01ReqAeGAcgDOUVuXlb46draWy0q1tbUqKiqy9R/lATqKy0pAD7n22ms1c+ZMFRUVKRgM9nQcoBXKAehBOTk5crlceumll3o6CtAKX7wHADBw5gAAMFAOAAAD5QAAMFAOAAAD5QAAMFAOAAAD5QAAMFAOAADD/wEE+lqTfGrUbgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_ROC_curve(W, classification_test_X, classification_test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.3.2\n",
    "\n",
    "The quality of the optimal kernel SVM is poor since all the models predict the same value of 1 regardless of the hyperparamter value. There is no ROC curve observed."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}