Helpers

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# Initialize Otter\n",
    "import otter\n",
    "grader = otter.Notebook(\"proj2a.ipynb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 2A: Spam/Ham Classification\n",
    "\n",
    "## Feature Engineering, Logistic Regression\n",
    "## Due Date: Thursday, August 4, 11:59PM PDT\n",
    "\n",
    "**Collaboration Policy**\n",
    "\n",
    "Data science is a collaborative activity. While you may talk with others about\n",
    "the homework, we ask that you **write your solutions individually**. If you do\n",
    "discuss the assignments with others please **include their names** at the top\n",
    "of your notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Collaborators**: *list collaborators here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "proj2",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "## This Assignment\n",
    "You will use what you've learned in class to create a classifier that can distinguish spam (junk or commercial or bulk) emails from ham (non-spam) emails. In addition to providing some skeleton code to fill in, we will evaluate your work based on your model's accuracy and your written responses in this notebook.\n",
    "\n",
    "After this homework, you should feel comfortable with the following:\n",
    "\n",
    "- Feature engineering with text data\n",
    "- Using `sklearn` libraries to process data and fit models\n",
    "- Validating the performance of your model and minimizing overfitting\n",
    "- Generating and analyzing precision-recall curves\n",
    "\n",
    "This first part of the project focuses on initial analysis. In the second part of this project (to be released next week),  you will build your own spam/ham classifier.\n",
    "\n",
    "## Warning\n",
    "This is a **real world** dataset – the emails you are trying to classify are actual spam and legitimate emails. As a result, some of the spam emails may be in poor taste or be considered inappropriate. We think the benefit of working with realistic data outweighs these innapropriate emails, and wanted to give a warning at the beginning of the homework so that you are made aware."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "# Run this cell to suppress all FutureWarnings\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=FutureWarning)\n",
    "\n",
    "# more readable exceptions\n",
    "%pip install --quiet iwut\n",
    "%load_ext iwut\n",
    "%wut on"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Score Breakdown\n",
    "Question | Points\n",
    "--- | ---\n",
    "1 | 2\n",
    "2 | 3\n",
    "3 | 3\n",
    "4 | 2\n",
    "5 | 2\n",
    "6a | 1\n",
    "6b | 1\n",
    "6c | 2\n",
    "6d | 2\n",
    "6e | 1\n",
    "6f | 3\n",
    "Total | 22"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "p1",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<br/><br/>\n",
    "<hr style=\"border: 5px solid #003262;\" />\n",
    "<hr style=\"border: 1px solid #fdb515;\" />\n",
    "\n",
    "# Part 1: Initial Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:41.341673Z",
     "start_time": "2019-04-03T20:17:41.330307Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "imports",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import seaborn as sns\n",
    "sns.set(style = \"whitegrid\", \n",
    "        color_codes = True,\n",
    "        font_scale = 1.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "loading",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Loading in the Data\n",
    "\n",
    "In email classification, our goal is to classify emails as spam or not spam (referred to as \"ham\") using features generated from the text in the email. \n",
    "\n",
    "The dataset is from [SpamAssassin](https://spamassassin.apache.org/old/publiccorpus/). It consists of email messages and their labels (0 for ham, 1 for spam). Your labeled training dataset contains 8348 labeled examples, and the unlabeled test set contains 1000 unlabeled examples.\n",
    "\n",
    "**Note:** The dataset is from 2004, so the contents of emails might be very different from those in 2022.\n",
    "\n",
    "Run the following cells to load the data into DataFrames.\n",
    "\n",
    "The `train` DataFrame contains labeled data that you will use to train your model. It contains four columns:\n",
    "\n",
    "1. `id`: An identifier for the training example\n",
    "1. `subject`: The subject of the email\n",
    "1. `email`: The text of the email\n",
    "1. `spam`: 1 if the email is spam, 0 if the email is ham (not spam)\n",
    "\n",
    "The `test` DataFrame contains 1000 unlabeled emails. You will predict labels for these emails and submit your predictions to the autograder for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "with zipfile.ZipFile('spam_ham_data.zip') as item:\n",
    "    item.extractall()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>subject</th>\n",
       "      <th>email</th>\n",
       "      <th>spam</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Subject: A&amp;L Daily to be auctioned in bankrupt...</td>\n",
       "      <td>url: http://boingboing.net/#85534171\\n date: n...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Subject: Wired: \"Stronger ties between ISPs an...</td>\n",
       "      <td>url: http://scriptingnews.userland.com/backiss...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Subject: It's just too small                  ...</td>\n",
       "      <td>&lt;html&gt;\\n &lt;head&gt;\\n &lt;/head&gt;\\n &lt;body&gt;\\n &lt;font siz...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Subject: liberal defnitions\\n</td>\n",
       "      <td>depends on how much over spending vs. how much...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Subject: RE: [ILUG] Newbie seeks advice - Suse...</td>\n",
       "      <td>hehe sorry but if you hit caps lock twice the ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                            subject  \\\n",
       "0   0  Subject: A&L Daily to be auctioned in bankrupt...   \n",
       "1   1  Subject: Wired: \"Stronger ties between ISPs an...   \n",
       "2   2  Subject: It's just too small                  ...   \n",
       "3   3                      Subject: liberal defnitions\\n   \n",
       "4   4  Subject: RE: [ILUG] Newbie seeks advice - Suse...   \n",
       "\n",
       "                                               email  spam  \n",
       "0  url: http://boingboing.net/#85534171\\n date: n...     0  \n",
       "1  url: http://scriptingnews.userland.com/backiss...     0  \n",
       "2  <html>\\n <head>\\n </head>\\n <body>\\n <font siz...     1  \n",
       "3  depends on how much over spending vs. how much...     0  \n",
       "4  hehe sorry but if you hit caps lock twice the ...     0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "original_training_data = pd.read_csv('train.csv')\n",
    "test = pd.read_csv('test.csv')\n",
    "\n",
    "# Convert the emails to lower case as a first step to processing the text\n",
    "original_training_data['email'] = original_training_data['email'].str.lower()\n",
    "test['email'] = test['email'].str.lower()\n",
    "\n",
    "original_training_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-34476156ed73b800",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "First, let's check if our data contains any missing values. We have filled in the cell below to print the number of NaN values in each column. If there are NaN values, we replace them with appropriate filler values (i.e., NaN values in the `subject` or `email` columns will be replaced with empty strings). Finally, we print the number of NaN values in each column after this modification to verify that there are no NaN values left.\n",
    "\n",
    "Note that while there are no NaN values in the `spam` column, we should be careful when replacing NaN labels. Doing so without consideration may introduce significant bias into our model when fitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:42.203231Z",
     "start_time": "2019-04-03T20:17:42.185104Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-b1fb39d9b651ca1b",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before imputation:\n",
      "id         0\n",
      "subject    6\n",
      "email      0\n",
      "spam       0\n",
      "dtype: int64\n",
      "------------\n",
      "After imputation:\n",
      "id         0\n",
      "subject    0\n",
      "email      0\n",
      "spam       0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('Before imputation:')\n",
    "print(original_training_data.isnull().sum())\n",
    "original_training_data = original_training_data.fillna('')\n",
    "print('------------')\n",
    "print('After imputation:')\n",
    "print(original_training_data.isnull().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "q1",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<br/>\n",
    "\n",
    "<hr style=\"border: 1px solid #fdb515;\" />\n",
    "\n",
    "### Question 1\n",
    "\n",
    "In the cell below, we have printed the text of the `email` field for the first ham and the first spam email in the original training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:42.247245Z",
     "start_time": "2019-04-03T20:17:42.228451Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "q1-answer",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "url: http://boingboing.net/#85534171\n",
      " date: not supplied\n",
      " \n",
      " arts and letters daily, a wonderful and dense blog, has folded up its tent due \n",
      " to the bankruptcy of its parent company. a&l daily will be auctioned off by the \n",
      " receivers. link[1] discuss[2] (_thanks, misha!_)\n",
      " \n",
      " [1] http://www.aldaily.com/\n",
      " [2] http://www.quicktopic.com/boing/h/zlfterjnd6jf\n",
      " \n",
      " \n",
      "\n",
      "<html>\n",
      " <head>\n",
      " </head>\n",
      " <body>\n",
      " <font size=3d\"4\"><b> a man endowed with a 7-8\" hammer is simply<br>\n",
      "  better equipped than a man with a 5-6\"hammer. <br>\n",
      " <br>would you rather have<br>more than enough to get the job done or fall =\n",
      " short. it's totally up<br>to you. our methods are guaranteed to increase y=\n",
      " our size by 1-3\"<br> <a href=3d\"http://209.163.187.47/cgi-bin/index.php?10=\n",
      " 004\">come in here and see how</a>\n",
      " </body>\n",
      " </html>\n",
      " \n",
      " \n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "first_ham = original_training_data.loc[original_training_data['spam'] == 0, 'email'].iloc[0]\n",
    "first_spam = original_training_data.loc[original_training_data['spam'] == 1, 'email'].iloc[0]\n",
    "print(first_ham)\n",
    "print(first_spam)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "Discuss one thing you notice that is different between the two emails that might relate to the identification of spam.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1\n",
    "manual: True\n",
    "points: 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ham seems to be typically addressed to no one in particular, and just lays out facts. Spam are addressed to the recipient in hopes of the recipient click on it out of curiousity. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-78513403ef52a957",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "\n",
    "\n",
    "## Training-Validation Split\n",
    "The training data we downloaded is all the data we have available for both training models and **validating** the models that we train.  We therefore need to split the training data into separate training and validation datsets.  You will need this **validation data** to assess the performance of your classifier once you are finished training. Note that we set the seed (`random_state`) to 42. This will produce a pseudo-random sequence of random numbers that is the same for every student. **Do not modify this random seed in the following questions, as our tests depend on it.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:42.317970Z",
     "start_time": "2019-04-03T20:17:42.294532Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-873194ed3e686dfb",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# This creates a 90/10 train-validation split on our labeled data\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train, val = train_test_split(original_training_data, test_size = 0.1, random_state = 42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "feat-eng",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<br/><br/>\n",
    "<hr style=\"border: 5px solid #003262;\" />\n",
    "<hr style=\"border: 1px solid #fdb515;\" />\n",
    "\n",
    "# Part 2: Basic Feature Engineering\n",
    "\n",
    "We would like to take the text of an email and predict whether the email is ham or spam. This is a *classification* problem, so we can use logistic regression to train a classifier. Recall that to train a logistic regression model we need a numeric feature matrix $X$ and a vector of corresponding binary labels $y$.  Unfortunately, our data are text, not numbers. To address this, we can create numeric features derived from the email text and use those features for logistic regression.\n",
    "\n",
    "Each row of $X$ is an email. Each column of $X$ contains one feature for all the emails. We'll guide you through creating a simple feature, and you'll create more interesting ones as you try to increase the accuracy of your model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "grade": false,
     "grade_id": "q2",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<br/>\n",
    "\n",
    "<hr style=\"border: 1px solid #fdb515;\" />\n",
    "\n",
    "### Question 2\n",
    "\n",
    "Create a function called `words_in_texts` that takes in a list of `words` and a pandas Series of email `texts`. It should output a 2-dimensional NumPy array containing one row for each email text. The row should contain either a 0 or a 1 for each word in the list: 0 if the word doesn't appear in the text and 1 if the word does. For example:\n",
    "\n",
    "```\n",
    ">>> words_in_texts(['hello', 'bye', 'world'], \n",
    "                   pd.Series(['hello', 'hello worldhello']))\n",
    "\n",
    "array([[1, 0, 0],\n",
    "       [1, 0, 1]])\n",
    "```\n",
    "\n",
    "*The provided tests make sure that your function works correctly, so that you can use it for future questions.*\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q2\n",
    "points: 3\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:42.337281Z",
     "start_time": "2019-04-03T20:17:42.320567Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "q2-answer",
     "locked": false,
     "schema_version": 2,
     "solution": true
    },
    "tags": [
     "student"
    ]
   },
   "outputs": [],
   "source": [
    "def words_in_texts(words, texts):\n",
    "    '''\n",
    "    Args:\n",
    "        words (list): words to find\n",
    "        texts (Series): strings to search in\n",
    "    \n",
    "    Returns:\n",
    "        NumPy array of 0s and 1s with shape (n, p) where n is the\n",
    "        number of texts and p is the number of words.\n",
    "    '''\n",
    "    indicator_array = []\n",
    "    for w in words:\n",
    "        indicator_array += [texts.str.contains(w).tolist()]\n",
    "    return np.array(indicator_array).T.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><strong><pre style='display: inline;'>q2</pre></strong> passed!</p>"
      ],
      "text/plain": [
       "q2 results: All test cases passed!"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check(\"q2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "eda",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<br/><br/>\n",
    "<hr style=\"border: 5px solid #003262;\" />\n",
    "<hr style=\"border: 1px solid #fdb515;\" />\n",
    "\n",
    "# Basic EDA\n",
    "\n",
    "We need to identify some features that allow us to distinguish spam emails from ham emails. One idea is to compare the distribution of a single feature in spam emails to the distribution of the same feature in ham emails. If the feature is itself a binary indicator, such as whether a certain word occurs in the text, this amounts to comparing the proportion of spam emails with the word to the proportion of ham emails with the word.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "q3a",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "The following plot (which was created using `sns.barplot`) compares the proportion of emails in each class containing a particular set of words. \n",
    "\n",
    "![training conditional proportions](images/training_conditional_proportions.png)\n",
    "\n",
    "You can use DataFrame's `.melt` method to \"unpivot\" a DataFrame. See the following code cell for an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:42.428419Z",
     "start_time": "2019-04-03T20:17:42.386697Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "> Our Original DataFrame has a `type` column and some columns corresponding to words. You can think of each row as a sentence, and the value of 1 or 0 indicates the number of occurences of the word in this sentence."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word_1</th>\n",
       "      <th>word_2</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>ham</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>ham</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>ham</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   word_1  word_2  type\n",
       "0       1       0  spam\n",
       "1       0       1   ham\n",
       "2       1       0   ham\n",
       "3       0       1   ham"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "> `melt` will turn columns into entries in a variable column. Notice how `word_1` and `word_2` become entries in `variable`; their values are stored in the value column."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type</th>\n",
       "      <th>variable</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>spam</td>\n",
       "      <td>word_1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>word_1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ham</td>\n",
       "      <td>word_1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>word_1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>spam</td>\n",
       "      <td>word_2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ham</td>\n",
       "      <td>word_2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ham</td>\n",
       "      <td>word_2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ham</td>\n",
       "      <td>word_2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   type variable  value\n",
       "0  spam   word_1      1\n",
       "1   ham   word_1      0\n",
       "2   ham   word_1      1\n",
       "3   ham   word_1      0\n",
       "4  spam   word_2      0\n",
       "5   ham   word_2      1\n",
       "6   ham   word_2      0\n",
       "7   ham   word_2      1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, Markdown\n",
    "df = pd.DataFrame({\n",
    "    'word_1': [1, 0, 1, 0],\n",
    "    'word_2': [0, 1, 0, 1],\n",
    "    'type': ['spam', 'ham', 'ham', 'ham']\n",
    "})\n",
    "display(Markdown(\"> Our Original DataFrame has a `type` column and some columns corresponding to words. You can think of each row as a sentence, and the value of 1 or 0 indicates the number of occurences of the word in this sentence.\"))\n",
    "display(df);\n",
    "display(Markdown(\"> `melt` will turn columns into entries in a variable column. Notice how `word_1` and `word_2` become entries in `variable`; their values are stored in the value column.\"))\n",
    "display(df.melt(\"type\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "<br/>\n",
    "\n",
    "<hr style=\"border: 1px solid #fdb515;\" />\n",
    "\n",
    "### Question 3\n",
    "\n",
    "Create a bar chart like the one above comparing the proportion of spam and ham emails containing certain words. Choose a set of words that are different from the ones above, but also have different proportions for the two classes. Make sure to only consider emails from `train`.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3\n",
    "manual: True\n",
    "format: image\n",
    "points: 3\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:43.145246Z",
     "start_time": "2019-04-03T20:17:42.430406Z"
    },
    "nbgrader": {
     "grade": true,
     "grade_id": "q3a-answer",
     "locked": false,
     "points": 2,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Frequency of Words in SPam/Ham Emails per Word')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuUAAAHNCAYAAAC5LQU4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABvp0lEQVR4nO3deXxM1/8/8NdkF5EFsURijZkgq6wiGiKxViWWWCK2qCqxU1qq37aUqKUVgqB2JY01qNbWqJJE7bEEUQRVkUhC9uX+/vCb+RiTxIRJJsvr+Xh4xNx7zrnve+/Mnfece+69IkEQBBARERERkdpoqDsAIiIiIqKajkk5EREREZGaMSknIiIiIlIzJuVERERERGrGpJyIiIiISM2YlBMRERERqRmTciKqEDExMfD394eDgwMkEgn27Nmj7pDeS2xsrErX4+HDh5BIJAgNDVVJe0Sq4uXlhcDAQLlpgYGB8PLyUlNEBAB79uyBRCJBbGysukMhFWFSTlWKNBEq6d+lS5fUHSIVIz09HRMnTkR2djZmz56NxYsXw9nZudiyS5YsgUQiwdmzZxXmrV27FhKJBEOHDlWYV1BQAAcHB/Tp00fl8Vd1ubm52Lp1K/r37w9XV1fY2dmhS5cuCAoKQnh4uFzZ0NBQuc+UlZUVXFxcMGrUKJw8eVJNayAvJSUFbdq0wc6dOwEAs2fPhkQiQWpqarHlpcnLkSNHKjJMpQUGBpZ6XAsLC1N3iDXK1KlTIZFI8ODBA4V58+bNg0QiwcyZMxXmJScnQyKRYNy4cRURJlVDWuoOgOhdfPjhh/jggw8Upjdt2lQN0dDbXL16FRkZGViwYAG6detWallXV1esW7cOMTEx6NChg9y8uLg4aGlp4cqVK8jOzkatWrXklpGVlQVXV9dyWYfy1qRJE1y5cgWampoqbbegoAAjRozAxYsX4enpiT59+kBfXx8PHz7E33//jTVr1mDs2LEK9SZNmgRzc3MUFhbi3r172LVrF8aNG4clS5ao/YfP8ePHIQgCunbtqtY4VElHRwfz588vdl6bNm0qOBp5lfXHTHlxdXXF4cOHERsbq/CdEhsbCy0trWJ7p6XTquoxiNSPSTlVSW3btkXfvn2VLl9YWIi8vDy5JI4qzrNnzwAARkZGby3r6OgIbW1txMXFyU0vKCjAhQsX8NFHH2HPnj24ePEi3N3dZfOl5V1cXFQS88uXL2FgYKCStpQhEomgq6ur8naPHz+OixcvYsSIEfjiiy8U5j958qTYeh988AFsbGxkr7t164b+/ftjzZo1ak/Kjx49Cnt7e5iamqo1DlXS0tIq0zGtIuno6Kg7BJUr7fMtTapjY2MxcOBA2fTk5GTcu3cP/fr1w549e3D//n00a9ZMNr+qH4NI/Th8haod6anqM2fOYNWqVfD29oatrS1+/fVXAIAgCNixYwf69esHOzs7ODg4IDAwEDExMQpt5ebmIiQkBB4eHrC1tcWAAQNw+vRp2eny1xU37hIoeexxXl4e1qxZg969e8PGxgZOTk4YN24crl+/XmL93bt3o3fv3rC2tkaXLl2wbt26YrfB9evXMWnSJLi7u8Pa2hqenp6YNm0aHjx4gLy8PLi5uWHIkCHF1l23bh0kEgn+/vvvkjfy/3fz5k1MmDABrq6usLGxQa9evbBu3ToUFhbKbZdZs2YBAIYPHy47JV8SfX19WFtb4+rVq8jOzpZNl/aEDx48GKampgo9VXFxcRCJRHLDYo4dO4bBgwfDwcEBDg4OGDx4MI4dO6awTOm+u379OoKCguDo6IiPPvpIrh1fX1/Y2NjA09MTP/74IwoKChTayc3NRWhoKLp37w47Ozs4OTmhT58+CAkJeeu2LG5M+evTTp48if79+8PGxgYeHh4ICQkpNoY33b9/HwAUzjpINWrU6K1tAIC1tTWMjY1l7f33339YtGgR+vbtC2dnZ9n+Dw8Pl9v/wP8+k2fPnsXKlSvRpUsX2NraYuDAgbIhZ3FxcRgyZAjs7e3h4eGBVatWFRvHy5cvcfbsWXh7eysVd0mKioqwevVqBAQEoGPHjrC2tkbnzp3x1Vdf4fnz53JlX98Phw8fRt++fWFrawsfHx/s3r0bAPD48WNMmjQJLi4ucHBwwIwZM/Dy5cv3irE40mPP8+fPMXv2bLi6usLBwQHjx49HcnIyAGDXrl3o2bMnbGxs0KNHj2Lf89u3b8fo0aPRqVMnWFtbw8PDAzNmzMDDhw8VypZ0bHvT7du3MWnSJFmbHTt2RGBgIP744w+l1ys1NRWfffYZXF1dYW9vjxEjRigcE6UOHz6MIUOGwMHBAXZ2dhg4cGCxvfoSiQSzZ8/G2bNnZeU//fTTEmNp0aIFGjZsqHCMkb4eN25csb3lsbGxMDQ0lDuz8csvv8DPzw+2trZwdHTE6NGjiz22vi3GX375BT169IC1tTV8fHywefNmCIJQ4jpQ1cSecqqSsrOzFcaP6ujoyPUqSJMWf39/1K5dGy1atAAAzJw5E4cOHUL37t3Rr18/5OXlISoqCqNHj0ZoaKjcKfFp06bh2LFj6NKlCzp16oQHDx5g4sSJMDc3f6/48/PzERQUhIsXL6Jv374ICAjAy5cvERERgSFDhmDbtm1yvZQAsHPnTjx79gwDBgyAoaEhDhw4gCVLlqBRo0ZyPZcnT57ExIkToa+vjwEDBqBZs2ZITk7G6dOncevWLTRt2hR+fn746aefkJiYiFatWsktZ8+ePWjevDmcnJxKXYerV68iMDAQWlpaCAgIQP369XHy5EksWbIEN2/exNKlSwEAX3zxBU6dOiUb/tCyZcu3bh9XV1dcvHgRFy5cQMeOHQG8Str09fXRrl07ODk5yX0hSnvRJRIJTExMALxKOr755hu0bNlS9uW2d+9eTJgwAd988w0GDRokt8zHjx9jxIgR6NGjB7p164asrCwAr3plJ06ciCZNmmDChAnQ1NTEnj17ik00vv76a+zevRu+vr6wt7dHUVER7t27994XYkVHR2PHjh0YPHgw+vfvj+PHj+Onn36CkZHRW8evWlhYAAAOHDiADh06QE9P751iSE1NRUZGBurXrw8ASEhIwO+//w4fHx80bdoU+fn5+PPPP7F06VI8fPgQ33zzjUIbS5YsQVFREYYPH478/Hz89NNPCAoKQkhICObMmQN/f3/06dMHv/76K1asWAFzc3OF3uPo6Gjk5+cXm5Snp6cXG7t0X74uPz8fGzZsQLdu3dC1a1fUqlULV69exe7du3HhwgXs3r1boYf45MmT2LlzJ4YMGQJjY2NERkbiiy++gLa2NpYvXw43NzdMnTpV1o6uri4WLFhQpm1cHENDQ2hpyX9djxkzBo0aNcKkSZPw4MEDbN26FcHBwfDx8UFERAQGDBgAHR0dbN26FZMnT8aRI0dk7wUA+Omnn2Bvb4/AwEAYGxvj1q1biIyMRExMDKKiomSfI2U9f/4cI0aMAAAMHjwYZmZmeP78OeLj43H58mV07txZqXbGjBkDIyMjBAcH49mzZ9i2bRsCAgKwa9cuiMViWbnly5djzZo16NSpEyZPngwNDQ0cPXoUkydPxrx58xAQECDXbnx8PH777Tf4+/vDz8/vrXG4uLggKioK9+7dQ/PmzQG8Oga1aNECzZo1Q9u2bREbGwt/f38AwNOnT3Hv3j107doVGhqv+ju///57rF+/Hra2tpg2bZrsGD9ixAiEhYXB09NTqRg3bdqEhQsXwsrKCtOmTUN2djY2bNiAevXqKbVNqQoRiKqQmJgYQSwWF/tvypQpgiAIwu7duwWxWCx069ZNyMrKkqv/+++/C2KxWNi5c6fc9Pz8fMHPz0/o0qWLUFRUJAiCIPz555+CWCwWZs2aJVf26NGjsmW+rkuXLsKwYcNKjHn37t2yaRs3bhTEYrFw6tQpubIvXrwQPD095dqR1u/YsaOQnp4um56VlSW4uroK/v7+CtPc3NyEJ0+eKMRSWFgoCIIg3L17VxCLxUJISIjc/L///lsQi8VCeHi4Qt03DRo0SGjTpo1w48YN2bSioiJh0qRJglgsFs6cOSObLt0nMTExb21XEAThzJkzglgsFpYuXSqbNnr0aGH06NGCIAjC9u3bhXbt2gmZmZmCIAjChQsXBLFYLMyfP18QBEFIS0sT7O3tBW9vb+HFixeyNl68eCF07dpVsLe3l9uWXbp0EcRisRARESEXR0FBgeDp6Sm4uLgIKSkpsukZGRlC586dFfars7OzMGbMGKXW8U1JSUmCWCwWVqxYoTDNzs5OSEpKkk0vKioSevfuLXTs2PGt7ebm5gp+fn6CWCwWHB0dhbFjxworV64U/vrrLyEvL0+h/IoVK2T7LyUlRXj69KkQFxcnDB48WBCLxcKSJUsEQRCE7Oxs2WfldTNmzBCsrKyE//77TzZNuv99fX2F3Nxc2fRjx44JYrFYaNOmjXD58mW5mDt27Cj33paaPHmy0Lt3b7lps2bNKvG48Pq/X3/9VW4bZmdnK7QfEREhiMVi4dChQ7Jpr++Hhw8fyqanpKQI1tbWgkQiEX766Se5diZMmCC0a9dOePnypcIy3jRs2LBS475y5YrCuv7f//2fXBvfffedIBaLBU9PT7n3/I0bN+T2m5T0s/M66efuzc9/cce2YcOGCV26dJG9lu7L17dbWUjXa8KECXLvq6tXrwoSiUT22RcEQYiPj1c4Pkh9+umngoODg9w2kG7Hv/76S+l4pO+D178runXrJnz55ZeCIAjC4sWLBQ8PD9m8qKgoQSwWCxs3bhQEQRASExMFiUQiDB48WO49/+TJE8HR0VHo0qWLUFBQ8NYY09PTBTs7O6Fnz55y32f//vuvYG9vX6bjKlV+HL5CVdKgQYOwceNGuX9vno4cMmSIwhjyAwcOoHbt2vD29kZqaqrsX0ZGBry8vPDo0SPcu3cPAGSnfIOCguTa8Pb2lvW6v6sDBw6gZcuWaNeunVwceXl5cHd3x/nz55GTkyNXp3///jA0NJS9rlWrFuzt7WXxAsDp06fx/PlzjBo1Cg0bNlRYrrQHp0WLFnBxccH+/fvlhkBERkZCS0vrrT1JKSkpuHjxIry8vGBlZSWbLhKJZD23R48eVX6DvMHBwQE6OjqyMZrSnnDpWE0XFxfk5+fjwoULAP43llM6FvSvv/5CVlYWAgMD5c6eGBgYYNiwYcjKysKZM2fklmlsbIx+/frJTbt27Rr+/fdf9OvXD3Xr1pVNr1OnDgYPHqwQt4GBAe7cuYNbt26987oXp2vXrnJnZ0QiEVxdXZGcnIzMzMxS60p7S6dMmQIzMzNER0djxYoVGDVqFDw9PXHgwIFi640cORIdOnSAh4cHhg0bhhs3bmDUqFGYPHkyAEBPTw8ikQjAq6FYaWlpSE1NhYeHB4qKihAfH6/Q5pAhQ+R6n6VnY+zs7GBraysXs42Njdx7W7qcU6dOlXiBZ2hoqMJxYePGjQqfYeDVNpSeNSgsLERGRgZSU1Ph5uYGALhy5YpCna5du6JJkyay13Xr1kWLFi2goaGh0DPr5OSE/Px8PHr0qNhY36Srq1ts7Bs3biz2eCPtlX59eQDQt29fufe8lZUVDAwMZMOOpPT19QG8Gsbz4sULpKamQiKRoE6dOsWu+9vUqVMHAPDnn3++17CdMWPGyN5XAGTDYM6ePSt7r0dFRUEkEsHX11fu+JmamgovLy9kZmYq3InLyspK7hqUt5G+D6THFmlP+OvHoKdPn+Kff/6RKyetJ70YecyYMXLv+YYNG8LPzw+PHj1SGJZTXIynT59GdnY2AgIC5L7P3jxDStUDh69QldSsWbO3HmCL+yJLTExEZmZmqXVTUlLQokULJCUlQUNDQ3bq8nWtWrWSHYzfRWJiInJyckoc5wu8Oh3cuHFj2evihswYGxsjLS1N9lqaxLRt2/atMfj7+2PGjBn4448/4O3tjczMTBw5cgSdO3eWDVEoiXTcqaWlpcK8Vq1aQUNDA0lJSW+NoSR6enqws7PDxYsXkZWVhYSEBGRlZcnGi1taWqJu3bqIjY2Fh4cH4uLioKGhIZsvja9169YKbUtPgb8Zn4WFhcKdT6Rlihty8+awH+DVUJ3PPvsMffr0gYWFBVxdXdGlSxd4eXnJfhC9i9eHHUgZGxsDANLS0lC7du1S69euXRuffvopPv30U7x8+RJXrlzBsWPHEBERgVmzZqFJkyZwdHSUqzNv3jy0aNECIpEIhoaGaNWqldzQl4KCAoSHh2P//v24f/++wvjWjIyMt66H9MLf4t7bRkZGcu9tALLEzMfHp9j1dHJykvvxJFXSxayHDx/Gxo0bcePGDeTn58vNK24oTHH7wcjICKampgpDXaQ/oN9ch5JoamqWKWl8Mxbp8kralm+Okz979izCwsJw+fJl5Obmys0raRhQaVxcXODr64s9e/YgKioK1tbWcHd3R69evYo9TpSkuM9Vq1atcPr0aTx+/BitW7dGYmIiBEFAz549S2xHenG5VHHH8dJYWFjAzMxMNvRMmnRLjzGOjo7Q1NREbGwsWrRogdjYWBgbG8uul1H2GPT6MMXiYpS2o+wxiKo2JuVUbRU3dlYQBNStW1c23rk4xR1Ei2tHWW9e9CatLxaL8fnnn5dY783kQplb5Unjer2nqSTdu3fH/PnzERkZCW9vbxw+fBhZWVlydxt423LKk6urK86dO4fz58/j+vXrqFWrltwXmLOzM+Li4mS96FZWVkrd3aUkxd2ZpyzbE3h1FuXEiROIjo7GuXPncObMGURGRsLJyQkbN25857tYlLbvy7ovDAwM4O7uDnd3d1hZWeHLL7/Enj17FJJyW1tbhesaXrdo0SJs3boVvXr1wrhx41C3bl1oa2vj2rVrsrHjbyrph4myt4E8evQoGjduDGtra6XKl+b333/H1KlTYWtriy+++AKNGzeGrq4uCgsLMWbMmGK3a0lxqnL/KOtdYpG6cuUKgoKC0LRpU0yfPh3m5uayMx9Tp05955hDQkIQFBSE6OhonD9/Hhs3bsSaNWvwxRdfYNiwYe/UJqC4DQVBgEgkwrp160pc3zd/CLzLnbdcXV2xd+9e3L17F3FxcWjWrJnsDKSBgQGsrKwQFxcHLy8v3Lt3Dz4+PrJjxbtsw9KOQcWpiOMwVSwm5VSjNGvWDPfu3YOdnd1bexctLCxkF+q9majfvXtXofybvdZSxfUYN2vWDM+fP4ebm9t79aC+Sdqbcv36ddkFkiXR0dGBr68vtm7div/++w+RkZFo2LAhOnXq9NblSHvp7ty5ozDv7t27KCoqKrZXsSxcXV2xcuVKxMbG4saNG7C3t4e2trZsvrOzMxYtWoTY2FiF+5NLl3379m2FsxHSmJWJT3qP4sTERIV5xU0DXr0P+vbti759+0IQBCxZsgTr16/H8ePHS+3ZUwc7OzsAr+6kUlb79++Hs7Mzli9fLjf9zWESqlJUVIQTJ06gV69eKmlv//790NXVxZYtW+SSoZL2a3Vy8OBBFBYWYt26dXKfg6ysrGLPcJSFWCyGWCzGxx9/jIyMDAwcOBBLly5FQECAUj9uExMTYW9vLzft7t270NTUhJmZGYBXPcp//vknzMzMyrW3WJqUx8bGIjY2VuGBZy4uLjh48KDC8Dngf8eO27dvK9zrvCzHIGmZu3fvKhzLivseoqqNY8qpRvH19UVRURGWLVtW7PzXT3lKx61u2LBBrsyxY8eKHbrSvHlz/PPPP3IJTl5eHrZv315sHMnJydi4ceNb4yiLjh07wsTEBBs3bsTTp08V5r/Zs+Lv74/CwkIsWbIEly5dgp+fn1I9bfXq1YODgwNOnjwpN35aEATZEyJLGmKgLHt7e+jq6uLMmTO4cOGCwheis7MzCgoKsHr1agDy9wbu2LEj9PX1sW3bNrnxrS9fvsS2bdugr6//1h8tANCuXTs0atQIe/bskbszxsuXL2VPk5SSjkt+nUgkkg0lepchAapw48aNYt8LwP+umyjL8AIpDQ0NhfdTVlYWNm3aVOa2lHHhwgWkpKS8960QpTQ1NSESieR69AVBkL2fqrOSPuNr164t9gyHMtLS0hTqGhoawtzcHNnZ2QpDZEqyfv16uffVtWvXcObMGXTo0EHWkSK9XemyZcuKPROZkpLyTuvwJmmSfejQIdy7d6/YY1BycrLsWPD6McjLywsikQgbNmyQGxr19OlT7NmzB02aNFFqmGHHjh2hp6eH7du3y90i9smTJ4iKinqv9aPKhz3lVKP06NED/fr1w7Zt23Dt2jV06dIFJiYmePLkCS5duoT79+/j+PHjAIBOnTqhS5cu2Lt3L9LS0tCpUyckJSXJbs315sV8AQEBOHToEEaOHInBgwcjPz8f+/fvL/aU5PDhw3HmzBksXrwYMTExcHNzg4GBAR4/foyYmBjZxXllVatWLSxYsACTJ09Gnz59ZLdETE1NxenTpzFy5Ei5pKZVq1ZwdHTEgQMHIBKJMGDAAKWXNWfOHAQGBiIgIABDhw6FqakpTp48idOnT+PDDz8sdby8MnR0dODg4CC7f/ybD+SQSCQwNjbGuXPn5MaTA6+SgRkzZuCbb76Ru73Y3r17cf/+fXzzzTeyC9NKo6mpic8//xxTpkzBwIED4e/vD01NTezevRvGxsZ4/PixrGxmZiY8PDzg5eWFtm3bom7dunj48CF+/vlnGBkZoUuXLu+1Pd7VmTNnsHz5cnTs2BHt27dH/fr18eLFC8TFxeHEiRMwNTXFqFGjytxu9+7dsWvXLkyZMgXu7u549uyZbLuUh6NHj8LY2Pitt+pUVvfu3fHbb79hxIgR8PX1RUFBAY4dOyaX+FSkgoIC7N+/v9h5FhYWaN++vcqW5e3tjU2bNuHjjz/GoEGDoK2tjb/++gsJCQllvhWi1L59+7B582Z4e3ujWbNm0NLSwrlz53D69Gn07NlT6VtxPn78GEFBQfDy8kJycjK2bdsGPT09ucfa29raYuLEiQgNDYWvry+6d++Ohg0b4unTp7h27RpOnTpV7IXGZWVmZgYLCwucO3cOgOIxyMnJCRoaGjh37hxMTEzkbtnYsmVLBAUFYf369Rg2bBh69uyJzMxMREREICsrC0uWLFGqA8TIyAiTJ09GSEgIBg8eDF9fX2RnZ2Pnzp1o3rx5ifdwp6qJSTnVOAsXLoSrqysiIiKwdu1a5Ofnw9TUFG3btsX06dPlyv7www/44YcfEBUVhTNnzqB169ZYsWIFDh06pJCUOzo6YtGiRVizZg2+//57NGjQAEOGDIG1tTVGjhwpV1ZbWxtr167Fjh07sH//ftkDYxo0aAAbGxul7qNbkq5du2LHjh1Ys2YNIiMjkZmZifr168PR0bHYh/b4+/vj/PnzcHV1LdOQExsbG+zcuRMrVqzAzz//jKysLFhYWGDGjBkYPXr0O8f/OldXV8TExEBXV1c21EJKJBLB0dERx48fR5s2bRSS7ICAADRo0AAbNmyQPYjGyspK9kApZfXo0QMrVqzAqlWrEBoainr16sHPzw/Ozs5y66mnp4cRI0bg7NmzsgsSGzRoAC8vL3zyySfF3g2nInTv3h15eXk4c+YMduzYgZSUFGhpaaFJkyYYOXIkgoKC3unJmJ9//jlq166NI0eO4Pjx42jcuDEGDRoEGxsbhfe7KkifF/Dm/brfVe/evZGZmYlNmzYhJCRE9sNp+vTpanlMel5eHj777LNi5/Xp00elSbmjoyNCQ0MRFhaGH3/8Ebq6unB3d8e2bdveeey3q6srbty4gT/++APJycnQ0NCAubk5Zs2aVaY2169fj4ULFyI0NBQ5OTmws7PDZ599JneXJwAIDg6GtbU1tm7dii1btiArKwv16tVD69ati31y7btydXVFUlISmjRpIhs+I2VkZASxWIybN2/CxcVFYXjOzJkz0axZM+zYsQNLly6FtrY27OzssHTp0jL9uBw9ejT09fWxceNGLF26FI0bN8bo0aNRp04dla4rqZ9IUOOVAk+ePMH69etx7do13Lx5E1lZWdiyZYvSB8QHDx7IxpQWFRXByckJs2bNeqdTsURlMXv2bOzduxcJCQnqDuW9HT58GFOnTsXSpUvx4YcfqjscIgU3b95E3759y/yDiqqO6nRMJXpXah1Tfv/+fRw6dAj6+vqye3sqKyUlBUOHDsWjR48QEhKCZcuWIT09HcOGDSvx9ldEpGjHjh0wMTFBt27d1B0KUbHy8vIQHBys1HUARERVlVqHrzg7O+Ps2bMAXp2aPHHihNJ1N2zYgIyMDOzevVt2Wtje3h5du3bF6tWr8fXXX5dLzETVQUpKCs6ePYu///4b586dw/Tp09/5dn1E5c3W1lbu4UJERNWRWnvK3+dWcMeOHYO7u7vcOE0TExN06dLlvZ4kSFQT3LlzB9OnT8ehQ4cwePDgd7rQj4iIiFSnSt4SMScnBw8ePJC70llKIpEgJSVFZbdEIirOokWLqvTYR1dXVyQkJODcuXP4+uuv5e7/TURU0ar6MZVIFark3VfS09MhCEKxT+97/dHT9erVe2tbRUVFyMzMhLa2ttJP7SMiIiIiKitBEJCfn4/atWsrjBipkkm5lCqS6MzMTIVb2xERERERlRexWKxwK98qmZQbGRlBJBIV+0hz6TRlH2AhPW0vFot5oRsRERERlZu8vDzcunWr2GGjVTIp19PTg4WFRbE93Ldu3ULdunWVGroC/K+3XUdHB7q6uiqNk4iIiIjoTcWN9qiSF3oCrx4TfObMGSQnJ8umpaWl4eTJk/Dx8VFjZEREREREZaP2nvIjR44AAK5evQoAOHfuHJ4/f45atWrB09MTABAYGIi4uDi5K7ODgoJw4MABjB07FhMmTICWlhZWr14NLS0tjBs3ruJXhIiIiIjoHak9KZ88ebLc69DQUABAkyZNSn2YUP369bF9+3aEhITgs88+gyAIcHR0xLZt22BmZlauMRMRERERqZJIEARB3UGoU25uLuLj42Ftbc0x5URERERUbkrLO6vsmHIiIiIioupC7cNXiIiIiEh5GRkZePr0KfLz89UdCv1/2traaNCgAQwNDd+5DSblRERERFVERkYG/vvvPzRp0gS1atXi08grAUEQkJ2djUePHgHAOyfmHL5CREREVEU8ffoUTZo0gb6+PhPySkIkEkFfXx9NmjTB06dP37kdJuVEREREVUR+fj5q1aql7jCoGLVq1XqvIUVMyomIiIiqEPaQV07vu1+YlBMRERERqRmTciIiIiIiNWNSTkRERESkZkzKiYiIiEhpN27cQGhoKB4+fKjuUKoVJuVEREREpLQbN25g5cqVsvtyk2owKSciIiIiUjMm5URERESklNDQUHz++ecAgOHDh0MikUAikaB9+/aQSCT45Zdfiq3Xu3dv+Pj4QBAEAEBgYCC8vLyQlJSETz/9FI6Ojmjfvj0mTJiApKQkhfqCIGDHjh3o168f7Ozs4ODggMDAQMTExJTfylYwJuVEREREpBQfHx8MGjQIADBu3DgsXrwYixcvRnh4OExNTREZGalQ59KlS7hz5w769+8vdy/vrKwsDB8+HNra2pg2bRoGDBiA6OhoDBkyBMnJyXJtzJw5E99++y2aNm2KmTNnYuLEiXj58iVGjx6N48ePl+9KVxAtdQdARERERFWDlZUV7O3tsWvXLri7u8PV1VU2r1+/fli7di3u3LkDS0tL2fTIyEhoamrCz89Prq3nz5+jT58+mDNnjmyas7MzgoODERoaim+++QYAcPToUURFReGbb76R/SAAXvXU+/v7Y8GCBfDy8qryD1ViTzkRERERvbeBAwdCJBLJ9ZZnZWXh8OHD+OCDD9CwYUOFOmPHjpV77ePjgxYtWsj1fh84cAC1a9eGt7c3UlNTZf8yMjLg5eWFR48e4d69e+W2XhWFPeVERERE9N4sLCzg7u6O/fv3Y/r06dDW1savv/6KzMxMDBgwQKG8oaEhTE1NFaa3atUKx44dQ1ZWFvT19ZGYmIjMzEy4u7uXuOyUlBS0aNFCpetT0ZiUExEREZFK+Pv7Y/LkyThx4gS6d++OyMhImJqaonPnzgplSxpuIr0Y9PXXdevWxdKlS0tcbuvWrd8r7sqASTkRERERKa20sdtdu3ZFvXr1EBkZidatW+PChQv4+OOPoaWlmHKmp6cjOTlZobf87t27qFevHvT19QEAzZo1w71792BnZ4fatWurdmUqEY4pJyIiIiKlSZPl9PR0hXna2trw8/PD6dOnsWrVKgAoduiKVHh4uNzro0eP4p9//oG3t7dsmq+vL4qKirBs2bJi23j27FmZ16EyYk85ERERESnNxsYGGhoaWLNmDdLT06Gvrw9zc3PY2dkBeDWEZcOGDTh48CBcXFzQvHnzYtsxMTHB0aNH8fTpU7i4uOD+/fvYsWMH6tevj+DgYFm5Hj16oF+/fti2bRuuXbuGLl26wMTEBE+ePMGlS5dw//79anFbRPaUExEREZHSzMzM8N133yEnJwdff/01pk2bhp9//lk2v1mzZrJbJfbv37/EdvT19bF161bk5eVh6dKl+OWXX9CpUyfs2LEDDRo0kCu7cOFChISEQENDA2vXrsW3336Lffv2QV9fH9OnTy+fFa1g7CknIiIiojLx8/NTuO/463R0dGBoaIgePXqU2o6FhQVWr16t1DJ9fX3h6+tbljCrFPaUExEREZHK3L9/H6dPn0bfvn2hp6en7nCqDPaUExEREdF7u3z5MhITE7F161Zoa2tj5MiR6g6pSmFSTkRERETv7eeff8a+fftgYWGBJUuWwNzcXN0hVSlMyomIiIjovS1atAiLFi1SquzWrVvLOZqqh2PKiYiIiIjUjEk5EREREZGaMSknIiIiIlIzJuVERERERGrGpJyIiIiISM2YlBMRERERqRmTciIiIiIiNWNSTkRERFTF5eUXqjuE94ohNDQUEokEGRkZKoyoauHDg4iIiIiqOB1tTQz9bLtaY9ixOECty6/q2FNORERERKRmTMqJiIiIqFJITk7GlClT0L59e7i7u+Pzzz/HixcvZPO3b9+OoUOHws3NDQ4ODujbty927NiBoqIiuXa8vLwwfvx4HDlyBB9++CFsbGzQp08fnDlzBgCwZcsWeHt7w8HBAcOHD0dSUlKFrmdxOHyFiIiIiCqF4OBg9OrVC/7+/khISMCyZcsAAAsXLgQAJCUloW/fvmjSpAk0NTVx5coVLF68GE+fPsWUKVPk2rp27RoePXqE8ePHQ19fH8uXL8eECRMwePBg3L59G1988QWysrKwcOFCTJo0CXv37q3o1ZXDpJyIiIiIKoVBgwZh5MiRAAB3d3c8ePAAu3fvxnfffQeRSITZs2fLyhYVFcHZ2RmFhYXYtGkTJk+eDJFIJJufnp6OyMhImJqaAgD09fURGBiIkydP4uDBg9DSepUGp6Sk4LvvvkNiYiJatWpVcSv7BiblRERERFQpeHl5yb2WSCTIzc3Fs2fPYGpqimvXrmH16tW4dOkSUlJS5IatpKSkoH79+rLX7dq1kyXkAGQJt4eHhywhf336o0ePmJQTERERERkbG8u91tHRAQDk5ubi4cOHCAgIQKtWrTBr1iw0adIE2traOHbsGNasWYOcnBy5ukZGRnKvtbW1S52el5enylUpMyblRERERFTpHT9+HNnZ2QgNDYWZmZnc9OqAd18hIiIiokpPOl5c2rMNvOpBP3DggLpCUin2lBMRERFRpdexY0doa2tj+vTpGDNmDDIzM7Fx40a58eFVGXvKiYiIiKjSa9WqFX744Qc8f/4cwcHB+P7779G5c2d8/PHH6g5NJUSCIAjqDkKdcnNzER8fD2tra+jq6qo7HCIiIqIS3bhxA23atFGYnpdfCB1tTTVEVLliULeS9o9UaXkne8qJiIiIqrjKkAxXhhiqMiblRERERERqxqSciIiIiEjNmJQTEREREakZk3IiIiIiIjVjUk5EREREpGZMyomIiIiI1IxJORERERGRmjEpJyIiIiJSMyblRERERERqxqSciIiIiEjNmJQTERERVXFFBfnqDqFSxFCVaak7ACIiIiJ6Pxpa2ji/eIxaY3D8bL1al1/VsaeciIiIiEjNmJQTERERkVqlpqZi7ty58PT0hLW1NTp06IBhw4bh8uXLAAAvLy+MHz8ehw4dQs+ePWFjY4MePXrgwIEDCu189dVX6NmzJ+zt7eHh4YExY8bg2rVrcuViY2MhkUhw6NAhzJ8/Hx06dICDgwOCg4ORlpaG1NRUTJ8+Hc7OzvDw8EBISAgKCgrKdRtw+AoRERERqdXMmTPx4MEDTJ06FU2aNEFaWhouX76MtLQ0WZn4+HgkJCRg4sSJMDIywq5duzBz5kxoaWmhV69eAIC0tDRoampi0qRJqFevHl68eIF9+/Zh8ODB2Lt3LywtLeWWu2TJEnTq1Anff/89/vnnHyxevBiff/45nj17hg8++AD9+/fH2bNnER4ejkaNGmHEiBHltg3UmpRnZmZi+fLlOHLkCDIyMmBpaYkJEyaga9eub63722+/YePGjUhMTAQAtGzZEiNGjJDtFCIiIiKqGi5cuIApU6bA19dXNs3Hx0euzNOnT3Ho0CG0atUKAODp6Yk+ffrgxx9/lOV/LVu2xLx582R1CgsL8cEHH+DDDz9EREQEvvjiC7k2ra2t8c033wAAPDw8cP36dezZsweTJ0/G+PHjAQDu7u74888/cfDgwXJNytU6fCU4OBhRUVGYPHky1q5dC0tLSwQHByM6OrrUenv37sWkSZPQoEEDLFmyBEuWLEHDhg0xdepUREZGVlD0RERERKQKtra2WLduHX766SfcuHEDhYWFCmWsrKxkCTkAaGhooGfPnrh37x6Sk5MBAIIg4JdffkG/fv3g5OSEtm3bwtraGvfu3cPdu3cV2vT09JR73bJly2Knt2rVCo8fP37v9SyN2nrKo6OjcebMGaxcuVL2S8jNzQ1JSUlYtGiRwsZ43Z49e9CkSRP88MMP0NB49buiU6dO8Pb2xv79+zFgwIAKWQciIiIien/Lly9HWFgYtmzZgpCQEBgbG6NXr16YOnUqDA0NAQD169dXqFevXj0Ar4atmJqaYsOGDfj+++8REBCAyZMnw8TEBCKRCHPnzkVubq5CfWNjY7nX2traAAAjIyOF6cXVVyW19ZQfPXoUderUkRuqIhKJ4Ofnh7t37+LOnTsl1tXS0oK+vr4sIQde/VrS19eHjo5OucZNRERERKpVt25dzJ07F3/88Qf++OMPfPrpp4iMjMT8+fNlZZ49e6ZQLyUlBcD/kuuDBw/C1dUV8+bNg6enJ2xtbWFjYyM3Nr2yUltSfvv2bVhaWsol1gAgkUgAALdu3SqxbkBAABITE7F69WqkpqYiNTUVq1evxj///FOuY32IiIiIqHw1btwYI0eOhL29PW7evCmbnpCQILuWEACKiorw66+/onnz5jA1NQXwqoNX2tstderUKTx58qRign8Pahu+kpaWhubNmytMl54uKO0Xjbe3N1avXo2ZM2fihx9+AADo6+vjxx9/xAcffFAO0RIRERFReXjx4gVGjBiBDz/8EC1btkStWrVw/vx5XLhwAaNGjZKVMzU1xSeffCK7+8rOnTtx584dLF++XFbG09MTa9asQWhoKJycnJCQkIDw8HA0bNhQHatWJmq9+4pIJHqneX/99RemT5+O3r17o3v37igsLERUVBSmTZuGFStWoHPnzmWOJT4+vsx1iIiIiCqSlpYWMjMzFabr6Wir/Ymahfl5yMnLL3O9goICtGnTBrt378a///6LwsJCmJmZ4dNPP8Xw4cORmZmJoqIitGnTBt27d0dYWBgePXoEMzMzfPvtt/D09JRtkxEjRiA9PR07d+7E+vXrIRaLsWjRIqxZswaFhYWycjk5ObK/r2/PvLw8AEB2drbc9IKCAgiCUOy2f11eXh7Onz9f5m0AACJBEIR3qvmeBg0aBJFIhJ07d8pNv3z5Mvz9/bF8+fJib28oCAI6deoEOzs7rFq1Sm5eYGAgHj9+jOPHjysdR25uLuLj42FtbQ1dXd13WxkiIiKiCnDjxg20adNG3WFUOC8vL1hZWSEsLEzdoZTqbfuntLxTbWPKLS0tkZiYiKKiIrnp0rHkYrG42HrPnj1DcnIyrK2tFeZZW1vj4cOH5X51LBERERGRKqktKffx8UFGRgZOnDghN33fvn1o0aKFwhOXpIyMjKCrq4srV64ozLt8+TKMjY3Z401EREREVYraxpR7enrC1dUVc+bMQVpaGszNzbFv3z6cP39e7tREYGAg4uLikJCQAADQ0dHB4MGDsXnzZsyZMwfdu3dHUVGRrO6UKVPUtEZEREREVB7e7MStjtSWlItEIoSFhWHZsmVYvnw5MjIyYGlpiZUrV8LLy6vUurNmzULLli0RERGB3377DRoaGmjevDkWL16Mjz76qILWgIiIiIhINdR2oWdlwQs9iYiIqKq4ceMGrKysSr1LHamHIAi4efNm1bvQk4iIiIjKRktLCwUFBeoOg4pRUFAALa13H4TCpJyIiIioitDT08PLly/VHQYV48WLF9DT03vn+kzKiYiIiKoIU1NTJCcnIysrCzV8BHKlIQgCsrKy8OzZM5iamr5zO2p9oicRERERKU9PTw8NGzbEkydP+FyWSkRXVxcNGzZ8r55yJuVEREREVYiRkRGMjIzUHQapGIevEBERERGpGZNyIiIiIiI1Y1JORERERKRmTMqJiIiIiNSMSTkRERERkZoxKSciIiIiUjMm5UREREREasaknIiIiIhIzZiUExERERGpGZNyIiIiIiI1Y1JORERERKRmTMqJiIiIiNSMSTkRERERkZppvU/lgoICHD9+HOnp6ejSpQtMTU1VFRcRERERUY2hdFK+ePFixMbGYvfu3QAAQRAwatQo/P333xAEAcbGxoiIiEDTpk3LLVgiIiIioupI6eErf/75J5ycnGSvT5w4gXPnziEoKAhLly4FAISHh6s+QiIiIiKiak7pnvInT56gWbNmstcnT56Eubk5ZsyYAQC4ffs2oqKiVB8hEREREVE1p3RPeX5+PjQ1NWWvY2Nj4e7uLnttYWGB5ORk1UZHRERERFQDKJ2UN2rUCJcuXQLwqlc8KSkJzs7OsvkpKSnQ19dXeYBERERERNWd0sNXevfujbCwMKSmpuL27dswMDCAp6enbP6NGzd4kScRERER0TtQuqf8k08+gZ+fHy5dugSRSISQkBAYGhoCAF68eIETJ06gQ4cO5RYoEREREVF1pXRPuY6ODr777rti59WuXRunT5+Gnp6eygIjIiIiIqop3uvhQVIaGhqoU6eOKpoiIiIiIqpxSkzKz507904Nvn7xJxERERERvV2JSXlgYCBEIpHSDQmCAJFIhBs3bqgkMCIiIiKimqLEpHzhwoUVGQcRERERUY1VYlLu5+dXkXEQEREREdVYSt8SkYiIiIiIykeJPeWPHz8GAJiZmcm9fhtpeSIiIiIiUk6JSbmXlxc0NDRw6dIl6OjowMvLS6kLP3mhJxERERFR2ZSYlE+YMAEikQhaWlpyr4mIiIiISLVEgiAI6g5CnXJzcxEfHw9ra2vo6uqqOxwiIiIiqqZKyzt5oScRERERkZqVOHylNJmZmXjx4gWKiooU5vFCTyIiIiKisilTUn7o0CGsXr0aiYmJJZbhhZ5ERERERGWj9PCVY8eOYfr06SgoKMCgQYMgCAJ69+6NHj16QEtLC23btsWECRPKM1YiIiIiompJ6Z7yDRs2oFWrVtizZw8yMzOxc+dO9O/fHx06dMCtW7cwZMgQWFlZlWesRERERETVktI95QkJCfD19YWuri40NF5Vk44pF4vF8Pf3R3h4ePlESURERERUjSmdlBcVFcHY2BgAoKenBwB48eKFbH7Lli1x+/Zt1UZHRERERFQDKJ2UN2zYEI8fPwbwKimvV68e4uPjZfPv3r2LWrVqqT5CIiIiIqJqTukx5e3bt8fZs2cxefJkAICXlxe2bNkCPT09CIKAHTt2oEuXLuUWKBERERFRdaV0Uj5kyBAcO3YMOTk50NPTw9SpU3HlyhWsXLkSANC6dWvMmjWr3AIlIiIiIqqulE7KbW1tYWtrK3tdt25d7N+/Hzdv3oSmpiZatWoluwCUiIiIiIiU905P9Hwdb4NIRERERPR+3ikpz87ORlpaGgRBUJhnZmb23kEREREREdUkSiflhYWFWLduHbZv345nz56VWO7GjRsqCYyIiIiIqKZQOilfuHAhtm3bhrZt26JHjx4wMjIqz7iIiIiIiGoMpZPyqKgodOvWDStWrCjPeIiIiIiIahylb5dSUFCAjh07lmcsREREREQ1ktJJuYODA+7cuVOesRARERER1UhKJ+UzZ87EwYMHcezYsfKMh4iIiIioxlF6TLlEIsG3336LSZMmoUGDBjA3N1d4WJBIJMLmzZtVHiQRERERUXWmdFIeHR2NKVOmoKioCC9fvsTjx4/LMy4iIiIiohpD6aR8yZIlaNy4MVauXAmJRFKeMRERERER1ShKjym/f/8+AgMDmZATEREREamY0km5mZkZcnNzyzMWIiIiIqIaSemkPDAwEJGRkcjMzCzPeIiIiIiIahylx5TXrl0bderUQa9evdCvXz+Ym5tDU1NToZyvr68q4yMiotfExMQgIiIC/v7+cHNzU3c4RESkIkon5bNnz5b9f/Xq1cWWEYlEZUrKMzMzsXz5chw5cgQZGRmwtLTEhAkT0LVr17fWFQQBERER2LVrFxITE6GtrY2WLVti9uzZaN++vdIxEBFVJZs2bcLt27eRlZXFpJyIqBpROinfsmWLyhceHByM69evY8aMGTA3N8fevXsRHByMNWvWwNPTs9S6c+bMwe+//44xY8bAwcEB2dnZiI+PR3Z2tsrjJCKqLLKysuT+EhFR9aB0Uu7i4qLSBUdHR+PMmTNYuXIlfHx8AABubm5ISkrCokWLSk3Kf/vtN+zduxc7duyAg4ODbHrnzp1VGiMRERERUUVQ+kJPZRQVFSld9ujRo6hTp47cUBWRSAQ/Pz/cvXsXd+7cKbHutm3b4OTkJJeQExERERFVVaUm5d27d8eJEydkr7OzszF//nzcu3dPoeyBAwfQrl07pRd8+/ZtWFpaQkNDPgTpfdBv3bpVbL38/HxcunQJEokEy5Ytg7u7O9q2bYvevXtj7969Si+fiIiIiKiyKDUpv3//vtwtEHNycrB9+3b8+++/773gtLQ0GBkZKUyXTktLSyuxXl5eHvbu3Yvjx4/jyy+/xLp16yAWizF79mxERES8d2xERERERBVJ6THlUoIgqGzhIpGozPOkQ2Ryc3MRHh6OJk2aAADc3d2RlJSEVatWwd/fv8yxxMfHl7kOEVFFkz7ELTc3F+fPn1dzNEREpCplTspVxdjYuNje8PT0dAAothddOl0kEqFly5ayhBx4lcR36tQJYWFhSElJQb169coUj7W1NXR1dctUh4iookmPU7q6unB0dFRzNEREVBa5ubkldgSr9ELPsrC0tERiYqLCxaHSseRisbjYenp6emjWrFmx86S9+KX1wBMRVZSignx1h1BmVTFmIqLqQG095T4+PoiMjMSJEyfg7e0tm75v3z60aNEClpaWpdbdtGkTHj58CHNzcwCvEvJTp07BwsICdevWLff4iYjeRkNLG+cXj1Fpm7nP/5P9VXXbAOD42XqVt0lERG/31qQ8Pj5edrpUetHn+fPn8eLFC7lyV69eLdOCPT094erqijlz5iAtLQ3m5ubYt28fzp8/j7CwMFm5wMBAxMXFISEhQTYtKCgIUVFRGDNmDIKDg1GnTh3s3r0b165dw/Lly8sUBxERERGRur01Kd+yZYvC0zxXrlypMEREEIQyDRsRiUQICwvDsmXLsHz5cmRkZMDS0hIrV66El5dXqXVNTEywfft2LF68GF9//TVycnIgFouxatUquV53IiIiIqKqoNSkfOHCheW6cAMDA8ybNw/z5s0rsczWrVuLnW5ubo4VK1aUV2hERERERBWm1KTcz8+vouIgIiIiIqqx1Hb3FSIiIiIieoVJORERERGRmjEpJyKqQnS1NOT+EhFR9cCjOhFRFdLN0gQtTfTQzdJE3aEQEZEKqe3hQUREVHZtTPXRxlRf3WEQEZGKldhTvnLlStkj7wHg8ePHyMnJqZCgiIiIiIhqklKT8tefotm1a1ccPXq0QoIiIiIiIqpJSkzKDQ0NkZGRIXstCEKFBEREREREVNOUOKa8TZs22LBhAwoKCmBkZAQA+Pvvv1FYWFhqg76+vioNkIiIiIiouhMJJXSB37x5E8HBwXj48OGrgiLRW3vLRSIRbty4ofooy1Fubi7i4+NhbW0NXV1ddYdDRNXM+cVj1B1CmTh+tl7dIRARVVul5Z0l9pRbWVnht99+Q1JSEpKTkxEYGIhx48bB3d293AMmIiIiIqpJSr0loqamJpo3b47mzZvD2dkZrq6ucHFxqajYiIiIiIjeW0xMDCIiIuDv7w83Nzd1h1Mspe9TvnXr1vKMg4iIiIioXGzatAm3b99GVlZW1U/KAaCoqAh79+7F0aNHZWPNzc3N0a1bN/j6+kJDgw8IrSmqwi9OIiIiIgDIysqS+1sZKZ2U5+Tk4OOPP8bff/8NkUgEU1NTAMCpU6cQHR2Nffv2Yd26dbxYsoaoCr84iYiIiKoKpbu2V69ejXPnzmHUqFE4e/YsoqOjER0djZiYGIwePRpxcXFYvXp1ecZKlUhV+MVJREREVFUonZQfPnwYPXv2xGeffSa7bznw6iFDM2fORM+ePXHo0KFyCZKIiIiIqDpTOil/8uRJqXdecXZ2xpMnT1QSFBERERFRTaJ0Um5oaIj79++XOP/BgwcwNDRUSVBERERERDWJ0km5u7s7duzYgT///FNh3unTp/Hzzz/Dw8NDpcEREREREdUESt99ZcqUKTh9+jTGjh2LNm3aoHXr1gCA27dv48aNGzAxMcGkSZPKLVAiIiIiqv6KCvKhoaWt7jCUpqp4lU7KmzRpgt27d2Pp0qU4efIkrl+/DgCoXbs2evfujWnTpsHMzOy9AyIiIiKimktDSxvnF49RaZu5z/+T/VV1246frVdJO2V6eJCZmRmWLl0KQRCQmpoKQRBQr149iEQilQRDRERERFQTlSkplxKJRKhXr56qY6FyUlNPAxERERFVFe+UlFPVUlNPAxERERFVFUrffYWIiIiIiMoHk3IiIiIiIjVjUk5EREREpGZMyomIiIiI1IxJORERERFVa7paGnJ/K6My3X0lKysLBw8exL1795CWlgZBEOTmi0QifPfddyoNkCqnqvDmJiIiIgKAbpYmiP4nHZ4tjNQdSomUTsqvXLmCsWPHIi0trcQyTMprjqrw5iYiIiICgDam+mhjqq/uMEqldFK+cOFCFBQU4IcffoCbmxuMjY3LMSyq7KrCm5uIiIioqlA6Kb927Ro++eQT9OjRozzjISIiIiKqcZQeEGxgYMDecSIiIiKicqB0Uu7j44PTp0+XZyxERERERDWS0kn5zJkzkZqaim+//RYPHjxQuPMKERERERG9G6XHlDs5OUEkEuHKlSvYsWNHsWVEIhGuX7+usuCIiIiIiGoCpZNyX19fiESi8oyFiIiIiKhGUjopX7RoUXnGQURERERUY/FxjEREREREaqZ0T7lUTEwMjh07hqSkJACAhYUFvL294ebmpvLgiIiIiIhqAqWT8qKiIsyaNQsHDx6EIAjQ0NCQTd++fTv69OmDkJAQjjsnIiIiIiojpZPyn376CVFRUejRowfGjRuHVq1aAQASExMRHh6OqKgoWFlZYfTo0eUWLBERERFRdaT0mPK9e/eiY8eO+OGHH2BlZQVtbW1oa2vDysoKy5Ytg7u7O3bv3l2esRIRERERVUtKJ+VJSUnw8vIqcb6Xl5dsnDkRERERESlP6aS8Vq1aePbsWYnzk5OTUatWLZUERURERERUkyidlDs5OWH79u24ffu2wrw7d+5gx44dcHZ2VmlwREREREQ1gdIXek6aNAmDBg2Cn58fvLy8YGlpCeBVQn7ixAloa2tj4sSJ5RYoEREREVF1pXRSLpFIsHXrVixYsAC///47fv/9d9k8BwcHzJkzBxKJpFyCJCIiIiKqzsr08CAbGxvs3LkTqampePjwIQRBgIWFBerWrVte8RERERERVXtlfqInANStW5eJOBERERGRiih9oScREREREZWPEnvKraysoKGhgUuXLkFHRwdWVlYQiUSlNiYSiXD9+nWVB0lEREREVJ2VmJT7+vpCJBJBU1NT7jUREREREalWiUn5okWLSn1NRERERESqofSY8nPnziE1NbXE+ampqTh37pxKgiIiIiIiqkmUTsqHDx+Ov/76q8T5MTExGD58uEqCIiIiIiKqSZROygVBKHV+YWEhNDR4MxciIiIiorIqUxZd2oWeFy9ehImJyXsHRERERERU05T68KDNmzdjy5Ytstffffcdli9frlAuIyMDL1++RP/+/VUfIRERERFRNVdqUm5oaAgzMzMAwKNHj2BsbIx69erJlRGJRGjdujXs7e0xYsSI8ouUiIiIiKiaKjUp9/Pzg5+fHwDAy8sL06dPR9euXSskMCIiIiKimkKpMeVZWVno168fdHR0VLrwzMxMzJ8/Hx4eHrC1tUW/fv1w/PjxMrUhCAKGDx8OiUSCBQsWqDQ+IiIiIqKKoFRSrq+vj/DwcDx58kSlCw8ODkZUVBQmT56MtWvXwtLSEsHBwYiOjla6jYiICNy9e1elcRERERERVSSl775iYWGB5ORklS04OjoaZ86cwfz58zFw4EB06NABISEhsLe3V/rpof/99x++//57fPnllyqLi4iIiIiooimdlA8dOhS//PILnj9/rpIFHz16FHXq1JEboy4SieDn54e7d+/izp07b23jq6++gpOTE7p3766SmIiIiIiI1KHUCz1fV7t2bRgZGaFHjx7w8/NDs2bNUKtWLYVyvr6+SrV3+/ZtWFpaKjxwSCKRAABu3boFS0vLEusfPHgQsbGxOHz4sLKrQERERERUKSmdlM+ePVv2/02bNhVbRiQSKZ2Up6WloXnz5grTjYyMZPNLkpqaigULFmDq1Klo3LixUst7m/j4eJW0Uxk5OjqqO4QyO3/+vLpDIHpvVfGzB/DzR0TqVRWPnao4biqdlL/+ECFVKe0JoaXNW7BgAczNzTFs2DCVxWJtbQ1dXV2VtUfvpyp+IImqC37+iIjKRtnjZm5ubokdwUon5S4uLsoWVYqxsXGxveHp6ekA/tdj/qa//voLhw8fxubNm/Hy5Uu5eXl5ecjIyIC+vj60tJReNSIiIiIitVL6Qs83paamIjU19Z0XbGlpicTERBQVFclNv3XrFgBALBYXW+/27dsoKipCYGAgnJ2dZf8AYOfOnXB2dsaZM2feOS4iIiIioopWpu7k//77D8uWLcPx48eRmZkJADAwMEDXrl0xdepUNGzYUOm2fHx8EBkZiRMnTsDb21s2fd++fWjRokWJF3n26NEDbdq0UZg+fPhwdO/eHQEBAbKLRYmIiIiIqgKlk/LHjx/D398fz549Q5s2bWRJc2JiIvbt24e//voLERERSl946enpCVdXV8yZMwdpaWkwNzfHvn37cP78eYSFhcnKBQYGIi4uDgkJCQCARo0aoVGjRsW22bBhQ7i6uiq7SkRERERElYLSSfmPP/6IjIwMrF27Fp6ennLzoqOjMXHiRPz4449KP/hHJBIhLCwMy5Ytw/Lly5GRkQFLS0usXLkSXl5eZVsLIiIiIqIqTOmk/K+//sLQoUMVEnLgVa/3kCFDcPDgwTIt3MDAAPPmzcO8efNKLLN161al2pL2pBMRERERVTVKX+iZnp6OZs2alTi/WbNmyMjIUElQREREREQ1idJJeaNGjRAXF1fi/L///rvEsd5ERERERFQypZPyHj164MiRI1i6dClevHghm/7y5UssW7YMv/76K3r16lUuQRIRERERVWdKjykfP348/v77b6xbtw4//fQTGjRoAAB4+vQpCgsL0b59e3z66aflFigRERERUXWldFJeq1YtbNu2Dbt378axY8fw8OFDCIIADw8PeHt7w8/Pj0/RJCIiIiJ6B2XKojU1NeHv7w9/f//yioeIiIiIqMZRekz5m3JycpCTk6PKWIiIiIiIaqQy9ZSnpKQgNDQUx44dQ0pKCgCgXr168Pb2RnBwMOrXr18uQRIRERERVWdKJ+VJSUkYOnQokpOT0aJFC9jb20MQBNy9exc7d+7E8ePHsWPHDlhYWJRnvERERERE1Y7SSXlISAjS0tKwcuVKeHt7y807evQopk2bhpCQEKxcuVLlQRIRERERVWdKjyk/e/YsAgICFBJyAPDx8cGQIUNw9uxZlQZHRERERFQTKJ2Ui0QiNGvWrMT5zZs3h0gkUklQREREREQ1idJJubOzM2JjY0ucHxcXBxcXF5UERURERERUkyidlH/xxRe4cuUKFi1aJLvzCvDqjiwLFy7ElStX8MUXX5RLkERERERE1ZnSF3qOHDkSOTk52Lx5MzZv3gxDQ0MAQEZGBgDAxMQEI0aMkKsjEolw7NgxFYZLRERERFT9KJ2Um5mZlWccRERERJVWTEwMIiIi4O/vDzc3N3WHQ9WQ0kn51q1byzMOIiIiokpr06ZNuH37NrKyspiUU7lQekw5ERERUU2VlZUl95dI1ZTuKZd68OABjh8/jqSkJACAhYUFunbtiqZNm6o8OCIiIiKimqBMSfkPP/yAdevWobCwUG76999/j08++QSTJ09WaXBERERERDWB0kl5ZGQk1qxZAwcHBwQFBUEsFgMAbt++jQ0bNmDNmjUwNzdH//79yy1YIiIiIqLqSOmkfMeOHbCzs8PWrVuhpfW/ak2bNoWnpycCAgKwfft2JuVERERERGWk9IWeiYmJ6NWrl1xCLqWlpYVevXohMTFRpcEREREREdUESifl2trapV5xnJmZCW1tbZUERURERERUkyidlNvY2GDXrl149uyZwryUlBRERETAzs5OpcEREREREdUESo8pHz9+PEaOHIlevXqhf//+sLS0BADcuXMHe/bsQWZmJpYsWVJugRIREREpo6ggHxpaVevsfVWMmVRL6aTc2dkZoaGh+Pbbb7Fx40a5eWZmZli0aBGcnJxUHiARERFRWWhoaeP84jEqbTP3+X+yv6puGwAcP1uv8japainTfcq9vLzQuXNnxMfH4+HDhwBePTyoXbt20NDgw0GJiIiIiN6FUkl5ZmYm+vbti2HDhmHkyJGwtbWFra1tecdGRERERFQjKNW9Xbt2baSlpaF27drlHQ8RERERUY2j9JgTOzs7XL16tTxjISIiIiKqkZROymfMmIEjR45g9+7dEAShPGMiIiIiIqpRlL7Qc+HChTA0NMTcuXPx/fffo2nTptDT05MrIxKJsHnzZpUHSURERERUnSmdlEvvttK4cWMAKPYhQkRERETVka6WhtxfIlVTOik/ceJEecZBREREVGl1szRB9D/p8GxhpO5QqJpSKilPTU1FUlISTExM0LRp0/KOiYiIiKhSaWOqjzam+uoOg6qxUpPyoqIi/N///R8iIyNlF3fa29tj1apVqFu3boUESERERERU3ZU6MGrbtm2IiIhA/fr14ePjA7FYjIsXL2LevHkVFR8RERERUbVXak/5vn370KpVK+zatQsGBgYAgLlz52Lv3r3IyMiAoaFhhQRJRKoVExODiIgI+Pv7w83NTd3hEBER1Xil9pT/888/8PPzkyXkADBs2DAUFhbi3r175R0bEZWTTZs24fLly9i0aZO6QyEiIiK8JSnPzs5GgwYN5KZJX2dlZZVfVERUrqSfX36OiYiIKoe33mxTJBIV+5pP9SQiIiIiUo233hIxOjpa7kFB2dnZEIlEOHLkCG7evClXViQSYeTIkSoPkoiIiIioOntrUn7w4EEcPHhQYfquXbsUpjEpJyIiIiIqu1KT8i1btlRUHERERERENVapSbmLi0tFxUFEREREVGO99UJPIlKfooJ8dYdQZlUxZqKKEhMTg2nTpiEmJkbdoRBRJfPWMeVEpD4aWto4v3iMytvNff6f7K+q23f8bL1K2yOqTjZt2oTbt28jKyuLD+4iIjnsKSciIqogfEYAEZWESTkRERERkZoxKSciIiIiUjMm5UREREREasaknIiIiIhIzZiUExERERGpGZNyohpIV0tD7i8RERGpF7+RiWqgbpYmaGmih26WJuoORe34MBciIqoM+PAgohqojak+2pjqqzuMSoEPc6GSFBXkQ0NLW91hKK2qxUtE8piUE1GNxoe5UEnK44m6fJouEZWEw1eIiIiIiNSMSTkRERERkZoxKSeiKiEvv1DdIRAREZUbjiknoipBR1sTQz/brvJ2nz17AQB48uyFytvfsThApe0REVH1xZ5yIiIiIiI1U2tPeWZmJpYvX44jR44gIyMDlpaWmDBhArp27VpqvV9++QXHjx9HQkICUlJS0KhRI3zwwQcYP3486tatW0HRExERlQ0f3EVEJVHrUSE4OBhRUVGYPHky1q5dC0tLSwQHByM6OrrUeitWrICBgQGmTZuG9evXY+TIkfj1118xYMAAZGRkVFD0RFQdiDS15f4SlSc+uIuISqK2nvLo6GicOXMGK1euhI+PDwDAzc0NSUlJWLRoETw9PUusu2/fPtSrV0/22sXFBZaWlggMDMT+/fsRGBhY7vETUfVgYNYemU+uonYjG3WHQjUAH9xFRCVRW0/50aNHUadOHbmhKiKRCH5+frh79y7u3LlTYt3XE3IpG5tXX6hPnjxRfbBEVG3pGlmgrqQXdI0s1B0KERHVYGpLym/fvg1LS0toaMiHIJFIAAC3bt0qU3sxMTEAgNatW6smQCIiIiKiCqK24StpaWlo3ry5wnQjIyPZ/LK0NX/+fDRv3hy9evV6p3ji4+PfqV5V4OjoqO4Qyuz8+fPqDqFSqIr7Diif/VdVt0VVxM/fK1XxPcd990pV3HcA959UVdx/qth3ar37ikgkeqd5r8vOzsaECROQnp6Obdu2QUdH551isba2hq6u7jvVJdWrih9I+h/uv6qN+6/q4r6r2rj/qi5l911ubm6JHcFqS8qNjY2L7Q1PT08H8L8e89Lk5OTg008/xfXr17FhwwZYWVmpOkwiIiIionKntjHllpaWSExMRFFRkdx06VhysVhcav3c3FyMHz8ely5dwtq1a9G+fftyi5WIiIiIqDypLSn38fFBRkYGTpw4ITd93759aNGiBSwtLUusm5eXh/Hjx+Pvv/9GWFgYXFxcyjtcIiIiIqJyo7bhK56ennB1dcWcOXOQlpYGc3Nz7Nu3D+fPn0dYWJisXGBgIOLi4pCQkCCbNmnSJJw+fRoTJkyAvr4+Ll26JJtXt25dNG3atCJXhYiIiIjovagtKReJRAgLC8OyZcuwfPlyZGRkwNLSEitXroSXl1epdU+ePAkAWLVqFVatWiU3z8/PD4sWLSq3uImIiIiIVE2td18xMDDAvHnzMG/evBLLbN26VWHa673mRERERERVndrGlBMRERER0StMyomIiIiI1IxJORERERGRmjEpJyIiIiJSMyblRERERERqxqSciIiIiEjNmJQTEREREakZk3IiIiIiIjVjUk5EREREpGZMyomIiIiI1IxJORERERGRmjEpJyIiIiJSMyblRERERERqxqSciIiIiEjNmJQTEREREakZk3IiIiIiIjVjUk5EREREpGZMyiuBmJgYTJs2DTExMeoOhYiIiIjUQEvdARCwadMm3L59G1lZWXBzc1N3OERERERUwdhTXglkZWXJ/SUiIiKimoVJORERERGRmjEpL6O8/EJ1h0BERERE1QzHlJeRjrYmhn62XaVtPnv2AgDw5NkLlbcNADsWB6i8TSIiIiJSHfaUExERERGpGZNyIiIiIiI1Y1JeCYg0teX+EhEREVHNwqS8EjAwaw9tg0YwMGuv7lCIiIiISsWHHpYPXuhZCegaWUDXyELdYRARERG9FR96WD7YU05ERERESuNDD8sHk3IiIiIiIjVjUk5ERERUTfGhh1UHx5QTERERVVN86GHVwZ5yovfEq9CJiIjofbGnnOg98Sp0IiKqSfh8lfLBnnKi98Sr0ImIqCbh81XKB3vKqUbJyy+EjramusMgIiKqsvh8lfLBpJxqlKp2wUtNvdiFiIiopuHwFSIiIiIiNWNSTvSeeMELERERvS8m5UTviRe8EBER0fvimHKi98QLXoiIiOh9saeciIiIKhwfvEYkjz3lREREVOH44DUieewpJyKiKok9rVUbH7xGJI895UREVCWxp5WIqhP2lBMRUZXEntaKk5dfqO4QiKo99pQTEVG5y8svhI62prrDoHfEpyETlT8m5UREVO6qWlIHMLErb3zwGpE8Dl8hIqIqiUld1cYHrxHJY085ERFVSQZm7ZH55CpqN7JRdyj0DvjgNSJ5TMqJiKhKYlJHRNUJh68QEREREakZk3IiIiIiIjVjUk5EREREpGZMyomIiIiI1IxJORERERGRmjEpJyIiIiJSMyblRERERERqxqSciIiIiEjNmJQTEREREakZk3IiIiIiIjVjUk5EREREpGZMyomIiIiI1EytSXlmZibmz58PDw8P2Nraol+/fjh+/LhSdR88eIDx48fD0dERDg4O+Pjjj3Hnzp1yjpiIiIiISPXUmpQHBwcjKioKkydPxtq1a2FpaYng4GBER0eXWi8lJQVDhw7Fo0ePEBISgmXLliE9PR3Dhg3DkydPKih6IiIiIiLV0FLXgqOjo3HmzBmsXLkSPj4+AAA3NzckJSVh0aJF8PT0LLHuhg0bkJGRgd27d6Nhw4YAAHt7e3Tt2hWrV6/G119/XSHrQERERESkCmrrKT969Cjq1KmDrl27yqaJRCL4+fnh7t27pQ5FOXbsGNzd3WUJOQCYmJigS5cuOHr0aLnGTURERESkamrrKb99+zYsLS2hoSH/u0AikQAAbt26BUtLS4V6OTk5ePDgAXr06KEwTyKR4ODBg0hJSUG9evWUikMQBABAXl6e0rEb6msrXbYyyM3NBfTqqDsMpeXm5pZr+1Vp/1W1fQeU7/6rSvsO4P57E/df+eKx83+q2r4D+Nl7XVXbf2XZd9J8U5p/vk4kFDe1AnTv3h3NmzfH2rVr5abfu3cP3bt3x1dffYWhQ4cq1Pvvv//wwQcf4LPPPkNQUJDcvIiICHz55Zc4fPgwWrVqpVQcL168wK1bt959RYiIiIiIykAsFqNOHfkfHmrrKQdeDVd5l3nKzFdW7dq1IRaLoa2trbI2iYiIiIjeJAgC8vPzUbt2bYV5akvKjY2NkZaWpjA9PT0dAGBkZFRsPSMjI4hEomLrSqcZGxsrHYeGhobCLxUiIiIiovKgp6dX7HS1XehpaWmJxMREFBUVyU2XDiURi8XF1tPT04OFhUWxQ05u3bqFunXrKj2enIiIiIioMlBbUu7j44OMjAycOHFCbvq+ffvQokWLYi/ylPL29saZM2eQnJwsm5aWloaTJ0/Kbq9IRERERFRVqO1CT0EQMGLECCQkJGDmzJkwNzfHvn37sG/fPoSFhcHLywsAEBgYiLi4OCQkJMjqPnv2DH379kWDBg0wYcIEaGlpYfXq1bh37x727t0LMzMzdawSEREREdE7UVtSDgAvX77EsmXL8NtvvyEjIwOWlpaYMGECvL29ZWWKS8qBV3dpCQkJQWxsLARBgKOjI2bNmoXWrVtX9GoQEREREb0XtSblRERERESkxjHlRERERET0CpNyIiIiIiI1Y1JeSYWGhkIikSAjI6NclyORSBAaGlquy6iKKmr7v87LywuzZ8+usOVR6ZR9D4SHh+PYsWMqX/6ePXsgkUjw8OFDlbdNVB1ERUWhV69esLGxkfusbNq0CV27dkW7du0gkUjUHGXNxNzi3aj1iZ5E9D8rV66EgYGBusOgMgoPD4e3t7fcBepEVL5SUlLw+eefo0uXLvj666+hra2NBg0a4Pr161i4cCGGDBmCDz/8EFpaTHOo6uC7laiSaNu2rbpDICKqEu7du4f8/Hz06dMHzs7Osul37twBAAwcOBDt2rVTybLy8vKgo6OjkraISsPhK5Xcw4cPMXbsWDg4OMDV1RVfffUVMjMzZfOzsrKwaNEidO7cGdbW1ujcuTNCQkKQk5Mj186LFy8wZ84cuLq6wsHBAaNHj8bdu3flykRFRUEikeDSpUsKcSxcuBAODg54+fJluaxnZVXa9n/48CEkEgn27NmjUO/NU3epqamYO3cuPD09YW1tjQ4dOmDYsGG4fPmyrMybw1ekwxfOnTuHefPmwdXVFa6urggODsZ///2nsMx9+/ahf//+sLOzg6OjI8aPH4979+7Jlbl27RrGjh2LDh06wNraGh988AHGjx+P9PR0WZkdO3agT58+sLe3R/v27dGrVy+Eh4e/8zas6pKTkzFlyhS0b98e7u7u+Pzzz/HixQsAr/bzixcvsHfvXkgkEkgkEtk+lA5/uXnzJj799FM4ODjAzc0NS5cuRVFREeLj4zFs2DDY29ujW7du2LdvnxrXsup43+1648YNjB07Fo6OjrC1tUX//v1x9OhRuTKq/ux9+eWXcHFxUTguA0C/fv0QEBCgkm1TncTExMj2o729vez2yAAwe/ZsDB06FAAwceJESCQSBAYGIjAwEDNnzgTwaru+/nkEgOjoaAwbNgwODg6wt7fHiBEjcOXKFbnlzp49G05OTrh27RoCAwNhb2+PefPmVdBaV37Sz9/169dLzU2Kk5iYiEmTJsHV1RXW1tbo06cPoqKi5Mqkpqbiq6++Qs+ePWFvbw8PDw+MGTMG165dkytXWFiIsLAwdO/eHba2tnB2doavry92794tV+7ixYsYM2YMnJycYGtrC39/f5w+fVo1G6McsKe8kgsODkafPn0wYsQIXL58GWFhYfj3338RHh6OoqIijBs3DhcuXMCECRNga2srK5OQkIANGzZAJBKhqKgIn376KS5fvoyJEyeiXbt2OH/+PD7++GO5ZfXo0QMhISHYsWMH7O3tZdNzcnKwd+9efPTRRzVueEVp278sZs6ciQcPHmDq1Klo0qQJ0tLScPnyZaSlpb21rvQU7dKlS/Hvv//i+++/x8yZM7FlyxZZmRUrVmD16tUYPHgwJk+ejJcvXyIsLAxDhgzB/v370aBBA2RmZmL06NFo27Ytvv32WxgaGuLp06f466+/kJubCwA4ePAg5s+fj08++QQuLi4oLCzEvXv38PTp0zKtb3USHByMXr16wd/fHwkJCVi2bBmAVz9Ud+3ahVGjRsHZ2Rnjx48HANStW1eu/pQpU9CvXz8EBgbixIkTCA8PR15eHv744w8EBQVh3Lhx2LZtG2bPng2JRII2bdpU+DpWRe+yXW/fvo2hQ4eicePG+Prrr6Gvr4+IiAgEBwcjJCQEvr6+cstQ1WcvMDAQEREROHjwIAYMGCCre+nSJVy7dg3Lly+vqM1WJZw9exZBQUGws7NDSEgIAGDjxo0YNWoUNmzYgPHjx8PGxgbffPMNZsyYAWdnZ9l308GDB7F69WqEhISgefPmss9jZGQk5s6di969e2P06NEoKirCxo0bZfvm9bHnubm5mDhxIgIDAzF+/Hhoa2tX/Eao5Mr63Xjz5k0MGTIELVu2xJdffgljY2McOXIEM2bMQE5ODgYOHAjg1dPZNTU1MWnSJNSrVw8vXrzAvn37MHjwYOzdu1f2tPf169cjPDwckyZNQrt27ZCTk4M7d+7IdTCdPn0a48aNg4uLC7777jvo6uoiIiICY8eORXh4ODw8PMp/Q5WVQJXSihUrBLFYLCxZskRu+tq1awWxWCxcuHBB+OOPPwSxWCxs3bpVrsymTZsEsVgsnDp1ShAEQVZu27ZtcuVWrlwpiMViYcWKFbJpoaGhgrW1tZCSkiKbFhERIYjFYiEhIUHVq1lpKbP9k5KSBLFYLOzevVuh/pvb1d7eXti0aVOpy+zSpYswa9Ys2evdu3cLYrFY+Pbbb+XKrVu3ThCLxcJ///0nCIIgPHr0SGjbtq0QEhIiV+7JkyeCnZ2dsGjRIkEQBOHKlSuCWCwWbty4UWIMX3/9tdC3b99S46wppO+BjRs3yk3/v//7P8HGxkYoKioSBEEQHB0d5fbbm/W3bNkim1ZUVCR4eXkJYrFYOHfunGz68+fPhTZt2giLFy+WTZPu/6SkJBWvWdX2Ptt18uTJgp2dnZCcnCxX18/PT/Dw8BAKCwsFQVD9Z08QBCEwMFDw8/OTKzdjxgyhY8eOQl5e3vtskmpnwIABQqdOnYScnBzZtOzsbKFjx46Cv7+/IAiCEBMTI4jFYuHo0aNydaX77vr167JpmZmZgpOTkzBp0iS5sllZWUKnTp2EiRMnyqbNmjVLEIvFwv79+8tj1ao8Zb4bBUHxO3DUqFFCly5dhMzMTLl648ePF9zd3WWfvTcVFBQIeXl5Qrdu3YQFCxbIpo8dO1YYP358qbF269ZNGDhwoFzb0s97v379lFvhCsbhK5Vc7969i3197tw5xMbGAgA++ugjuTLS3h7pfOkpvz59+siVe7MeAAwePBiCIMidAvr555/h7OwMsVj8HmtSNZW2/cvC1tYW69atw08//YQbN26gsLBQ6bpdu3aVey3t0Xn8+DEA4K+//kJBQQE++ugjFBQUyP7Vq1cPbdu2lcXavHlzGBkZYc6cOdi7dy+SkpKKjfPmzZuYN28eTp8+LRumUZN5eXnJvZZIJMjNzcWzZ8+Uqt+5c2fZ/0UiEVq2bInatWvDyclJNt3Y2Bh169aV7VN6u3fZrrGxsXB3d0f9+vXl6n700Ud4+vQp/vnnH7llqOqzB7x6OvW1a9dkwyVSU1Nx5MgR+Pv7syf2NVlZWbh69Sq6d+8OXV1d2XQ9PT306NEDV65cQXZ2dpnavHTpEjIyMhT2k7a2NlxcXIo9nvv4+Lz3ulRnZfluzM3NRWxsLLp16wYdHR25feDp6Ylnz57JhtMKgoBffvkF/fr1g5OTE9q2bQtra2vcu3dPbsitra0toqOjsXjxYpw7d07hPXH//n3cu3cPH374IYqKimTLKywsRKdOnXDt2rW3DrdRBw5fqeRe//J4/fXz58+RkZEBHR0dGBoaypUxMjKCjo6ObGhEWloadHV1Fco1aNCg2OX17NkTO3fuRFBQEK5cuVKjT6+Wtv3LYvny5QgLC8OWLVsQEhICY2Nj9OrVC1OnTlXYL28yNjaWey294Eg65ESaHPbt27fY+ubm5gCAOnXqYOvWrQgLC8N3332HjIwMmJubIyAgAKNGjYJIJIKvry/y8/MRGRmJyMhIiEQiODo6Yvr06bCzsyvTOlcXb9v+b2NkZCT3WltbW6FNabvKtknvtl3T09MVPtMAYGpqCgAKw8lU9dkDXv24a9KkCXbs2AFbW1tERkaiqKgIgwYNKmENa6aMjAwIgiDbJ69r0KABioqKynyrWul+kg4xe5NIJJJ7bWBggFq1apVpGTVNWb4b09LSUFBQgI0bN2Ljxo3Ftiett2HDBnz//fcICAjA5MmTYWJiApFIhLlz58odHz/55BPo6enhwIED+Omnn6CtrQ0PDw/MmDEDrVq1ku3zBQsWYMGCBcUuMz09HbVr1y77ypcjJuWV3LNnz+Te/NI3momJCTQ0NJCXl4eMjAy5xC49PR15eXmyLxRjY2Pk5uYqlCtpnHBgYCAOHDiAP//8E4cOHYKpqWmN7TUobftLe3Hy8vLk6hR3UKpbty7mzp2LuXPn4t9//8Vvv/2GpUuXIjMzE4sXL36vGE1MTAAAq1atQsOGDRXmv37XAIlEgh9//BGCICAhIQE///wzQkJCYGhoKBvrOnDgQAwcOBDZ2dmIjY3FsmXLMHr0aJw4cUIhESKqSoyNjYs9w5GcnCybXxZl+expampiyJAhCA0NxWeffYadO3eia9euxdaryQwNDSESiWT75HVPnz6FhobGWzsy3iTdT1999RVsbGzeWv7NJJ0Ulfbd+CZDQ0NoaGigX79+GDx4cLHttWjRAsCrawJcXV0VLq5NS0uT2+9aWloICgpCUFAQMjIycPr0aSxZsgRjxozBiRMnZHGMHz9e4WynVHE/0NWNSXkld+jQIVhZWcm9BgBnZ2e8fPkS69evx4EDBzBs2DBZmQMHDgAA3NzcAACurq5Yv349oqKi5K7yl5Z7k62tLezs7LBmzRrEx8dj7NixNfb0amnbv379+tDV1UVCQoJcnePHj5faZuPGjTFy5EgcP34cN2/efO8YO3bsCE1NTTx8+FDpe2WLRCJYWVlh3rx5iIyMLDaOWrVqoXPnznj+/Dlmz56NR48eMSkvho6OTrF31aDKx83NDSdOnJBLKARBQFRUFBo2bIiWLVuWqb2yfvYGDhyIlStXYtq0aXj06BEWLlz4TutRnenr68POzg6//fYbZsyYIev8yM3Nxe+//w47O7sy92K3b98eBgYGuHv3ruyuLfR+SvtufFOtWrXg7OyMGzduoE2bNqXeO14kEinkG6dOncKTJ0/QtGnTYusYGhqiV69euHv3LkJDQ5GRkYEWLVrAwsICCQkJmDx58rusolowKa/kDh06BE1NTTg7O+PKlStYtWoVPD094eDgAEEQ4O7ujpCQELx48QK2trayMh4eHujYsSMAwMPDA87OzggJCUFmZqbs7iv79+8vcbnDhg3DzJkzoaWlBX9//4pa3UqntO0PvBqnv3v3bjRt2hRWVla4cuUKDh48KNfGixcvMGLECHz44Ydo2bIlatWqhfPnz+PChQsYNWrUe8doYWGB8ePHY8mSJUhKSoK7uzsMDAyQnJyMCxcuoGXLlhg2bBhOnjyJn3/+Gd7e3jA3N0dhYSGioqJkY+wAYO7cudDT00P79u1hamqKJ0+eYO3atTAzM5Nd9U7yWrdujXPnzuGPP/5AvXr1YGJiIjdsgSqPCRMm4OTJkxgxYgTGjx8PfX197Nq1C/Hx8ViyZEmZe0iV/exJGRsbo0+fPvjll19gaWkJV1dXVa9itTB16lQEBQVh5MiRGD16NARBwMaNG5GamoqlS5eWub3atWvjiy++wNy5c5Geng4fHx/ZWZP4+Hjo6OhgypQpql+Rauxt341v+uKLLxAQEIDAwEAMGjQIZmZmyMjIwN27d3H16lXZLYQ9PT2xZs0ahIaGwsnJCQkJCQgPD1c4ozRu3DiIxWJYW1vDxMQE9+/fx86dO2FnZyfrPPq///s/jBs3Dp988gn69u0LU1NTPH/+HAkJCUhOTsY333xTvhvpHTApr+RWrVqFZcuWYfPmzdDW1ka/fv0wa9YsAK9+UYaFhWHFihXYtWsXVq5cCVNTUwwfPhyTJk2SfcFoaGhg9erVWLhwIdatW4f8/Hy0b98e69atQ8+ePYtdbvfu3fHFF1/U+NOrpW1/4NUt00QiEdavX4+srCy4urpizZo1cqfLdHV1YWNjg7179+LRo0coKipCkyZNMHnyZAQFBakkzuDgYLRq1Qrbtm3Dnj17UFhYiAYNGsDe3h62trYAgGbNmsHAwADh4eF4+vQpdHV10apVK/z444/w9PQEADg5OWHPnj04fPgwMjIyULduXbi6umLSpEl8eEYJPv/8c3z11VeYNGkScnNz4efnh0WLFqk7LCpGq1at8PPPP2PZsmX48ssvkZ+fD4lEglWrVr3zE1mV+ey9rnfv3vjll1/YY1sKNzc3/PTTTwgNDZXdd9zGxgabNm2Su5C3LPr37w8zMzOsX78ec+bMQU5ODurXrw9ra+sSh1RQyd723fgmKysr7NmzB6tWrcKSJUuQlpYGIyMjtGzZUi4PGT9+PLKysrBr1y6sX78ebdq0wQ8//CD33A8AcHFxwe+//45du3YhMzMTDRo0gLe3NyZOnCgr4+HhgZ07d2LNmjX45ptv8PLlS5iYmMDKygp+fn6q3ygqIBIEQVB3EFT5/P7775g4cSK2bt0KFxcXdYdDRFQtzJ8/H3v27MGpU6dq3HMfqOoLDQ3FypUrce7cuTKP7ae3Y085ybl79y4eP36MxYsXw97engk5EZEKXL16FYmJidi1axdGjhzJhJyIFDApJzlfffUVLl68iHbt2smepEZERO9nwIABqFWrFrp27VrirfmIqGbj8BUiIiIiIjXjEz2JiIiIiNSMSTkRERERkZoxKSciIiIiUjMm5UREVOEkEglmz56t7jCIiCoNJuVERNXIqVOnIJFI8MMPPyjMu3jxIiQSCaytrZGdna0wPygoCFZWVkhNTa2ASImI6HVMyomIqhFHR0doaWkhNjZWYV5cXBy0tLSQn5+Pixcvys0rKCjAhQsX0Lp1a9StW7eiwiUiov+PSTkRUTVSu3Zt2NjY4OrVqwq94XFxcXB3d4epqalC0n716lVkZWXB1dX1vWPIyclBQUHBe7dDRFSTMCknIqpmXF1dkZ+fjwsXLsimSXvCnZ2d4ezsrJCUx8XFyeoCwM2bNzFhwgS4urrCxsYGvXr1wrp161BYWChXb/bs2ZBIJEhNTcXnn38Od3d32Nvb48mTJwCA27dvIygoSPaE4BkzZiAlJaXYuPft24cBAwbAyckJ9vb26Nq1K6ZPn87hNERUI/CJnkRE1YyrqyvWrFmDuLg4dOzYEcD/esJdXFxgYGCA7777DllZWdDX1wfwKikXiURwdnbG1atXERgYCC0tLQQEBKB+/fo4efIklixZgps3b2Lp0qUKyxw1ahTq16+P8ePHy9pNSkpCQEAA8vLyEBAQgMaNG+PkyZMYM2aMQv39+/dj1qxZcHJywqRJk6Cnp4fHjx/j1KlTSElJ4ZAaIqr2mJQTEVUz7du3h7a2tlxveFxcHPT19WFtbY06derIetI9PDxkvegSiQTGxsYYN24c8vLysHPnTlhZWQEAhg0bhilTpuDgwYMYMGAAOnToILfM1q1bY8mSJXLTFixYgPT0dGzevBlubm4AgICAAAQHB+P69etyZY8ePYratWtj8+bN0NL631fTlClTVLlpiIgqLQ5fISKqZvT09GBnZ4f4+HhkZWUBeJWUt2/fHlpaWmjVqhXq1asnG7Ly+njylJQUXLx4EV5eXrKEHABEIhHGjRsH4FUC/aagoCC510VFRThx4gSsra1lCbm0neJ6yuvUqYOcnBz88ccfEATh/TcCEVEVw6SciKgako4rP3/+vNx4ciknJydZT7o0OXdxccHDhw8BAJaWlgpttmrVChoaGkhKSlKY17x5c7nXKSkpyMrKQsuWLRXKFtf2J598AjMzM0yYMAFubm6YOHEifvnlF7x8+VL5lSYiqsKYlBMRVUPSCzbj4uJkPeGvJ+UuLi6Ij49HZmYm4uLioKGhAWdn53fupa5Vq5bca2k7IpFIqfrNmzfH4cOHER4eDj8/Pzx69Ahz585Fz5498eDBg3eKiYioKuGYciKiasjBwQG6urqIjY2FgYEB9PT0YGNjI5vv7OyMgoICxMXF4cKFC2jTpg2MjIxgYWEBALhz545Cm3fv3kVRUZGsTGnq1asHfX19JCYmKswrrm0A0NHRgaenJzw9PQEA0dHRGDt2LDZu3IivvvpKqfUmIqqq2FNORFQN6ejowN7eHteuXcPJkydhb28PHR0d2XyxWAxjY2Ns2LBBdlcW4FUy7eDggJMnT+LWrVuy8oIgIDw8HADg4+Pz1uVramqiS5cuiI+PR0xMjFw769evVyhf3G0P27ZtCwBIT09Xcq2JiKou9pQTEVVTrq6uiI2NxcWLFzFx4kS5eSKRCE5OTjh27JisrNScOXMQGBiIgIAADB06FKampjh58iROnz6NDz/8UOHOKyWZMmUKTp06hXHjxmHYsGFo1KgRTp48WWwCHhQUBAMDAzg7O6Nx48bIyMjA3r17IRKJ0Ldv3/fYCkREVQOTciKiaur1RFvaE/46Z2dnHDt2DJqamnBycpJNt7Gxwc6dO7FixQr8/PPPyMrKgoWFBWbMmIHRo0crvfymTZti+/btCAkJwbZt26Cjo4NOnTph8eLFcHd3lys7ZMgQ/Prrr9i1axfS09NhbGyMNm3aYO7cuXJ3byEiqq5EAu89RURERESkVhxTTkRERESkZkzKiYiIiIjUjEk5EREREZGaMSknIiIiIlIzJuVERERERGrGpJyIiIiISM2YlBMRERERqRmTciIiIiIiNWNSTkRERESkZkzKiYiIiIjU7P8BmMWvKlQk2R4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x504 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train = train.reset_index(drop=True) # We must do this in order to preserve the ordering of emails to labels for words_in_texts\n",
    "\n",
    "words = [\"body\", \"business\", \"html\", \"money\", \"offer\", \"please\"]\n",
    "text = train[\"email\"]\n",
    "indicator = words_in_texts(words, text)\n",
    "\n",
    "df = pd.DataFrame(data = indicator, columns = words)\n",
    "df['type'] = train['spam'].map({0: \"ham\", 1: \"spam\"})\n",
    "melted_df = df.melt('type')\n",
    "\n",
    "plt.figure(figsize = (12, 7))\n",
    "sns.barplot(x=\"variable\", y=\"value\", hue=\"type\", data=melted_df)\n",
    "plt.ylim(0, 1)\n",
    "plt.xlabel('Words')\n",
    "plt.ylabel('Proportion of Emails')\n",
    "plt.title('Frequency of Words in SPam/Ham Emails per Word')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "q3b",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "\n",
    "\n",
    "When the feature is binary, it makes sense to compare its proportions across classes (as in the previous question). Otherwise, if the feature can take on numeric values, we can compare the distributions of these values for different classes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "classification",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<br/><br/>\n",
    "<hr style=\"border: 5px solid #003262;\" />\n",
    "<hr style=\"border: 1px solid #fdb515;\" />\n",
    "\n",
    "# Part 3: Basic Classification\n",
    "\n",
    "Notice that the output of `words_in_texts(words, train['email'])` is a numeric matrix containing features for each email. This means we can use it directly to train a classifier!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "grade": false,
     "grade_id": "q4",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<br/>\n",
    "\n",
    "<hr style=\"border: 1px solid #fdb515;\" />\n",
    "\n",
    "### Question 4\n",
    "\n",
    "We've given you 5 words that might be useful as features to distinguish spam/ham emails. Use these words as well as the `train` DataFrame to create two NumPy arrays: `X_train` and `Y_train`.\n",
    "\n",
    "`X_train` should be a matrix of 0s and 1s created by using your `words_in_texts` function on all the emails in the training set.\n",
    "\n",
    "`Y_train` should be a vector of the correct labels for each email in the training set.\n",
    "\n",
    "*The provided tests check that the dimensions of your feature matrix (X) are correct, and that your features and labels are binary (i.e. consists of only 0's and 1's). It does not check that your function is correct; that was verified in a previous question.*\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q4\n",
    "points: 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>subject</th>\n",
       "      <th>email</th>\n",
       "      <th>spam</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7657</td>\n",
       "      <td>Subject: Patch to enable/disable log\\n</td>\n",
       "      <td>while i was playing with the past issues, it a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6911</td>\n",
       "      <td>Subject: When an engineer flaps his wings\\n</td>\n",
       "      <td>url: http://diveintomark.org/archives/2002/10/...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6074</td>\n",
       "      <td>Subject: Re: [Razor-users] razor plugins for m...</td>\n",
       "      <td>no, please post a link!\\n \\n fox\\n ----- origi...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4376</td>\n",
       "      <td>Subject: NYTimes.com Article: Stop Those Press...</td>\n",
       "      <td>this article from nytimes.com \\n has been sent...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5766</td>\n",
       "      <td>Subject: What's facing FBI's new CIO? (Tech Up...</td>\n",
       "      <td>&lt;html&gt;\\n &lt;head&gt;\\n &lt;title&gt;tech update today&lt;/ti...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7508</th>\n",
       "      <td>5734</td>\n",
       "      <td>Subject: [Spambayes] understanding high false ...</td>\n",
       "      <td>&gt;&gt;&gt;&gt;&gt; \"tp\" == tim peters &lt;tim.one@comcast.net&gt;...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7509</th>\n",
       "      <td>5191</td>\n",
       "      <td>Subject: Reach millions on the internet!!\\n</td>\n",
       "      <td>\\n dear consumers, increase your business sale...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7510</th>\n",
       "      <td>5390</td>\n",
       "      <td>Subject: Facts about sex.\\n</td>\n",
       "      <td>\\n forwarded-by: flower\\n \\n did you know that...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7511</th>\n",
       "      <td>860</td>\n",
       "      <td>Subject: Re: Zoot apt/openssh &amp; new DVD playin...</td>\n",
       "      <td>on tue, oct 08, 2002 at 04:36:13pm +0200, matt...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7512</th>\n",
       "      <td>7270</td>\n",
       "      <td>Subject: Re: Internet radio - example from a c...</td>\n",
       "      <td>chris haun wrote:\\n &gt; \\n &gt; we would need someo...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7513 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id                                            subject  \\\n",
       "0     7657             Subject: Patch to enable/disable log\\n   \n",
       "1     6911        Subject: When an engineer flaps his wings\\n   \n",
       "2     6074  Subject: Re: [Razor-users] razor plugins for m...   \n",
       "3     4376  Subject: NYTimes.com Article: Stop Those Press...   \n",
       "4     5766  Subject: What's facing FBI's new CIO? (Tech Up...   \n",
       "...    ...                                                ...   \n",
       "7508  5734  Subject: [Spambayes] understanding high false ...   \n",
       "7509  5191        Subject: Reach millions on the internet!!\\n   \n",
       "7510  5390                        Subject: Facts about sex.\\n   \n",
       "7511   860  Subject: Re: Zoot apt/openssh & new DVD playin...   \n",
       "7512  7270  Subject: Re: Internet radio - example from a c...   \n",
       "\n",
       "                                                  email  spam  \n",
       "0     while i was playing with the past issues, it a...     0  \n",
       "1     url: http://diveintomark.org/archives/2002/10/...     0  \n",
       "2     no, please post a link!\\n \\n fox\\n ----- origi...     0  \n",
       "3     this article from nytimes.com \\n has been sent...     0  \n",
       "4     <html>\\n <head>\\n <title>tech update today</ti...     0  \n",
       "...                                                 ...   ...  \n",
       "7508  >>>>> \"tp\" == tim peters <tim.one@comcast.net>...     0  \n",
       "7509  \\n dear consumers, increase your business sale...     1  \n",
       "7510  \\n forwarded-by: flower\\n \\n did you know that...     0  \n",
       "7511  on tue, oct 08, 2002 at 04:36:13pm +0200, matt...     0  \n",
       "7512  chris haun wrote:\\n > \\n > we would need someo...     0  \n",
       "\n",
       "[7513 rows x 4 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:43.726012Z",
     "start_time": "2019-04-03T20:17:43.498088Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "q4-answer",
     "locked": false,
     "schema_version": 2,
     "solution": true
    },
    "tags": [
     "student"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[False, False, False, False, False],\n",
       "        [False, False, False, False, False],\n",
       "        [False, False, False, False, False],\n",
       "        [False, False, False, False, False],\n",
       "        [False, False, False,  True, False]]),\n",
       " 0    0\n",
       " 1    0\n",
       " 2    0\n",
       " 3    0\n",
       " 4    0\n",
       " Name: spam, dtype: int64)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "some_words = ['drug', 'bank', 'prescription', 'memo', 'private']\n",
    "\n",
    "X_train = np.array(words_in_texts(some_words, train['email']))\n",
    "Y_train = train['spam']\n",
    "\n",
    "X_train[:5], Y_train[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><strong><pre style='display: inline;'>q4</pre></strong> passed!</p>"
      ],
      "text/plain": [
       "q4 results: All test cases passed!"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check(\"q4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "grade": false,
     "grade_id": "q5",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<br/>\n",
    "\n",
    "<hr style=\"border: 1px solid #fdb515;\" />\n",
    "\n",
    "### Question 5\n",
    "\n",
    "Now that we have matrices, we can build a model with `scikit-learn`! Using the [`LogisticRegression`](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) classifier, train a logistic regression model using `X_train` and `Y_train`. Then, output the model's training accuracy below. You should get an accuracy of around $0.75$\n",
    "\n",
    "*The provided test checks that you initialized your logistic regression model correctly.*\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q5\n",
    "points: 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:44.593918Z",
     "start_time": "2019-04-03T20:17:43.783872Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "q5-answer",
     "locked": false,
     "schema_version": 2,
     "solution": true
    },
    "tags": [
     "student"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy:  0.7576201251164648\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "model = LogisticRegression(random_state=0).fit(X_train, Y_train)\n",
    "training_accuracy = model.score(X_train, Y_train)\n",
    "print(\"Training Accuracy: \", training_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><strong><pre style='display: inline;'>q5</pre></strong> passed!</p>"
      ],
      "text/plain": [
       "q5 results: All test cases passed!"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check(\"q5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "<hr style=\"border: 5px solid #003262;\" />\n",
    "<hr style=\"border: 1px solid #fdb515;\" />\n",
    "\n",
    "# Part 4: Evaluating Classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That doesn't seem too shabby! But the classifier you made above isn't as good as the accuracy would make you believe. First, we are evaluating accuracy on the training set, which may provide a misleading accuracy measure. Accuracy on the training set doesn't always translate to accuracy in the real world (on the test set). In future parts of this analysis, we will hold out some of our data for model validation and comparison.\n",
    "\n",
    "Presumably, our classifier will be used for **filtering**, i.e. preventing messages labeled `spam` from reaching someone's inbox. There are two kinds of errors we can make:\n",
    "- False positive (FP): a ham email gets flagged as spam and filtered out of the inbox.\n",
    "- False negative (FN): a spam email gets mislabeled as ham and ends up in the inbox.\n",
    "\n",
    "To be clear, we label spam emails as 1 and ham emails as 0. These definitions depend both on the true labels and the predicted labels. False positives and false negatives may be of differing importance, leading us to consider more ways of evaluating a classifier, in addition to overall accuracy:\n",
    "\n",
    "**Precision** measures the proportion $\\frac{\\text{TP}}{\\text{TP} + \\text{FP}}$ of emails flagged as spam that are actually spam.\n",
    "\n",
    "**Recall** measures the proportion $\\frac{\\text{TP}}{\\text{TP} + \\text{FN}}$ of spam emails that were correctly flagged as spam. \n",
    "\n",
    "**False-alarm rate** measures the proportion $\\frac{\\text{FP}}{\\text{FP} + \\text{TN}}$ of ham emails that were incorrectly flagged as spam. \n",
    "\n",
    "The below graphic (modified slightly from [Wikipedia](https://en.wikipedia.org/wiki/Precision_and_recall)) may help you understand precision and recall visually:\n",
    "![precision_recall](images/precision_recall.png)\n",
    "\n",
    "Note that a true positive (TP) is a spam email that is classified as spam, and a true negative (TN) is a ham email that is classified as ham."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "grade": false,
     "grade_id": "q6",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<br/>\n",
    "\n",
    "<hr style=\"border: 1px solid #fdb515;\" />\n",
    "\n",
    "### Question 6\n",
    "\n",
    "### Question 6a\n",
    "\n",
    "Suppose we have a classifier `zero_predictor` that always predicts 0 (never predicts positive). How many false positives and false negatives would this classifier have if it were evaluated on the training set and its results were compared to `Y_train`? Fill in the variables below (feel free to hard code your answers for this part):\n",
    "\n",
    "*Tests in Question 6 only check that you have assigned appropriate types of values to each response variable, but do not check that your answers are correct.*\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q6a\n",
    "points: 1\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:20:13.853633Z",
     "start_time": "2019-04-03T20:20:13.825724Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "q6a-answer",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 1918)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zero_predictor_fp = 0\n",
    "zero_predictor_fn = Y_train.sum()\n",
    "zero_predictor_fp, zero_predictor_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><strong><pre style='display: inline;'>q6a</pre></strong> passed!</p>"
      ],
      "text/plain": [
       "q6a results: All test cases passed!"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check(\"q6a\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<br/>\n",
    "\n",
    "---\n",
    "\n",
    "### Question 6b\n",
    "\n",
    "What is the accuracy and recall of `zero_predictor` (classifies every email as ham) on the training set? Do **NOT** use any `sklearn` functions.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q6b\n",
    "points: 1\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:23:21.553134Z",
     "start_time": "2019-04-03T20:23:21.548219Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 0)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zero_predictor_acc = 0\n",
    "zero_predictor_recall = 0\n",
    "zero_predictor_acc, zero_predictor_recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><strong><pre style='display: inline;'>q6b</pre></strong> passed!</p>"
      ],
      "text/plain": [
       "q6b results: All test cases passed!"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check(\"q6b\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "<br/>\n",
    "\n",
    "---\n",
    "\n",
    "### Question 6c\n",
    "\n",
    "Comment on the results from 6a and 6b. For **each** of FP, FN, accuracy, and recall, briefly explain why we see the result that we do.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q6c\n",
    "manual: True\n",
    "points: 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FP: It would be zero since the predictor never predicts a positive, therefore there is 0 possibility that it will predict a false positive if it never predicts a positive at all.  \n",
    "FN: False negative would occur whenever it predicts a positive to be a negative. Since our predictor will predict all positives to be a negative (because it only predicts negatives), the amount of false negatives will be however many positives there are in the entire dataset.  \n",
    "Accuracy: It will be zero, since TP will be 0 due to the fact that there is no positive predictions since there is no positive predictions. 0/anything is 0.   \n",
    "Recall: Same reason as above, since TP = 0, TP/anything will also be zero.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "<br/>\n",
    "\n",
    "---\n",
    "\n",
    "### Question 6d\n",
    "\n",
    "Compute the precision, recall, and false-alarm rate of the `LogisticRegression` classifier created and trained in Question 5. Do **NOT** use any `sklearn` functions, with the exception of the `.predict` method of your model object.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q6d\n",
    "points: 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_pred = model.predict(X_train)\n",
    "model.score(X_train, Y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "TP = np.count_nonzero((Y_pred == Y_train) & (Y_pred == 1))\n",
    "TN = np.count_nonzero((Y_pred == Y_train) & (Y_pred == 0))\n",
    "FP = np.count_nonzero((Y_pred != Y_train) & (Y_pred == 1))\n",
    "FN = np.count_nonzero((Y_pred != Y_train) & (Y_pred == 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:37:54.875265Z",
     "start_time": "2019-04-03T20:37:54.720667Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6422287390029325\n",
      "0.11418143899895725\n",
      "0.021805183199285077\n"
     ]
    }
   ],
   "source": [
    "logistic_predictor_precision =  TP/(TP + FP)\n",
    "logistic_predictor_recall = TP/(TP + FN)\n",
    "logistic_predictor_far = FP/(FP + TN)\n",
    "print(logistic_predictor_precision)\n",
    "print(logistic_predictor_recall)\n",
    "print(logistic_predictor_far)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><strong><pre style='display: inline;'>q6d</pre></strong> passed!</p>"
      ],
      "text/plain": [
       "q6d results: All test cases passed!"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check(\"q6d\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "<br/>\n",
    "\n",
    "---\n",
    "\n",
    "### Question 6e\n",
    "\n",
    "Are there more false positives or false negatives when using the logistic regression classifier from Question 5?\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q6e\n",
    "manual: True\n",
    "points: 1\n",
    "-->"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "There are definitly more false positives on account of the fact that there are positives. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "<br/>\n",
    "\n",
    "---\n",
    "\n",
    "### Question 6f\n",
    "\n",
    "1. Our logistic regression classifier got 75.76% prediction accuracy (number of correct predictions / total). How does this compare with predicting 0 for every email?\n",
    "1. Given the word features we gave you above, name one reason this classifier is performing poorly. Hint: Think about how prevalent these words are in the email set.\n",
    "1. Which of these two classifiers would you prefer for a spam filter and why? Describe your reasoning and relate it to at least one of the evaluation metrics you have computed so far.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q6f\n",
    "manual: True\n",
    "points: 3\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. The Zero predictor has a 0% accuracy, so I'd say the logistic regression classifier is doing better\n",
    "2. There are some words that were given that has low effectiveness, ie not being clear on whether it would belong in a ham email or a spam email \n",
    "3. I would choose the logistical regression classifier, as the zero predictor would not be effective at all with a 0% accuracy. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "\n",
    "\n",
    "<br/><br/>\n",
    "\n",
    "## Congratulations! You have finished Project 2A!\n",
    "\n",
    "In Project 2B, you will focus on building a spam/ham email classifier with logistic regression. You will be well-prepared to build such a model: you have considered what is in this data set, what it can be used for, and engineered some features that should be useful for prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "---\n",
    "\n",
    "To double-check your work, the cell below will rerun all of the autograder tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "q2 results: All test cases passed!\n",
       "\n",
       "q4 results: All test cases passed!\n",
       "\n",
       "q5 results: All test cases passed!\n",
       "\n",
       "q6a results: All test cases passed!\n",
       "\n",
       "q6b results: All test cases passed!\n",
       "\n",
       "q6d results: All test cases passed!"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "## Submission\n",
    "\n",
    "Make sure you have run all cells in your notebook in order before running the cell below, so that all images/graphs appear in the output. The cell below will generate a zip file for you to submit. **Please save before exporting!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <p>Your submission has been exported. Click <a href=\"proj2a_2022_08_11T21_49_03_052260.zip\" download=\"proj2a_2022_08_11T21_49_03_052260.zip\" target=\"_blank\">here</a>\n",
       "            to download the zip file.</p>\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Save your notebook first, then run this cell to export your submission.\n",
    "grader.export()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}