notebooks/13_natural_language_processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural Language Processing (NLP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "*Adapted from [NLP Crash Course](http://files.meetup.com/7616132/DC-NLP-2013-09%20Charlie%20Greenbacker.pdf) by Charlie Greenbacker and [Introduction to NLP](http://spark-public.s3.amazonaws.com/nlp/slides/intro.pdf) by Dan Jurafsky*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is NLP?\n",
    "\n",
    "- Using computers to process (analyze, understand, generate) natural human languages\n",
    "- Most knowledge created by humans is unstructured text, and we need a way to make sense of it\n",
    "- Build probabilistic model using data about a language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What are some of the higher level task areas?\n",
    "\n",
    "- **Information retrieval**: Find relevant results and similar results\n",
    "    - [Google](https://www.google.com/)\n",
    "- **Information extraction**: Structured information from unstructured documents\n",
    "    - [Events from Gmail](https://support.google.com/calendar/answer/6084018?hl=en)\n",
    "- **Machine translation**: One language to another\n",
    "    - [Google Translate](https://translate.google.com/)\n",
    "- **Text simplification**: Preserve the meaning of text, but simplify the grammar and vocabulary\n",
    "    - [Rewordify](https://rewordify.com/)\n",
    "    - [Simple English Wikipedia](https://simple.wikipedia.org/wiki/Main_Page)\n",
    "- **Predictive text input**: Faster or easier typing\n",
    "    - [My application](https://justmarkham.shinyapps.io/textprediction/)\n",
    "    - [A much better application](https://farsite.shinyapps.io/swiftkey-cap/)\n",
    "- **Sentiment analysis**: Attitude of speaker\n",
    "    - [Hater News](http://haternews.herokuapp.com/)\n",
    "- **Automatic summarization**: Extractive or abstractive summarization\n",
    "    - [autotldr](https://www.reddit.com/r/technology/comments/35brc8/21_million_people_still_use_aol_dialup/cr2zzj0)\n",
    "- **Natural Language Generation**: Generate text from data\n",
    "    - [How a computer describes a sports match](http://www.bbc.com/news/technology-34204052)\n",
    "    - [Publishers withdraw more than 120 gibberish papers](http://www.nature.com/news/publishers-withdraw-more-than-120-gibberish-papers-1.14763)\n",
    "- **Speech recognition and generation**: Speech-to-text, text-to-speech\n",
    "    - [Google's Web Speech API demo](https://www.google.com/intl/en/chrome/demos/speech.html)\n",
    "    - [Vocalware Text-to-Speech demo](https://www.vocalware.com/index/demo)\n",
    "- **Question answering**: Determine the intent of the question, match query with knowledge base, evaluate hypotheses\n",
    "    - [How did supercomputer Watson beat Jeopardy champion Ken Jennings?](http://blog.ted.com/how-did-supercomputer-watson-beat-jeopardy-champion-ken-jennings-experts-discuss/)\n",
    "    - [IBM's Watson Trivia Challenge](http://www.nytimes.com/interactive/2010/06/16/magazine/watson-trivia-game.html)\n",
    "    - [The AI Behind Watson](http://www.aaai.org/Magazine/Watson/watson.php)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What are some of the lower level components?\n",
    "\n",
    "- **Tokenization**: breaking text into tokens (words, sentences, n-grams)\n",
    "- **Stopword removal**: a/an/the\n",
    "- **Stemming and lemmatization**: root word\n",
    "- **TF-IDF**: word importance\n",
    "- **Part-of-speech tagging**: noun/verb/adjective\n",
    "- **Named entity recognition**: person/organization/location\n",
    "- **Spelling correction**: \"New Yrok City\"\n",
    "- **Word sense disambiguation**: \"buy a mouse\"\n",
    "- **Segmentation**: \"New York City subway\"\n",
    "- **Language detection**: \"translate this page\"\n",
    "- **Machine learning**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why is NLP hard?\n",
    "\n",
    "- **Ambiguity**:\n",
    "    - Hospitals are Sued by 7 Foot Doctors\n",
    "    - Juvenile Court to Try Shooting Defendant\n",
    "    - Local High School Dropouts Cut in Half\n",
    "- **Non-standard English**: text messages\n",
    "- **Idioms**: \"throw in the towel\"\n",
    "- **Newly coined words**: \"retweet\"\n",
    "- **Tricky entity names**: \"Where is A Bug's Life playing?\"\n",
    "- **World knowledge**: \"Mary and Sue are sisters\", \"Mary and Sue are mothers\"\n",
    "\n",
    "NLP requires an understanding of the **language** and the **world**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Reading in the Yelp Reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- \"corpus\" = collection of documents\n",
    "- \"corpora\" = plural form of corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import scipy as sp\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn import metrics\n",
    "from textblob import TextBlob, Word\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# read yelp.csv into a DataFrame\n",
    "url = 'https://raw.githubusercontent.com/JamesByers/GA-SEA-DAT2/master/data/yelp.csv'\n",
    "yelp = pd.read_csv(url)\n",
    "\n",
    "# create a new DataFrame that only contains the 5-star and 1-star reviews\n",
    "yelp_best_worst = yelp[(yelp.stars==5) | (yelp.stars==1)]\n",
    "\n",
    "# define X and y\n",
    "X = yelp_best_worst.text\n",
    "y = yelp_best_worst.stars\n",
    "\n",
    "# split the new DataFrame into training and testing sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **What:** Separate text into units such as sentences or words\n",
    "- **Why:** Gives structure to previously unstructured text\n",
    "- **Notes:** Relatively easy with English language text, not easy with some languages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# use CountVectorizer to create document-term matrices from X_train and X_test\n",
    "vect = CountVectorizer()\n",
    "X_train_dtm = vect.fit_transform(X_train)\n",
    "X_test_dtm = vect.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# rows are documents, columns are terms (aka \"tokens\" or \"features\")\n",
    "X_train_dtm.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# last 50 features\n",
    "print vect.get_feature_names()[-50:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# show vectorizer options\n",
    "vect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[CountVectorizer documentation](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Parameter **lowercase:** boolean, True by default\n",
    "    - If True, Convert all characters to lowercase before tokenizing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# We will not convert to lowercase this time\n",
    "vect = CountVectorizer(lowercase=False)\n",
    "X_train_dtm = vect.fit_transform(X_train)\n",
    "X_train_dtm.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# last 50 features\n",
    "print vect.get_feature_names()[-50:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Parameter **ngram_range:** tuple (min_n, max_n)\n",
    "    - The lower and upper boundary of the range of n-values for different n-grams to be extracted. All values of n such that min_n <= n <= max_n will be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# include 1-grams and 2-grams\n",
    "vect = CountVectorizer(ngram_range=(1, 2))\n",
    "X_train_dtm = vect.fit_transform(X_train)\n",
    "X_train_dtm.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# last 50 features\n",
    "print vect.get_feature_names()[-50:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Predicting the star rating:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# use default options for CountVectorizer\n",
    "vect = CountVectorizer()\n",
    "\n",
    "# create document-term matrices\n",
    "X_train_dtm = vect.fit_transform(X_train)\n",
    "X_test_dtm = vect.transform(X_test)\n",
    "\n",
    "# use Naive Bayes to predict the star rating\n",
    "nb = MultinomialNB()\n",
    "nb.fit(X_train_dtm, y_train)\n",
    "y_pred_class = nb.predict(X_test_dtm)\n",
    "\n",
    "# calculate accuracy\n",
    "print metrics.accuracy_score(y_test, y_pred_class)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# calculate null accuracy\n",
    "y_test_category = pd.DataFrame(np.where(y_test==5, 'best', 'worst'), columns = ['rating'])\n",
    "y_test_category.rating.value_counts() / len(y_test_category.rating)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# alternate way to calculate null accuracy\n",
    "y_test_binary = np.where(y_test==5, 1, 0)\n",
    "max(y_test_binary.mean(), 1 - y_test_binary.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define a function that accepts a vectorizer and calculates the accuracy\n",
    "def tokenize_test(vect):\n",
    "    X_train_dtm = vect.fit_transform(X_train)\n",
    "    print 'Features: ', X_train_dtm.shape[1]\n",
    "    X_test_dtm = vect.transform(X_test)\n",
    "    nb = MultinomialNB()\n",
    "    nb.fit(X_train_dtm, y_train)\n",
    "    y_pred_class = nb.predict(X_test_dtm)\n",
    "    print 'Accuracy: ', metrics.accuracy_score(y_test, y_pred_class)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vect = CountVectorizer()\n",
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# include 1-grams and 2-grams\n",
    "vect = CountVectorizer(ngram_range=(1, 2))\n",
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Stopword Removal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **What:** Remove common words that will likely appear in any text\n",
    "- **Why:** They don't tell you much about your text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# show vectorizer options\n",
    "vect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **stop_words:** string {'english'}, list, or None (default)\n",
    "    - If 'english', a built-in stop word list for English is used.\n",
    "    - If a list, that list is assumed to contain stop words, all of which will be removed from the resulting tokens.\n",
    "    - If None, no stop words will be used. max_df can be set to a value in the range [0.7, 1.0) to automatically detect and filter stop words based on intra corpus document frequency of terms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# remove English stop words\n",
    "vect = CountVectorizer(stop_words='english')\n",
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# set of stop words\n",
    "print vect.get_stop_words()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Other CountVectorizer Options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **max_features:** int or None, default=None\n",
    "- If not None, build a vocabulary that only consider the top max_features ordered by term frequency across the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# remove English stop words and only keep 100 features\n",
    "vect = CountVectorizer(stop_words='english', max_features=100)\n",
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# all 100 features\n",
    "print vect.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# include 1-grams and 2-grams, and limit the number of features\n",
    "vect = CountVectorizer(ngram_range=(1, 2), max_features=100000)\n",
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# include 1-grams and 2-grams\n",
    "vect = CountVectorizer(ngram_range=(1, 2))\n",
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# include 1-grams and 2-grams, and limit the number of features\n",
    "vect = CountVectorizer(ngram_range=(1, 2), max_features=100000)\n",
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# include 1-grams and 2-grams, and limit the number of features\n",
    "vect = CountVectorizer(ngram_range=(1, 2), max_features=30000)\n",
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **min_df:** float in range [0.0, 1.0] or int, default=1\n",
    "    - When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold. This value is also called cut-off in the literature. If float, the parameter represents a proportion of documents, integer absolute counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# include 1-grams and 2-grams, and only include terms that appear at least 2 times\n",
    "vect = CountVectorizer(ngram_range=(1, 2),  max_features=30000, min_df=2)\n",
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# include 1-grams and 2-grams, and only include terms that appear at least 2 times\n",
    "vect = CountVectorizer(ngram_range=(1, 2),  max_features=30000, min_df=2)\n",
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 5: Introduction to TextBlob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TextBlob: \"Simplified Text Processing\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# print the first review\n",
    "print yelp_best_worst.text[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# save it as a TextBlob object\n",
    "review = TextBlob(yelp_best_worst.text[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# list the words\n",
    "review.words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# list the sentences\n",
    "review.sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# some string methods are available\n",
    "review.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 6: Stemming and Lemmatization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Stemming:**\n",
    "\n",
    "- **What:** Reduce a word to its base/stem/root form\n",
    "- **Why:** Often makes sense to treat related words the same way\n",
    "- **Notes:**\n",
    "    - Uses a \"simple\" and fast rule-based approach\n",
    "    - Stemmed words are usually not shown to users (used for analysis/indexing)\n",
    "    - Some search engines treat words with the same stem as synonyms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# initialize stemmer\n",
    "stemmer = SnowballStemmer('english')\n",
    "\n",
    "# stem each word\n",
    "print [stemmer.stem(word) for word in review.words]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lemmatization**\n",
    "\n",
    "- **What:** Derive the canonical form ('lemma') of a word\n",
    "- **Why:** Can be better than stemming\n",
    "- **Notes:** Uses a dictionary-based approach (slower than stemming)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# assume every word is a noun\n",
    "print [word.lemmatize() for word in review.words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# assume every word is a verb\n",
    "print [word.lemmatize(pos='v') for word in review.words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define a function that accepts text and returns a list of lemmas\n",
    "def split_into_lemmas(text):\n",
    "    text = unicode(text, 'utf-8').lower()\n",
    "    words = TextBlob(text).words\n",
    "    return [word.lemmatize() for word in words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# use split_into_lemmas as the feature extraction function (WARNING: SLOW!)\n",
    "vect = CountVectorizer(analyzer=split_into_lemmas)\n",
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# last 50 features\n",
    "print vect.get_feature_names()[-50:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 7: Term Frequency-Inverse Document Frequency (TF-IDF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **What:** Computes \"relative frequency\" that a word appears in a document compared to its frequency across all documents\n",
    "- **Why:** More useful than \"term frequency\" for identifying \"important\" words in each document (high frequency in that document, low frequency in other documents)\n",
    "- **Notes:** Used for search engine scoring, text summarization, document clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# example documents\n",
    "simple_train = ['call you tonight', 'Call me a cab', 'please call me... PLEASE!']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Term Frequency\n",
    "vect = CountVectorizer()\n",
    "tf = pd.DataFrame(vect.fit_transform(simple_train).toarray(), columns=vect.get_feature_names())\n",
    "tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Document Frequency\n",
    "vect = CountVectorizer(binary=True)\n",
    "df = vect.fit_transform(simple_train).toarray().sum(axis=0)\n",
    "pd.DataFrame(df.reshape(1, 6), columns=vect.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Term Frequency-Inverse Document Frequency (simple version)\n",
    "tf/df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# TfidfVectorizer\n",
    "vect = TfidfVectorizer()\n",
    "pd.DataFrame(vect.fit_transform(simple_train).toarray(), columns=vect.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**More details:** [TF-IDF is about what matters](http://planspace.org/20150524-tfidf_is_about_what_matters/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 8: Using TF-IDF to Summarize a Yelp Review\n",
    "\n",
    "Reddit's autotldr uses the [SMMRY](http://smmry.com/about) algorithm, which is based on TF-IDF!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# create a document-term matrix using TF-IDF\n",
    "vect = TfidfVectorizer(stop_words='english')\n",
    "dtm = vect.fit_transform(yelp.text)\n",
    "features = vect.get_feature_names()\n",
    "dtm.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def summarize():\n",
    "    \n",
    "    # choose a random review that is at least 300 characters\n",
    "    review_length = 0\n",
    "    while review_length < 300:\n",
    "        review_id = np.random.randint(0, len(yelp))\n",
    "        review_text = unicode(yelp.text[review_id], 'utf-8')\n",
    "        review_length = len(review_text)\n",
    "    \n",
    "    # create a dictionary of words and their TF-IDF scores\n",
    "    word_scores = {}\n",
    "    for word in TextBlob(review_text).words:\n",
    "        word = word.lower()\n",
    "        if word in features:\n",
    "            word_scores[word] = dtm[review_id, features.index(word)]\n",
    "    \n",
    "    # print words with the top 5 TF-IDF scores\n",
    "    print 'TOP SCORING WORDS:'\n",
    "    top_scores = sorted(word_scores.items(), key=lambda x: x[1], reverse=True)[:5]\n",
    "    for word, score in top_scores:\n",
    "        print word\n",
    "    \n",
    "    # print 5 random words\n",
    "    print '\\n' + 'RANDOM WORDS:'\n",
    "    random_words = np.random.choice(word_scores.keys(), size=5, replace=False)\n",
    "    for word in random_words:\n",
    "        print word\n",
    "    \n",
    "    # print the review\n",
    "    print '\\n' + review_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "summarize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# create a document-term matrix using TF-IDF\n",
    "vect = TfidfVectorizer(stop_words='english',max_features=10000)\n",
    "dtm = vect.fit_transform(yelp.text)\n",
    "tokenize_test(vect)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 9: Sentiment Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# polarity ranges from -1 (most negative) to 1 (most positive)\n",
    "review.sentiment.polarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# understanding the apply method\n",
    "yelp['length'] = yelp.text.apply(len)\n",
    "yelp.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define a function that accepts text and returns the polarity\n",
    "def detect_sentiment(text):\n",
    "    return TextBlob(text.decode('utf-8')).sentiment.polarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# create a new DataFrame column for sentiment (WARNING: SLOW!)\n",
    "yelp['sentiment'] = yelp.text.apply(detect_sentiment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# box plot of sentiment grouped by stars\n",
    "yelp.boxplot(column='sentiment', by='stars')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# reviews with most positive sentiment\n",
    "yelp[yelp.sentiment == 1].text.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# reviews with most negative sentiment\n",
    "yelp[yelp.sentiment == -1].text.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# widen the column display\n",
    "pd.set_option('max_colwidth', 500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# negative sentiment in a 5-star review\n",
    "yelp[(yelp.stars == 5) & (yelp.sentiment < -0.3)].head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# positive sentiment in a 1-star review\n",
    "yelp[(yelp.stars == 1) & (yelp.sentiment > 0.5)].head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# reset the column display width\n",
    "pd.reset_option('max_colwidth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus: Adding Features to a Document-Term Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create a DataFrame that only contains the 5-star and 1-star reviews\n",
    "yelp_best_worst = yelp[(yelp.stars==5) | (yelp.stars==1)]\n",
    "\n",
    "# define X and y\n",
    "feature_cols = ['text', 'sentiment', 'cool', 'useful', 'funny']\n",
    "X = yelp_best_worst[feature_cols]\n",
    "y = yelp_best_worst.stars\n",
    "\n",
    "# split into training and testing sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# use CountVectorizer with text column only\n",
    "vect = CountVectorizer()\n",
    "X_train_dtm = vect.fit_transform(X_train.text)\n",
    "X_test_dtm = vect.transform(X_test.text)\n",
    "print X_train_dtm.shape\n",
    "print X_test_dtm.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# shape of other four feature columns\n",
    "X_train.drop('text', axis=1).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# cast other feature columns to float and convert to a sparse matrix\n",
    "extra = sp.sparse.csr_matrix(X_train.drop('text', axis=1).astype(float))\n",
    "extra.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# combine sparse matrices\n",
    "X_train_dtm_extra = sp.sparse.hstack((X_train_dtm, extra))\n",
    "X_train_dtm_extra.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# repeat for testing set\n",
    "extra = sp.sparse.csr_matrix(X_test.drop('text', axis=1).astype(float))\n",
    "X_test_dtm_extra = sp.sparse.hstack((X_test_dtm, extra))\n",
    "X_test_dtm_extra.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# use logistic regression with text column only\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train_dtm, y_train)\n",
    "y_pred_class = logreg.predict(X_test_dtm)\n",
    "print metrics.accuracy_score(y_test, y_pred_class)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# use logistic regression with all features\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train_dtm_extra, y_train)\n",
    "y_pred_class = logreg.predict(X_test_dtm_extra)\n",
    "print metrics.accuracy_score(y_test, y_pred_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus: Fun TextBlob Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# spelling correction\n",
    "TextBlob('15 minuets late').correct()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# spellcheck\n",
    "Word('parot').spellcheck()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# definitions\n",
    "Word('bank').define('v')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# language identification\n",
    "TextBlob('Hola amigos').detect_language()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "- NLP is a gigantic field\n",
    "- Understanding the basics broadens the types of data you can work with\n",
    "- Simple techniques go a long way\n",
    "- Use scikit-learn for NLP whenever possible"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}