machine_translation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Artificial Intelligence Nanodegree\n",
    "## Machine Translation Project\n",
    "## Introduction\n",
    "In this notebook, you will build a deep neural network that functions as part of an end-to-end machine translation pipeline. Your completed pipeline will accept English text as input and return the French translation.\n",
    "\n",
    "- **Preprocess** - You'll convert text to sequence of integers.\n",
    "- **Models** Create models which accepts a sequence of integers as input and returns a probability distribution over possible translations. After learning about the basic types of neural networks that are often used for machine translation, you will engage in your own investigations, to design your own model!\n",
    "- **Prediction** Run the model on English text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%aimport helper, tests\n",
    "%autoreload 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import collections\n",
    "\n",
    "import helper\n",
    "import numpy as np\n",
    "from matplotlib import pyplot\n",
    "from pandas import DataFrame\n",
    "import project_tests as tests\n",
    "\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.models import Model, Sequential\n",
    "from keras.layers import GRU, Input, Dense, TimeDistributed, Activation, RepeatVector, Bidirectional, LSTM, Dropout\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.optimizers import Adam, rmsprop\n",
    "from keras.losses import sparse_categorical_crossentropy, categorical_crossentropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify access to the GPU\n",
    "The following test applies only if you expect to be using a GPU, e.g., while running in a Udacity Workspace or using an AWS instance with GPU support. Run the next cell, and verify that the device_type is \"GPU\".\n",
    "- If the device is not GPU & you are running from a Udacity Workspace, then save your workspace with the icon at the top, then click \"enable\" at the bottom of the workspace.\n",
    "- If the device is not GPU & you are running from an AWS instance, then refer to the cloud computing instructions in the classroom to verify your setup steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[name: \"/cpu:0\"\n",
      "device_type: \"CPU\"\n",
      "memory_limit: 268435456\n",
      "locality {\n",
      "}\n",
      "incarnation: 15495246828385944096\n",
      ", name: \"/gpu:0\"\n",
      "device_type: \"GPU\"\n",
      "memory_limit: 357433344\n",
      "locality {\n",
      "  bus_id: 1\n",
      "}\n",
      "incarnation: 11437601037674840354\n",
      "physical_device_desc: \"device: 0, name: Tesla K80, pci bus id: 0000:00:04.0\"\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.python.client import device_lib\n",
    "print(device_lib.list_local_devices())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "We begin by investigating the dataset that will be used to train and evaluate your pipeline.  The most common datasets used for machine translation are from [WMT](http://www.statmt.org/).  However, that will take a long time to train a neural network on.  We'll be using a dataset we created for this project that contains a small vocabulary.  You'll be able to train your model in a reasonable time with this dataset.\n",
    "### Load Data\n",
    "The data is located in `data/small_vocab_en` and `data/small_vocab_fr`. The `small_vocab_en` file contains English sentences with their French translations in the `small_vocab_fr` file. Load the English and French data from these files from running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset Loaded\n"
     ]
    }
   ],
   "source": [
    "# Load English data\n",
    "english_sentences = helper.load_data('data/small_vocab_en')\n",
    "# Load French data\n",
    "french_sentences = helper.load_data('data/small_vocab_fr')\n",
    "\n",
    "print('Dataset Loaded')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Files\n",
    "Each line in `small_vocab_en` contains an English sentence with the respective translation in each line of `small_vocab_fr`.  View the first two lines from each file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "small_vocab_en Line 1:  new jersey is sometimes quiet during autumn , and it is snowy in april .\n",
      "small_vocab_fr Line 1:  new jersey est parfois calme pendant l' automne , et il est neigeux en avril .\n",
      "small_vocab_en Line 2:  the united states is usually chilly during july , and it is usually freezing in november .\n",
      "small_vocab_fr Line 2:  les états-unis est généralement froid en juillet , et il gèle habituellement en novembre .\n"
     ]
    }
   ],
   "source": [
    "for sample_i in range(2):\n",
    "    print('small_vocab_en Line {}:  {}'.format(sample_i + 1, english_sentences[sample_i]))\n",
    "    print('small_vocab_fr Line {}:  {}'.format(sample_i + 1, french_sentences[sample_i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "there are 137861 of sentences in English\n",
      "there are 137861 of sentences in French\n"
     ]
    }
   ],
   "source": [
    "print('there are {} of sentences in English'.format(len(english_sentences)))\n",
    "print('there are {} of sentences in French'.format(len(french_sentences)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From looking at the sentences, you can see they have been preprocessed already.  The puncuations have been delimited using spaces. All the text have been converted to lowercase.  This should save you some time, but the text requires more preprocessing.\n",
    "### Vocabulary\n",
    "The complexity of the problem is determined by the complexity of the vocabulary.  A more complex vocabulary is a more complex problem.  Let's look at the complexity of the dataset we'll be working with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1823250 English words.\n",
      "227 unique English words.\n",
      "10 Most common words in the English dataset:\n",
      "\"is\" \",\" \".\" \"in\" \"it\" \"during\" \"the\" \"but\" \"and\" \"sometimes\"\n",
      "\n",
      "1961295 French words.\n",
      "355 unique French words.\n",
      "10 Most common words in the French dataset:\n",
      "\"est\" \".\" \",\" \"en\" \"il\" \"les\" \"mais\" \"et\" \"la\" \"parfois\"\n"
     ]
    }
   ],
   "source": [
    "english_words_counter = collections.Counter([word for sentence in english_sentences for word in sentence.split()])\n",
    "french_words_counter = collections.Counter([word for sentence in french_sentences for word in sentence.split()])\n",
    "\n",
    "print('{} English words.'.format(len([word for sentence in english_sentences for word in sentence.split()])))\n",
    "print('{} unique English words.'.format(len(english_words_counter)))\n",
    "print('10 Most common words in the English dataset:')\n",
    "print('\"' + '\" \"'.join(list(zip(*english_words_counter.most_common(10)))[0]) + '\"')\n",
    "print()\n",
    "print('{} French words.'.format(len([word for sentence in french_sentences for word in sentence.split()])))\n",
    "print('{} unique French words.'.format(len(french_words_counter)))\n",
    "print('10 Most common words in the French dataset:')\n",
    "print('\"' + '\" \"'.join(list(zip(*french_words_counter.most_common(10)))[0]) + '\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For comparison, _Alice's Adventures in Wonderland_ contains 2,766 unique words of a total of 15,500 words.\n",
    "## Preprocess\n",
    "For this project, you won't use text data as input to your model. Instead, you'll convert the text into sequences of integers using the following preprocess methods:\n",
    "1. Tokenize the words into ids\n",
    "2. Add padding to make all the sequences the same length.\n",
    "\n",
    "Time to start preprocessing the data...\n",
    "### Tokenize (IMPLEMENTATION)\n",
    "For a neural network to predict on text data, it first has to be turned into data it can understand. Text data like \"dog\" is a sequence of ASCII character encodings.  Since a neural network is a series of multiplication and addition operations, the input data needs to be number(s).\n",
    "\n",
    "We can turn each character into a number or each word into a number.  These are called character and word ids, respectively.  Character ids are used for character level models that generate text predictions for each character.  A word level model uses word ids that generate text predictions for each word.  Word level models tend to learn better, since they are lower in complexity, so we'll use those.\n",
    "\n",
    "Turn each sentence into a sequence of words ids using Keras's [`Tokenizer`](https://keras.io/preprocessing/text/#tokenizer) function. Use this function to tokenize `english_sentences` and `french_sentences` in the cell below.\n",
    "\n",
    "Running the cell will run `tokenize` on sample data and show output for debugging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'the': 1, 'quick': 2, 'a': 3, 'brown': 4, 'fox': 5, 'jumps': 6, 'over': 7, 'lazy': 8, 'dog': 9, 'by': 10, 'jove': 11, 'my': 12, 'study': 13, 'of': 14, 'lexicography': 15, 'won': 16, 'prize': 17, 'this': 18, 'is': 19, 'short': 20, 'sentence': 21}\n",
      "\n",
      "Sequence 1 in x\n",
      "  Input:  The quick brown fox jumps over the lazy dog .\n",
      "  Output: [1, 2, 4, 5, 6, 7, 1, 8, 9]\n",
      "Sequence 2 in x\n",
      "  Input:  By Jove , my quick study of lexicography won a prize .\n",
      "  Output: [10, 11, 12, 2, 13, 14, 15, 16, 3, 17]\n",
      "Sequence 3 in x\n",
      "  Input:  This is a short sentence .\n",
      "  Output: [18, 19, 3, 20, 21]\n"
     ]
    }
   ],
   "source": [
    "def tokenize(x):\n",
    "    \"\"\"\n",
    "    Tokenize x\n",
    "    :param x: List of sentences/strings to be tokenized\n",
    "    :return: Tuple of (tokenized x data, tokenizer used to tokenize x)\n",
    "    \"\"\"\n",
    "    x_tk = Tokenizer()\n",
    "    x_tk.fit_on_texts(x)\n",
    "    \n",
    "    return x_tk.texts_to_sequences(x), x_tk\n",
    "\n",
    "tests.test_tokenize(tokenize)\n",
    "\n",
    "# Tokenize Example output\n",
    "text_sentences = [\n",
    "    'The quick brown fox jumps over the lazy dog .',\n",
    "    'By Jove , my quick study of lexicography won a prize .',\n",
    "    'This is a short sentence .']\n",
    "text_tokenized, text_tokenizer = tokenize(text_sentences)\n",
    "print(text_tokenizer.word_index)\n",
    "print()\n",
    "for sample_i, (sent, token_sent) in enumerate(zip(text_sentences, text_tokenized)):\n",
    "    print('Sequence {} in x'.format(sample_i + 1))\n",
    "    print('  Input:  {}'.format(sent))\n",
    "    print('  Output: {}'.format(token_sent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Padding (IMPLEMENTATION)\n",
    "When batching the sequence of word ids together, each sequence needs to be the same length.  Since sentences are dynamic in length, we can add padding to the end of the sequences to make them the same length.\n",
    "\n",
    "Make sure all the English sequences have the same length and all the French sequences have the same length by adding padding to the **end** of each sequence using Keras's [`pad_sequences`](https://keras.io/preprocessing/sequence/#pad_sequences) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequence 1 in x\n",
      "  Input:  [1 2 4 5 6 7 1 8 9]\n",
      "  Output: [1 2 4 5 6 7 1 8 9 0]\n",
      "Sequence 2 in x\n",
      "  Input:  [10 11 12  2 13 14 15 16  3 17]\n",
      "  Output: [10 11 12  2 13 14 15 16  3 17]\n",
      "Sequence 3 in x\n",
      "  Input:  [18 19  3 20 21]\n",
      "  Output: [18 19  3 20 21  0  0  0  0  0]\n"
     ]
    }
   ],
   "source": [
    "def pad(x, length=None):\n",
    "    \"\"\"\n",
    "    Pad x\n",
    "    :param x: List of sequences.\n",
    "    :param length: Length to pad the sequence to.  If None, use length of longest sequence in x.\n",
    "    :return: Padded numpy array of sequences\n",
    "    \"\"\"\n",
    "    if length == None:\n",
    "        length = max([len(text) for text in x])\n",
    "        \n",
    "    return pad_sequences(x, maxlen=length, padding=\"post\")\n",
    "\n",
    "tests.test_pad(pad)\n",
    "\n",
    "# Pad Tokenized output\n",
    "test_pad = pad(text_tokenized)\n",
    "for sample_i, (token_sent, pad_sent) in enumerate(zip(text_tokenized, test_pad)):\n",
    "    print('Sequence {} in x'.format(sample_i + 1))\n",
    "    print('  Input:  {}'.format(np.array(token_sent)))\n",
    "    print('  Output: {}'.format(pad_sent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess Pipeline\n",
    "Your focus for this project is to build neural network architecture, so we won't ask you to create a preprocess pipeline.  Instead, we've provided you with the implementation of the `preprocess` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Preprocessed\n",
      "Max English sentence length: 15\n",
      "Max French sentence length: 21\n",
      "English vocabulary size: 199\n",
      "French vocabulary size: 344\n"
     ]
    }
   ],
   "source": [
    "def preprocess(x, y):\n",
    "    \"\"\"\n",
    "    Preprocess x and y\n",
    "    :param x: Feature List of sentences\n",
    "    :param y: Label List of sentences\n",
    "    :return: Tuple of (Preprocessed x, Preprocessed y, x tokenizer, y tokenizer)\n",
    "    \"\"\"\n",
    "    preprocess_x, x_tk = tokenize(x)\n",
    "    preprocess_y, y_tk = tokenize(y)\n",
    "\n",
    "    preprocess_x = pad(preprocess_x)\n",
    "    preprocess_y = pad(preprocess_y)\n",
    "\n",
    "    # Keras's sparse_categorical_crossentropy function requires the labels to be in 3 dimensions\n",
    "    preprocess_y = preprocess_y.reshape(*preprocess_y.shape, 1)\n",
    "\n",
    "    return preprocess_x, preprocess_y, x_tk, y_tk\n",
    "\n",
    "preproc_english_sentences, preproc_french_sentences, english_tokenizer, french_tokenizer =\\\n",
    "    preprocess(english_sentences, french_sentences)\n",
    "    \n",
    "max_english_sequence_length = preproc_english_sentences.shape[1]\n",
    "max_french_sequence_length = preproc_french_sentences.shape[1]\n",
    "english_vocab_size = len(english_tokenizer.word_index)\n",
    "french_vocab_size = len(french_tokenizer.word_index)\n",
    "\n",
    "print('Data Preprocessed')\n",
    "print(\"Max English sentence length:\", max_english_sequence_length)\n",
    "print(\"Max French sentence length:\", max_french_sequence_length)\n",
    "print(\"English vocabulary size:\", english_vocab_size)\n",
    "print(\"French vocabulary size:\", french_vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example of output and input format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 4],\n",
       "       [32],\n",
       "       [31],\n",
       "       [ 1],\n",
       "       [12],\n",
       "       [19],\n",
       "       [ 2],\n",
       "       [49],\n",
       "       [ 6],\n",
       "       [ 3],\n",
       "       [95],\n",
       "       [69],\n",
       "       [ 2],\n",
       "       [51],\n",
       "       [ 0],\n",
       "       [ 0],\n",
       "       [ 0],\n",
       "       [ 0],\n",
       "       [ 0],\n",
       "       [ 0],\n",
       "       [ 0]], dtype=int32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preproc_french_sentences[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 5, 20, 21,  1,  9, 62,  4, 43,  7,  3,  1,  9, 51,  2, 45], dtype=int32)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preproc_english_sentences[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models\n",
    "In this section, you will experiment with various neural network architectures.\n",
    "You will begin by training four relatively simple architectures.\n",
    "- Model 1 is a simple RNN\n",
    "- Model 2 is a RNN with Embedding\n",
    "- Model 3 is a Bidirectional RNN\n",
    "- Model 4 is an optional Encoder-Decoder RNN\n",
    "\n",
    "After experimenting with the four simple architectures, you will construct a deeper architecture that is designed to outperform all four models.\n",
    "### Ids Back to Text\n",
    "The neural network will be translating the input to words ids, which isn't the final form we want.  We want the French translation.  The function `logits_to_text` will bridge the gab between the logits from the neural network to the French translation.  You'll be using this function to better understand the output of the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "`logits_to_text` function loaded.\n"
     ]
    }
   ],
   "source": [
    "def logits_to_text(logits, tokenizer):\n",
    "    \"\"\"\n",
    "    Turn logits from a neural network into text using the tokenizer\n",
    "    :param logits: Logits from a neural network\n",
    "    :param tokenizer: Keras Tokenizer fit on the labels\n",
    "    :return: String that represents the text of the logits\n",
    "    \"\"\"\n",
    "    index_to_words = {id: word for word, id in tokenizer.word_index.items()}\n",
    "    index_to_words[0] = '<PAD>'\n",
    "\n",
    "    return ' '.join([index_to_words[prediction] for prediction in np.argmax(logits, 1)])\n",
    "\n",
    "print('`logits_to_text` function loaded.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "`source_to_text` function loaded.\n"
     ]
    }
   ],
   "source": [
    "def source_to_text(text, tokenizer):\n",
    "    \"\"\"\n",
    "    Reverse translate the source text from integer back into English using the tokenizer.\n",
    "    :param sentences: Text from the source\n",
    "    :param tokenizer: Keras Tokenizer fit on the labels\n",
    "    :return: String that represents the text of the preprocess text\n",
    "    \"\"\"\n",
    "    index_to_words = {id: word for word, id in tokenizer.word_index.items()}\n",
    "    index_to_words[0] = '<PAD>'\n",
    "    \n",
    "    return ' '.join([index_to_words[reverse_translate] for reverse_translate in text])\n",
    "\n",
    "print('`source_to_text` function loaded.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a DataFrame for storing models' loss information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = DataFrame()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 1: RNN (IMPLEMENTATION)\n",
    "![RNN](images/rnn.png)\n",
    "A basic RNN model is a good baseline for sequence data.  In this model, you'll build a RNN that translates English to French."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Inputs (InputLayer)          (None, 21, 1)             0         \n",
      "_________________________________________________________________\n",
      "RNN (GRU)                    (None, 21, 512)           789504    \n",
      "_________________________________________________________________\n",
      "Outputs (TimeDistributed)    (None, 21, 344)           176472    \n",
      "_________________________________________________________________\n",
      "softmax_activation (Activati (None, 21, 344)           0         \n",
      "=================================================================\n",
      "Total params: 965,976\n",
      "Trainable params: 965,976\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "\n",
      "Train on 110288 samples, validate on 27573 samples\n",
      "Epoch 1/10\n",
      "110288/110288 [==============================] - 19s 176us/step - loss: 2.2333 - acc: 0.5160 - val_loss: nan - val_acc: 0.5914\n",
      "Epoch 2/10\n",
      "110288/110288 [==============================] - 19s 169us/step - loss: 1.4322 - acc: 0.6112 - val_loss: nan - val_acc: 0.6269\n",
      "Epoch 3/10\n",
      "110288/110288 [==============================] - 19s 169us/step - loss: 1.2587 - acc: 0.6366 - val_loss: nan - val_acc: 0.6332\n",
      "Epoch 4/10\n",
      "110288/110288 [==============================] - 19s 169us/step - loss: 1.1518 - acc: 0.6559 - val_loss: nan - val_acc: 0.6637\n",
      "Epoch 5/10\n",
      "110288/110288 [==============================] - 19s 169us/step - loss: 1.0758 - acc: 0.6699 - val_loss: nan - val_acc: 0.6754\n",
      "Epoch 6/10\n",
      "110288/110288 [==============================] - 19s 169us/step - loss: 1.0181 - acc: 0.6774 - val_loss: nan - val_acc: 0.6791\n",
      "Epoch 7/10\n",
      "110288/110288 [==============================] - 19s 169us/step - loss: 0.9721 - acc: 0.6848 - val_loss: nan - val_acc: 0.6847\n",
      "Epoch 8/10\n",
      "110288/110288 [==============================] - 19s 169us/step - loss: 0.9316 - acc: 0.6928 - val_loss: nan - val_acc: 0.6996\n",
      "Epoch 9/10\n",
      "110288/110288 [==============================] - 19s 169us/step - loss: 0.8941 - acc: 0.7024 - val_loss: nan - val_acc: 0.7107\n",
      "Epoch 10/10\n",
      "110288/110288 [==============================] - 19s 169us/step - loss: 0.8585 - acc: 0.7140 - val_loss: nan - val_acc: 0.7261\n",
      "new jersey est parfois chaud en l' et il et il est en en <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "def simple_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):\n",
    "    \"\"\"\n",
    "    Build and train a basic RNN on x and y\n",
    "    :param input_shape: Tuple of input shape\n",
    "    :param output_sequence_length: Length of output sequence\n",
    "    :param english_vocab_size: Number of unique English words in the dataset\n",
    "    :param french_vocab_size: Number of unique French words in the dataset\n",
    "    :return: Keras model built, but not trained\n",
    "    \"\"\"\n",
    "    # simple params\n",
    "    learning_rate = 1e-3\n",
    "    \n",
    "    # loss: 1.0069 - acc: 0.6808 - val_loss: nan - val_acc: 0.6841\n",
    "    X = Input(shape=input_shape[1:], name=\"Inputs\")  \n",
    "    RNN = GRU(512, return_sequences=True, name=\"RNN\")(X)\n",
    "    y = TimeDistributed(Dense(french_vocab_size), name=\"Outputs\")(RNN)\n",
    "    \n",
    "    model = Model(X, Activation(\"softmax\", name=\"softmax_activation\")(y))\n",
    "    model.compile(loss=sparse_categorical_crossentropy,\n",
    "                  optimizer=Adam(learning_rate),\n",
    "                  metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "tests.test_simple_model(simple_model)\n",
    "\n",
    "# Reshaping the input to work with a basic RNN\n",
    "tmp_x = pad(preproc_english_sentences, max_french_sequence_length)\n",
    "tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2], 1))\n",
    "\n",
    "# # Train the neural network\n",
    "simple_rnn_model = simple_model(\n",
    "    tmp_x.shape,\n",
    "    max_french_sequence_length,\n",
    "    english_vocab_size,\n",
    "    french_vocab_size)\n",
    "\n",
    "print(simple_rnn_model.summary())\n",
    "print()\n",
    "hist1 = simple_rnn_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=10, validation_split=0.2)\n",
    "\n",
    "# # Print prediction(s)\n",
    "print(logits_to_text(simple_rnn_model.predict(tmp_x[:1])[0], french_tokenizer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "results[\"simpleRNN\"] = hist1.history['loss']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 2: Embedding (IMPLEMENTATION)\n",
    "![RNN](images/embedding.png)\n",
    "You've turned the words into ids, but there's a better representation of a word.  This is called word embeddings.  An embedding is a vector representation of the word that is close to similar words in n-dimensional space, where the n represents the size of the embedding vectors.\n",
    "\n",
    "In this model, you'll create a RNN model using embedding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Inputs (InputLayer)          (None, 21)                0         \n",
      "_________________________________________________________________\n",
      "Embeddings (Embedding)       (None, 21, 256)           50944     \n",
      "_________________________________________________________________\n",
      "RNN (GRU)                    (None, 21, 512)           1181184   \n",
      "_________________________________________________________________\n",
      "Outputs (TimeDistributed)    (None, 21, 344)           176472    \n",
      "_________________________________________________________________\n",
      "Activation (Activation)      (None, 21, 344)           0         \n",
      "=================================================================\n",
      "Total params: 1,408,600\n",
      "Trainable params: 1,408,600\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Train on 110288 samples, validate on 27573 samples\n",
      "Epoch 1/10\n",
      "110288/110288 [==============================] - 24s 214us/step - loss: 2.8042 - acc: 0.4815 - val_loss: nan - val_acc: 0.5898\n",
      "Epoch 2/10\n",
      "110288/110288 [==============================] - 23s 210us/step - loss: 1.2854 - acc: 0.6941 - val_loss: nan - val_acc: 0.7723\n",
      "Epoch 3/10\n",
      "110288/110288 [==============================] - 23s 210us/step - loss: 0.7110 - acc: 0.8092 - val_loss: nan - val_acc: 0.8378\n",
      "Epoch 4/10\n",
      "110288/110288 [==============================] - 23s 210us/step - loss: 0.4758 - acc: 0.8580 - val_loss: nan - val_acc: 0.8753\n",
      "Epoch 5/10\n",
      "110288/110288 [==============================] - 23s 210us/step - loss: 0.3659 - acc: 0.8858 - val_loss: nan - val_acc: 0.8939\n",
      "Epoch 6/10\n",
      "110288/110288 [==============================] - 23s 210us/step - loss: 0.3091 - acc: 0.9010 - val_loss: nan - val_acc: 0.9049\n",
      "Epoch 7/10\n",
      "110288/110288 [==============================] - 23s 210us/step - loss: 0.2729 - acc: 0.9110 - val_loss: nan - val_acc: 0.9132\n",
      "Epoch 8/10\n",
      "110288/110288 [==============================] - 23s 210us/step - loss: 0.2485 - acc: 0.9178 - val_loss: nan - val_acc: 0.9195\n",
      "Epoch 9/10\n",
      "110288/110288 [==============================] - 23s 210us/step - loss: 0.2299 - acc: 0.9231 - val_loss: nan - val_acc: 0.9225\n",
      "Epoch 10/10\n",
      "110288/110288 [==============================] - 23s 210us/step - loss: 0.2163 - acc: 0.9270 - val_loss: nan - val_acc: 0.9260\n",
      "new jersey est parfois calme au l' automne et il est neigeux en avril <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "from keras.utils import plot_model\n",
    "\n",
    "def embed_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):\n",
    "    \"\"\"\n",
    "    Build and train a RNN model using word embedding on x and y\n",
    "    :param input_shape: Tuple of input shape\n",
    "    :param output_sequence_length: Length of output sequence\n",
    "    :param english_vocab_size: Number of unique English words in the dataset\n",
    "    :param french_vocab_size: Number of unique French words in the dataset\n",
    "    :return: Keras model built, but not trained\n",
    "    \"\"\"\n",
    "    learning_rate = 1e-3\n",
    "        \n",
    "    X = Input(shape=input_shape[1:], name=\"Inputs\")\n",
    "    embedding = Embedding(english_vocab_size, 256, input_length=input_shape[1], name=\"Embeddings\")(X)\n",
    "    RNN = GRU(512, return_sequences=True, name=\"RNN\")(embedding)\n",
    "    y = TimeDistributed(Dense(french_vocab_size), name=\"Outputs\")(RNN)\n",
    "    \n",
    "    model = Model(X, Activation(\"softmax\", name=\"Activation\")(y))\n",
    "    \n",
    "    model.compile(loss=sparse_categorical_crossentropy,\n",
    "                  optimizer=Adam(learning_rate),\n",
    "                  metrics=['accuracy'])\n",
    "    \n",
    "    return model\n",
    "\n",
    "tests.test_embed_model(embed_model)\n",
    "\n",
    "# Inputs preparation using max french length\n",
    "tmp_x = pad(preproc_english_sentences, max_french_sequence_length)\n",
    "\n",
    "# Train the neural network\n",
    "embed_rnn_model = embed_model(\n",
    "    tmp_x.shape,\n",
    "    max_french_sequence_length,\n",
    "    english_vocab_size,\n",
    "    french_vocab_size)\n",
    "\n",
    "print(embed_rnn_model.summary())\n",
    "\n",
    "hist2 = embed_rnn_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=10, validation_split=0.2)\n",
    "\n",
    "# Print prediction(s)\n",
    "print(logits_to_text(embed_rnn_model.predict(tmp_x[:1])[0], french_tokenizer))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "results[\"embedRNN\"] = hist2.history['loss']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 3: Bidirectional RNNs (IMPLEMENTATION)\n",
    "![RNN](images/bidirectional.png)\n",
    "One restriction of a RNN is that it can't see the future input, only the past.  This is where bidirectional recurrent neural networks come in.  They are able to see the future data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bidirectional RNN using \"Sum\" Merge mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Inputs (InputLayer)          (None, 21, 1)             0         \n",
      "_________________________________________________________________\n",
      "Bi-Dir_Sum (Bidirectional)   (None, 21, 512)           1579008   \n",
      "_________________________________________________________________\n",
      "TimeDistributed_Outputs (Tim (None, 21, 344)           176472    \n",
      "_________________________________________________________________\n",
      "Activation (Activation)      (None, 21, 344)           0         \n",
      "=================================================================\n",
      "Total params: 1,755,480\n",
      "Trainable params: 1,755,480\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "\n",
      "Train on 110288 samples, validate on 27573 samples\n",
      "Epoch 1/10\n",
      "110288/110288 [==============================] - 33s 302us/step - loss: 1.8873 - acc: 0.5714 - val_loss: nan - val_acc: 0.6263\n",
      "Epoch 2/10\n",
      "110288/110288 [==============================] - 32s 294us/step - loss: 1.2549 - acc: 0.6432 - val_loss: nan - val_acc: 0.6566\n",
      "Epoch 3/10\n",
      "110288/110288 [==============================] - 32s 294us/step - loss: 1.1132 - acc: 0.6715 - val_loss: nan - val_acc: 0.6769\n",
      "Epoch 4/10\n",
      "110288/110288 [==============================] - 32s 295us/step - loss: 1.0160 - acc: 0.6892 - val_loss: nan - val_acc: 0.6942\n",
      "Epoch 5/10\n",
      "110288/110288 [==============================] - 32s 294us/step - loss: 0.9431 - acc: 0.7017 - val_loss: nan - val_acc: 0.7010\n",
      "Epoch 6/10\n",
      "110288/110288 [==============================] - 32s 294us/step - loss: 0.8908 - acc: 0.7099 - val_loss: nan - val_acc: 0.7038\n",
      "Epoch 7/10\n",
      "110288/110288 [==============================] - 32s 294us/step - loss: 0.8477 - acc: 0.7182 - val_loss: nan - val_acc: 0.7223\n",
      "Epoch 8/10\n",
      "110288/110288 [==============================] - 32s 294us/step - loss: 0.8151 - acc: 0.7240 - val_loss: nan - val_acc: 0.7270\n",
      "Epoch 9/10\n",
      "110288/110288 [==============================] - 32s 294us/step - loss: 0.7791 - acc: 0.7333 - val_loss: nan - val_acc: 0.7389\n",
      "Epoch 10/10\n",
      "110288/110288 [==============================] - 32s 294us/step - loss: 0.7419 - acc: 0.7428 - val_loss: nan - val_acc: 0.7502\n",
      "new jersey est parfois calme au mois de mai et il est en en <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "def bd_sum_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):\n",
    "    \"\"\"\n",
    "    Build and train a bidirectional RNN model on x and y\n",
    "    :param input_shape: Tuple of input shape\n",
    "    :param output_sequence_length: Length of output sequence\n",
    "    :param english_vocab_size: Number of unique English words in the dataset\n",
    "    :param french_vocab_size: Number of unique French words in the dataset\n",
    "    :return: Keras model built, but not trained\n",
    "    \"\"\"\n",
    "    learning_rate = 1e-3\n",
    "    \n",
    "    X = Input(shape=input_shape[1:], name=\"Inputs\")\n",
    "    Bi_RNN = Bidirectional(GRU(512, return_sequences=True, name=\"RNN\"), merge_mode=\"sum\", name=\"Bi-Dir_Sum\")(X)\n",
    "    y = TimeDistributed(Dense(french_vocab_size), name=\"TimeDistributed_Outputs\")(Bi_RNN)\n",
    "    \n",
    "    model = Model(X, Activation(\"softmax\", name=\"Activation\")(y))\n",
    "    \n",
    "    model.compile(loss=sparse_categorical_crossentropy,\n",
    "                  optimizer=Adam(learning_rate),\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    return model\n",
    "\n",
    "tests.test_bd_model(bd_sum_model)\n",
    "\n",
    "# Reshaping the input to work with a basic RNN\n",
    "tmp_x = pad(preproc_english_sentences, max_french_sequence_length)\n",
    "tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2], 1))\n",
    "\n",
    "# Train the neural network\n",
    "bd_sum_rnn_model = bd_sum_model(\n",
    "    tmp_x.shape,\n",
    "    max_french_sequence_length,\n",
    "    english_vocab_size,\n",
    "    french_vocab_size)\n",
    "\n",
    "print(bd_sum_rnn_model.summary())\n",
    "print()\n",
    "\n",
    "hist3 = bd_sum_rnn_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=10, validation_split=0.2)\n",
    "\n",
    "# Print prediction(s)\n",
    "print(logits_to_text(bd_sum_rnn_model.predict(tmp_x[:1])[0], french_tokenizer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "results[\"bd_sum\"] = hist3.history['loss']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bidirectional RNN using \"Concatenation\" Merge mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Inputs (InputLayer)          (None, 21, 1)             0         \n",
      "_________________________________________________________________\n",
      "Bi-Dir_Concat (Bidirectional (None, 21, 1024)          1579008   \n",
      "_________________________________________________________________\n",
      "TimeDistributed_Outputs (Tim (None, 21, 344)           352600    \n",
      "_________________________________________________________________\n",
      "Activation (Activation)      (None, 21, 344)           0         \n",
      "=================================================================\n",
      "Total params: 1,931,608\n",
      "Trainable params: 1,931,608\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "\n",
      "Train on 110288 samples, validate on 27573 samples\n",
      "Epoch 1/10\n",
      "110288/110288 [==============================] - 35s 316us/step - loss: 1.8661 - acc: 0.5713 - val_loss: nan - val_acc: 0.6277\n",
      "Epoch 2/10\n",
      "110288/110288 [==============================] - 34s 308us/step - loss: 1.2394 - acc: 0.6447 - val_loss: nan - val_acc: 0.6613\n",
      "Epoch 3/10\n",
      "110288/110288 [==============================] - 34s 308us/step - loss: 1.1002 - acc: 0.6717 - val_loss: nan - val_acc: 0.6851\n",
      "Epoch 4/10\n",
      "110288/110288 [==============================] - 34s 308us/step - loss: 1.0024 - acc: 0.6898 - val_loss: nan - val_acc: 0.6952\n",
      "Epoch 5/10\n",
      "110288/110288 [==============================] - 34s 308us/step - loss: 0.9343 - acc: 0.7016 - val_loss: nan - val_acc: 0.7116\n",
      "Epoch 6/10\n",
      "110288/110288 [==============================] - 34s 308us/step - loss: 0.8787 - acc: 0.7133 - val_loss: nan - val_acc: 0.7212\n",
      "Epoch 7/10\n",
      "110288/110288 [==============================] - 34s 308us/step - loss: 0.8299 - acc: 0.7231 - val_loss: nan - val_acc: 0.7315\n",
      "Epoch 8/10\n",
      "110288/110288 [==============================] - 34s 308us/step - loss: 0.7852 - acc: 0.7358 - val_loss: nan - val_acc: 0.7435\n",
      "Epoch 9/10\n",
      "110288/110288 [==============================] - 34s 308us/step - loss: 0.7369 - acc: 0.7529 - val_loss: nan - val_acc: 0.7526\n",
      "Epoch 10/10\n",
      "110288/110288 [==============================] - 34s 308us/step - loss: 0.6916 - acc: 0.7707 - val_loss: nan - val_acc: 0.7739\n",
      "new jersey est parfois calme au mois de il et il en en <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "def bd_concat_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):\n",
    "    \"\"\"\n",
    "    Build and train a bidirectional RNN model on x and y\n",
    "    :param input_shape: Tuple of input shape\n",
    "    :param output_sequence_length: Length of output sequence\n",
    "    :param english_vocab_size: Number of unique English words in the dataset\n",
    "    :param french_vocab_size: Number of unique French words in the dataset\n",
    "    :return: Keras model built, but not trained\n",
    "    \"\"\"\n",
    "    learning_rate = 1e-3\n",
    "    \n",
    "    X = Input(shape=input_shape[1:], name=\"Inputs\")\n",
    "    Bi_RNN = Bidirectional(GRU(512, return_sequences=True), merge_mode=\"concat\", name=\"Bi-Dir_Concat\")(X)\n",
    "    y = TimeDistributed(Dense(french_vocab_size), name=\"TimeDistributed_Outputs\")(Bi_RNN)\n",
    "    \n",
    "    model = Model(X, Activation(\"softmax\", name=\"Activation\")(y))\n",
    "    \n",
    "    model.compile(loss=sparse_categorical_crossentropy,\n",
    "                  optimizer=Adam(learning_rate),\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    return model\n",
    "\n",
    "tests.test_bd_model(bd_concat_model)\n",
    "\n",
    "# Reshaping the input to work with a basic RNN\n",
    "tmp_x = pad(preproc_english_sentences, max_french_sequence_length)\n",
    "tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2], 1))\n",
    "\n",
    "# Train the neural network\n",
    "bd_concat_rnn_model = bd_concat_model(\n",
    "    tmp_x.shape,\n",
    "    max_french_sequence_length,\n",
    "    english_vocab_size,\n",
    "    french_vocab_size)\n",
    "\n",
    "print(bd_concat_rnn_model.summary())\n",
    "print()\n",
    "\n",
    "hist4 = bd_concat_rnn_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=10, validation_split=0.2)\n",
    "\n",
    "# Print prediction(s)\n",
    "print(logits_to_text(bd_concat_rnn_model.predict(tmp_x[:1])[0], french_tokenizer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "results[\"bd_concat\"] = hist4.history['loss']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 4: Encoder-Decoder (OPTIONAL)\n",
    "Time to look at encoder-decoder models.  This model is made up of an encoder and decoder. The encoder creates a matrix representation of the sentence.  The decoder takes this matrix as input and predicts the translation as output.\n",
    "\n",
    "Create an encoder-decoder model in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Inputs (InputLayer)             (None, 15)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Source_Embedding (Embedding)    (None, 15, 256)      50944       Inputs[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "Encoder (LSTM)                  [(None, 512), (None, 1574912     Source_Embedding[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "Decoder_Inputs (RepeatVector)   (None, 21, 512)      0           Encoder[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "Decoder (LSTM)                  [(None, 21, 512), (N 2099200     Decoder_Inputs[0][0]             \n",
      "                                                                 Encoder[0][1]                    \n",
      "                                                                 Encoder[0][2]                    \n",
      "__________________________________________________________________________________________________\n",
      "Outputs (TimeDistributed)       (None, 21, 344)      176472      Decoder[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "Activation (Activation)         (None, 21, 344)      0           Outputs[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 3,901,528\n",
      "Trainable params: 3,901,528\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n",
      "None\n",
      "\n",
      "Train on 110288 samples, validate on 27573 samples\n",
      "Epoch 1/10\n",
      "110288/110288 [==============================] - 51s 460us/step - loss: 2.7021 - acc: 0.4674 - val_loss: nan - val_acc: 0.5313\n",
      "Epoch 2/10\n",
      "110288/110288 [==============================] - 50s 451us/step - loss: 1.5696 - acc: 0.5909 - val_loss: nan - val_acc: 0.6304\n",
      "Epoch 3/10\n",
      "110288/110288 [==============================] - 50s 451us/step - loss: 1.2494 - acc: 0.6519 - val_loss: nan - val_acc: 0.6761\n",
      "Epoch 4/10\n",
      "110288/110288 [==============================] - 50s 451us/step - loss: 1.0808 - acc: 0.6893 - val_loss: nan - val_acc: 0.7075\n",
      "Epoch 5/10\n",
      "110288/110288 [==============================] - 50s 451us/step - loss: 0.9509 - acc: 0.7227 - val_loss: nan - val_acc: 0.7410\n",
      "Epoch 6/10\n",
      "110288/110288 [==============================] - 50s 451us/step - loss: 0.8423 - acc: 0.7502 - val_loss: nan - val_acc: 0.7657\n",
      "Epoch 7/10\n",
      "110288/110288 [==============================] - 50s 451us/step - loss: 0.7373 - acc: 0.7777 - val_loss: nan - val_acc: 0.7895\n",
      "Epoch 8/10\n",
      "110288/110288 [==============================] - 50s 451us/step - loss: 0.6383 - acc: 0.8060 - val_loss: nan - val_acc: 0.8180\n",
      "Epoch 9/10\n",
      "110288/110288 [==============================] - 50s 451us/step - loss: 0.5935 - acc: 0.8171 - val_loss: nan - val_acc: 0.8387\n",
      "Epoch 10/10\n",
      "110288/110288 [==============================] - 50s 451us/step - loss: 0.4571 - acc: 0.8592 - val_loss: nan - val_acc: 0.8718\n",
      "new jersey est parfois chaud au cours et l' automne et en est <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "def encdec_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):\n",
    "    \"\"\"\n",
    "    Build and train an encoder-decoder model on x and y\n",
    "    :param input_shape: Tuple of input shape\n",
    "    :param output_sequence_length: Length of output sequence\n",
    "    :param english_vocab_size: Number of unique English words in the dataset\n",
    "    :param french_vocab_size: Number of unique French words in the dataset\n",
    "    :return: Keras model built, but not trained\n",
    "    \"\"\"\n",
    "    learning_rate=1e-3\n",
    "    \n",
    "    # Encoder\n",
    "    X = Input(shape=input_shape[1:], name=\"Inputs\")\n",
    "    embedding = Embedding(english_vocab_size, 256, name=\"Source_Embedding\")(X)\n",
    "    encoder, state_h, state_c = LSTM(512, return_state=True, name=\"Encoder\")(embedding)\n",
    "    \n",
    "    # Decoder\n",
    "    decoder_inputs = RepeatVector(output_sequence_length, name=\"Decoder_Inputs\")(encoder)    \n",
    "    decoder, _, _ = LSTM(512, return_sequences=True, return_state=True, name=\"Decoder\")(decoder_inputs, initial_state=[state_h, state_c])\n",
    "    decoder_y = TimeDistributed(Dense(french_vocab_size), name=\"Outputs\")(decoder)\n",
    "    \n",
    "    model = Model(X, Activation(\"softmax\", name=\"Activation\")(decoder_y))\n",
    "    \n",
    "    model.compile(loss=sparse_categorical_crossentropy,\n",
    "                  optimizer=Adam(learning_rate),\n",
    "                  metrics=['accuracy'])\n",
    "    \n",
    "    return model\n",
    "\n",
    "# tests.test_encdec_model(encdec_model)\n",
    "\n",
    "# Reshaping the input to work with a basic RNN\n",
    "tmp_x = pad(preproc_english_sentences, max_english_sequence_length)\n",
    "\n",
    "# Train the neural network\n",
    "encdec_rnn_model = encdec_model(\n",
    "    tmp_x.shape,\n",
    "    max_french_sequence_length,\n",
    "    english_vocab_size,\n",
    "    french_vocab_size)\n",
    "\n",
    "print(encdec_rnn_model.summary())\n",
    "print()\n",
    "\n",
    "hist5 = encdec_rnn_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=10, validation_split=0.2)\n",
    "\n",
    "# Print prediction(s)\n",
    "print(logits_to_text(encdec_rnn_model.predict(tmp_x[:1])[0], french_tokenizer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "results[\"encdec\"] = hist5.history['loss']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "new jersey is sometimes quiet during autumn and it is snowy in april <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "print(source_to_text(preproc_english_sentences[0], english_tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 5: Custom (IMPLEMENTATION)\n",
    "Use everything you learned from the previous models to create a model that incorporates embedding and a bidirectional rnn into one model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bidirectional and embedding RNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final Model Loaded\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Inputs (InputLayer)          (None, 21)                0         \n",
      "_________________________________________________________________\n",
      "Embeddings (Embedding)       (None, 21, 256)           50944     \n",
      "_________________________________________________________________\n",
      "BD_RNN (Bidirectional)       (None, 21, 1024)          2362368   \n",
      "_________________________________________________________________\n",
      "Outputs (TimeDistributed)    (None, 21, 344)           352600    \n",
      "_________________________________________________________________\n",
      "Activation (Activation)      (None, 21, 344)           0         \n",
      "=================================================================\n",
      "Total params: 2,765,912\n",
      "Trainable params: 2,765,912\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "\n",
      "Train on 110288 samples, validate on 27573 samples\n",
      "Epoch 1/10\n",
      "110288/110288 [==============================] - 44s 400us/step - loss: 2.5675 - acc: 0.5000 - val_loss: nan - val_acc: 0.6405\n",
      "Epoch 2/10\n",
      "110288/110288 [==============================] - 43s 390us/step - loss: 1.0877 - acc: 0.7338 - val_loss: nan - val_acc: 0.7958\n",
      "Epoch 3/10\n",
      "110288/110288 [==============================] - 43s 390us/step - loss: 0.6145 - acc: 0.8290 - val_loss: nan - val_acc: 0.8621\n",
      "Epoch 4/10\n",
      "110288/110288 [==============================] - 43s 390us/step - loss: 0.3989 - acc: 0.8841 - val_loss: nan - val_acc: 0.9035\n",
      "Epoch 5/10\n",
      "110288/110288 [==============================] - 43s 390us/step - loss: 0.2888 - acc: 0.9141 - val_loss: nan - val_acc: 0.9225\n",
      "Epoch 6/10\n",
      "110288/110288 [==============================] - 43s 390us/step - loss: 0.2209 - acc: 0.9334 - val_loss: nan - val_acc: 0.9377\n",
      "Epoch 7/10\n",
      "110288/110288 [==============================] - 43s 390us/step - loss: 0.1825 - acc: 0.9447 - val_loss: nan - val_acc: 0.9483\n",
      "Epoch 8/10\n",
      "110288/110288 [==============================] - 43s 390us/step - loss: 0.1512 - acc: 0.9542 - val_loss: nan - val_acc: 0.9577\n",
      "Epoch 9/10\n",
      "110288/110288 [==============================] - 43s 390us/step - loss: 0.1292 - acc: 0.9611 - val_loss: nan - val_acc: 0.9619\n",
      "Epoch 10/10\n",
      "110288/110288 [==============================] - 43s 390us/step - loss: 0.1140 - acc: 0.9656 - val_loss: nan - val_acc: 0.9654\n",
      "new jersey est parfois calme pendant cours automne et il est neigeux en avril <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "def model_final(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):\n",
    "    \"\"\"\n",
    "    Build and train a model that incorporates embedding, encoder-decoder, and bidirectional RNN on x and y\n",
    "    :param input_shape: Tuple of input shape\n",
    "    :param output_sequence_length: Length of output sequence\n",
    "    :param english_vocab_size: Number of unique English words in the dataset\n",
    "    :param french_vocab_size: Number of unique French words in the dataset\n",
    "    :return: Keras model built, but not trained\n",
    "    \"\"\"\n",
    "    learning_rate = 1e-3\n",
    "    \n",
    "    # Building Bidrectional RNN with Embedding Model\n",
    "    X = Input(shape=input_shape[1:], name=\"Inputs\")\n",
    "    embedding = Embedding(english_vocab_size, 256, input_length=input_shape[1], name=\"Embeddings\")(X)\n",
    "    BiDirect_RNN = Bidirectional(GRU(512, return_sequences=True), name=\"BD_RNN\")(embedding)\n",
    "    y = TimeDistributed(Dense(french_vocab_size), name=\"Outputs\")(BiDirect_RNN)\n",
    "    \n",
    "    model = Model(X, Activation(\"softmax\", name=\"Activation\")(y))\n",
    "    \n",
    "    model.compile(loss=sparse_categorical_crossentropy,\n",
    "                  optimizer=Adam(learning_rate),\n",
    "                  metrics=['accuracy'])  \n",
    "    \n",
    "    return model\n",
    "\n",
    "tests.test_model_final(model_final)\n",
    "\n",
    "print('Final Model Loaded')\n",
    "\n",
    "# Reshaping the input to work with a basic RNN\n",
    "tmp_x = pad(preproc_english_sentences, max_french_sequence_length)\n",
    "\n",
    "# Train the neural network\n",
    "final_rnn_model = model_final(\n",
    "    tmp_x.shape,\n",
    "    max_french_sequence_length,\n",
    "    english_vocab_size,\n",
    "    french_vocab_size)\n",
    "\n",
    "print(final_rnn_model.summary())\n",
    "print()\n",
    "\n",
    "hist6 = final_rnn_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=10, validation_split=0.2)\n",
    "\n",
    "# Print prediction(s)\n",
    "print(logits_to_text(final_rnn_model.predict(tmp_x[:1])[0], french_tokenizer))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "results[\"final_model\"] = hist6.history['loss']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f9e80101550>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xd8FGX+wPHPbLK72fTeeyFACgESCCBNEEEUG+r95Gx3HqjHyXnnCYKKDcuJeqKoJ4qcytk4RUFEkC49gVBCEkggIb33tm1+f2xYCIQkQAobnvfrta/dnXnmmWcifufZZ575jiTLMoIgCELfoujtBgiCIAhdTwR3QRCEPkgEd0EQhD5IBHdBEIQ+SAR3QRCEPkgEd0EQhD5IBHdBEIQ+SAR3QRCEPkgEd0EQhD7Iurd27O7uLgcHB/fW7gVBECxScnJymSzLHh2V67XgHhwcTFJSUm/tXhAEwSJJkpTTmXJiWEYQBKEPEsFdEAShDxLBXRAEoQ/qtTF3QRAsh06nIy8vj6ampt5uyjXDxsYGf39/lErlZW0vgrsgCB3Ky8vDwcGB4OBgJEnq7eb0ebIsU15eTl5eHiEhIZdVhxiWEQShQ01NTbi5uYnA3kMkScLNze2KfimJ4C4IQqeIwN6zrvTvbXnBvSQd1s8HfXNvt0QQBOGqZXnBveo07FkK2Tt6uyWCIPSyhx9+mGPHjnVJXfb29h2WsbKyIi4ujujoaG655RaqqqoAyM7ORpIk3n33XXPZ2bNns2LFCgAefPBB/Pz8aG42dUrLysro7jv0LS+4h4wBpS1k/NzbLREEoZd9/PHHDBw4sMf2p9FoSElJ4ejRo7i6urJ06VLzOk9PT9555x20Wm2b21pZWbF8+fKeaqoFBnelDYRdDxnrQZZ7uzWCIPSQ+vp6pk6dyqBBg4iOjubrr79m3Lhx5jQm9vb2zJ07l6FDhzJx4kT27dvHuHHjCA0N5ccffwRgxYoV3HrrrUyePJnIyEheeOGFNvf1xhtvkJCQQGxsLAsXLmyzzIgRI8jPzzd/9/DwYMKECfznP/9ps/xf//pX3n77bfR6/ZX8GTrNMqdCRk6B9LVQdAR8Ynu7NYJwTXlhTSrHCmq6tM6Bvo4svCWq3TLr16/H19eXn376CYDq6mo++OAD8/r6+nrGjRvH66+/zu23384zzzzDxo0bOXbsGA888ADTpk0DYN++fRw9ehRbW1sSEhKYOnUq8fHx5no2bNjAiRMn2LdvH7IsM23aNLZv386YMWPMZQwGA5s2beKPf/xjqzbOmzePKVOm8Ic//OGC9gcGBnLdddfx+eefc8stt1z6H+kSWV7PHSDiRkCC4+t7uyWCIPSQmJgYfv31V+bOncuOHTtwcnJqtV6lUjF58mRz2bFjx6JUKomJiSE7O9tc7oYbbsDNzQ2NRsMdd9zBb7/91qqeDRs2sGHDBgYPHsyQIUNIT0/nxIkTADQ2NhIXF4ebmxsVFRXccMMNrbYNCQlh2LBh/Pe//23zGObPn88bb7yB0Wi80j9Hhyyz527vAf4JkLEOxj7V260RhGtKRz3s7tKvXz+Sk5NZt24dTz/9NJMmTWq1XqlUmqcPKhQK1Gq1+fO5QyHnTzE8/7ssyzz99NPMmjXrgjacGXOvrq7m5ptvZunSpTz++OOtysyfP5/p06e36umfER4eTlxcHN98880lHPnlsciee1OdzjQ0U3AQagp7uzmCIPSAgoICbG1t+f3vf8+TTz7JgQMHLquejRs3UlFRQWNjI6tXr2bUqFGt1t94440sX76curo6APLz8ykpKWlVxsnJiSVLlrB48WJ0Ol2rdf3792fgwIGsXbu2zf0vWLCAxYsXX1bbL4XFBffj+4v49KnfqPZo+TkkhmYE4Zpw5MgRhg0bRlxcHIsWLeKZZ565rHquu+467rvvPuLi4rjzzjtbjbcDTJo0iXvvvZcRI0YQExPD9OnTqa2tvaCewYMHM2jQIL766qsL1i1YsIC8vLw29x8VFcWQIUMuq+2XQpJ7acZJfHy8fDkP66irbOazBbsYPDGAEbl3gnskzOj+nziCcC1LS0tjwIABvd2MK7ZixQqSkpJ47733erspndLW312SpGRZluMvsomZxfXc7V3UhMS6k7a7EEPYTXBqG2jre7tZgiAIVxWLC+4AUWN8aazVcVKaBPomOLm1t5skCIIFePDBBy2m136lLDK4B/R3xdHdhqPpjqB2Ms2aEQRBEMwsMrhLComo0X4UnKimwvcOOP4L9MC8UUEQBEthkcEdYMBIHxRWEqn1k6C+FPKTe7tJgiAIVw2LDe4aBxVhQzzJOGGHDg0cF4nEBEEQzrDY4A4QPcaX5kYDmbb3iyyRgiBc1IoVK5g9e3aXbP/888/j5+dHXFwcAwcO5MsvvzSXay+1b0dpgbuaxQV3WZYpz88FwCfcGRdvW1JrxkLJMajM7t3GCYJwTXjiiSdISUnhhx9+YNasWa3uUm0vtW9HaYG7ksUF99Rtm1jxt0cpz89FkiSixvhRXKahVBdiSgMsCEKf9cUXX5jvUp01axYGg6FTqX4BcnNz20z121adAJ9++in9+vVj7Nix7Ny5s832REREYGtrS2VlpXlZe6l9O0oL3JUsLnFYUEwcAJn79+DmF0DkcG/2fJ9FqvFuxmWsg8RHermFgtDH/TzPlG67K3nHwJTX2i2SlpbG119/zc6dO1EqlTz22GOsXLnyilL92tnZtVnnDTfcwMKFC0lOTsbJyYnx48czePDgC9p04MABIiIi8PT0NC/rKLVve2mBu1KHwV2SpADgM8AbMAIfybL8znllxgE/AKdaFn0ny/KLXdtUEwc3d7zDIshK2sPw2+7Cxk5JeIIXGXsTGHnqfVRN1WDj1HFFgiBYlE2bNpGcnExCQgJgSr/r6el5QapftVrdbqpfwJzq19raus069+7dy7hx4/Dw8ADgnnvu4fjx4+a63n77bZYtW8bJkydZv/7CEYP58+czbdo0pk6desG6jtICd5XO9Nz1wN9lWT4gSZIDkCxJ0kZZls9/cOEOWZZv7vomXigsPpGdX39OXWUF9i6uRI/2I31XIccbRhKd+StE39kTzRCEa1MHPezuIssyDzzwAK+++mqr5YsXL77sVL8Xq3P16tUXlD/XE088wZNPPsl3333H/fffT1ZWFjY2Nub1HaX2bS8tcFfpcMxdluVCWZYPtHyuBdIAv25rUSeEJyQCkJW0FwDPYAfcA+w52nQzcroYdxeEvmjChAmsWrXKnH63oqKCnJycTm/fVqrfi9U5fPhwtm7dSnl5OTqdjm+//bbNOu+44w7i4+PbHENvL7VvR2mBu8IlXVCVJCkYGAzsbWP1CEmSDkmS9LMkSd2azd/NPxBnLx+ykvacaRfRY/wo1/pTfDQTDLoOahAEwdIMHDiQl19+mUmTJhEbG8sNN9xAYWHnn+fQVqrfi9Xp4+PD888/z4gRI5g4cWK7KXqfe+453nrrrQuertRRat/20gJ3hU6n/JUkyR7YBiySZfm789Y5AkZZluskSboJeEeW5Yg26pgJzAQIDAwceiln3fNt/fwTUtav4dFl/0Vta4u2Sc+Kf2wj1GorE+dMgpDRl123IAit9ZWUv5am21P+SpKkBP4HrDw/sAPIslwjy3Jdy+d1gFKSJPc2yn0ky3K8LMvxZy5UXK7whEQMej3Zh0xpB1Q21kQO8yKzaRRNR3+9oroFQRAsXYfBXTJdVfgESJNl+a2LlPFuKYckScNa6i3vyoaez7dffzSOTmTu32NeFjU+CAMq0pOqoJceQiIIgnA16MxsmVHAfcARSZJSWpbNBwIBZFn+EJgOPCpJkh5oBH4nd/MjnhQKK8KGDuPE3l0Y9HqsrK1x93fA27OJ1LJ4BpUeR/KM7M4mCIIgXLU6DO6yLP8GXHxOkKnMe0CPZ8APi0/k6JaN5B07SlCs6eamqHHBbPrGhvzt2/GfLoK7IAjXJotLP3CuoNg4rNVqMpN2m5eFXxeJ2qqB1ANixowgCNcuiw7uSpWa4NghZCbt5cwokLXKiv5h1ZysCKehqPPTpARBEPoSiw7uYJo1U1deRsmpLPOyqImRGLEmbd2+XmyZIAhC77H44B46JAFJUpC5/+zQjEvMUPw06aQeBqNRzJoRhL4gOzub6Ojodsts3bqVm2/ukSwoVz2LD+4aB0f8B0S1mhKJJBE9oJbaJgdyjxT3XuMEQRB6icWl/G1LWHwiWz9bRlVRIc7ePgCEjB6MJqWSoxtSCRrk3cstFIS+4/V9r5Nekd6ldfZ37c/cYXM7LKfX63nggQc4ePAg/fr147PPPmP79u389a9/xd3dvd3b/QG2bdvGnDlzAFPaku3bt5OcnMzixYvNeV5mz55NfHw8Dz74IMHBwdx7771s2bIFnU7HRx99xNNPP01mZib/+Mc/eOSRqzfFuMX33AHCE4YDkJl0tvduFT6GgXbbyMmC2oqm3mqaIAhdKCMjg5kzZ3L48GEcHR156623+NOf/sSaNWvYsWMHRUVF7W6/ePFili5dSkpKCjt27ECj0XS4z4CAAHbv3s3o0aN58MEHWbVqFXv27OG5557rqsPqFn2i5+7k6Y1HYDBZSXuJv/l200KlDQMHNpO8V+bYbwUMnxbau40UhD6iMz3s7hIQEMCoUaMA+P3vf8+SJUsICQkhIiLCvOyjjz666PajRo3ib3/7GzNmzOCOO+7A39+/w32eedhHTEwMdXV1ODg44ODggI2NDVVVVTg7O3fBkXW9PtFzBwhLGEF++jEaaqrNyxzjRhOkPsCx7acxGIztbC0IgiU4P8d6dXV1u3nXzzdv3jw+/vhjGhsbSUxMJD09HWtr61YZHZuaWv/SPzc//JnPZ7639Si9q0WfCe7hCYnIspGTyedMf4y4kSjNBhrqjOQc7tZUN4Ig9IDTp0+ze7dpZtyXX37JxIkTOXXqFFlZWeZl7cnKyiImJoa5c+cSHx9Peno6QUFBHDt2jObmZqqrq9m0aVO3H0dP6DPB3TM4FAd3DzKTzkk1b+9BUJgCe2U1R3fk917jBEHoEgMGDOA///kPsbGxVFRU8MQTT/DRRx8xdepUrrvuOoKCgtrd/l//+hfR0dEMGjQIjUbDlClTCAgI4O677yY2NpYZM2a0+axUS9TpfO5dLT4+Xk5KSurSOjd/+m+ObN7AYx+vRKlueeTVjjfZ/0Ma++ru5fcvJeLkYdul+xSEa4HI5947uj2fu6UIT0hEr20m+/DBswsjb2Kg5lckSSZ1R0HvNU4QBKEHWVxwzyypY+6qwzTrDRes8+sfhdrOjqz95wzNePTHzt2JENdTpO0qxKATF1YFoa/79NNPiYuLa/X685//3NvN6lEWNxWyoKqRr5NyCfWwY9bYsFbrrKytCR0yjKwD+zAaDCisrECSoN8Uoku+5GTdArJSSuiXIG5qEoS+7KGHHuKhhx7q7Wb0KovruY/p58GE/p68uzmT0trmC9aHxw+nqbaGgoy0swsjp+BvnYyjk5HU7WJoRhCEvs/igjvAgqkDaNYbeHNDxgXrggcNwUqpbJXjnaCRSDaORHmnUnCiioqC+h5srSAIQs+zyOAe6mHPAyOC+Topl6P51a3WqTS2BMXEkbl/jznHO1ZKiJjIgOZPUVhLpIppkYIg9HEWGdwB/jIhAhdbFS+uOcb50znD4odTXVJMWW7O2YX9pqBpOkVYpBXpe4rQaS+8ICsIgtBXWGxwd9Io+fukfuzLrmDdkdbJgsKGDgdJapXjnYiJIFkR7XEAbaOezCSRClgQLIkl53N/5ZVXenyfFhvcAX6XEEh/bwdeWZdGk+5sT9zO2QXfiP6tc7xrXCBoJD7lX+PiY8fRbWJoRhCEntEbwd3ipkKey0oh8dwtA7l32V4+3nGS2ddHmNeFxQ9nx39XUFNWiqO7h2lh5BSkX+YTnWDDjjXllOTU4Bnk2EutFwTLVPTKKzSndW0+d/WA/njPn99huSvN515XV8df/vIXkpKSkCSJhQsXcuedd/Lll1/yyiuvIMsyU6dO5fXXXwfA3t6eOXPmsHbtWjQaDT/88ANeXl4UFxfzyCOPcPLkSQA++OADRo4cyW233UZubi5NTU3MmTOHmTNnMm/ePBobG4mLiyMqKoqVK1de+R+sEyy65w4wMsydG6O8eH9rFsU1Z7O5hSeMACDrnBzvRE4xvdnvxVqpEHesCoKFudJ87i+99BJOTk4cOXKEw4cPc/3111NQUMDcuXPZvHkzKSkp7N+/n9WrVwNQX19PYmIihw4dYsyYMSxbtgyAxx9/nLFjx3Lo0CEOHDhAVFQUAMuXLyc5OZmkpCSWLFlCeXk5r732GhqNhpSUlB4L7GDhPfczFtw0kIlvbeP19em8dXccAK6+frj6+pOZtJfBk28xFXQNBfdI1NlriUh4neP7ixl5ZzhqTZ/4MwhCj+hMD7u7XGk+919//ZWvvvrK/N3FxYXt27czbtw4PDxMv/BnzJjB9u3bue2221CpVOYx/KFDh7Jx40YANm/ezGeffQaAlZUVTk5OACxZsoTvv/8egNzcXE6cOIGbm1tX/gk6zeJ77gCBbrb84boQvjuQT0pulXl5eEIieceO0FRXd7Zw5BTI2UlUohP6ZgPH97Z/phcE4epxpfncZVm+oHx7yROVSqW5vJWVVbv527du3cqvv/7K7t27OXToEIMHD74gN3xP6hPBHWD29eG426t5cU2q+T9WeMIIjAYDpw7uP1swcgoY9Xg278Yj0IHUHfnt/scVBOHqcaX53CdNmsR7771n/l5ZWcnw4cPZtm0bZWVlGAwGvvzyS8aOHdtuPRMmTOCDDz4AwGAwUFNTQ3V1NS4uLtja2pKens6ePWeHhJVKJTqd7rKO+XL1meBur7bmqRsjOXC6ih8PmcbSvcMisHNxbZ3j3T8BbN2Qjv9M1GhfyvPrKTpZ00utFgThUlxpPvdnnnmGyspKc073LVu24OPjw6uvvsr48eMZNGgQQ4YM4dZbb223nnfeeYctW7YQExPD0KFDSU1NZfLkyej1emJjY3n22WdJTEw0l585c6Y5X3xP6VP53I1GmVuX7qSsrplNfx+LrcqajcveI+23bTy2bCXWKpWp4OrHIH0t2r8cZ8WCvYQO8mDiQwO7tC2C0JeIfO69o1vzuUuSFCBJ0hZJktIkSUqVJGlOG2UkSZKWSJKUKUnSYUmS2p+P1E0ULVMjC6ub+Pc20xSl8PhEdE2NnE49dLZgv8nQVI2qZD/9h3uTmVxCU13P/mQSBEHoTp0ZltEDf5dleQCQCPxZkqTzu7lTgIiW10zggy5t5SVICHbl5lgf/r09i/yqRgKiB6G00bTO8R52PVip4Ph6osb4YdAbSd9T2FtNFgShi4l87p0I7rIsF8qyfKDlcy2QBvidV+xW4DPZZA/gLEmST5e3tpOevmkAsgyv/ZyOtVJJyOB4spL3Ip95wrnaHkLGQMY63Hzt8Alz4uh2cWFVEPqKhx56iJSUlFavpUuX9nazetQlXVCVJCkYGAzsPW+VH5B7zvc8LjwBIEnSTEmSkiRJSiotLb20ll4CP2cNs8aEsuZQAUnZFYTHD6e+qpLCzONnC0VOgYqTUHaCqDF+VJc0kp9R2W1tEgRB6EmdDu6SJNkD/wP+Ksvy+dNL2ppoekE3WJblj2RZjpdlOf7MDQPd5ZFxYXg72vDCmmMEDRqKwsqKzHPvVu032fSesY6wIR6o7aw5Kh7kIQhCH9Gp4C5JkhJTYF8py/J3bRTJAwLO+e4P9GqktFVZM3dKJEfyq/kpo4qAqNjWicSc/ME7FjJ+xlppxYARPpxKKaW++sKnOwmCIFiazsyWkYBPgDRZlt+6SLEfgftbZs0kAtWyLPf6FcpbB/kRF+DMP3/JwD8ugcqCPMrzzxk9irwJ8vZBfRlRo/0wGmXSdvV6swVB6CHPP/88ixcv7u1mdIvO9NxHAfcB10uSlNLyukmSpEckSXqkpcw64CSQCSwDHuue5l4ahUJi4S0DKa1tZnujJwBZ597QFDkZZCOc2ICzly3+/V04tqMAo1FcWBUEwbJ1ZrbMb7IsS7Isx8qyHNfyWifL8oeyLH/YUkaWZfnPsiyHybIcI8ty196ddAUGB7pw+2A/Pk6pxCUwtPUDPHziwMEHMn4GIGq0H7UVTZxOLe+l1gqC0J4vvviCYcOGERcXx6xZszAYDNjb27NgwQIGDRpEYmIixcWmB/EUFxdz++23M2jQIAYNGsSuXbsAWLRoEZGRkUycOJGMjLPPYc7KymLy5MkMHTqU0aNHk56e3m49V7trIh3i3Mn9WX+0iExNMG7Ht1BXWYG9iytIkunC6pFvQd9MSJw7to4qUncUEBzj3tvNFoSr0o5vjlOWW9dxwUvgHmDP6Lv7tVsmLS2Nr7/+mp07d6JUKnnsscdYuXKlOS3vokWLeOqpp1i2bBnPPPOMOS3v999/j8FgoK6ujuTkZL766isOHjyIXq9nyJAhDB06FDClCPjwww+JiIhg7969PPbYY2zevLnNeiyBxQV3WZZJq0hjoFvn0wV4O9nw6LgwVvxUwgxZ5mTyPmIntsyWibwJkj+F7B1YhU9kwCgfDqzPobaiCQdXm246CkEQLtWmTZtITk4mISEBgMbGRjw9PS8pLe+OHTu4/fbbsbW1BWDatGmA6SEeu3bt4q677jLvr7m5+aL1WAKLC+6rM1fz3K7n+PcN/2ak78hObzdzTChf7ztNQ5kzJ/bvORvcQ8aA0tY0NBM+kYHX+ZK8PodjvxUwfFpoNx2FIFiujnrY3UWWZR544AFeffXVVssXL17c6bS8cGHaYACj0YizszMpKSld1+BeZnFZISdYDWTeL7Y8v2U+5Y2dHxu3UVox76YBZKiDyD6SgraxwbRCaWNKR5CxHmQZRzcNQdFuHNtZgMFg7KajEAThUk2YMIFVq1ZRUlICQEVFBTk5Oe2WPz8t75gxY/j+++9pbGyktraWNWvWAODo6EhISAjffvstYDqRHDp06KL1WAKLC+6K/BKGHKjh5nXlPLPzGYxy5wPwzbE+qMJiwKDn2P59Z1dEToGaPCg6AkD0aD8aqrVkHy7r6uYLgnCZBg4cyMsvv8ykSZOIjY3lhhtuoLDw4lOX20rLO2TIEO655x7i4uK48847GT16tLn8ypUr+eSTTxg0aBBRUVH88MMPF63HElhkyt/iN96g4pPlLL5DwdgZc7k/6v5Ob3skt5IfnvoTquABPPXqS6aFdaWwOALGPQ3j5mI0yny+YBcu3rZMmzP4stooCH2JSPnbO7o15e/VyHPOHGxiY/jLzwo+3/IWx8qPdXrbmAAXCByIPjuVrOJq00J7D9NDPI6bpkQqFBJRo33JTaukqqShOw5BEAShW1lkcJdUKvzefBMbKzV//UFm3pZ/0KDrfBCeMnUiaqOWd75Yf3Zh5GQoOAg1pqwJA0b5Iikkju0Q+WYEQbA8FhncAVQBAfi++CKhuVoS12WzaO+iTm8bO3wYWCupSj/IjhMt2SkjbzK9HzcFfDsnNaGD3EnbVYhBJy6sCoJgWSw2uAM43nQTznfdxW27jZz6dTVrT67t1HZKtQ2hcUOIaMzhpTWp6A1G8OgPzkGmWTMtosb40VSvI+tgSXcdgiAIQrew6OAO4DX/adShYTzxkxVLfn2R3JrcjjcCIhJGYKurpTI3m//uO226WzXyJji5FbT1APhHuuDkoeHo9vxuPAJBEISuZ/HBXaHR4Pf2W9hrJR7+oZGntv4DnaHj56GGDklAkhSMURXx1sbjVDVoTVMiDc2mAA9IComo0X4UZlZTXmAZtxwLgiBAHwjuADb9+uE9fwExWXpCfj7CuynvdriNraMTfv0HEtmcQ02jjn/9egKCRoLaCTLWmcv1H+mNwloiVTzIQxB61ZIlSxgwYAAuLi689tprl12Pvb19F7bqyvfVXe3pE8EdwPnuu3CYPJkZ22V2/LKcXfkdZ24LT0iktjCXewfa8/meHDLLmyBiIhz/BVqet6qxVxE+xJOMPYXomg3dfRiCIFzE+++/z7p166isrGTevHm93ZyrXp8J7pIk4fPiC6i8fXhyjYKXNs6jrLH9O0zD4hMBmGRXiq3KipfWpkG/KVBfCvnJ5nJRY/zQNhk4kVTcrccgCELbHnnkEU6ePMm0adN4++23mT17NgAPPvggjz/+OCNHjiQ0NJRVq1YBpkRgEyZMYMiQIcTExJjvNu3I1q1bGTt2LHfffTf9+vVj3rx5rFy5kmHDhhETE0NWVhYAOTk5TJgwgdjYWCZMmMDp06cBOHXqFCNGjCAhIYFnn322Vd1vvPEGCQkJxMbGsnDhwq7601yUxSUOa4+VoyP+b76JfsbvueeHCp7xW8D7Ez9AIbV9DnP28sY9MJjCw0nMmTCTl39KY0d8HKMlK9PQTIAp+5xPmBOuvnakbs9n4CjfnjwkQbjqbFnxESU5J7u0Ts+gUMY/OPOi6z/88EPWr1/Pli1bWLu29ay4wsJCfvvtN9LT05k2bRrTp0/HxsaG77//HkdHR8rKykhMTGTatGltJg0736FDh0hLS8PV1ZXQ0FAefvhh9u3bxzvvvMO7777Lv/71L2bPns3999/PAw88wPLly3n88cdZvXo1c+bM4dFHH+X+++9n6dKl5jo3bNjAiRMn2LdvH7IsM23aNLZv386YMWMu/4/WgT7Tcz9DExeH5xN/ZXiaAZt1v/H5sc/bLR+ekEh++jHuinYl1N2OhRvzMQaOMM93B9OvgqjRfpTk1FKSYxlJgwThWnHbbbehUCgYOHCg+UEdsiwzf/58YmNjmThxIvn5+eZ1HUlISMDHxwe1Wk1YWBiTJk0CICYmhuzsbAB2797NvffeC8B9993Hb7/9BsDOnTv5v//7P/PyMzZs2MCGDRsYPHgwQ4YMIT09nRMnTnTJ8V9Mn+q5n+H6hz9Qv3sPf/x1N08HvEW8dzxRblFtlg2PT2TP/74i91ASC6bG8Mf/JLHPeziJOW9CZTa4BAMQmejN7u8zSd2ej+d9jj13MIJwlWmvh90b1Gq1+fOZXFkrV66ktLSU5ORklEolwcHBNDU1XXJ9CoXC/F1W03e1AAAgAElEQVShUFw0nfC5vwja+nUgyzJPP/00s2bN6lQbukKf67kDSAoFvq+/hsrRmSdWG1mw8UnqdfVtlvUMCcPBzYOspD1c39+TMf08eOFEkGnlOTc0qTXW9Evw4vj+Ypob288XLQhC76qursbT0xOlUsmWLVvaTQ18OUaOHMlXX30FmE4k1113HQCjRo1qtfyMG2+8keXLl5uf4pSfn29OXdxd+mRwB7B2d8f/jTfwKTUwYfVpXtn7SpvlJEkiLH442YcOotc28+zUARzXelCiDm41JRJMF1b1WiPH9xb1wBEIgnC5ZsyYQVJSEvHx8axcuZL+/ft3af1Llizh008/JTY2ls8//5x33nkHMKUHXrp0KQkJCVRXV5vLT5o0iXvvvZcRI0YQExPD9OnTqa2t7dI2nc8iU/5eipK33qb8o494+zYFt/zpNW4Ju+WCMjmHU1i16BluffIZwhMSef7HVLz3vcos1c9IT50Em7OP1fr21f3odUZ+9+ywTl2cEYS+QKT87R3XXMrfS+Hxl9nYxA3i0fUSH/78IqdrTl9Qxn9gNGpbOzKT9gDw14kR7FEOQzLqkU/82qps1Bg/KgrqKcqqvqAeQRCEq0WfD+6SUonf4jfRKDU89n0j87ZcmJ7Aytqa0CEJZCXvw2gw4Gyr4vqJUymXHSjc/12rshHxXqhsrDi6Q+SbEQRLdeTIEeLi4lq9hg8f3tvN6lJ9PrgDqPz98H15EaH5BmK/P8K7By9MTxAWn0hTbQ0FGWkA3JsYQrIqAYfTW2huPnuVXam2IjLRh6zkUhrrtD12DIIgdJ2YmBhSUlJavfbu3dvbzepS10RwB3C8cRLO//c7bt0jc2DNJ+zM39lqfUjcEKysrc1DM9ZWCvwS78SBen75ufXdbVGjfTHojaTvFhdWhWtHb12fu1Zd6d/7mgnuAF5z56KMiGDOWonXfp7bKj2BSmNLYEwcmUl7zH/UqOtuQ4eSioM/UlrbbC7r5mePT7gTR7bkUVPW2OPHIQg9zcbGhvLychHge4gsy5SXl2NjY3PZdfT52TLna87K4uSdd3LUW8emJ67j/Uln0xMc/nU9G5e9xwNvvId7YDAADctvozTnGO9Hf8vrdw0y15N/vJJ17x9GBkbf3Y/+I7zF7Bmhz9LpdOTl5XX6RiDhytnY2ODv749SqWy1vLOzZTq8Q1WSpOXAzUCJLMvRbawfB/wAnGpZ9J0syy92ou29Qh0Whs+zz8KCZzj83W987v85D0Q9AEBY/HA2fryUzP17zMHdNuZmgk5vIfngXo6ODCbazzQt0q+fC/c8O4xNK9LY/Fka2YfLGDcjEo2DqrcOTRC6jVKpJCQkpLebIVyCzgzLrAAmd1BmhyzLcS2vqzawn+F0xx04Tp3K73bI/PTjW6SWpQJg5+yCT3g/87g7AP1Mh36L+hAvrjnW6mepo5uG254YzMg7w8k+WsaXL+0j+0j7mSgFQRB6QofBXZbl7UBFD7Slx0iShPcLz6P09WXOagPP/fI3c3qC8IQRFJ/MpKas5cHZTv7gHcv/OaeyL7uCdUdaX0SVFBKDbwjk7qcTsHVQ8dPSw2z9b4bI/S4IQq/qqguqIyRJOiRJ0s+SJLWdoesqY2VvT8Db/8KlHm75JpdFu18GTFkiAbKSz5kWFTkFj6pDDPc08sq6NJp0FwZuNz977poXz+BJgaTuyOfrl/dRdErc6CQIQu/oiuB+AAiSZXkQ8C6w+mIFJUmaKUlSkiRJSaWlpV2w6yujiYnG6+9PMjxDpul/P7Imaw2uvv64+PqTuf+coZnIKUiykZejC8ivamTZ9rZzWVspFYy8I5zb/zYYo0HmuzcOsHfNSQwGYw8dkSAIgskVB3dZlmtkWa5r+bwOUEqS5H6Rsh/Jshwvy3K8h4fHle66S7g+cD92Y0bz0CaZFT+8QE5NDuEJieQdO0JTfctDsX3iwMGHiMrfmBzlzftbsyiqvvisAd8I08XWyGFeJP2UzXf/TKayqO2slIIgCN3hioO7JEneUsscQEmShrXUWX6l9fYUSaHA97XXUDq78ufvGlmw8UmCBw/FaDBw6mDLVE1JMl1YzdzE/EkhGIwy/1yf3m69ao01Ex4cyOSZ0dSUNfHNov0c2Zon5gkLgtAjOgzukiR9CewGIiVJypMk6Y+SJD0iSdIjLUWmA0clSToELAF+J1tYBLN2dSVg8Zt4l8uM+DqVb2rXY+vkTGbSuePuN4GunsCaA/xxdAjfHczn4OnKDusOG+LJ754bhm8/F7Z/dZy17x6ivqq5w+0EQRCuxDV3E1N7Spcsoez9D3hnmoJRPrdTmZLBYx//F2ulEnRN8M8QiLuXuomvM37xVtzsVCyYOoCRYe5YKdq/gUmWZVK357NzVSZWSgXjZvQnfKhnDx2ZIAh9hUj5exncH3sM9ZDBzPoFttXtQNfUSO7RQ6aVShsIux4y1mOvsmLRbdEUVDVy3yf7GP36Zhb/kkF22cXH1SVJInqsP/c8MwwnDw2/LDvKxk9TaW7QXXQbQRCEyyWC+zkka2sC3nwTG7Utt20ux2gtceL8G5pq8qDoCJOivNm3YCLv3TuYCC8H3t+aybjFW7nrw118sz+Xuua2H8Xn7GXLHU8NJeHmEE7sL+Grl/aRl9Hx8I4gCMKlEMMybajdtIm8P89m7WAvrGzdeWLZN0gKBdSVwOJ+MO5pGDe31TZF1U18dzCPVUl5nCyrR6O0YkqMN3cNDWB4iCuKNoZtik/VsPHTVKpLGhk0MYDEW0OxVlr11GEKgmCBOjssI4L7RRS9/DJH1/7AoSAvhj/5GNcl3GRa8fENYNDCrG1tbifLMgdOV7EqOZe1hwqpbdYT4KrhziH+3DnEnwBX21bldc0Gdn2XydFt+bj62jHxoYF4BDh09+EJgmChRHC/QsbmZtLuvotf1AYKQo08/+K32KvsYcebsOlF+FsaOPq2W0ej1sAvqUWsSs5jZ1YZsgwjQt2YPtSfKTHe2KrO5m3LOVrO5s/SaKrXMXxaKHE3BLbZ2xcE4domgnsXaD55ii8f/xPVNgqK/jKYV8a8hlSaDu8nws1vQ/wfOl1XflUj3yXnsepAHjnlDdiprJga68Nd8QHEB7kgSRKNdVq2rcwg62ApPuFOTHxwII7umm48QkEQLI0I7l1k1z8XsTt5N3V2eQxe8CLTQm+BdwaBR3+Y8c0l1yfLMvuzK/k2KZefjhTSoDUQ7GbL9KH+3DHEHx8nG47vLWL7V8eRZRh9TwT9R/iIXPGCIAAiuHeZmrJSlv35IfoVlvP1lGZefuw7gnd9CEnLYe4pUNlddt31zXp+PlrEquRc9pysQJLgunB3pg/1Z6SPM799kUHBiSpC4zxErnhBEAAR3LvU50/9BV1WFpG5OXz0t0iWxT+G6os74Xf/hf5Tu2Qfp8sb+N+BPFYl55Ff1YiD2pqbY30YZVCRu70Qta2S6+/rT3BMm2l7BEG4RoibmLpQ+PCRVFpbYau1ZsLKdN4pTwK1E2Ss67J9BLrZ8sQN/djx1Hj++6fh3DDQi+9T8pl9IIsNvtAoyaZc8SvT0Ta1PYdeEAThDBHcOyE8YQQg0zT9dhJOyJR88Tk7QuIh42do7NobkBQKiZFh7rx1Txz7F0zk9TtjsHZV8xY17FfrOLqjgE8X7ub0cXHjkyAIFyeCeye4BwTh5OlFoTXYjhvL/Ztl/p2fR6muDv73MBi756lLDjZK7kkI5NtHRvLrP8YRfVMwG71kymua+eGtA7z+xh5ScipFpklBEC4gxtw7aetny0jZsI6Zbywl5+67KTJU8NmjPtxXmMqIuIfRTHq5R9phMMpsTy1m96pMnIq1FFkZSfFWEBHpSkKwK/FBLoR52Is58oLQR4kLql0s99gRvnnhaW55Yh7+VmqyH3iAgxHWrLzOSImbkRGuAxkfdS9j/cfipnHrkTYd2VPIji8zkJuNlChlDlvrSFMasLFXMjTQhfhgV+KDXYjxc8JGpDUQhD5BBPcuZjQY+GDWfYTEDeWm2X+n/OOPKVn8JgD1ThJJYTLb+ylIC1QQ7R3H+MDxjAsYR6hTaLe2q7FWS/qeIjL2FFKeXw8KaHBTcchax+6GBgwSqKwUxPg7ER9kCvhDg1xwtRPTKgXBEong3g3Wv/8vMpN28+hHK7GytkZXUkLd1q3UbfyF+l07kQ0Sehsl6f1s2RJYy8EwCXfvEMYHjGd84Hhi3WOxUnRfD7osr5aMPUUc31dMQ40Wla01mjAHCpwU7K2p43B+NTqD6b93mIcd8UGmnn18sCvBbrbiRilBsAAiuHeDE/t38+PiRUx/5mWCYuJarTOe2EH9G/dQVxNIbZ4SQ1kZskIiP8SBbcEN7AuTafZzY6z/WMYFjGOE7wg01t2TWsBoMJKbXknGniJOpZSi1xlx8tQQluCFIUDD4ap6krMrScqppLrRlE/e3V7F0CAXc8CP8nVCZS2utwvC1UYE926ga27i/YdnEHP9JK5/aNaFBZJXwJo5yCPm0OR7F7WbN1O3ZSvN6abnrVZ72rEnVM+uMD05QTYM8x/B9QHXM8Z/TLeN02sb9WQdLCFjTxH5x6sA8I1wJjLRm9A4D3LrmtifXUlSTgVJ2ZWcrmgAQG2tIC7A2dyzHxLogpNG2S1tFASh80Rw7yar33iZklNZ/Gnp8raHMdb8FZI/hemfQvQdAOjy86ndupW6LVup37MH9Hq0dioOh1uzI7iJQ6EK+gWYxunHB4wnxCmkW9peU97I8b3FZOwtoqq4ASulgtBB7kQm+hAwwAWFlYKSmiaScyrZn11Jck4FRwtqMBhl0zPCPR1agr2ph+/vohFDOYLQw0Rw7yZHt2zklw/f4fev/guv0PALC+i18J+boegI/HEjeEe3Wm2oq6d+507qNm+mbts2DFVVGK0UnArRsD24kaQICbvAEHOg745xelmWKcmuJWNPIceTimmu16NxVNFvmBf9E71x9z+bT75Bqyclt4qklmGcAzmV5qdMeTmqz47bB7nS38cBpZUYyhGE7iSCezdpqKnmw5n3MfyOexh194y2C9UWwUfjwEoFM7eCrWubxWSDgcaUFOq2bKF2y1a0WVkAlPnYsjOkmX1hUBHqypjAcYwPGE+ib2KXj9Mb9EZyjpaTsaeI7CNlGA0ybn52RA73od8wL+yc1a3LG2UyimrNwzhJ2RUUVDcBYK2QCHG3I8LLnghPB/N7sLstamsxFVMQuoII7t3oq4Vz0TbUc/8b7128UO5+WHETBI2CGavAyvriZVtoc3Ko3bKFus1baEhOBoOBRgcVyWGwO1TP8XANg4O7b5y+qU5HZnIx6XuKKD5VgyRBwABX+g03jc8r1W0H6IKqRpJyKkkvrOF4cR2ZJbXkVDRw5p+WlUIi2M3WHPDDPe3p5+VAiLudmH8vCJdIBPdulLT2e7Z9/gkPv/sxTp7eFy+Y/B9Y8ziMfBwmvXRJ+zBUV1O3fQd1W7ZQt2MHxtpaDEoF6SEqdoVoORCuIDB8MOMDxjPKbxRhTmFdOnxTVdxAxt4iMvYUUVvRhFJtRdgQDyITffCLcEbq4A7YJp2Bk6X1nCip5URxnfk9u7weY8s/OYUEQW52RHjam3v54Z6m4C+CviC0TQT3blRVVMgnc/7EuPv/xNCpt7ZfeO3fIOkTuPMTiJl+WfuTdToakpNNwzebt6DLzQWgwM+GnSHNpIQoKPbX0M8rmhj3GKLdTe/edt5XfMFTNsoUZlWRvqeIzOQSdE0G7F3U9BvuTf9Eb1y8Ly2ffbPewKmyelPAL67lREkdJ0rqyC6rR98S9SUJAl1tW4K+g+nd04EwT7tWjyYUhGuRCO7d7D9P/hkbBwfuWfha+wX1WvhsGhSkwMMbwTvmivYryzLarCxqN2+hbssWGlNSQJYxWkkU+WhI9dJy3NvICT8JrY8b0Z6x5mAf5R6Fk9rpsvet1xo4daiMjL1FnD5WgWyU8QxyIDLRh4gETzT2l3/Xq1ZvJLu8/mwvv8QU/E+V1ZtvvJIk8HfRmIZ3zgn84Z722KlF0BeuDSK4d7Pd//uSXd+sZMIfHyNu0k3tF64tbrnAag0zt130Auvl0JeX03DgAE2HD9N4+AiNR44gN5jmqms11uT4qTji2USmD5zwlXD2DTYH+2j3aPq79kdtpe5gLxeqr27mxH7TtMqy3DoUCgm//i4EDnQlKNoNZ6+uueNVZzCSU97Qqpd/oriWk6X1aA1Gczk/Z03L0I49gW52+LtoCHDR4Odsi0YlhniEvkME925m0Ov48a1XOZm8j0mzHifm+kntb5CXBJ9OgaCRMON/nbrAejlkgwHtyZOmQH/4MI1HDtOccRwMprTEtS42ZPrAUS8tmb4Sp32sCfKObDWcE+IUgkLq/JTG8vw682ybyiLTicXB1YbAKFcCo9zw7++CyqZrj1dvMHK6ooETJXVkltRxvNg0pp9VWkez3tiqrLu9Cj8XWwJcNPi72OLvoml5mT6L8X3Bkojg3gP0Oh0/LH6Z7EMHmPLYEwwcc337Gxz4HH6cDSP/Aj2UIhjA2NhIU1oajYcP09QS9HV5eQDIkkSZj4Y0Lz1p3noyfSQqfO0Y6BHTqofvZefVqX3VlDeSe6yCnKPl5KVXoms2oFBI+IQ7ERjlRmCUK25+9t1285PRKFNa10xeZQN5lY0tL9Pn3IoG8qsazcM8Z7jbqy8I+CL4C1erLgvukiQtB24GSmRZjm5jvQS8A9wENAAPyrJ8oKMd94XgDqDTNrP69RfJTT3CTY8/Sf+RY9rf4Ke/w/6Pr+gCa1fQV1TQdOQIjYcO03jEFPCN1dWmdSor8v1sOOLZxHEfmUxfCYWXJ9EeMcS0BP0otygcVA7t7sOgN1J0sprTqRXkpJZTnlcHgK2TyhToB7oSMMAVG7ueS2tgNMqU1J4b/FufBNoK/h4O6osGfj9nEfyFntWVwX0MUAd8dpHgfhPwF0zBfTjwjizLwzvacV8J7gC6pib+9+pCCo6nccsT84gYNvLihc+9wPrHDeAT23MNbYcsy+hOnzYP5zQdPkxTWhqyVgtAo6OaU35WHPJoItMXsnwkvLxCWw3nRLhEtDt+X1/VzOljFZw+Vk7usQqaG/RIEniFOLb06t3wDHTocJpldzIYZUpqm84G/oqWwF9lOgkUXELw93a0wdNBjbOtUqRpELpMlw7LSJIUDKy9SHD/N7BVluUvW75nAONkWS5sr86+FNwBtI0NrHrlOYqzMpn29/mEDR128cJ1JfDvsaZx9z9tBbueebjHpZK1WpoyjtN45DBNLT187cmT5vWVXrZkeBtJ9TSN3+d6SHi6+BPiFEKoUyihzqHmz+fP0jEajJTk1JKTWs7p1ApKcmpABht7JQEDTBdlAwa4Yut4deWdbyv4557T+y+oajRP6TxDZaXAw0GNl6MaTwcb03tL4PdytMHTUY2Xg404CQid0pPBfS3wmizLv7V83wTMlWW53cjd14I7QHNDPd++9Axlp09x21PPETxoyMUL5yWbLrAGDofff99tF1i7mqGmhqajR89esD18CENZOQCyBPUuGorcFJxybCbPxUihKxS6Shg8XQlyPRvsz7x723mjkBQ01mrJTavgdKqpZ99Ya0pF7BHoQGCUK0FRbniFOKK4ynPXGIwyxTWm4F9S20RxTTMltU2UtLwX1zRTXNNEbZP+gm3FSUDojJ4M7j8Br54X3J+SZTm5jbIzgZkAgYGBQ3Nycjrct6VprKvl2xfnU1mQz+3znicwup1hl4NfwA9/hhGz4cZFPdfILiTLMvqiIhoPHab5ZBba7Gy02TloT53CWFtrLmewUlDlpiLX2UCus54CV4lCV6j01ODsG0Kwc0tv3ymUYIdg7GrcKUyr4fSxcopO1iAbZVQaawIGuLSM17th73LpUzivFo1agyno15qCfUlNM8XnnQRKapqoEScB4TxiWKYXNdRU880LT1NdWsyd81/Ev3/UxQv/9CTsXwZ3fAyxd/VcI7uZLMsYKitNwf5UdkvQN72aT+dAs9ZcVqtSUOxmRa6znkIXKHCVKHZTIAX64eMdQYhNON5VoSjzXanLMtJQberVu/nZETjQNAPHJ9wZqz74cJFGrYHSWlPgP/ckUHrOyaC4g5OAu70Kd3s1buZ30zKPcz4726qwEg9Vtwg9GdynArM5e0F1iSzL7Qw4m/Tl4A5QX1XJ18/Po76qgukLXsYnIrLtggYdfHYr5Ce3XGAd1LMN7QWy0Yi+qMgU6M8N+qdOoc8vAOPZeer1dlbku8jku8gUukoUuoDCNRIPVSIe1eEoi53AKGGlkgjo70ZQlCt+kS5ddhOVpWjSGVr1/otrTL8KSmqaKKvXUl7XTFldM+V12guuCYApz4+r3dkTgbu9qiXwm04KHq1ODiqR5bMXdeVsmS+BcYA7UAwsBJQAsix/2DIV8j1gMqapkA91NN4OfT+4A9RWlPHN80/TWFvDXc8uajv/O0BdKXw0FiQrU4rgq/QCa0+QtVq0eXlnh3eys2nOPkXTqZPIpeWtypY7SuS6qSlx70ej/UBUiijUBtPdv7KNDqWvHucgFf79XAkL98fT3uOSbs7qi4xGmZomHWV1zZTVac0B/9zv5y5r0BrarMfBxvqCgH/mV4HHeb8Q7NXW19SJtruJm5iuEjVlJXz9/Dy0jY3c9ewiPIND2y6YfwCWT7a4C6w9yVBXj+50ztke/6lsGk5losvOQaprQAYaNR5UOodT6hZOtWMYBpUHAEa01CmzqXUpojGgAXWoNV7OHnjbeeNt6216t/PGWe0sAtE5GrT6VsH/zC+Atk4ElQ26NutQWytws1Phaq/C1U5t+nzey838rsZRI04G7RHB/SpSVVzE1y/Mw6DVcs/zr+HmH9h2wZT/wupHIfHPMPmVnm2kBZNlGUNVlXlsX5efj64gn8a8XGqLqqjSOlPtEEK1Uxi19v4gKUA2oGrKRzZkUW+VRandSUqcaqlxUSP5eOHg4YuX/dmg72PnYz4J2Kvse/uQr0p6g5GKem2bgb+sTktFfTMVDTrTe52W+ov8KrBWSDjbng34rvaqi5wQ1LjaqXCxVWJ9lc+i6koiuF9lKgvz+fr5eQDcvfA1XH392i647inY92+4/SMYdE8PtrDvkg0G9GVl6PILqM8poDirkuIiPaW1aiplV4wK0x2ymoYSnKuzcKrOwq7+JA3qckocDJQ6QamjRJkTlDpJNLjZYePlg4dj617/ub8CbKxtevmor35NOgMV9Voq6rWU17cE//qW4F+vpbxO22p9dWPbvwwAnG2VuNqeE/jtz3xW42qnvOAXgyXfVSyC+1WoPO80X7/wNFbW1tyz8DWcvX0uLGTQwWe3QX4S/OEX8I3r+YZeQwx6IyU5NRSkFlOQXkZxXhPNWtOQgJomnJvzcSxLx6noKPZ1eShk08Veo0KixsmaEkcodNBT5ghlThKljqYTgN7DGSdHD9w17q1ebho302cb03cntZMYgugkncFIVYOuJdg3nw3855wEzj0ZVDZoMbRx8RjARqnA1VaFS0uwd7E1/QI497urncp00mj5frWcEERwv0qV5pzimxfno7Sx4XfPv46jh+eFhepKTSmCJanlAqt7D7fy2iUbZSqLGyjMrKIws5rCrCpqylqeEauU8HADd5s6XPTFOFZlIRfmos3Px1BS0mqWD0CjnZIaR2vKHWSKbXWU2RupcJAod4AKB4kKe2iys8btYieAMy8b0zJbpW1v/Eks1pmLx+XnnASqGrRUNGiprNdSUa+jssF0EqhsKdPWlNIzbFVWppOAndIc/E0nBRWudi0nhpYTxply3TGrSAT3q1jxyUy+fXkBNvamh304uLURvM9cYA0YBvd9D1Y9l1xLaK2uspnCLFOwL8isojy/DmSQFBIeAfb4RDjjE2yPu4MWZU0JuoIC06u4GH1xCfriYnTFxRjKyy+o26C0osHZpuUkAMV2Ogo0zZTby6YTgANU2oPBSsLW2vbiJ4Azy2zccdW4olSIfy+X48yvg6oGU7CvbDjnJFB/zonhnDJt3W18hp3KyvxrwNlWhWvLr4Mx/TwYH9lGx64TRHC/yhVmZrDq5Wewc3bl7oWvYu/SxgM8Ur6E1Y/A8EdhSgdPfBJ6THOjnqKT1RSeqKIwq5riUzUYWnLIO3vZ4hPuhE+YMz5hTjh5aszDLrJWi760FF1xCfqS4pagX4K+qAhdydkTwZlkbWfIkoTOUUODi4YaR2sqHKDYVk+Bpok820bzr4BGNaZfe4CL2gU3jRtuGjec1c44qZxwUptejipH82cnlRPONs44qhxRWV1deXwshVZvpKpRS2UbJ4HKBt15JwUtVfU6HhoVzN8mXeTelw6I4G4B8tOP8b9XnsPRw5O7F76KrWMbj8D7eS7s/RBu/zcM+l3PN1LokEFnpOR0bctQjingNzeYenMaByXeoU54hznhE+qER5AD1u2M3Z6Z+aMvaenxFxWZgn5JcatfAoaqqgu2Ndqo0bnZ0+DcchJwlCi11VNiq6PQpol8VQOldgZ0yrbH+DXWGnPANwf/87+rnHBUm04OzmpnnNROl/Ukr2udLMuXfa1FBHcLkZt6mO9eewEXbx/ueu4VNA6OrQu0usC6HnwH905DhU6TjTIVRfUUZVVTlFVNYVY11aWNACisJTwDHfAONfXuvcOcLivzpbG52XQCKCoy9f6Li1tOAC2fi4vRlZSAvo0hA3s7ZDdn9C4OaJ1taXBSU+egpMpeotzOSKmtniJNMyVSHdW6Gqqaq9AbLz70YGNlYw74FzsxnDkRuKhdcLZxxlntjLVC3MtxOURwtyDZhw+y+p8v4uYfyF3PLsLG7rx51PVlpgussmy6wGrv0QutFK5EQ42WopOmYF90sprinBqMetP/e44eGnzO9O7DnHD1seuSnPay0YihshJ9aanpVVJ69nNZ2dnPpRObigEAABlrSURBVKXITU0XbP//7Z15dBzHfec/NfcBzAxukABB8BJvSZQoSpRkOZZki7ZkUZLp3Si7dpy17Odkfa5eDjm7ebvZ58heJ46dxNnEq9hJvNrEtiRblGVLjlaSJa8YkdRJgofEAyTuewYYYO6u/aMbgxlgAIIgwAEGv897/aq6u6pRaBLfX/Wvqn6l3G4c1dU4ampQ1ZWmMajwEwt6iAacRMpsDPo1/Z4kkdQIkUSESDJCJBFhOGEahaSRLNAyk3JXeVbsK9wVhNwhKjzTpO4KAu7Asl9hDCLuS44zbxziya9/hbo16/jIH/533L5JMyM634Tv3gENO+HjP5EB1iVOJmXQ1zZC1ylT7LtOh7Nhjl1eB/VrAtSvMwW/rjkw73vQ5qK1xohGJwzAJOHPPYzh4akPsNtxVFbiqKkxj1oztVdVY1QFiYe8jAScRMoU4UyUocQQ4Xg4P02EGYoPMRQfmtYg2JQtO0YwG2MQ8oQocy7clo7FQsR9CfLuoQP89M+/yooNG7nvof+Gy+PNL/DWD+DHn4brPwMf/FpxGiksCFprhvtjWTdO95kIA52j5qwcBVWNZWbvfn2Q+rVByis9RREtIx4n3T9Auq+3sAGwDENmYHDK1FAAe1UVjrpanLV1OOrqcNTW4Kwbz5vnCb+LSDJS2AhMMgbhRJhwPExaF3YbOZQj6wYaF/5KT+WUfIW7Ips6F3nHScR9iXLywK94+lv/g8Yt27j39/8Ip3vSSsdnHoJ//Wu452/g6vuL00jhspAYS9Fzdjgr9t1nh0knzCX7/pDb8tubvfvqVWXYF9ESfJ1Okx4czHEJ9ZpfBT09eTODMkNDU+oqtxtHbW2+EairNY1Aba1lCGqxucyxCq010VS0oPBnDcD4l0HC/DqIJCJoCmtfubPc/DrwVFDptgyBJ5TN5xqDSk8lXof3shpaEfclzPGXX+Bn3/4Gq7dfzT2/+19wuHIG3DJp+P490HbQHGBtmGG3J6GkMDIGAx2jE2J/OsLI4PgCKxu1zYGs375+bfCybjw+V4xk0hT97NTQnLUBvT1Zg6ATiSl17RUV+b3/2hwjYBkAe0VFQeHNGBnz6yA+xGB8MGsExvPjRmD8GEwMTjuo7La7878Ccr8ELAORaywudexAxH2Jc+SFX/CLv/kL1l5zHXc/+GXsjpw/1OwAq2ENsM5tMYSw9IkOxek+M0zX6TDdpyP0t0UxrCX3FfW+rM++bk2QypV+bEtwQw6tNUYkMml9QM+UL4FCi8SU0znR26+rxVlbOzE2UFNj3qupwRYIzNj71lozmhrNE/7B+GDWVVQoP5oaLfgsm7LxyW2f5PPXfH5O70PEvQR4619+xnOP/DUbdt3InV/4PeyOnEG17ADrtfDxJ2WAVQAglczQ25rjyjkTITFq9jgdbju1TeXUrQmYR3NwSW9VOJnsIrHe3oJrA8a/BHQsNqVudmbQNOKfHSQOhVC22fW6E5lE1lU0GB/My++o3cHNDTfP6fcUcS8RXv/5fl74+++w8cZb+NDnHsRmy1kA8/YP4YlPwbaPwB0PQ3ld8RoqLEq01kR6Y/S0DtNzdpiesxH626MYGfPv3h9yW0JvCn7t6gBO9+IIkLUQaK0xRkfzp4X2Fhgc7u3FiEanPsDpzE4PNY+pBsFZW4u9shJlX5j3KOJeQhza/zgvPfo9ttxyK3t++4v5PYcXvwa//BrYXbDrAbjpixJoTJiRdCpDf1vUFPtWU/DHg6MpBZUry3J69wEqVixNd86lYsRihQ1Ab9/EbKHePjKRyNTKNhuOqqppvwDcGzbgappmX4cLIOJeYhx4/J945YePsv3WD/D+T302X+AHTpsC//YPwemDGz4Duz8LvgLxagShALGR5ETvvnWY3tbhbAgFp8dO7epAXg/fHywdd86lYiSTZMbdQXm9//yvgczgoLkQEaj61APUPvjgnH6eiHsJ8qt//j6v/vgHXPWBO7ntP3xm6gBQ30l48WFo+TG4A3DD78Du3wFPgZg1gjAD2tCEe8dy3DnDDLRPDNaWVbqpaw5me/g1TeU4XaXrzpkPdCpFemCAdF8f9lAI16pVc3qOiHsJorXmpUe/x+GnnuDaO/fy3o89UHiEv6cFXvgTOPFT8ITgxs+ZC5/csj2cMHfSyQx9bVF6zkayoj8yYLlzbIqqBj91a4LZ3n1FnW9ewigI+Yi4lyhaa174h+/wxs+fYtfefdx8/29OP4Wr801T5N99FnxVpj/+ugfAJZs+CPPD2HAy67fvOWu6c5Jxc6GVy+ugdrU5O6e6sRx/0IUv6MYfdOGQXv6cEXEvYbTWPPfIt3n7uWfYve9+bvzov5u5QtshePFP4PTz4K+F9/wnuPa3wCn7fArzy/hOVrmDtQMdo+hJ2925vA5L7F34Am58QRf+8TTHCLi8jpKLDXOpiLiXONowePZv/4KWF5/j2rvu5Yb7/u3UaJKTOfeK2ZNvfRnKV8ItD8KOj4NDNmkQFo5UMkOkd4zRSJKxSJKx4YSVTzAWSTJqpelUgVg0Tpsp9gF3VvRzDYAv4MIfdOMtcy4bF5CI+zLAMDI897++zZHnf4HL6+XK2z/ItR/aS1ll1cwVz/wSXvgKtL0KwSZ47+/CVffLQiihaGitScYzjEUmhD9rAIaTefnxWTy5KJvCV+7M9vgnjMCEAfAFXfhD7iU/rVPEfRnR23qGQ/sf5+QrL2Oz29jy3tu47sP3UbGiYfpKWsPp/wvPfwU6X4eKNfBrfwDbPwo28YcKi5d0MpMn+Nl0ON8YxEaSTI4N5vY7WL21iubt1azaUrkk4u9MRsR9GRLu6ebwU09w9MV/IZNOc8X1N7Fr7z7q1q6fvpLW8M4zZk+++whUbTBFfut9MMtl1oKwGDEyBrGRVNbtMxpJ0H06QuvRAeLRFMqmWLEuSPP2alZvr6Ki3rck/Psi7suY0fAQbzzzFG8++zSJsVGatl/Nrr37aNp21fT/eQ0DTjwFLzwMfcehdgv82kOw+cPZTZcFoRQwDE1v6zCtR/ppPTLAQLsZZiBQ7aF5ezXN26tZuSGE3bk4Ozci7gKJsTHefu7nvPb0TxgND1G3dgO77tnH+utuyI9Rk4thQMsT8OJXYeBdqL8S3veHcMUdIvJCSTIyGOfc0QHOHemn7cQQmZSB021n1ZZKmrdXsXpb9Zz2uV0o5lXclVJ7gG8BduARrfVXJ93/BPB1oMO69Fda60dmeqaI++UjnUxy7OXnObT/ccLdXVSsaOC6uz/C5ve8D4dzGp9jJg1HfgS//CoMtZrb+73vy7DuVhF5oWRJJTN0nByi9Ygp9tEhM4587epymq80e/XVq4q7dd+8ibtSyg68A7wfaAcOAfdrrY/llPkEsFNr/dnZNlDE/fJjGBneffUAB5/8Eb1nT1NWUck1d97DVbfvweWdZmFTJgVv/h946esQaYOm3abIr7nl8jZeEC4zWmsGOqK0vj3AuaP9dJ8dBg3+oIvV26tp3l5F46bKyx5Fcz7FfTfwX7XWd1jnDwForR/OKfMJRNyXDFprzh95i4NP/ojzR9/C7fdz9Qfu4poPfhhfMFS4UjoJb/wjvPRnMNIJze+BW/8zNN1weRsvCEUiNpLkXMsArW8P0HZsgGQ8g91ho2FjyByU3VZFoNp74QddIvMp7vuAPVrrB6zzjwHX5wq5Je4PA32Yvfwvaa3bZnquiPvioPvUOxzc/xjvHjyAw+Fk263vZ+dd9xKsrS9cIRWH174HL38DRnth3W2mT77x2svbcEEoIpm0QdepsOm+OTpAuGcMgMqV/uzsm/o1AWwLsK/tfIr7R4E7Jon7Lq3153LKVAFRrXVCKfUZ4N9orW8t8KxPA58GaGpquvbcuXMX8zsJC8hgZzuH9j/BsZeeR2uDTTfewnV3f4Sa1WsKV0iOwaFH4P99E8YG4Io9sPOT0HS9RKEUlh3hnrHs7Juud8MYhl6wOfWX1S0zqbwdGNRaz/gXLj33xcnIYD+vPf0kbz/3DKl4jDU7drJr7z4aN28rXCExAq/+LbzylxAPg7JB3VZouhFW7zZT2SFKWEYkYmnajg3SeqSfcwswp34+xd2B6Wq5DXM2zCHgN7TWLTllVmitu6z8vcDva61ndMaKuC9uYtER3nr2aV7/+X5iI8OsvGIzu+7Zx9od1xXeQzI5Bu2H4PwBM4ZN+yFImZ+qVK61xN4S/Io1MuNGWBZMN6f+qttXcfO+DXN65nxPhfwQ8E3MqZDf1Vp/RSn1x8BhrfV+pdTDwN1AGhgEfltrfWKmZ4q4Lw1SiThHX3yOw089wXBfL1WNTezau4+NN96Sv2H3ZDIp6HrLFPrzB8wjNmTeK6uf6NWv3m0umJKQB8IyYHxOfeVKPyvXTzN54QLIIiZhXsmk07xz4GUOPvkY/W3nKK+uYedd97H91vfjdM8idLBhQP/JCbE/dwCG28177qDpq2/aDatvgpU7JFKlIEyDiLuwIGitOfvGYQ4++SM6ThzDWx5gx54Pc/Weu/CWlV/cw8LnTbEfF/z+d8zrDo+5aGr1blPwV+0C90U+WxBKFBF3YcHpOHGMg/sf48xrB3G6PazbeT2Nm7fRuHkrlQ2rLn7AaLR/old//hXTraMNUHao32767Jt2m6m/emF+KUFY5Ii4C5eN/vOtHH76J7S++RqjYdOv7ikP0LBxC42bt9K4aSu1a9Zhs1+kXz0xAm0HJwS/4zCkzT07qb5iQuibdkOoSQZphWWBiLtw2dFaE+7pouN4C+3HW+g40UK4pwsAp9vDiis2ZcW+fsNGnC73xf2AdMLcF/b8K1bv/l8hETHvBRossd8NNZsguMq8Zp9h0FcQliAi7sKiIDo4QMfJY7QfP0rH8Rb62s6B1tjsDurWrTfdOJu2snLj5gtvEzgZIwO9xybcOOcOQLR74r6yQ2Cl2asPNZmCH2qCkJUGGmXgVlhyiLgLi5J4NErnO8dpP36U9hMt9Jw+hZFJg1LUNDXTsGkrjZu30rBpK2UVlRf3cK3NQdqhs2YabrPS82bQs+EO04efRUH5inzBzxqAJgg2gnPhY4UIwsUg4i4sCVKJON2n3qH9eAvtJ1roeucEqYTpVw/VrzDFftNWGjZvJVS34tJCrWZSpsBPFv1svh10Jr9OWd3UHn9otXVtFbj8l/DbC8LFI+IuLEky6TR9rWesnv0xOk4eIz4yDIC/otIS+y00bNpKTVNz4dWyc//hMNKVI/htED6XYwTawEjl1/FVTe3xj5/7a8BXKRuPC/OKiLtQEmjDYLCz3RT74y10nDjGyEAfAG6/n4aNptA3bNpK/br12B0LKKSGYfr0sz3/c5MMwXnIJKbWcwdNkfdVmVM4fVUT59kj57onJPvXCtMi4i6ULMN9vVmffcfxFgY7zZWuDpebFeuvoHp1MxX1K6lY0UDFigbKq6un31ZwPjEMGO2bEPyxARgbtNJ+K7WujfYXNgRgBl/z5oi/v2qSIShwuPwyFXSZIOIuLBvGImE6Thyj/UQLnSePMdDRTioey963O52E6lZQsWJC8MfzvmCoOFumaW0GVhsbMIU+awRyj8nXB6eOCYxjdxc2Bt5Kc3Wvu8xMXeWFz2XW0JJBxF1YtmitGQ0PEe7qZLCrg6GuDoa6Ohnq6iDS00Umnc6WdXm9UwS/on4lFSsbcPsW2WCpYZjz+sd7/lOMQYGvhHhkds+2uyzRL2QAyibuZcuUgTtQuIyMMSwoIu6CUADDyDDS38dQZweDluAPdXUQ7u4k0tdr9qgtfMFQVvBD9SuptAxAsH7FxS/AKhZGBpJRc7VvwkqTIxc4Hy8/kn8+HsL5Qjg8Uw2Cq8ycVur0gcs3kc8eXuv6dNe84PTLojRmL+7ypoRlhc1mJ1hbT7C2nuar87cGTCeTRHq7zd5+pyn4Q12dnH3jcDasAgBKEaiuIWT59Stz3D2BmtqLD7OwkNjs5s5Y87E7ViZtCn2esRiewXjkXIv2QCpmHaNWOktjkff7OKcxEJMNh7+wMXF4rMNlpnY3OHIPj/kV4/CY50s4FLWIuyBYOFwuqhqbqGpsmnIvMTZGuNt084RzevwnfvUiibHRbDmb3UGwrp6K+hX4QxV4A0F81uG1Dl/QPF/QmT0Lgd0B3pB5zAdaFxb85NhEPnvErOtjk8rHIGnlY0M5963nTDdoPVtsjgsbAId7ahl7TtlChqR2C6y4cn7e4zSIuAvCLHD7fNStXU/d2vV517XWxIYjWZ/+uH8/3N1Jz5lTxEaGMTKFB0HdPj/eQMAyACF8hfLBEN5AYGkagwuhlNmrdvmAqoX5GUYmR+xHzfhE6ThkkmaattJMwrpnHdnzeIFrk85jQ9M/Jx0HCri+b/6SiLsgLGaUUviCIXzBEA2btky5rw2D+NgoseEIY8MRYhErtc7H85HebrpPnWRsOII2jAI/acIY+AIhywgETPEvN78G8r8SAqVnDOaCzW4N+l5k3KL5Qmsw0vlin0mYg9ELjIi7ICwgymbDW1aOt6ycypWNFyyfZwwiYWLDw5YRmMjHhsNEervpevcEsZHhCxoDT1k5Hn8Zbn9ZNu8pKzOvlZXh9ZfjLpu45nC5izM9tBRRypw9ZHdedgMj4i4Ii4i5GgPTEEQsAxC2jIBpDBKjUeLREcI9XcRHR0lEo2hd2CAA2B0OPGXlOcbAbxmEnGuWIfCUmUbDa92bcV9d4bIi/xKCsITJNQY0rJpVHW0YJOMx4tER4tEo8dEo8WjUNAKWIRhPE6NRooOD9LedIx6NkozNPMPF6fZkDYK7rAyP3zQEbp8fp8eLy+vF5fHg9Hhxejy4PF5cOXmndV9cSpeOiLsgLDOUzYbb58ft8xOsvbi6RiZDfNQyBOOGYdwQ5BiK8Wvh7k6r/Gg22udssNkdphHw5oq/B6fHZ6WmgTCNw3jeg9M7cd+sN1F/uX1VLK/fVhCES8Jmt2cHbS8WbRikkglS8TjJ2BjJeJxUPGaex2MkrXz+/TjJ+JiVxomN9GTLJeMx0onZT3W0Oxw43R7sLhcOpxOHy43dSs1zFw6ny7xv5R0uF3bnRHmHq0C98XI59exOJ06XG7vLeXniGhVAxF0QhMuCstmybhh/qGJenmkYGVLxBKl4LGsscsU/GRs3GOb1dDJJOpkgnUqRTibIWGk6mSQ+GiWdTJJJJc1yqSSZZIp0KnlJbbTZHflGweniytv3sPOue+flHUyHiLsgCEsWm82O2+fD7fMt2M/QhkEmnc4Kfp4BmHwtaywmGY9UKq+OPzhPC8FmQMRdEARhBpTNZrpbXEsrcqbsCCAIglCCiLgLgiCUICLugiAIJcisxF0ptUcpdVIpdUop9QcF7ruVUj+w7r+qlGqe74YKgiAIs+eC4q6UsgPfBj4IbAHuV0pNjpD0SWBIa70e+HPga/PdUEEQBGH2zKbnvgs4pbU+o7VOAv8M7J1UZi/wD1b+MeA2JZGHBEEQisZsxL0BaMs5b7euFSyjtU4DEQoEaFZKfVopdVgpdbivr29uLRYEQRAuyGzEvVAPfHL0+dmUQWv9Ha31Tq31zpqamtm0TxAEQZgDs1nE1A7khptrBDqnKdOulHIAQWBwpoe+9tpr/UqpcxfR1lyqgf451i1F5H3kI+9jAnkX+ZTC+1g9m0KzEfdDwAal1BqgA/h14DcmldkP/CZwANgHPK+1LrC31ARa6zl33ZVSh2ez+/dyQd5HPvI+JpB3kc9yeh8XFHetdVop9VngWcAOfFdr3aKU+mPgsNZ6P/B3wPeVUqcwe+y/vpCNFgRBEGZmVrFltNY/A3426dof5eTjwEfnt2mCIAjCXFmqK1S/U+wGLDLkfeQj72MCeRf5LJv3oS7gGhcEQRCWIEu15y4IgiDMwJIT9wvFuVlOKKVWKaVeUEodV0q1KKW+UOw2FRullF0p9YZS6qfFbkuxUUqFlFKPKaVOWP9Hdhe7TcVCKfUl62/kqFLqn5RSnmK3aaFZUuI+yzg3y4k08KDWejNwA/Afl/n7APgCcLzYjVgkfAt4Rmu9CbiKZfpelFINwOeBnVrrbZiz/kp+Rt+SEndmF+dm2aC17tJav27lRzD/eCeHhlg2KKUagTuBR4rdlmKjlAoAt2BOU0ZrndRah4vbqqLiALzWIksfUxdilhxLTdxnE+dmWWKFWd4BvFrclhSVbwK/BxjFbsgiYC3QB3zPclM9opTyF7tRxUBr3QH8KXAe6AIiWutfFLdVC89SE/dZxbBZbiilyoDHgS9qrYeL3Z5ioJS6C+jVWr9W7LYsEhzANcD/1FrvAEaBZTlGpZSqwPzCXwOsBPxKqX9f3FYtPEtN3GcT52ZZoZRyYgr7o1rrJ4rdniJyE3C3UqoV0113q1Lqfxe3SUWlHWjXWo9/yT2GKfbLkduBs1rrPq11CngCuLHIbVpwlpq4Z+PcKKVcmIMi+4vcpqJhxcz/O+C41vobxW5PMdFaP6S1btRaN2P+v3hea13yvbPp0Fp3A21KqY3WpduAY0VsUjE5D9yglPJZfzO3sQwGl2cVfmCxMF2cmyI3q5jcBHwMOKKUetO69mUrXIQgfA541OoInQF+q8jtKQpa61eVUo8Br2POMHuDZbBSVVaoCoIglCBLzS0jCIIgzAIRd0EQhBJExF0QBKEEEXEXBEEoQUTcBUEQShARd0EQhBJExF0QBKEEEXEXBEEoQf4/Kdlo8GFkyAwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f9e81a06ef0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "results.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bidirectional Encoder-Decoder RNN using \"Sum\" Merge mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final Model Loaded\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Inputs (InputLayer)          (None, 15)                0         \n",
      "_________________________________________________________________\n",
      "Embeddings (Embedding)       (None, 15, 344)           68456     \n",
      "_________________________________________________________________\n",
      "BD_RNN (Bidirectional)       (None, 512)               923136    \n",
      "_________________________________________________________________\n",
      "repeat_vector_1 (RepeatVecto (None, 21, 512)           0         \n",
      "_________________________________________________________________\n",
      "bidirectional_1 (Bidirection (None, 21, 512)           1181184   \n",
      "_________________________________________________________________\n",
      "Outputs (TimeDistributed)    (None, 21, 344)           176472    \n",
      "_________________________________________________________________\n",
      "Activation (Activation)      (None, 21, 344)           0         \n",
      "=================================================================\n",
      "Total params: 2,349,248\n",
      "Trainable params: 2,349,248\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "\n",
      "Train on 110288 samples, validate on 27573 samples\n",
      "Epoch 1/20\n",
      "110288/110288 [==============================] - 48s 435us/step - loss: 2.5659 - acc: 0.4700 - val_loss: nan - val_acc: 0.5444\n",
      "Epoch 2/20\n",
      "110288/110288 [==============================] - 45s 411us/step - loss: 1.5785 - acc: 0.5963 - val_loss: nan - val_acc: 0.6512\n",
      "Epoch 3/20\n",
      "110288/110288 [==============================] - 45s 412us/step - loss: 1.2168 - acc: 0.6697 - val_loss: nan - val_acc: 0.7090\n",
      "Epoch 4/20\n",
      "110288/110288 [==============================] - 45s 412us/step - loss: 0.9961 - acc: 0.7184 - val_loss: nan - val_acc: 0.7501\n",
      "Epoch 5/20\n",
      "110288/110288 [==============================] - 46s 413us/step - loss: 0.8484 - acc: 0.7524 - val_loss: nan - val_acc: 0.7842\n",
      "Epoch 6/20\n",
      "110288/110288 [==============================] - 45s 409us/step - loss: 0.7225 - acc: 0.7862 - val_loss: nan - val_acc: 0.8223\n",
      "Epoch 7/20\n",
      "110288/110288 [==============================] - 45s 410us/step - loss: 0.6107 - acc: 0.8197 - val_loss: nan - val_acc: 0.8546\n",
      "Epoch 8/20\n",
      "110288/110288 [==============================] - 45s 409us/step - loss: 0.5179 - acc: 0.8481 - val_loss: nan - val_acc: 0.8820\n",
      "Epoch 9/20\n",
      "110288/110288 [==============================] - 45s 409us/step - loss: 0.4370 - acc: 0.8727 - val_loss: nan - val_acc: 0.9054\n",
      "Epoch 10/20\n",
      "110288/110288 [==============================] - 45s 411us/step - loss: 0.3635 - acc: 0.8942 - val_loss: nan - val_acc: 0.9249\n",
      "Epoch 11/20\n",
      "110288/110288 [==============================] - 45s 411us/step - loss: 0.3048 - acc: 0.9124 - val_loss: nan - val_acc: 0.9404\n",
      "Epoch 12/20\n",
      "110288/110288 [==============================] - 45s 411us/step - loss: 0.2596 - acc: 0.9259 - val_loss: nan - val_acc: 0.9488\n",
      "Epoch 13/20\n",
      "110288/110288 [==============================] - 45s 411us/step - loss: 0.2255 - acc: 0.9356 - val_loss: nan - val_acc: 0.9550\n",
      "Epoch 14/20\n",
      "110288/110288 [==============================] - 45s 411us/step - loss: 0.1975 - acc: 0.9435 - val_loss: nan - val_acc: 0.9607\n",
      "Epoch 15/20\n",
      "110288/110288 [==============================] - 46s 414us/step - loss: 0.1807 - acc: 0.9483 - val_loss: nan - val_acc: 0.9587\n",
      "Epoch 16/20\n",
      "110288/110288 [==============================] - 46s 413us/step - loss: 0.1678 - acc: 0.9516 - val_loss: nan - val_acc: 0.9659\n",
      "Epoch 17/20\n",
      "110288/110288 [==============================] - 46s 413us/step - loss: 0.1515 - acc: 0.9561 - val_loss: nan - val_acc: 0.9663\n",
      "Epoch 18/20\n",
      "110288/110288 [==============================] - 46s 413us/step - loss: 0.1402 - acc: 0.9590 - val_loss: nan - val_acc: 0.9689\n",
      "Epoch 19/20\n",
      "110288/110288 [==============================] - 46s 413us/step - loss: 0.1307 - acc: 0.9615 - val_loss: nan - val_acc: 0.9711\n",
      "Epoch 20/20\n",
      "110288/110288 [==============================] - 46s 413us/step - loss: 0.1221 - acc: 0.9641 - val_loss: nan - val_acc: 0.9720\n",
      "new jersey est parfois calme pendant l' automne et il et il en avril <PAD> avril <PAD> <PAD> <PAD> <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "def bd_encdec_model_final(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):\n",
    "    \"\"\"\n",
    "    Build and train a model that incorporates embedding, encoder-decoder, and bidirectional RNN on x and y\n",
    "    :param input_shape: Tuple of input shape\n",
    "    :param output_sequence_length: Length of output sequence\n",
    "    :param english_vocab_size: Number of unique English words in the dataset\n",
    "    :param french_vocab_size: Number of unique French words in the dataset\n",
    "    :return: Keras model built, but not trained\n",
    "    \"\"\"\n",
    "    learning_rate = 1e-3\n",
    "    \n",
    "    # Model\n",
    "    X = Input(shape=input_shape[1:], name=\"Inputs\")\n",
    "    embedding = Embedding(english_vocab_size, french_vocab_size, input_length=input_shape[1], name=\"Embeddings\")(X)\n",
    "    BiDirect_RNN = Bidirectional(GRU(256, dropout=0.2, recurrent_dropout=0.4), name=\"BD_RNN\")(embedding)\n",
    "    decoder_input = RepeatVector(output_sequence_length)(BiDirect_RNN)\n",
    "    BiDirect_Decode_RNN = Bidirectional(GRU(256, return_sequences=True, dropout=0.2, recurrent_dropout=0.4), merge_mode=\"concat\")(decoder_input)\n",
    "    y = TimeDistributed(Dense(french_vocab_size), name=\"Outputs\")(BiDirect_Decode_RNN)\n",
    "    \n",
    "    model = Model(X, Activation(\"softmax\", name=\"Activation\")(y))\n",
    "    \n",
    "    model.compile(loss=sparse_categorical_crossentropy,\n",
    "                  optimizer=Adam(learning_rate),\n",
    "                  metrics=['accuracy'])  \n",
    "    \n",
    "    return model\n",
    "\n",
    "# tests.test_model_final(bd_encdec_model_final)\n",
    "\n",
    "print('Final Model Loaded')\n",
    "# TODO: Train the final model\n",
    "\n",
    "# Reshaping the input to work with a basic RNN\n",
    "tmp_x = pad(preproc_english_sentences, max_english_sequence_length)\n",
    "# tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2], 1))\n",
    "\n",
    "# Train the neural network\n",
    "final_bd_encdec_model = bd_encdec_model_final(\n",
    "    tmp_x.shape,\n",
    "    max_french_sequence_length,\n",
    "    english_vocab_size,\n",
    "    french_vocab_size)\n",
    "\n",
    "print(final_bd_encdec_model.summary())\n",
    "print()\n",
    "\n",
    "hist7 = final_bd_encdec_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=20, validation_split=0.2)\n",
    "\n",
    "# Print prediction(s)\n",
    "print(logits_to_text(final_bd_encdec_model.predict(tmp_x[:1])[0], french_tokenizer))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction (IMPLEMENTATION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Inputs (InputLayer)          (None, 15)                0         \n",
      "_________________________________________________________________\n",
      "Embeddings (Embedding)       (None, 15, 344)           68456     \n",
      "_________________________________________________________________\n",
      "BD_RNN (Bidirectional)       (None, 512)               923136    \n",
      "_________________________________________________________________\n",
      "repeat_vector_2 (RepeatVecto (None, 21, 512)           0         \n",
      "_________________________________________________________________\n",
      "bidirectional_2 (Bidirection (None, 21, 512)           3148800   \n",
      "_________________________________________________________________\n",
      "Outputs (TimeDistributed)    (None, 21, 344)           176472    \n",
      "_________________________________________________________________\n",
      "Activation (Activation)      (None, 21, 344)           0         \n",
      "=================================================================\n",
      "Total params: 4,316,864\n",
      "Trainable params: 4,316,864\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Train on 110288 samples, validate on 27573 samples\n",
      "Epoch 1/20\n",
      "110288/110288 [==============================] - 69s 628us/step - loss: 2.4463 - acc: 0.4827 - val_loss: nan - val_acc: 0.5813\n",
      "Epoch 2/20\n",
      "110288/110288 [==============================] - 67s 605us/step - loss: 1.4346 - acc: 0.6256 - val_loss: nan - val_acc: 0.6745\n",
      "Epoch 3/20\n",
      "110288/110288 [==============================] - 67s 605us/step - loss: 1.0861 - acc: 0.6978 - val_loss: nan - val_acc: 0.7436\n",
      "Epoch 4/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.8518 - acc: 0.7502 - val_loss: nan - val_acc: 0.7851\n",
      "Epoch 5/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.6891 - acc: 0.7919 - val_loss: nan - val_acc: 0.8326\n",
      "Epoch 6/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.5509 - acc: 0.8330 - val_loss: nan - val_acc: 0.8749\n",
      "Epoch 7/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.4321 - acc: 0.8693 - val_loss: nan - val_acc: 0.9081\n",
      "Epoch 8/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.3358 - acc: 0.8987 - val_loss: nan - val_acc: 0.9300\n",
      "Epoch 9/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.2656 - acc: 0.9204 - val_loss: nan - val_acc: 0.9439\n",
      "Epoch 10/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.2181 - acc: 0.9344 - val_loss: nan - val_acc: 0.9525\n",
      "Epoch 11/20\n",
      "110288/110288 [==============================] - 67s 605us/step - loss: 0.1857 - acc: 0.9439 - val_loss: nan - val_acc: 0.9584\n",
      "Epoch 12/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.1639 - acc: 0.9501 - val_loss: nan - val_acc: 0.9638\n",
      "Epoch 13/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.1470 - acc: 0.9549 - val_loss: nan - val_acc: 0.9667\n",
      "Epoch 14/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.1323 - acc: 0.9595 - val_loss: nan - val_acc: 0.9692\n",
      "Epoch 15/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.1216 - acc: 0.9626 - val_loss: nan - val_acc: 0.9708\n",
      "Epoch 16/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.1129 - acc: 0.9650 - val_loss: nan - val_acc: 0.9724\n",
      "Epoch 17/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.1039 - acc: 0.9676 - val_loss: nan - val_acc: 0.9740\n",
      "Epoch 18/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.0968 - acc: 0.9699 - val_loss: nan - val_acc: 0.9755\n",
      "Epoch 19/20\n",
      "110288/110288 [==============================] - 67s 605us/step - loss: 0.0902 - acc: 0.9719 - val_loss: nan - val_acc: 0.9770\n",
      "Epoch 20/20\n",
      "110288/110288 [==============================] - 67s 604us/step - loss: 0.0859 - acc: 0.9732 - val_loss: nan - val_acc: 0.9779\n",
      "Test score: nan\n",
      "Test accuracy: 0.98302327353\n",
      "Sample 1:\n",
      "il a vu un vieux camion jaune <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n",
      "Il a vu un vieux camion jaune\n",
      "Sample 2:\n",
      "new jersey est parfois calme pendant l' automne et il est neigeux en avril <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n",
      "new jersey est parfois calme pendant l' automne et il est neigeux en avril <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "def final_predictions(x, y, x_tk, y_tk):\n",
    "    \"\"\"\n",
    "    Gets predictions using the final model\n",
    "    :param x: Preprocessed English data\n",
    "    :param y: Preprocessed French data\n",
    "    :param x_tk: English tokenizer\n",
    "    :param y_tk: French tokenizer\n",
    "    \"\"\"\n",
    "    x = pad(x, x.shape[1])\n",
    "    \n",
    "    model = bd_encdec_model_final(x.shape, y.shape[1], english_vocab_size, french_vocab_size)\n",
    "    print(model.summary())\n",
    "    model.fit(x, y, batch_size=1024, epochs=20, validation_split=0.2)\n",
    "    \n",
    "    score = model.evaluate(x, y, verbose=0)\n",
    "    print('Test score:', score[0])\n",
    "    print('Test accuracy:', score[1])\n",
    "    \n",
    "    ## DON'T EDIT ANYTHING BELOW THIS LINE\n",
    "    y_id_to_word = {value: key for key, value in y_tk.word_index.items()}\n",
    "    y_id_to_word[0] = '<PAD>'\n",
    "\n",
    "    sentence = 'he saw a old yellow truck'\n",
    "    sentence = [x_tk.word_index[word] for word in sentence.split()]\n",
    "    # Not using encoder-decoder model, so change the max length to longest target output length.\n",
    "    sentence = pad_sequences([sentence], maxlen= x.shape[1], padding='post')\n",
    "    sentences = np.array([sentence[0], x[0]])\n",
    "    predictions = model.predict(sentences,len(sentences))\n",
    "    \n",
    "\n",
    "    print('Sample 1:')\n",
    "    print(' '.join([y_id_to_word[np.argmax(x)] for x in predictions[0]]))\n",
    "    print('Il a vu un vieux camion jaune')\n",
    "    print('Sample 2:')\n",
    "    print(' '.join([y_id_to_word[np.argmax(x)] for x in predictions[1]]))\n",
    "    print(' '.join([y_id_to_word[np.max(x)] for x in y[0]]))\n",
    "\n",
    "\n",
    "final_predictions(preproc_english_sentences, preproc_french_sentences, english_tokenizer, french_tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission\n",
    "When you're ready to submit, complete the following steps:\n",
    "1. Review the [rubric](https://review.udacity.com/#!/rubrics/1004/view) to ensure your submission meets all requirements to pass\n",
    "2. Generate an HTML version of this notebook\n",
    "\n",
    "  - Run the next cell to attempt automatic generation (this is the recommended method in Workspaces)\n",
    "  - Navigate to **FILE -> Download as -> HTML (.html)**\n",
    "  - Manually generate a copy using `nbconvert` from your shell terminal\n",
    "```\n",
    "$ pip install nbconvert\n",
    "$ python -m nbconvert machine_translation.ipynb\n",
    "```\n",
    "  \n",
    "3. Submit the project\n",
    "\n",
    "  - If you are in a Workspace, simply click the \"Submit Project\" button (bottom towards the right)\n",
    "  \n",
    "  - Otherwise, add the following files into a zip archive and submit them \n",
    "  - `helper.py`\n",
    "  - `machine_translation.ipynb`\n",
    "  - `machine_translation.html`\n",
    "    - You can export the notebook by navigating to **File -> Download as -> HTML (.html)**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate the html\n",
    "\n",
    "**Save your notebook before running the next cell to generate the HTML output.** Then submit your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[NbConvertApp] Converting notebook machine_translation.ipynb to html',\n",
       " '[NbConvertApp] Writing 449188 bytes to machine_translation.html']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Save before you run this cell!\n",
    "!!jupyter nbconvert *.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional Enhancements\n",
    "\n",
    "This project focuses on learning various network architectures for machine translation, but we don't evaluate the models according to best practices by splitting the data into separate test & training sets -- so the model accuracy is overstated. Use the [`sklearn.model_selection.train_test_split()`](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function to create separate training & test datasets, then retrain each of the models using only the training set and evaluate the prediction accuracy using the hold out test set. Does the \"best\" model change?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}