starter_notebook_gdrive_from_English.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/bamtak/masakhane/blob/master/starter_notebook_gdrive.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Igc5itf-xMGj"
   },
   "source": [
    "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x4fXCKCf36IK"
   },
   "source": [
    "## Note before beginning:\n",
    "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
    "\n",
    "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
    "\n",
    "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
    "\n",
    "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
    "\n",
    "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to masakhanetranslation@gmail.com\n",
    "\n",
    "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l929HimrxS0a"
   },
   "source": [
    "## Retrieve your data & make a parallel corpus\n",
    "\n",
    "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
    "\n",
    "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "id": "oGRmDELn7Az0",
    "outputId": "ee3c1dfa-8e85-4e8d-af68-a1b2da787446"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
      "\n",
      "Enter your authorization code:\n",
      "··········\n",
      "Mounted at /content/drive\n"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Cn3tgQLzUxwn"
   },
   "outputs": [],
   "source": [
    "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
    "# These will also become the suffix's of all vocab and corpus files used throughout\n",
    "import os\n",
    "source_language = \"en\"\n",
    "target_language = \"xh\" \n",
    "lc = False  # If True, lowercase the data.\n",
    "seed = 42  # Random seed for shuffling.\n",
    "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
    "\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "os.environ[\"tag\"] = tag\n",
    "\n",
    "# This will save it to a folder in our gdrive instead! \n",
    "!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n",
    "g_drive_path = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)\n",
    "os.environ[\"gdrive_path\"] = g_drive_path\n",
    "models_path = '%s/models/%s%s_transformer'% (g_drive_path, source_language, target_language)\n",
    "# model temporary directory for training\n",
    "model_temp_dir = \"/content/drive/My Drive/masakhane/model-temp\"\n",
    "# model permanent storage on the drive\n",
    "!mkdir -p \"$gdrive_path/models/${src}${tgt}_transformer/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "kBSgJHEw7Nvx",
    "outputId": "5bd42959-2ef0-4f81-8e90-9e3ae81ad6e6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/content/drive/My Drive/masakhane/en-xh-baseline\n"
     ]
    }
   ],
   "source": [
    "!echo $gdrive_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "id": "gA75Fs9ys8Y9",
    "outputId": "e0f08e8d-1662-4da5-b24c-8cc6f6bf79ec"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting opustools-pkg\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6c/9f/e829a0cceccc603450cd18e1ff80807b6237a88d9a8df2c0bb320796e900/opustools_pkg-0.0.52-py3-none-any.whl (80kB)\n",
      "\r",
      "\u001b[K     |████                            | 10kB 25.7MB/s eta 0:00:01\r",
      "\u001b[K     |████████                        | 20kB 33.7MB/s eta 0:00:01\r",
      "\u001b[K     |████████████▏                   | 30kB 38.2MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████▏               | 40kB 38.9MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████▎           | 51kB 40.7MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▎       | 61kB 43.3MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▎   | 71kB 44.3MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████████| 81kB 11.1MB/s \n",
      "\u001b[?25hInstalling collected packages: opustools-pkg\n",
      "Successfully installed opustools-pkg-0.0.52\n"
     ]
    }
   ],
   "source": [
    "#TODO: Skip for retrain\n",
    "# Install opus-tools\n",
    "! pip install opustools-pkg "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "xq-tDZVks7ZD",
    "outputId": "af093b69-c9f9-4041-b17e-a999ae9d40da"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-xh.xml.gz not found. The following files are available for downloading:\n",
      "\n",
      "   7 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-xh.xml.gz\n",
      " 263 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en.zip\n",
      "  78 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/xh.zip\n",
      "\n",
      " 348 MB Total size\n",
      "./JW300_latest_xml_en-xh.xml.gz ... 100% of 7 MB\n",
      "./JW300_latest_xml_en.zip ... 100% of 263 MB\n",
      "./JW300_latest_xml_xh.zip ... 100% of 78 MB\n"
     ]
    }
   ],
   "source": [
    "#TODO: Skip for retrain\n",
    "# Downloading our corpus\n",
    "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
    "\n",
    "# extract the corpus file\n",
    "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 578
    },
    "colab_type": "code",
    "id": "n48GDRnP8y2G",
    "outputId": "ac518c91-f4b8-44d9-da4a-fa879d29a514"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-01-31 11:19:59--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 277791 (271K) [text/plain]\n",
      "Saving to: ‘test.en-any.en’\n",
      "\n",
      "\r",
      "test.en-any.en        0%[                    ]       0  --.-KB/s               \r",
      "test.en-any.en      100%[===================>] 271.28K  --.-KB/s    in 0.01s   \n",
      "\n",
      "2020-01-31 11:19:59 (23.4 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
      "\n",
      "--2020-01-31 11:20:04--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-xh.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 206162 (201K) [text/plain]\n",
      "Saving to: ‘test.en-xh.en’\n",
      "\n",
      "test.en-xh.en       100%[===================>] 201.33K  --.-KB/s    in 0.007s  \n",
      "\n",
      "2020-01-31 11:20:04 (29.4 MB/s) - ‘test.en-xh.en’ saved [206162/206162]\n",
      "\n",
      "--2020-01-31 11:20:12--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-xh.xh\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 211374 (206K) [text/plain]\n",
      "Saving to: ‘test.en-xh.xh’\n",
      "\n",
      "test.en-xh.xh       100%[===================>] 206.42K  --.-KB/s    in 0.004s  \n",
      "\n",
      "2020-01-31 11:20:12 (55.4 MB/s) - ‘test.en-xh.xh’ saved [211374/211374]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#TODO: Skip for retrain\n",
    "# Download the global test set.\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
    "  \n",
    "# And the specific test set for this language pair.\n",
    "os.environ[\"trg\"] = target_language \n",
    "os.environ[\"src\"] = source_language \n",
    "\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
    "! mv test.en-$trg.en test.en\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
    "! mv test.en-$trg.$trg test.$trg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "NqDG-CI28y2L",
    "outputId": "ac7f26d3-d8ae-4532-cd4c-d52b82d8ae2f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 3571 global test sentences to filter from the training/dev data.\n"
     ]
    }
   ],
   "source": [
    "#TODO: Skip for retrain\n",
    "# Read the test data to filter from train and dev splits.\n",
    "# Store english portion in set for quick filtering checks.\n",
    "en_test_sents = set()\n",
    "filter_test_sents = \"test.en-any.en\"\n",
    "j = 0\n",
    "with open(filter_test_sents) as f:\n",
    "  for line in f:\n",
    "    en_test_sents.add(line.strip())\n",
    "    j += 1\n",
    "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 159
    },
    "colab_type": "code",
    "id": "3CNdwLBCfSIl",
    "outputId": "87da24e1-3f3b-48f9-fc87-b680e0ca1338"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded data and skipped 7158/876188 lines since contained in test set.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_sentence</th>\n",
       "      <th>target_sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How One Marriage Was Saved</td>\n",
       "      <td>Indlela Owasindiswa Ngayo Lo Mtshato</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>“ The application of the counsel in the book M...</td>\n",
       "      <td>Omnye umfundi onoxabiso nowaseMzantsi Afrika w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>“ Chapter 5 , ‘ A Wife Who Is Dearly Loved , ’...</td>\n",
       "      <td>“ Isahluko 5 , esithi ‘ Umfazi Othandwa Kunene...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     source_sentence                                    target_sentence\n",
       "0                         How One Marriage Was Saved               Indlela Owasindiswa Ngayo Lo Mtshato\n",
       "1  “ The application of the counsel in the book M...  Omnye umfundi onoxabiso nowaseMzantsi Afrika w...\n",
       "2  “ Chapter 5 , ‘ A Wife Who Is Dearly Loved , ’...  “ Isahluko 5 , esithi ‘ Umfazi Othandwa Kunene..."
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#TODO: Skip for retrain\n",
    "import pandas as pd\n",
    "\n",
    "# TMX file to dataframe\n",
    "source_file = 'jw300.' + source_language\n",
    "target_file = 'jw300.' + target_language\n",
    "\n",
    "source = []\n",
    "target = []\n",
    "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
    "with open(source_file) as f:\n",
    "    for i, line in enumerate(f):\n",
    "        # Skip sentences that are contained in the test set.\n",
    "        if line.strip() not in en_test_sents:\n",
    "            source.append(line.strip())\n",
    "        else:\n",
    "            skip_lines.append(i)             \n",
    "with open(target_file) as f:\n",
    "    for j, line in enumerate(f):\n",
    "        # Only add to corpus if corresponding source was not skipped.\n",
    "        if j not in skip_lines:\n",
    "            target.append(line.strip())\n",
    "    \n",
    "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
    "    \n",
    "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
    "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
    "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YkuK3B4p2AkN"
   },
   "source": [
    "## Pre-processing and export\n",
    "\n",
    "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
    "\n",
    "In addition we will split our data into dev/test/train and export to the filesystem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "id": "M_2ouEOH1_1q",
    "outputId": "5f6331f2-2d8e-49be-e5c1-35b67551e2b6"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:6: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  import sys\n"
     ]
    }
   ],
   "source": [
    "#TODO: Skip for retrain\n",
    "# drop duplicate translations\n",
    "df_pp = df.drop_duplicates()\n",
    "\n",
    "# drop conflicting translations\n",
    "# (this is optional and something that you might want to comment out \n",
    "# depending on the size of your corpus)\n",
    "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
    "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
    "\n",
    "# Shuffle the data to remove bias in dev set selection.\n",
    "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "Z_1BwAApEtMk",
    "outputId": "a4c76676-678c-4742-f958-966efe999c0a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting fuzzywuzzy\n",
      "  Downloading https://files.pythonhosted.org/packages/d8/f1/5a267addb30ab7eaa1beab2b9323073815da4551076554ecc890a3595ec9/fuzzywuzzy-0.17.0-py2.py3-none-any.whl\n",
      "Installing collected packages: fuzzywuzzy\n",
      "Successfully installed fuzzywuzzy-0.17.0\n",
      "Collecting python-Levenshtein\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz (48kB)\n",
      "\u001b[K     |████████████████████████████████| 51kB 6.0MB/s \n",
      "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein) (42.0.2)\n",
      "Building wheels for collected packages: python-Levenshtein\n",
      "  Building wheel for python-Levenshtein (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.0-cp36-cp36m-linux_x86_64.whl size=144675 sha256=c603320b54916b69dc5d26f92c63f9a5b5df8a2a0c4926898fe11de36cad266c\n",
      "  Stored in directory: /root/.cache/pip/wheels/de/c2/93/660fd5f7559049268ad2dc6d81c4e39e9e36518766eaf7e342\n",
      "Successfully built python-Levenshtein\n",
      "Installing collected packages: python-Levenshtein\n",
      "Successfully installed python-Levenshtein-0.12.0\n",
      "00:00:00.26 0.00 percent complete\n",
      "00:00:18.30 0.13 percent complete\n",
      "00:00:36.75 0.25 percent complete\n",
      "00:00:55.49 0.38 percent complete\n",
      "00:01:13.98 0.51 percent complete\n",
      "00:01:32.77 0.63 percent complete\n",
      "00:01:50.78 0.76 percent complete\n",
      "00:02:08.95 0.88 percent complete\n",
      "00:02:27.15 1.01 percent complete\n",
      "00:02:45.84 1.14 percent complete\n",
      "00:03:04.76 1.26 percent complete\n",
      "00:03:23.44 1.39 percent complete\n",
      "00:03:41.86 1.52 percent complete\n",
      "00:04:00.52 1.64 percent complete\n",
      "00:04:18.59 1.77 percent complete\n",
      "00:04:36.64 1.89 percent complete\n",
      "00:04:55.39 2.02 percent complete\n",
      "00:05:13.84 2.15 percent complete\n",
      "00:05:32.23 2.27 percent complete\n",
      "00:05:50.62 2.40 percent complete\n",
      "00:06:08.43 2.53 percent complete\n",
      "00:06:26.29 2.65 percent complete\n",
      "00:06:44.77 2.78 percent complete\n",
      "00:07:02.86 2.90 percent complete\n",
      "00:07:21.18 3.03 percent complete\n",
      "00:07:39.21 3.16 percent complete\n",
      "00:07:57.71 3.28 percent complete\n",
      "00:08:16.02 3.41 percent complete\n",
      "00:08:34.72 3.54 percent complete\n",
      "00:08:53.78 3.66 percent complete\n",
      "00:09:12.26 3.79 percent complete\n",
      "00:09:31.28 3.91 percent complete\n",
      "00:09:50.27 4.04 percent complete\n",
      "00:10:08.58 4.17 percent complete\n",
      "00:10:27.10 4.29 percent complete\n",
      "00:10:46.28 4.42 percent complete\n",
      "00:11:05.16 4.55 percent complete\n",
      "00:11:23.28 4.67 percent complete\n",
      "00:11:41.82 4.80 percent complete\n",
      "00:12:00.12 4.92 percent complete\n",
      "00:12:18.84 5.05 percent complete\n",
      "00:12:37.06 5.18 percent complete\n",
      "00:12:56.10 5.30 percent complete\n",
      "00:13:14.16 5.43 percent complete\n",
      "00:13:32.85 5.56 percent complete\n",
      "00:13:51.92 5.68 percent complete\n",
      "00:14:09.63 5.81 percent complete\n",
      "00:14:28.49 5.93 percent complete\n",
      "00:14:47.13 6.06 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '← ←']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:15:05.54 6.19 percent complete\n",
      "00:15:23.96 6.31 percent complete\n",
      "00:15:41.57 6.44 percent complete\n",
      "00:16:00.46 6.57 percent complete\n",
      "00:16:18.03 6.69 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '→ →']\n",
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '↓']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:16:35.78 6.82 percent complete\n",
      "00:16:54.71 6.94 percent complete\n",
      "00:17:12.70 7.07 percent complete\n",
      "00:17:30.78 7.20 percent complete\n",
      "00:17:49.67 7.32 percent complete\n",
      "00:18:08.25 7.45 percent complete\n",
      "00:18:26.55 7.58 percent complete\n",
      "00:18:45.16 7.70 percent complete\n",
      "00:19:04.12 7.83 percent complete\n",
      "00:19:22.51 7.95 percent complete\n",
      "00:19:40.97 8.08 percent complete\n",
      "00:19:59.66 8.21 percent complete\n",
      "00:20:18.00 8.33 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '● ● ● ● ●']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:20:35.81 8.46 percent complete\n",
      "00:20:54.57 8.59 percent complete\n",
      "00:21:12.73 8.71 percent complete\n",
      "00:21:30.86 8.84 percent complete\n",
      "00:21:49.29 8.96 percent complete\n",
      "00:22:07.06 9.09 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '⇩']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:22:25.55 9.22 percent complete\n",
      "00:22:43.84 9.34 percent complete\n",
      "00:23:02.31 9.47 percent complete\n",
      "00:23:20.83 9.60 percent complete\n",
      "00:23:39.24 9.72 percent complete\n",
      "00:23:57.72 9.85 percent complete\n",
      "00:24:16.26 9.97 percent complete\n",
      "00:24:33.77 10.10 percent complete\n",
      "00:24:52.62 10.23 percent complete\n",
      "00:25:10.83 10.35 percent complete\n",
      "00:25:28.97 10.48 percent complete\n",
      "00:25:47.46 10.61 percent complete\n",
      "00:26:06.37 10.73 percent complete\n",
      "00:26:24.78 10.86 percent complete\n",
      "00:26:43.19 10.98 percent complete\n",
      "00:27:01.43 11.11 percent complete\n",
      "00:27:19.17 11.24 percent complete\n",
      "00:27:37.29 11.36 percent complete\n",
      "00:27:55.97 11.49 percent complete\n",
      "00:28:14.30 11.62 percent complete\n",
      "00:28:33.50 11.74 percent complete\n",
      "00:28:52.37 11.87 percent complete\n",
      "00:29:10.65 11.99 percent complete\n",
      "00:29:28.52 12.12 percent complete\n",
      "00:29:47.24 12.25 percent complete\n",
      "00:30:05.14 12.37 percent complete\n",
      "00:30:23.28 12.50 percent complete\n",
      "00:30:41.87 12.63 percent complete\n",
      "00:31:00.26 12.75 percent complete\n",
      "00:31:18.06 12.88 percent complete\n",
      "00:31:36.79 13.00 percent complete\n",
      "00:31:55.37 13.13 percent complete\n",
      "00:32:13.30 13.26 percent complete\n",
      "00:32:31.21 13.38 percent complete\n",
      "00:32:49.14 13.51 percent complete\n",
      "00:33:07.20 13.64 percent complete\n",
      "00:33:25.65 13.76 percent complete\n",
      "00:33:44.25 13.89 percent complete\n",
      "00:34:02.41 14.01 percent complete\n",
      "00:34:20.65 14.14 percent complete\n",
      "00:34:39.05 14.27 percent complete\n",
      "00:34:57.30 14.39 percent complete\n",
      "00:35:15.94 14.52 percent complete\n",
      "00:35:34.37 14.65 percent complete\n",
      "00:35:52.91 14.77 percent complete\n",
      "00:36:11.10 14.90 percent complete\n",
      "00:36:29.55 15.02 percent complete\n",
      "00:36:48.46 15.15 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '↓ ↓ ↓ ↓']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:37:07.16 15.28 percent complete\n",
      "00:37:25.51 15.40 percent complete\n",
      "00:37:44.10 15.53 percent complete\n",
      "00:38:02.99 15.66 percent complete\n",
      "00:38:21.59 15.78 percent complete\n",
      "00:38:39.70 15.91 percent complete\n",
      "00:38:58.74 16.03 percent complete\n",
      "00:39:17.00 16.16 percent complete\n",
      "00:39:35.44 16.29 percent complete\n",
      "00:39:54.08 16.41 percent complete\n",
      "00:40:12.45 16.54 percent complete\n",
      "00:40:30.75 16.67 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '□ ․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:40:49.68 16.79 percent complete\n",
      "00:41:07.97 16.92 percent complete\n",
      "00:41:25.93 17.04 percent complete\n",
      "00:41:44.23 17.17 percent complete\n",
      "00:42:02.37 17.30 percent complete\n",
      "00:42:21.19 17.42 percent complete\n",
      "00:42:39.64 17.55 percent complete\n",
      "00:42:58.47 17.68 percent complete\n",
      "00:43:15.92 17.80 percent complete\n",
      "00:43:34.28 17.93 percent complete\n",
      "00:43:52.47 18.05 percent complete\n",
      "00:44:09.96 18.18 percent complete\n",
      "00:44:27.89 18.31 percent complete\n",
      "00:44:46.09 18.43 percent complete\n",
      "00:45:04.85 18.56 percent complete\n",
      "00:45:23.01 18.69 percent complete\n",
      "00:45:41.24 18.81 percent complete\n",
      "00:45:59.34 18.94 percent complete\n",
      "00:46:17.64 19.06 percent complete\n",
      "00:46:36.20 19.19 percent complete\n",
      "00:46:54.85 19.32 percent complete\n",
      "00:47:12.73 19.44 percent complete\n",
      "00:47:30.85 19.57 percent complete\n",
      "00:47:49.37 19.70 percent complete\n",
      "00:48:07.32 19.82 percent complete\n",
      "00:48:25.65 19.95 percent complete\n",
      "00:48:43.78 20.07 percent complete\n",
      "00:49:02.24 20.20 percent complete\n",
      "00:49:19.99 20.33 percent complete\n",
      "00:49:38.06 20.45 percent complete\n",
      "00:49:56.98 20.58 percent complete\n",
      "00:50:15.16 20.71 percent complete\n",
      "00:50:33.81 20.83 percent complete\n",
      "00:50:52.73 20.96 percent complete\n",
      "00:51:10.57 21.08 percent complete\n",
      "00:51:28.27 21.21 percent complete\n",
      "00:51:46.66 21.34 percent complete\n",
      "00:52:04.82 21.46 percent complete\n",
      "00:52:22.25 21.59 percent complete\n",
      "00:52:40.95 21.72 percent complete\n",
      "00:52:59.93 21.84 percent complete\n",
      "00:53:18.31 21.97 percent complete\n",
      "00:53:36.28 22.09 percent complete\n",
      "00:53:54.79 22.22 percent complete\n",
      "00:54:12.68 22.35 percent complete\n",
      "00:54:30.85 22.47 percent complete\n",
      "00:54:49.28 22.60 percent complete\n",
      "00:55:07.54 22.73 percent complete\n",
      "00:55:25.19 22.85 percent complete\n",
      "00:55:43.71 22.98 percent complete\n",
      "00:56:01.80 23.10 percent complete\n",
      "00:56:19.49 23.23 percent complete\n",
      "00:56:37.96 23.36 percent complete\n",
      "00:56:56.33 23.48 percent complete\n",
      "00:57:14.48 23.61 percent complete\n",
      "00:57:32.94 23.74 percent complete\n",
      "00:57:51.94 23.86 percent complete\n",
      "00:58:10.33 23.99 percent complete\n",
      "00:58:28.06 24.11 percent complete\n",
      "00:58:46.30 24.24 percent complete\n",
      "00:59:04.49 24.37 percent complete\n",
      "00:59:22.01 24.49 percent complete\n",
      "00:59:40.35 24.62 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '→ → →']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00:59:58.41 24.75 percent complete\n",
      "01:00:16.83 24.87 percent complete\n",
      "01:00:34.36 25.00 percent complete\n",
      "01:00:52.84 25.12 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '↓ ↓']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:01:11.30 25.25 percent complete\n",
      "01:01:29.91 25.38 percent complete\n",
      "01:01:48.57 25.50 percent complete\n",
      "01:02:06.23 25.63 percent complete\n",
      "01:02:23.88 25.76 percent complete\n",
      "01:02:42.35 25.88 percent complete\n",
      "01:03:00.93 26.01 percent complete\n",
      "01:03:19.04 26.13 percent complete\n",
      "01:03:37.06 26.26 percent complete\n",
      "01:03:56.18 26.39 percent complete\n",
      "01:04:13.94 26.51 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '▴']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:04:32.16 26.64 percent complete\n",
      "01:04:50.83 26.77 percent complete\n",
      "01:05:09.59 26.89 percent complete\n",
      "01:05:27.84 27.02 percent complete\n",
      "01:05:45.93 27.14 percent complete\n",
      "01:06:04.32 27.27 percent complete\n",
      "01:06:23.24 27.40 percent complete\n",
      "01:06:41.47 27.52 percent complete\n",
      "01:06:59.85 27.65 percent complete\n",
      "01:07:17.89 27.78 percent complete\n",
      "01:07:35.87 27.90 percent complete\n",
      "01:07:54.36 28.03 percent complete\n",
      "01:08:11.95 28.15 percent complete\n",
      "01:08:30.44 28.28 percent complete\n",
      "01:08:49.12 28.41 percent complete\n",
      "01:09:07.68 28.53 percent complete\n",
      "01:09:25.67 28.66 percent complete\n",
      "01:09:44.01 28.79 percent complete\n",
      "01:10:02.76 28.91 percent complete\n",
      "01:10:21.23 29.04 percent complete\n",
      "01:10:39.75 29.16 percent complete\n",
      "01:10:57.40 29.29 percent complete\n",
      "01:11:15.44 29.42 percent complete\n",
      "01:11:33.31 29.54 percent complete\n",
      "01:11:51.73 29.67 percent complete\n",
      "01:12:09.58 29.80 percent complete\n",
      "01:12:26.81 29.92 percent complete\n",
      "01:12:45.80 30.05 percent complete\n",
      "01:13:03.85 30.17 percent complete\n",
      "01:13:21.86 30.30 percent complete\n",
      "01:13:39.68 30.43 percent complete\n",
      "01:13:58.06 30.55 percent complete\n",
      "01:14:15.91 30.68 percent complete\n",
      "01:14:34.33 30.81 percent complete\n",
      "01:14:52.02 30.93 percent complete\n",
      "01:15:09.89 31.06 percent complete\n",
      "01:15:27.64 31.18 percent complete\n",
      "01:15:46.34 31.31 percent complete\n",
      "01:16:04.96 31.44 percent complete\n",
      "01:16:22.50 31.56 percent complete\n",
      "01:16:40.50 31.69 percent complete\n",
      "01:16:59.05 31.82 percent complete\n",
      "01:17:16.60 31.94 percent complete\n",
      "01:17:35.34 32.07 percent complete\n",
      "01:17:53.51 32.19 percent complete\n",
      "01:18:11.24 32.32 percent complete\n",
      "01:18:29.46 32.45 percent complete\n",
      "01:18:48.05 32.57 percent complete\n",
      "01:19:05.90 32.70 percent complete\n",
      "01:19:24.17 32.83 percent complete\n",
      "01:19:42.42 32.95 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '\\']\n",
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '”']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:20:00.72 33.08 percent complete\n",
      "01:20:18.63 33.20 percent complete\n",
      "01:20:37.14 33.33 percent complete\n",
      "01:20:56.16 33.46 percent complete\n",
      "01:21:14.58 33.58 percent complete\n",
      "01:21:32.73 33.71 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '↓ ↓ ↓']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:21:51.66 33.84 percent complete\n",
      "01:22:09.48 33.96 percent complete\n",
      "01:22:28.31 34.09 percent complete\n",
      "01:22:46.79 34.21 percent complete\n",
      "01:23:04.50 34.34 percent complete\n",
      "01:23:22.32 34.47 percent complete\n",
      "01:23:40.73 34.59 percent complete\n",
      "01:23:59.13 34.72 percent complete\n",
      "01:24:17.16 34.85 percent complete\n",
      "01:24:35.73 34.97 percent complete\n",
      "01:24:54.31 35.10 percent complete\n",
      "01:25:12.66 35.22 percent complete\n",
      "01:25:30.98 35.35 percent complete\n",
      "01:25:50.48 35.48 percent complete\n",
      "01:26:08.32 35.60 percent complete\n",
      "01:26:27.02 35.73 percent complete\n",
      "01:26:45.44 35.86 percent complete\n",
      "01:27:04.11 35.98 percent complete\n",
      "01:27:22.33 36.11 percent complete\n",
      "01:27:40.74 36.23 percent complete\n",
      "01:27:59.24 36.36 percent complete\n",
      "01:28:17.61 36.49 percent complete\n",
      "01:28:36.28 36.61 percent complete\n",
      "01:28:54.62 36.74 percent complete\n",
      "01:29:13.68 36.87 percent complete\n",
      "01:29:31.89 36.99 percent complete\n",
      "01:29:50.00 37.12 percent complete\n",
      "01:30:08.60 37.24 percent complete\n",
      "01:30:26.32 37.37 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:30:44.93 37.50 percent complete\n",
      "01:31:03.32 37.62 percent complete\n",
      "01:31:21.59 37.75 percent complete\n",
      "01:31:40.35 37.88 percent complete\n",
      "01:31:58.43 38.00 percent complete\n",
      "01:32:16.47 38.13 percent complete\n",
      "01:32:34.71 38.25 percent complete\n",
      "01:32:53.11 38.38 percent complete\n",
      "01:33:11.32 38.51 percent complete\n",
      "01:33:29.91 38.63 percent complete\n",
      "01:33:48.34 38.76 percent complete\n",
      "01:34:06.36 38.89 percent complete\n",
      "01:34:24.82 39.01 percent complete\n",
      "01:34:42.96 39.14 percent complete\n",
      "01:35:01.25 39.26 percent complete\n",
      "01:35:19.47 39.39 percent complete\n",
      "01:35:37.79 39.52 percent complete\n",
      "01:35:56.58 39.64 percent complete\n",
      "01:36:14.98 39.77 percent complete\n",
      "01:36:33.70 39.90 percent complete\n",
      "01:36:51.85 40.02 percent complete\n",
      "01:37:09.75 40.15 percent complete\n",
      "01:37:27.85 40.27 percent complete\n",
      "01:37:46.31 40.40 percent complete\n",
      "01:38:04.71 40.53 percent complete\n",
      "01:38:22.84 40.65 percent complete\n",
      "01:38:41.33 40.78 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '*']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:38:59.77 40.91 percent complete\n",
      "01:39:17.96 41.03 percent complete\n",
      "01:39:36.03 41.16 percent complete\n",
      "01:39:54.89 41.28 percent complete\n",
      "01:40:12.62 41.41 percent complete\n",
      "01:40:30.91 41.54 percent complete\n",
      "01:40:49.59 41.66 percent complete\n",
      "01:41:07.75 41.79 percent complete\n",
      "01:41:25.55 41.92 percent complete\n",
      "01:41:43.54 42.04 percent complete\n",
      "01:42:01.90 42.17 percent complete\n",
      "01:42:19.32 42.29 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '← ◯']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:42:37.35 42.42 percent complete\n",
      "01:42:55.14 42.55 percent complete\n",
      "01:43:12.62 42.67 percent complete\n",
      "01:43:30.48 42.80 percent complete\n",
      "01:43:48.69 42.93 percent complete\n",
      "01:44:06.85 43.05 percent complete\n",
      "01:44:25.16 43.18 percent complete\n",
      "01:44:43.78 43.30 percent complete\n",
      "01:45:02.07 43.43 percent complete\n",
      "01:45:20.62 43.56 percent complete\n",
      "01:45:39.07 43.68 percent complete\n",
      "01:45:57.55 43.81 percent complete\n",
      "01:46:15.46 43.94 percent complete\n",
      "01:46:33.88 44.06 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '. .']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:46:52.47 44.19 percent complete\n",
      "01:47:10.97 44.31 percent complete\n",
      "01:47:29.27 44.44 percent complete\n",
      "01:47:47.58 44.57 percent complete\n",
      "01:48:05.53 44.69 percent complete\n",
      "01:48:24.14 44.82 percent complete\n",
      "01:48:42.30 44.95 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '←']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:49:00.60 45.07 percent complete\n",
      "01:49:18.26 45.20 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '” ?']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:49:36.70 45.32 percent complete\n",
      "01:49:54.50 45.45 percent complete\n",
      "01:50:12.20 45.58 percent complete\n",
      "01:50:30.86 45.70 percent complete\n",
      "01:50:49.28 45.83 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '⇧']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:51:07.34 45.96 percent complete\n",
      "01:51:25.43 46.08 percent complete\n",
      "01:51:43.33 46.21 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '“ . . .']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01:52:02.06 46.33 percent complete\n",
      "01:52:20.58 46.46 percent complete\n",
      "01:52:39.06 46.59 percent complete\n",
      "01:52:57.72 46.71 percent complete\n",
      "01:53:16.06 46.84 percent complete\n",
      "01:53:34.55 46.97 percent complete\n",
      "01:53:52.84 47.09 percent complete\n",
      "01:54:11.16 47.22 percent complete\n",
      "01:54:29.73 47.34 percent complete\n",
      "01:54:48.02 47.47 percent complete\n",
      "01:55:05.99 47.60 percent complete\n",
      "01:55:23.97 47.72 percent complete\n",
      "01:55:41.84 47.85 percent complete\n",
      "01:56:00.41 47.98 percent complete\n",
      "01:56:19.21 48.10 percent complete\n",
      "01:56:37.38 48.23 percent complete\n",
      "01:56:56.06 48.35 percent complete\n",
      "01:57:14.87 48.48 percent complete\n",
      "01:57:33.83 48.61 percent complete\n",
      "01:57:52.18 48.73 percent complete\n",
      "01:58:10.20 48.86 percent complete\n",
      "01:58:28.81 48.99 percent complete\n",
      "01:58:46.96 49.11 percent complete\n",
      "01:59:05.29 49.24 percent complete\n",
      "01:59:23.44 49.36 percent complete\n",
      "01:59:42.58 49.49 percent complete\n",
      "02:00:01.06 49.62 percent complete\n",
      "02:00:19.01 49.74 percent complete\n",
      "02:00:37.00 49.87 percent complete\n",
      "02:00:55.86 50.00 percent complete\n",
      "02:01:13.49 50.12 percent complete\n",
      "02:01:32.34 50.25 percent complete\n",
      "02:01:50.90 50.37 percent complete\n",
      "02:02:09.28 50.50 percent complete\n",
      "02:02:27.40 50.63 percent complete\n",
      "02:02:45.42 50.75 percent complete\n",
      "02:03:03.81 50.88 percent complete\n",
      "02:03:21.97 51.01 percent complete\n",
      "02:03:40.40 51.13 percent complete\n",
      "02:03:58.36 51.26 percent complete\n",
      "02:04:16.06 51.38 percent complete\n",
      "02:04:34.50 51.51 percent complete\n",
      "02:04:52.84 51.64 percent complete\n",
      "02:05:10.95 51.76 percent complete\n",
      "02:05:29.48 51.89 percent complete\n",
      "02:05:48.30 52.02 percent complete\n",
      "02:06:05.78 52.14 percent complete\n",
      "02:06:23.75 52.27 percent complete\n",
      "02:06:42.23 52.39 percent complete\n",
      "02:07:00.75 52.52 percent complete\n",
      "02:07:18.87 52.65 percent complete\n",
      "02:07:36.78 52.77 percent complete\n",
      "02:07:54.98 52.90 percent complete\n",
      "02:08:12.68 53.03 percent complete\n",
      "02:08:31.78 53.15 percent complete\n",
      "02:08:49.44 53.28 percent complete\n",
      "02:09:07.26 53.40 percent complete\n",
      "02:09:26.06 53.53 percent complete\n",
      "02:09:44.04 53.66 percent complete\n",
      "02:10:02.64 53.78 percent complete\n",
      "02:10:20.60 53.91 percent complete\n",
      "02:10:38.96 54.04 percent complete\n",
      "02:10:57.45 54.16 percent complete\n",
      "02:11:15.15 54.29 percent complete\n",
      "02:11:33.86 54.41 percent complete\n",
      "02:11:52.01 54.54 percent complete\n",
      "02:12:09.84 54.67 percent complete\n",
      "02:12:28.19 54.79 percent complete\n",
      "02:12:46.29 54.92 percent complete\n",
      "02:13:04.60 55.05 percent complete\n",
      "02:13:23.06 55.17 percent complete\n",
      "02:13:42.04 55.30 percent complete\n",
      "02:14:00.18 55.42 percent complete\n",
      "02:14:18.53 55.55 percent complete\n",
      "02:14:36.84 55.68 percent complete\n",
      "02:14:55.07 55.80 percent complete\n",
      "02:15:13.59 55.93 percent complete\n",
      "02:15:32.03 56.06 percent complete\n",
      "02:15:50.49 56.18 percent complete\n",
      "02:16:08.96 56.31 percent complete\n",
      "02:16:26.84 56.43 percent complete\n",
      "02:16:44.79 56.56 percent complete\n",
      "02:17:02.97 56.69 percent complete\n",
      "02:17:20.93 56.81 percent complete\n",
      "02:17:39.29 56.94 percent complete\n",
      "02:17:57.57 57.07 percent complete\n",
      "02:18:15.79 57.19 percent complete\n",
      "02:18:34.46 57.32 percent complete\n",
      "02:18:53.15 57.44 percent complete\n",
      "02:19:11.55 57.57 percent complete\n",
      "02:19:29.43 57.70 percent complete\n",
      "02:19:47.81 57.82 percent complete\n",
      "02:20:05.82 57.95 percent complete\n",
      "02:20:23.74 58.08 percent complete\n",
      "02:20:42.73 58.20 percent complete\n",
      "02:21:01.20 58.33 percent complete\n",
      "02:21:19.38 58.45 percent complete\n",
      "02:21:37.95 58.58 percent complete\n",
      "02:21:56.78 58.71 percent complete\n",
      "02:22:14.46 58.83 percent complete\n",
      "02:22:32.90 58.96 percent complete\n",
      "02:22:50.99 59.09 percent complete\n",
      "02:23:09.38 59.21 percent complete\n",
      "02:23:27.74 59.34 percent complete\n",
      "02:23:46.13 59.46 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '— ― ― ― ― ― ― ―']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "02:24:04.49 59.59 percent complete\n",
      "02:24:23.45 59.72 percent complete\n",
      "02:24:41.83 59.84 percent complete\n",
      "02:24:59.95 59.97 percent complete\n",
      "02:25:18.12 60.10 percent complete\n",
      "02:25:36.83 60.22 percent complete\n",
      "02:25:55.70 60.35 percent complete\n",
      "02:26:13.94 60.47 percent complete\n",
      "02:26:32.21 60.60 percent complete\n",
      "02:26:50.83 60.73 percent complete\n",
      "02:27:09.13 60.85 percent complete\n",
      "02:27:26.57 60.98 percent complete\n",
      "02:27:45.01 61.11 percent complete\n",
      "02:28:03.29 61.23 percent complete\n",
      "02:28:21.60 61.36 percent complete\n",
      "02:28:39.73 61.48 percent complete\n",
      "02:28:58.26 61.61 percent complete\n",
      "02:29:16.97 61.74 percent complete\n",
      "02:29:35.42 61.86 percent complete\n",
      "02:29:53.92 61.99 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '●']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "02:30:12.10 62.12 percent complete\n",
      "02:30:30.69 62.24 percent complete\n",
      "02:30:49.54 62.37 percent complete\n",
      "02:31:07.55 62.49 percent complete\n",
      "02:31:26.06 62.62 percent complete\n",
      "02:31:45.09 62.75 percent complete\n",
      "02:32:03.63 62.87 percent complete\n",
      "02:32:22.35 63.00 percent complete\n",
      "02:32:40.79 63.13 percent complete\n",
      "02:32:58.94 63.25 percent complete\n",
      "02:33:17.32 63.38 percent complete\n",
      "02:33:36.80 63.50 percent complete\n",
      "02:33:54.42 63.63 percent complete\n",
      "02:34:13.00 63.76 percent complete\n",
      "02:34:31.95 63.88 percent complete\n",
      "02:34:50.72 64.01 percent complete\n",
      "02:35:08.84 64.14 percent complete\n",
      "02:35:27.01 64.26 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "02:35:44.65 64.39 percent complete\n",
      "02:36:03.03 64.51 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '↓ ↑']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "02:36:21.10 64.64 percent complete\n",
      "02:36:39.41 64.77 percent complete\n",
      "02:36:57.59 64.89 percent complete\n",
      "02:37:15.07 65.02 percent complete\n",
      "02:37:32.83 65.15 percent complete\n",
      "02:37:50.77 65.27 percent complete\n",
      "02:38:07.73 65.40 percent complete\n",
      "02:38:25.38 65.52 percent complete\n",
      "02:38:44.25 65.65 percent complete\n",
      "02:39:02.33 65.78 percent complete\n",
      "02:39:20.31 65.90 percent complete\n",
      "02:39:38.82 66.03 percent complete\n",
      "02:39:57.42 66.16 percent complete\n",
      "02:40:15.64 66.28 percent complete\n",
      "02:40:34.11 66.41 percent complete\n",
      "02:40:52.17 66.53 percent complete\n",
      "02:41:10.34 66.66 percent complete\n",
      "02:41:29.19 66.79 percent complete\n",
      "02:41:47.32 66.91 percent complete\n",
      "02:42:05.59 67.04 percent complete\n",
      "02:42:23.62 67.17 percent complete\n",
      "02:42:42.28 67.29 percent complete\n",
      "02:43:00.80 67.42 percent complete\n",
      "02:43:18.28 67.54 percent complete\n",
      "02:43:36.47 67.67 percent complete\n",
      "02:43:55.61 67.80 percent complete\n",
      "02:44:13.83 67.92 percent complete\n",
      "02:44:31.81 68.05 percent complete\n",
      "02:44:50.07 68.18 percent complete\n",
      "02:45:07.57 68.30 percent complete\n",
      "02:45:25.80 68.43 percent complete\n",
      "02:45:43.83 68.55 percent complete\n",
      "02:46:02.04 68.68 percent complete\n",
      "02:46:19.70 68.81 percent complete\n",
      "02:46:38.19 68.93 percent complete\n",
      "02:46:56.34 69.06 percent complete\n",
      "02:47:14.42 69.19 percent complete\n",
      "02:47:32.93 69.31 percent complete\n",
      "02:47:51.19 69.44 percent complete\n",
      "02:48:09.05 69.56 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "02:48:27.04 69.69 percent complete\n",
      "02:48:44.80 69.82 percent complete\n",
      "02:49:03.40 69.94 percent complete\n",
      "02:49:21.55 70.07 percent complete\n",
      "02:49:39.76 70.20 percent complete\n",
      "02:49:57.71 70.32 percent complete\n",
      "02:50:15.72 70.45 percent complete\n",
      "02:50:34.68 70.57 percent complete\n",
      "02:50:53.20 70.70 percent complete\n",
      "02:51:11.28 70.83 percent complete\n",
      "02:51:29.77 70.95 percent complete\n",
      "02:51:48.00 71.08 percent complete\n",
      "02:52:05.98 71.21 percent complete\n",
      "02:52:24.07 71.33 percent complete\n",
      "02:52:42.52 71.46 percent complete\n",
      "02:53:00.69 71.58 percent complete\n",
      "02:53:18.79 71.71 percent complete\n",
      "02:53:37.42 71.84 percent complete\n",
      "02:53:56.24 71.96 percent complete\n",
      "02:54:14.67 72.09 percent complete\n",
      "02:54:32.49 72.22 percent complete\n",
      "02:54:50.67 72.34 percent complete\n",
      "02:55:08.66 72.47 percent complete\n",
      "02:55:27.20 72.59 percent complete\n",
      "02:55:45.92 72.72 percent complete\n",
      "02:56:04.09 72.85 percent complete\n",
      "02:56:22.66 72.97 percent complete\n",
      "02:56:41.12 73.10 percent complete\n",
      "02:56:59.38 73.23 percent complete\n",
      "02:57:17.85 73.35 percent complete\n",
      "02:57:36.14 73.48 percent complete\n",
      "02:57:53.81 73.60 percent complete\n",
      "02:58:12.18 73.73 percent complete\n",
      "02:58:30.74 73.86 percent complete\n",
      "02:58:49.12 73.98 percent complete\n",
      "02:59:07.79 74.11 percent complete\n",
      "02:59:26.47 74.24 percent complete\n",
      "02:59:45.78 74.36 percent complete\n",
      "03:00:04.06 74.49 percent complete\n",
      "03:00:22.33 74.61 percent complete\n",
      "03:00:40.84 74.74 percent complete\n",
      "03:00:59.14 74.87 percent complete\n",
      "03:01:17.07 74.99 percent complete\n",
      "03:01:35.86 75.12 percent complete\n",
      "03:01:54.58 75.25 percent complete\n",
      "03:02:13.56 75.37 percent complete\n",
      "03:02:32.47 75.50 percent complete\n",
      "03:02:51.23 75.62 percent complete\n",
      "03:03:09.03 75.75 percent complete\n",
      "03:03:27.37 75.88 percent complete\n",
      "03:03:45.58 76.00 percent complete\n",
      "03:04:03.78 76.13 percent complete\n",
      "03:04:22.06 76.26 percent complete\n",
      "03:04:40.79 76.38 percent complete\n",
      "03:04:59.11 76.51 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '→']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "03:05:16.83 76.63 percent complete\n",
      "03:05:35.27 76.76 percent complete\n",
      "03:05:53.67 76.89 percent complete\n",
      "03:06:12.39 77.01 percent complete\n",
      "03:06:30.77 77.14 percent complete\n",
      "03:06:48.72 77.27 percent complete\n",
      "03:07:06.81 77.39 percent complete\n",
      "03:07:24.74 77.52 percent complete\n",
      "03:07:42.87 77.64 percent complete\n",
      "03:08:01.24 77.77 percent complete\n",
      "03:08:19.07 77.90 percent complete\n",
      "03:08:38.10 78.02 percent complete\n",
      "03:08:56.50 78.15 percent complete\n",
      "03:09:14.79 78.28 percent complete\n",
      "03:09:33.18 78.40 percent complete\n",
      "03:09:52.12 78.53 percent complete\n",
      "03:10:10.23 78.65 percent complete\n",
      "03:10:28.14 78.78 percent complete\n",
      "03:10:46.57 78.91 percent complete\n",
      "03:11:04.83 79.03 percent complete\n",
      "03:11:22.86 79.16 percent complete\n",
      "03:11:41.38 79.29 percent complete\n",
      "03:11:59.66 79.41 percent complete\n",
      "03:12:18.06 79.54 percent complete\n",
      "03:12:36.72 79.66 percent complete\n",
      "03:12:54.78 79.79 percent complete\n",
      "03:13:12.84 79.92 percent complete\n",
      "03:13:31.36 80.04 percent complete\n",
      "03:13:50.07 80.17 percent complete\n",
      "03:14:07.68 80.30 percent complete\n",
      "03:14:25.77 80.42 percent complete\n",
      "03:14:43.82 80.55 percent complete\n",
      "03:15:02.45 80.67 percent complete\n",
      "03:15:20.97 80.80 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "03:15:39.36 80.93 percent complete\n",
      "03:15:58.13 81.05 percent complete\n",
      "03:16:16.43 81.18 percent complete\n",
      "03:16:34.61 81.31 percent complete\n",
      "03:16:53.10 81.43 percent complete\n",
      "03:17:11.20 81.56 percent complete\n",
      "03:17:29.56 81.68 percent complete\n",
      "03:17:48.24 81.81 percent complete\n",
      "03:18:06.22 81.94 percent complete\n",
      "03:18:24.32 82.06 percent complete\n",
      "03:18:42.65 82.19 percent complete\n",
      "03:19:00.73 82.32 percent complete\n",
      "03:19:18.77 82.44 percent complete\n",
      "03:19:37.67 82.57 percent complete\n",
      "03:19:56.34 82.69 percent complete\n",
      "03:20:14.17 82.82 percent complete\n",
      "03:20:32.90 82.95 percent complete\n",
      "03:20:51.53 83.07 percent complete\n",
      "03:21:09.49 83.20 percent complete\n",
      "03:21:27.69 83.33 percent complete\n",
      "03:21:46.19 83.45 percent complete\n",
      "03:22:04.13 83.58 percent complete\n",
      "03:22:22.27 83.70 percent complete\n",
      "03:22:40.83 83.83 percent complete\n",
      "03:22:59.86 83.96 percent complete\n",
      "03:23:18.19 84.08 percent complete\n",
      "03:23:36.76 84.21 percent complete\n",
      "03:23:55.00 84.34 percent complete\n",
      "03:24:13.42 84.46 percent complete\n",
      "03:24:31.11 84.59 percent complete\n",
      "03:24:49.13 84.71 percent complete\n",
      "03:25:07.71 84.84 percent complete\n",
      "03:25:26.00 84.97 percent complete\n",
      "03:25:44.09 85.09 percent complete\n",
      "03:26:02.39 85.22 percent complete\n",
      "03:26:20.61 85.35 percent complete\n",
      "03:26:38.44 85.47 percent complete\n",
      "03:26:56.20 85.60 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '” *']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "03:27:14.59 85.72 percent complete\n",
      "03:27:32.88 85.85 percent complete\n",
      "03:27:51.27 85.98 percent complete\n",
      "03:28:09.32 86.10 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '← ← ← ← ← ← ← ← ← ←']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "03:28:27.87 86.23 percent complete\n",
      "03:28:45.90 86.36 percent complete\n",
      "03:29:04.25 86.48 percent complete\n",
      "03:29:22.24 86.61 percent complete\n",
      "03:29:40.38 86.73 percent complete\n",
      "03:29:58.83 86.86 percent complete\n",
      "03:30:16.88 86.99 percent complete\n",
      "03:30:35.07 87.11 percent complete\n",
      "03:30:54.09 87.24 percent complete\n",
      "03:31:11.99 87.37 percent complete\n",
      "03:31:29.91 87.49 percent complete\n",
      "03:31:47.80 87.62 percent complete\n",
      "03:32:05.70 87.74 percent complete\n",
      "03:32:23.91 87.87 percent complete\n",
      "03:32:42.28 88.00 percent complete\n",
      "03:33:00.62 88.12 percent complete\n",
      "03:33:18.49 88.25 percent complete\n",
      "03:33:36.64 88.38 percent complete\n",
      "03:33:54.61 88.50 percent complete\n",
      "03:34:12.63 88.63 percent complete\n",
      "03:34:30.96 88.75 percent complete\n",
      "03:34:49.25 88.88 percent complete\n",
      "03:35:07.86 89.01 percent complete\n",
      "03:35:26.65 89.13 percent complete\n",
      "03:35:45.36 89.26 percent complete\n",
      "03:36:03.59 89.39 percent complete\n",
      "03:36:21.08 89.51 percent complete\n",
      "03:36:39.09 89.64 percent complete\n",
      "03:36:57.04 89.76 percent complete\n",
      "03:37:14.77 89.89 percent complete\n",
      "03:37:32.94 90.02 percent complete\n",
      "03:37:50.85 90.14 percent complete\n",
      "03:38:08.87 90.27 percent complete\n",
      "03:38:26.71 90.40 percent complete\n",
      "03:38:44.79 90.52 percent complete\n",
      "03:39:02.66 90.65 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․ ․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "03:39:20.52 90.77 percent complete\n",
      "03:39:38.44 90.90 percent complete\n",
      "03:39:56.86 91.03 percent complete\n",
      "03:40:15.22 91.15 percent complete\n",
      "03:40:33.33 91.28 percent complete\n",
      "03:40:51.69 91.41 percent complete\n",
      "03:41:09.49 91.53 percent complete\n",
      "03:41:27.66 91.66 percent complete\n",
      "03:41:46.10 91.78 percent complete\n",
      "03:42:04.25 91.91 percent complete\n",
      "03:42:22.88 92.04 percent complete\n",
      "03:42:41.57 92.16 percent complete\n",
      "03:42:59.68 92.29 percent complete\n",
      "03:43:17.98 92.42 percent complete\n",
      "03:43:36.06 92.54 percent complete\n",
      "03:43:54.54 92.67 percent complete\n",
      "03:44:13.04 92.79 percent complete\n",
      "03:44:31.23 92.92 percent complete\n",
      "03:44:50.10 93.05 percent complete\n",
      "03:45:08.41 93.17 percent complete\n",
      "03:45:27.13 93.30 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '$ $ $']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "03:45:45.34 93.43 percent complete\n",
      "03:46:04.25 93.55 percent complete\n",
      "03:46:22.11 93.68 percent complete\n",
      "03:46:39.82 93.80 percent complete\n",
      "03:46:58.31 93.93 percent complete\n",
      "03:47:16.92 94.06 percent complete\n",
      "03:47:34.82 94.18 percent complete\n",
      "03:47:53.48 94.31 percent complete\n",
      "03:48:11.41 94.44 percent complete\n",
      "03:48:29.59 94.56 percent complete\n",
      "03:48:47.86 94.69 percent complete\n",
      "03:49:05.70 94.81 percent complete\n",
      "03:49:24.02 94.94 percent complete\n",
      "03:49:42.04 95.07 percent complete\n",
      "03:50:00.65 95.19 percent complete\n",
      "03:50:18.86 95.32 percent complete\n",
      "03:50:37.31 95.45 percent complete\n",
      "03:50:55.82 95.57 percent complete\n",
      "03:51:13.92 95.70 percent complete\n",
      "03:51:32.28 95.82 percent complete\n",
      "03:51:50.32 95.95 percent complete\n",
      "03:52:08.59 96.08 percent complete\n",
      "03:52:26.93 96.20 percent complete\n",
      "03:52:45.17 96.33 percent complete\n",
      "03:53:03.67 96.46 percent complete\n",
      "03:53:22.27 96.58 percent complete\n",
      "03:53:40.63 96.71 percent complete\n",
      "03:53:59.02 96.83 percent complete\n",
      "03:54:17.40 96.96 percent complete\n",
      "03:54:35.42 97.09 percent complete\n",
      "03:54:53.90 97.21 percent complete\n",
      "03:55:11.71 97.34 percent complete\n",
      "03:55:30.01 97.47 percent complete\n",
      "03:55:47.80 97.59 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '* * *']\n",
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "03:56:05.88 97.72 percent complete\n",
      "03:56:23.84 97.84 percent complete\n",
      "03:56:42.22 97.97 percent complete\n",
      "03:57:00.68 98.10 percent complete\n",
      "03:57:18.45 98.22 percent complete\n",
      "03:57:36.79 98.35 percent complete\n",
      "03:57:55.08 98.48 percent complete\n",
      "03:58:13.94 98.60 percent complete\n",
      "03:58:32.58 98.73 percent complete\n",
      "03:58:50.85 98.85 percent complete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '$ ․ ․ ․ ․ ․']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "03:59:09.34 98.98 percent complete\n",
      "03:59:28.37 99.11 percent complete\n",
      "03:59:46.59 99.23 percent complete\n",
      "04:00:04.88 99.36 percent complete\n",
      "04:00:22.93 99.49 percent complete\n",
      "04:00:41.14 99.61 percent complete\n",
      "04:00:59.34 99.74 percent complete\n",
      "04:01:17.44 99.86 percent complete\n",
      "04:01:35.66 99.99 percent complete\n"
     ]
    }
   ],
   "source": [
    "#TODO: Skip for retrain\n",
    "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
    "# test and training sets.\n",
    "! pip install fuzzywuzzy\n",
    "! pip install python-Levenshtein\n",
    "import time\n",
    "from fuzzywuzzy import process\n",
    "import numpy as np\n",
    "\n",
    "# reset the index of the training set after previous filtering\n",
    "df_pp.reset_index(drop=False, inplace=True)\n",
    "\n",
    "# Remove samples from the training data set if they \"almost overlap\" with the\n",
    "# samples in the test set.\n",
    "\n",
    "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
    "# within a certain length of characters of the given sample.\n",
    "def fuzzfilter(sample, candidates, pad):\n",
    "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
    "  if len(candidates) > 0:\n",
    "    return process.extractOne(sample, candidates)[1]\n",
    "  else:\n",
    "    return np.nan\n",
    "\n",
    "# NOTE - This might run slow depending on the size of your training set. We are\n",
    "# printing some information to help you track how long it would take. \n",
    "scores = []\n",
    "start_time = time.time()\n",
    "for idx, row in df_pp.iterrows():\n",
    "  scores.append(fuzzfilter(row['source_sentence'], list(en_test_sents), 5))\n",
    "  if idx % 1000 == 0:\n",
    "    hours, rem = divmod(time.time() - start_time, 3600)\n",
    "    minutes, seconds = divmod(rem, 60)\n",
    "    print(\"{:0>2}:{:0>2}:{:05.2f}\".format(int(hours),int(minutes),seconds), \"%0.2f percent complete\" % (100.0*float(idx)/float(len(df_pp))))\n",
    "\n",
    "# Filter out \"almost overlapping samples\"\n",
    "df_pp['scores'] = scores\n",
    "df_pp = df_pp[df_pp['scores'] < 95]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 819
    },
    "colab_type": "code",
    "id": "hxxBOCA-xXhy",
    "outputId": "2f101d21-f906-4ea0-a275-e60c71572361"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==> train.en <==\n",
      "Not long after my baptism , I married one of Jehovah’s Witnesses .\n",
      "Yet , he could not take sole blame for the horrors to come .\n",
      "How do we gain the help of holy spirit ?\n",
      "Faith moved a prostitute to risk her life so as to protect two Israelite spies .\n",
      "At some points , this wall stood 165 feet ( 50 m ) high .\n",
      "He loved me dearly , something I failed to realize at the time .\n",
      "His sister , Christine Antisdel , who had recently become one of Jehovah’s Witnesses , offered to move in with him and help him care for his two very young children .\n",
      "However , they should contain not only requests but also expressions of faith .\n",
      "At that point , any interest I had in winning the approval of my schoolmates would fade away .\n",
      "Jesus ’ brothers now urge him : “ Pass on over from here and go into Judea . ”\n",
      "\n",
      "==> train.xh <==\n",
      "Kungekudala ndibhaptiziwe , ndatshata omnye wamaNgqina kaYehova .\n",
      "Kodwa ke , yayingenguye yedwa owabangela iziphendu ezaziza kulandela .\n",
      "Silufumana njani uncedo lomoya oyingcwele ?\n",
      "Ukholo lwabangela unongogo ukuba abeke ubomi bakhe esichengeni ukuze akhusele iintlola ezimbini ezingamaSirayeli .\n",
      "Ezi ndonga zaziphakeme ngeyona ndlela .\n",
      "Wayendithanda kakhulu , nto leyo ndandingayiqondi ngelo xesha .\n",
      "Udadewalo , uChristine Antisdel , owayesandul ’ ukuba liNgqina likaYehova waya kuhlala nalo , ukuze alincedise ekunyamekeleni abantwana balo ababini ababesebancinane kakhulu .\n",
      "Noko ke , imithandazo yethu ayifanele ibe zizicelo kuphela kodwa ifanele ibonise nokholo esinalo .\n",
      "Oko kwakuyiphelisa tu into yokufuna ukukholisa abantwana besikolo .\n",
      "Ngoku abantakwaboYesu bayambongoza besithi : “ Gqitha apha , uye kwelakwaYuda . ”\n",
      "==> dev.en <==\n",
      "I was moved to fill out an application for the full - time preaching work .\n",
      "While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "It is fundamental in forging and maintaining good relations with people and with God .\n",
      "What kind of faith pleases Jehovah ?\n",
      "In fact , the risk is higher because most people are buying for the first time , they are bereaved and they must act quickly . ”\n",
      "Such unpredictability can make your life an emotional seesaw .\n",
      "The work of the Masoretes continues to benefit us today .\n",
      "They are not interested in learning about Jehovah or in serving him .\n",
      "Their motive was to look holier than their fellow men . ​ — Matthew 23 : 5 ; Numbers 15 : 38 - 40 .\n",
      "\n",
      "==> dev.xh <==\n",
      "Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "Lukholo olunjani olukholisa uYehova ?\n",
      "Enyanisweni , ingozi inkulu gqitha ngenxa yokuba inkoliso yabantu ithenga okwesihlandlo sokuqala , ifelwe ngumntu othandekayo yaye imele yenze izinto ngokukhawuleza . ”\n",
      "Ukungaqiniseki okunjalo kunokukubangela ungazinzi ngokweemvakalelo .\n",
      "Umsebenzi wamaMasorete uyaqhubeka usizisela iingenelo nanamhlanje .\n",
      "Kanye njengoyise , uSathana , abawexuki balwa nabantu abagcina ingqibelelo .\n",
      "Injongo yabo yayikukuba bakhangeleke bengcwele kunabanye abantu . — Mateyu 23 : 5 ; INumeri 15 : 38 - 40 .\n"
     ]
    }
   ],
   "source": [
    "#TODO: Skip for retrain\n",
    "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
    "# We use 1000 dev test and the given test set.\n",
    "import csv\n",
    "\n",
    "# Do the split between dev/train and create parallel corpora\n",
    "num_dev_patterns = 1000\n",
    "\n",
    "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
    "if lc:  # Julia: making lowercasing optional\n",
    "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
    "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
    "\n",
    "# Julia: test sets are already generated\n",
    "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
    "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
    "\n",
    "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in stripped.iterrows():\n",
    "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
    "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
    "    \n",
    "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in dev.iterrows():\n",
    "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
    "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
    "\n",
    "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
    "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
    "\n",
    "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
    "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
    "\n",
    "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
    "! head train.*\n",
    "! head dev.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "epeCydmCyS8X"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "## Installation of JoeyNMT\n",
    "\n",
    "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "iBRMm4kMxZ8L",
    "outputId": "a0577349-c596-45e3-bb1b-a990ba6256cb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'joeynmt'...\n",
      "remote: Enumerating objects: 72, done.\u001b[K\n",
      "remote: Counting objects: 100% (72/72), done.\u001b[K\n",
      "remote: Compressing objects: 100% (54/54), done.\u001b[K\n",
      "remote: Total 2256 (delta 43), reused 32 (delta 18), pack-reused 2184\u001b[K\n",
      "Receiving objects: 100% (2256/2256), 2.63 MiB | 17.50 MiB/s, done.\n",
      "Resolving deltas: 100% (1564/1564), done.\n",
      "Processing /content/joeynmt\n",
      "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.16.0)\n",
      "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (6.2.2)\n",
      "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.17.5)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (42.0.2)\n",
      "Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.3.1)\n",
      "Requirement already satisfied: tensorflow>=1.14 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.15.0)\n",
      "Requirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.3.1)\n",
      "Collecting sacrebleu>=1.3.6\n",
      "  Downloading https://files.pythonhosted.org/packages/45/31/1a135b964c169984b27fb2f7a50280fa7f8e6d9d404d8a9e596180487fd1/sacrebleu-1.4.3-py3-none-any.whl\n",
      "Collecting subword-nmt\n",
      "  Downloading https://files.pythonhosted.org/packages/74/60/6600a7bc09e7ab38bc53a48a20d8cae49b837f93f5842a41fe513a694912/subword_nmt-0.3.7-py2.py3-none-any.whl\n",
      "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (3.1.2)\n",
      "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.9.0)\n",
      "Collecting pyyaml>=5.1\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/3d/d9/ea9816aea31beeadccd03f1f8b625ecf8f645bd66744484d162d84803ce5/PyYAML-5.3.tar.gz (268kB)\n",
      "\u001b[K     |████████████████████████████████| 276kB 9.4MB/s \n",
      "\u001b[?25hCollecting pylint\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e9/59/43fc36c5ee316bb9aeb7cf5329cdbdca89e5749c34d5602753827c0aa2dc/pylint-2.4.4-py3-none-any.whl (302kB)\n",
      "\u001b[K     |████████████████████████████████| 307kB 66.7MB/s \n",
      "\u001b[?25hRequirement already satisfied: six==1.12 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.12.0)\n",
      "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.9.0)\n",
      "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.10.0)\n",
      "Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
      "Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.2.2)\n",
      "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.0.8)\n",
      "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
      "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.33.6)\n",
      "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.1.8)\n",
      "Requirement already satisfied: tensorflow-estimator==1.15.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.1)\n",
      "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.11.2)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n",
      "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
      "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.8.1)\n",
      "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (4.28.1)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (2.21.0)\n",
      "Requirement already satisfied: typing in /usr/local/lib/python3.6/dist-packages (from sacrebleu>=1.3.6->joeynmt==0.0.1) (3.6.6)\n",
      "Collecting portalocker\n",
      "  Downloading https://files.pythonhosted.org/packages/91/db/7bc703c0760df726839e0699b7f78a4d8217fdc9c7fcb1b51b39c5a22a4e/portalocker-1.5.2-py2.py3-none-any.whl\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.6.1)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (1.1.0)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.4.6)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (0.10.0)\n",
      "Requirement already satisfied: pandas>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (0.25.3)\n",
      "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (1.4.1)\n",
      "Collecting astroid<2.4,>=2.3.0\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ad/ae/86734823047962e7b8c8529186a1ac4a7ca19aaf1aa0c7713c022ef593fd/astroid-2.3.3-py3-none-any.whl (205kB)\n",
      "\u001b[K     |████████████████████████████████| 215kB 63.8MB/s \n",
      "\u001b[?25hCollecting isort<5,>=4.2.5\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e5/b0/c121fd1fa3419ea9bfd55c7f9c4fedfec5143208d8c7ad3ce3db6c623c21/isort-4.3.21-py2.py3-none-any.whl (42kB)\n",
      "\u001b[K     |████████████████████████████████| 51kB 9.5MB/s \n",
      "\u001b[?25hCollecting mccabe<0.7,>=0.6\n",
      "  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (0.16.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.1)\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.8->tensorflow>=1.14->joeynmt==0.0.1) (2.8.0)\n",
      "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (1.24.3)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2.8)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2019.11.28)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (3.0.4)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn->joeynmt==0.0.1) (2018.9)\n",
      "Collecting typed-ast<1.5,>=1.4.0; implementation_name == \"cpython\" and python_version < \"3.8\"\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/90/ed/5459080d95eb87a02fe860d447197be63b6e2b5e9ff73c2b0a85622994f4/typed_ast-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (737kB)\n",
      "\u001b[K     |████████████████████████████████| 747kB 56.3MB/s \n",
      "\u001b[?25hCollecting lazy-object-proxy==1.4.*\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0b/dd/b1e3407e9e6913cf178e506cd0dee818e58694d9a5cd1984e3f6a8b9a10f/lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55kB)\n",
      "\u001b[K     |████████████████████████████████| 61kB 11.3MB/s \n",
      "\u001b[?25hBuilding wheels for collected packages: joeynmt, pyyaml\n",
      "  Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for joeynmt: filename=joeynmt-0.0.1-cp36-none-any.whl size=72567 sha256=2afdf7a94c529b64e8f4d8f110b3a8edd4c3a8012bae7f0d7aa48108022c66b2\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-9w69c53m/wheels/db/01/db/751cc9f3e7f6faec127c43644ba250a3ea7ad200594aeda70a\n",
      "  Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for pyyaml: filename=PyYAML-5.3-cp36-cp36m-linux_x86_64.whl size=44229 sha256=6ca44824b48b45a4a1c55501abc4911c0390adcc3b9860cc07234ef1c1d60549\n",
      "  Stored in directory: /root/.cache/pip/wheels/e4/76/4d/a95b8dd7b452b69e8ed4f68b69e1b55e12c9c9624dd962b191\n",
      "Successfully built joeynmt pyyaml\n",
      "Installing collected packages: portalocker, sacrebleu, subword-nmt, pyyaml, typed-ast, lazy-object-proxy, astroid, isort, mccabe, pylint, joeynmt\n",
      "  Found existing installation: PyYAML 3.13\n",
      "    Uninstalling PyYAML-3.13:\n",
      "      Successfully uninstalled PyYAML-3.13\n",
      "Successfully installed astroid-2.3.3 isort-4.3.21 joeynmt-0.0.1 lazy-object-proxy-1.4.3 mccabe-0.6.1 portalocker-1.5.2 pylint-2.4.4 pyyaml-5.3 sacrebleu-1.4.3 subword-nmt-0.3.7 typed-ast-1.4.1\n"
     ]
    }
   ],
   "source": [
    "# Install JoeyNMT\n",
    "! git clone https://github.com/joeynmt/joeynmt.git\n",
    "! cd joeynmt; pip3 install .\n",
    "# Install Pytorch with GPU support v1.7.1.\n",
    "! pip install torch==1.7.1+cu101 -f https://download.pytorch.org/whl/torch_stable.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AaE77Tcppex9"
   },
   "source": [
    "# Preprocessing the Data into Subword BPE Tokens\n",
    "\n",
    "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
    "\n",
    "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
    "\n",
    "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 428
    },
    "colab_type": "code",
    "id": "H-TyjtmXB1mL",
    "outputId": "0f67d86a-f355-454e-81ec-dc1ba573a98e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bpe.codes.4000\tdev.en\t     test.bpe.xh     test.xh\t   train.en\n",
      "dev.bpe.en\tdev.xh\t     test.en\t     train.bpe.en  train.xh\n",
      "dev.bpe.xh\ttest.bpe.en  test.en-any.en  train.bpe.xh  vocab.txt\n",
      "bpe.codes.4000\tdev.en\t     test.bpe.xh     test.xh\t   train.en\n",
      "dev.bpe.en\tdev.xh\t     test.en\t     train.bpe.en  train.xh\n",
      "dev.bpe.xh\ttest.bpe.en  test.en-any.en  train.bpe.xh\n",
      "BPE Xhosa Sentences\n",
      "Oku kwa@@ ph@@ umela ekubeni nd@@ id@@ ume njengom@@ ntu ong@@ any@@ anis@@ ekanga .\n",
      "Xa nd@@ afunda inyaniso , and@@ izange nd@@ iph@@ inde nd@@ iv@@ ume ukuq@@ hubeka ndis@@ enza oko , naku@@ beni ndand@@ ihl@@ awul@@ wa um@@ v@@ uzo on@@ c@@ um@@ isayo .\n",
      "Nd@@ ing@@ umzekelo om@@ hle ko@@ onyana bam abab@@ ini yaye nd@@ iye nd@@ af@@ anel@@ ekela amalung@@ elo ang@@ akumbi ebandleni .\n",
      "Nd@@ az@@ iwa njengom@@ ntu on@@ yanisekileyo ngabantu end@@ ish@@ ish@@ ina kunye nabo kwan@@ abahl@@ ol@@ i boku@@ m@@ a kwem@@ ali ye@@ enk@@ amp@@ ani ( t@@ a@@ x a@@ ud@@ it@@ ors ) . ”\n",
      "U@@ R@@ ute wa@@ f@@ ud@@ ukela kwa@@ Sirayeli apho way@@ eza ku@@ kwazi uku@@ nq@@ ula oy@@ ena Thixo w@@ okwenyaniso .\n",
      "Combined BPE Vocab\n",
      "!@@\n",
      "Ο@@\n",
      "ȁ@@\n",
      "¥\n",
      "Ù@@\n",
      "̆\n",
      "ģ@@\n",
      "ź@@\n",
      "Í\n",
      "õ\n"
     ]
    }
   ],
   "source": [
    "#TODO: Skip for retrain\n",
    "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
    "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
    "\n",
    "# Do subword NMT\n",
    "from os import path\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "\n",
    "# Learn BPEs on the training data.\n",
    "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman! \n",
    "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt\n",
    "\n",
    "# Apply BPE splits to the development and test data.\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
    "\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
    "\n",
    "# Create directory, move everyone we care about to the correct location\n",
    "! mkdir -p $data_path\n",
    "! cp train.* $data_path\n",
    "! cp test.* $data_path\n",
    "! cp dev.* $data_path\n",
    "! cp bpe.codes.4000 $data_path\n",
    "! ls $data_path\n",
    "\n",
    "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
    "! cp train.* \"$gdrive_path\"\n",
    "! cp test.* \"$gdrive_path\"\n",
    "! cp dev.* \"$gdrive_path\"\n",
    "! cp bpe.codes.4000 \"$gdrive_path\"\n",
    "! ls \"$gdrive_path\"\n",
    "\n",
    "# Create that vocab using build_vocab\n",
    "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
    "! joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path \"$gdrive_path/vocab.txt\"\n",
    "\n",
    "# Some output\n",
    "! echo \"BPE Xhosa Sentences\"\n",
    "! tail -n 5 test.bpe.$tgt\n",
    "! echo \"Combined BPE Vocab\"\n",
    "! tail -n 10 \"$gdrive_path/vocab.txt\"  # Herman"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ixmzi60WsUZ8"
   },
   "source": [
    "# Creating the JoeyNMT Config\n",
    "\n",
    "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
    "\n",
    "- We used Transformer architecture \n",
    "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
    "\n",
    "Things worth playing with:\n",
    "- The batch size (also recommended to change for low-resourced languages)\n",
    "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
    "- The decoder options (beam_size, alpha)\n",
    "- Evaluation metrics (BLEU versus Crhf4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Wc47fvWqyxbd"
   },
   "outputs": [],
   "source": [
    "def get_last_checkpoint(directory):\n",
    "  last_checkpoint = ''\n",
    "  try:\n",
    "    for filename in os.listdir(directory):\n",
    "      if not 'best' in filename and filename.endswith(\".ckpt\"):\n",
    "          if not last_checkpoint or int(filename.split('.')[0]) > int(last_checkpoint.split('.')[0]):\n",
    "            last_checkpoint = filename\n",
    "  except FileNotFoundError as e:\n",
    "    print('Error Occur ', e)\n",
    "  return last_checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "x_ffEoFdy1Qo",
    "outputId": "bf68cd42-11fb-4074-c68a-e661eb94fdc5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last checkpoint : 32000.ckpt\n"
     ]
    }
   ],
   "source": [
    "# Copy the created models from the temporary storage to main storage on google drive for persistant storage \n",
    "# the content of te folder will be overwrite when you start trainin\n",
    "!cp -r \"/content/drive/My Drive/masakhane/model-temp/\"* \"$gdrive_path/models/${src}${tgt}_transformer/\"\n",
    "last_checkpoint = get_last_checkpoint(models_path)\n",
    "print('Last checkpoint :',last_checkpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PIs1lY2hxMsl"
   },
   "outputs": [],
   "source": [
    "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
    "# (You can of course play with all the parameters if you'd like!)\n",
    "\n",
    "name = '%s%s' % (source_language, target_language)\n",
    "gdrive_path = os.environ[\"gdrive_path\"]\n",
    "\n",
    "# Create the config\n",
    "config = \"\"\"\n",
    "name: \"{name}_transformer\"\n",
    "\n",
    "data:\n",
    "    src: \"{source_language}\"\n",
    "    trg: \"{target_language}\"\n",
    "    train: \"{gdrive_path}/train.bpe\"\n",
    "    dev:   \"{gdrive_path}/dev.bpe\"\n",
    "    test:  \"{gdrive_path}/test.bpe\"\n",
    "    level: \"bpe\"\n",
    "    lowercase: False\n",
    "    max_sent_length: 100\n",
    "    src_vocab: \"{gdrive_path}/vocab.txt\"\n",
    "    trg_vocab: \"{gdrive_path}/vocab.txt\"\n",
    "\n",
    "testing:\n",
    "    beam_size: 5\n",
    "    alpha: 1.0\n",
    "\n",
    "training:\n",
    "    #load_model: \"{gdrive_path}/models/{name}_transformer/{last_checkpoint}\" # uncommented to load a pre-trained model from last checkpoint\n",
    "    random_seed: 42\n",
    "    optimizer: \"adam\"\n",
    "    normalization: \"tokens\"\n",
    "    adam_betas: [0.9, 0.999] \n",
    "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
    "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
    "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
    "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
    "    decrease_factor: 0.7\n",
    "    loss: \"crossentropy\"\n",
    "    learning_rate: 0.0003\n",
    "    learning_rate_min: 0.00000001\n",
    "    weight_decay: 0.0\n",
    "    label_smoothing: 0.1\n",
    "    batch_size: 4096\n",
    "    batch_type: \"token\"\n",
    "    eval_batch_size: 3600\n",
    "    eval_batch_type: \"token\"\n",
    "    batch_multiplier: 1\n",
    "    early_stopping_metric: \"ppl\"\n",
    "    epochs: 30                     # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
    "    validation_freq: 1000          # TODO: Set to at least once per epoch.\n",
    "    logging_freq: 100\n",
    "    eval_metric: \"bleu\"\n",
    "    model_dir: \"{model_temp_dir}\"\n",
    "    overwrite: True               # TODO: Set to True if you want to overwrite possibly existing models. \n",
    "    shuffle: True\n",
    "    use_cuda: True\n",
    "    max_output_length: 100\n",
    "    print_valid_sents: [0, 1, 2, 3]\n",
    "    keep_last_ckpts: 3\n",
    "\n",
    "model:\n",
    "    initializer: \"xavier\"\n",
    "    bias_initializer: \"zeros\"\n",
    "    init_gain: 1.0\n",
    "    embed_initializer: \"xavier\"\n",
    "    embed_init_gain: 1.0\n",
    "    tied_embeddings: True\n",
    "    tied_softmax: True\n",
    "    encoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "    decoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language, model_temp_dir=model_temp_dir, last_checkpoint=last_checkpoint)\n",
    "with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
    "    f.write(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pIifxE3Qzuvs"
   },
   "source": [
    "# Train the Model\n",
    "\n",
    "This single line of joeynmt runs the training using the config we made above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "6ZBPFwT94WpI",
    "outputId": "b950060a-8472-474e-92b4-3df26c42164b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-01-31 18:29:28,118 Hello! This is Joey-NMT.\n",
      "2020-01-31 18:29:29,295 Total params: 12200448\n",
      "2020-01-31 18:29:29,296 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n",
      "2020-01-31 18:29:39,430 cfg.name                           : enxh_transformer\n",
      "2020-01-31 18:29:39,431 cfg.data.src                       : en\n",
      "2020-01-31 18:29:39,431 cfg.data.trg                       : xh\n",
      "2020-01-31 18:29:39,431 cfg.data.train                     : /content/drive/My Drive/masakhane/en-xh-baseline/train.bpe\n",
      "2020-01-31 18:29:39,431 cfg.data.dev                       : /content/drive/My Drive/masakhane/en-xh-baseline/dev.bpe\n",
      "2020-01-31 18:29:39,431 cfg.data.test                      : /content/drive/My Drive/masakhane/en-xh-baseline/test.bpe\n",
      "2020-01-31 18:29:39,432 cfg.data.level                     : bpe\n",
      "2020-01-31 18:29:39,432 cfg.data.lowercase                 : False\n",
      "2020-01-31 18:29:39,432 cfg.data.max_sent_length           : 100\n",
      "2020-01-31 18:29:39,432 cfg.data.src_vocab                 : /content/drive/My Drive/masakhane/en-xh-baseline/vocab.txt\n",
      "2020-01-31 18:29:39,432 cfg.data.trg_vocab                 : /content/drive/My Drive/masakhane/en-xh-baseline/vocab.txt\n",
      "2020-01-31 18:29:39,432 cfg.testing.beam_size              : 5\n",
      "2020-01-31 18:29:39,432 cfg.testing.alpha                  : 1.0\n",
      "2020-01-31 18:29:39,433 cfg.training.random_seed           : 42\n",
      "2020-01-31 18:29:39,433 cfg.training.optimizer             : adam\n",
      "2020-01-31 18:29:39,433 cfg.training.normalization         : tokens\n",
      "2020-01-31 18:29:39,433 cfg.training.adam_betas            : [0.9, 0.999]\n",
      "2020-01-31 18:29:39,433 cfg.training.scheduling            : plateau\n",
      "2020-01-31 18:29:39,433 cfg.training.patience              : 5\n",
      "2020-01-31 18:29:39,434 cfg.training.learning_rate_factor  : 0.5\n",
      "2020-01-31 18:29:39,434 cfg.training.learning_rate_warmup  : 1000\n",
      "2020-01-31 18:29:39,434 cfg.training.decrease_factor       : 0.7\n",
      "2020-01-31 18:29:39,434 cfg.training.loss                  : crossentropy\n",
      "2020-01-31 18:29:39,434 cfg.training.learning_rate         : 0.0003\n",
      "2020-01-31 18:29:39,434 cfg.training.learning_rate_min     : 1e-08\n",
      "2020-01-31 18:29:39,434 cfg.training.weight_decay          : 0.0\n",
      "2020-01-31 18:29:39,435 cfg.training.label_smoothing       : 0.1\n",
      "2020-01-31 18:29:39,435 cfg.training.batch_size            : 4096\n",
      "2020-01-31 18:29:39,435 cfg.training.batch_type            : token\n",
      "2020-01-31 18:29:39,435 cfg.training.eval_batch_size       : 3600\n",
      "2020-01-31 18:29:39,435 cfg.training.eval_batch_type       : token\n",
      "2020-01-31 18:29:39,435 cfg.training.batch_multiplier      : 1\n",
      "2020-01-31 18:29:39,435 cfg.training.early_stopping_metric : ppl\n",
      "2020-01-31 18:29:39,436 cfg.training.epochs                : 30\n",
      "2020-01-31 18:29:39,436 cfg.training.validation_freq       : 1000\n",
      "2020-01-31 18:29:39,436 cfg.training.logging_freq          : 100\n",
      "2020-01-31 18:29:39,436 cfg.training.eval_metric           : bleu\n",
      "2020-01-31 18:29:39,436 cfg.training.model_dir             : /content/drive/My Drive/masakhane/model-temp\n",
      "2020-01-31 18:29:39,436 cfg.training.overwrite             : True\n",
      "2020-01-31 18:29:39,437 cfg.training.shuffle               : True\n",
      "2020-01-31 18:29:39,437 cfg.training.use_cuda              : True\n",
      "2020-01-31 18:29:39,437 cfg.training.max_output_length     : 100\n",
      "2020-01-31 18:29:39,437 cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
      "2020-01-31 18:29:39,437 cfg.training.keep_last_ckpts       : 3\n",
      "2020-01-31 18:29:39,437 cfg.model.initializer              : xavier\n",
      "2020-01-31 18:29:39,437 cfg.model.bias_initializer         : zeros\n",
      "2020-01-31 18:29:39,438 cfg.model.init_gain                : 1.0\n",
      "2020-01-31 18:29:39,438 cfg.model.embed_initializer        : xavier\n",
      "2020-01-31 18:29:39,438 cfg.model.embed_init_gain          : 1.0\n",
      "2020-01-31 18:29:39,438 cfg.model.tied_embeddings          : True\n",
      "2020-01-31 18:29:39,438 cfg.model.tied_softmax             : True\n",
      "2020-01-31 18:29:39,438 cfg.model.encoder.type             : transformer\n",
      "2020-01-31 18:29:39,439 cfg.model.encoder.num_layers       : 6\n",
      "2020-01-31 18:29:39,439 cfg.model.encoder.num_heads        : 4\n",
      "2020-01-31 18:29:39,439 cfg.model.encoder.embeddings.embedding_dim : 256\n",
      "2020-01-31 18:29:39,439 cfg.model.encoder.embeddings.scale : True\n",
      "2020-01-31 18:29:39,439 cfg.model.encoder.embeddings.dropout : 0.2\n",
      "2020-01-31 18:29:39,439 cfg.model.encoder.hidden_size      : 256\n",
      "2020-01-31 18:29:39,439 cfg.model.encoder.ff_size          : 1024\n",
      "2020-01-31 18:29:39,440 cfg.model.encoder.dropout          : 0.3\n",
      "2020-01-31 18:29:39,440 cfg.model.decoder.type             : transformer\n",
      "2020-01-31 18:29:39,440 cfg.model.decoder.num_layers       : 6\n",
      "2020-01-31 18:29:39,440 cfg.model.decoder.num_heads        : 4\n",
      "2020-01-31 18:29:39,440 cfg.model.decoder.embeddings.embedding_dim : 256\n",
      "2020-01-31 18:29:39,440 cfg.model.decoder.embeddings.scale : True\n",
      "2020-01-31 18:29:39,441 cfg.model.decoder.embeddings.dropout : 0.2\n",
      "2020-01-31 18:29:39,441 cfg.model.decoder.hidden_size      : 256\n",
      "2020-01-31 18:29:39,441 cfg.model.decoder.ff_size          : 1024\n",
      "2020-01-31 18:29:39,441 cfg.model.decoder.dropout          : 0.3\n",
      "2020-01-31 18:29:39,441 Data set sizes: \n",
      "\ttrain 786371,\n",
      "\tvalid 1000,\n",
      "\ttest 2711\n",
      "2020-01-31 18:29:39,441 First training example:\n",
      "\t[SRC] Not long after my bap@@ t@@ ism , I married one of Jehovah’s Witnesses .\n",
      "\t[TRG] Kung@@ ekudala nd@@ ib@@ hap@@ ti@@ zi@@ we , nd@@ atshat@@ a omnye wama@@ Ngqina kaYehova .\n",
      "2020-01-31 18:29:39,442 First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) the (7) to (8) of (9) a\n",
      "2020-01-31 18:29:39,442 First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) the (7) to (8) of (9) a\n",
      "2020-01-31 18:29:39,442 Number of Src words (types): 4454\n",
      "2020-01-31 18:29:39,443 Number of Trg words (types): 4454\n",
      "2020-01-31 18:29:39,443 Model(\n",
      "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
      "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
      "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4454),\n",
      "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4454))\n",
      "2020-01-31 18:29:39,491 EPOCH 1\n",
      "2020-01-31 18:29:56,146 Epoch   1 Step:      100 Batch Loss:     5.903441 Tokens per Sec:    15574, Lr: 0.000300\n",
      "2020-01-31 18:30:11,313 Epoch   1 Step:      200 Batch Loss:     5.949011 Tokens per Sec:    16335, Lr: 0.000300\n",
      "2020-01-31 18:30:26,869 Epoch   1 Step:      300 Batch Loss:     5.827965 Tokens per Sec:    16164, Lr: 0.000300\n",
      "2020-01-31 18:30:42,679 Epoch   1 Step:      400 Batch Loss:     5.656576 Tokens per Sec:    16112, Lr: 0.000300\n",
      "2020-01-31 18:30:58,413 Epoch   1 Step:      500 Batch Loss:     5.102027 Tokens per Sec:    16027, Lr: 0.000300\n",
      "2020-01-31 18:31:14,584 Epoch   1 Step:      600 Batch Loss:     4.956975 Tokens per Sec:    15723, Lr: 0.000300\n",
      "2020-01-31 18:31:30,853 Epoch   1 Step:      700 Batch Loss:     5.407192 Tokens per Sec:    15660, Lr: 0.000300\n",
      "2020-01-31 18:31:46,972 Epoch   1 Step:      800 Batch Loss:     4.736966 Tokens per Sec:    15818, Lr: 0.000300\n",
      "2020-01-31 18:32:03,231 Epoch   1 Step:      900 Batch Loss:     4.191297 Tokens per Sec:    15851, Lr: 0.000300\n",
      "2020-01-31 18:32:19,295 Epoch   1 Step:     1000 Batch Loss:     4.963958 Tokens per Sec:    16099, Lr: 0.000300\n",
      "2020-01-31 18:32:46,673 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 18:32:46,674 Saving new checkpoint.\n",
      "2020-01-31 18:32:47,936 Example #0\n",
      "2020-01-31 18:32:47,936 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 18:32:47,936 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 18:32:47,936 \tHypothesis: Ubazi ukuba ndithela ukuba ndithela .\n",
      "2020-01-31 18:32:47,936 Example #1\n",
      "2020-01-31 18:32:47,937 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 18:32:47,937 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 18:32:47,937 \tHypothesis: KaliMti , uMari - 1900 , uMeri - 1900 , uMera , uMera , uMera , uMara , uMqa .\n",
      "2020-01-31 18:32:47,937 Example #2\n",
      "2020-01-31 18:32:47,937 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 18:32:47,937 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 18:32:47,937 \tHypothesis: UMari : “ Uthela ukuba ndilana , ” — YiMara .\n",
      "2020-01-31 18:32:47,937 Example #3\n",
      "2020-01-31 18:32:47,937 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 18:32:47,938 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 18:32:47,938 \tHypothesis: Ubazi ukuba ndenza ukuba ndenza ukuba ndenza .\n",
      "2020-01-31 18:32:47,938 Validation result (greedy) at epoch   1, step     1000: bleu:   0.00, loss: 132836.3125, ppl: 116.6621, duration: 28.6423s\n",
      "2020-01-31 18:33:04,226 Epoch   1 Step:     1100 Batch Loss:     4.803463 Tokens per Sec:    15695, Lr: 0.000300\n",
      "2020-01-31 18:33:20,227 Epoch   1 Step:     1200 Batch Loss:     4.381655 Tokens per Sec:    15872, Lr: 0.000300\n",
      "2020-01-31 18:33:36,257 Epoch   1 Step:     1300 Batch Loss:     4.465355 Tokens per Sec:    15772, Lr: 0.000300\n",
      "2020-01-31 18:33:52,461 Epoch   1 Step:     1400 Batch Loss:     4.501562 Tokens per Sec:    15831, Lr: 0.000300\n",
      "2020-01-31 18:34:08,425 Epoch   1 Step:     1500 Batch Loss:     4.561132 Tokens per Sec:    15380, Lr: 0.000300\n",
      "2020-01-31 18:34:24,696 Epoch   1 Step:     1600 Batch Loss:     4.107214 Tokens per Sec:    15939, Lr: 0.000300\n",
      "2020-01-31 18:34:40,741 Epoch   1 Step:     1700 Batch Loss:     3.824157 Tokens per Sec:    15780, Lr: 0.000300\n",
      "2020-01-31 18:34:56,817 Epoch   1 Step:     1800 Batch Loss:     4.245836 Tokens per Sec:    15816, Lr: 0.000300\n",
      "2020-01-31 18:35:13,128 Epoch   1 Step:     1900 Batch Loss:     4.006897 Tokens per Sec:    15899, Lr: 0.000300\n",
      "2020-01-31 18:35:29,076 Epoch   1 Step:     2000 Batch Loss:     3.852657 Tokens per Sec:    15313, Lr: 0.000300\n",
      "2020-01-31 18:36:12,283 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 18:36:12,283 Saving new checkpoint.\n",
      "2020-01-31 18:36:13,580 Example #0\n",
      "2020-01-31 18:36:13,581 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 18:36:13,581 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 18:36:13,582 \tHypothesis: Ndiyayinakuphendula ukuba ndiye ndiye ndiye ndiye ndibone .\n",
      "2020-01-31 18:36:13,582 Example #1\n",
      "2020-01-31 18:36:13,583 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 18:36:13,583 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 18:36:13,583 \tHypothesis: Xa uGari , uGari , uGari wan ’ uGari , wathetha ukuba wayeza kuhlala ukufundwa kwindlela yokubonisa ngayo .\n",
      "2020-01-31 18:36:13,583 Example #2\n",
      "2020-01-31 18:36:13,584 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 18:36:13,584 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 18:36:13,584 \tHypothesis: UTari wathetha ukuba uGari uthi : “ Kuba lona lona lililindele lona liphila . ” — Thelekisa 2 : 1 - 5 .\n",
      "2020-01-31 18:36:13,584 Example #3\n",
      "2020-01-31 18:36:13,585 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 18:36:13,585 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 18:36:13,585 \tHypothesis: Unokubonisa ukuba ubulumko bokomoya bokomoya .\n",
      "2020-01-31 18:36:13,585 Validation result (greedy) at epoch   1, step     2000: bleu:   1.61, loss: 104823.5234, ppl:  42.7614, duration: 44.5095s\n",
      "2020-01-31 18:36:30,097 Epoch   1 Step:     2100 Batch Loss:     3.934059 Tokens per Sec:    15639, Lr: 0.000300\n",
      "2020-01-31 18:36:46,101 Epoch   1 Step:     2200 Batch Loss:     3.888976 Tokens per Sec:    15871, Lr: 0.000300\n",
      "2020-01-31 18:37:02,237 Epoch   1 Step:     2300 Batch Loss:     3.348117 Tokens per Sec:    15733, Lr: 0.000300\n",
      "2020-01-31 18:37:18,212 Epoch   1 Step:     2400 Batch Loss:     3.845164 Tokens per Sec:    15800, Lr: 0.000300\n",
      "2020-01-31 18:37:34,405 Epoch   1 Step:     2500 Batch Loss:     3.348575 Tokens per Sec:    15728, Lr: 0.000300\n",
      "2020-01-31 18:37:50,753 Epoch   1 Step:     2600 Batch Loss:     3.575891 Tokens per Sec:    15466, Lr: 0.000300\n",
      "2020-01-31 18:38:06,908 Epoch   1 Step:     2700 Batch Loss:     3.506172 Tokens per Sec:    15504, Lr: 0.000300\n",
      "2020-01-31 18:38:23,141 Epoch   1 Step:     2800 Batch Loss:     3.131526 Tokens per Sec:    16022, Lr: 0.000300\n",
      "2020-01-31 18:38:39,195 Epoch   1 Step:     2900 Batch Loss:     3.320538 Tokens per Sec:    15788, Lr: 0.000300\n",
      "2020-01-31 18:38:55,195 Epoch   1 Step:     3000 Batch Loss:     3.841508 Tokens per Sec:    16031, Lr: 0.000300\n",
      "2020-01-31 18:39:32,996 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 18:39:32,996 Saving new checkpoint.\n",
      "2020-01-31 18:39:34,129 Example #0\n",
      "2020-01-31 18:39:34,129 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 18:39:34,130 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 18:39:34,130 \tHypothesis: Ndayisoloko ndafumana ukuba ndiqhuba ixesha elide .\n",
      "2020-01-31 18:39:34,130 Example #1\n",
      "2020-01-31 18:39:34,130 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 18:39:34,130 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 18:39:34,130 \tHypothesis: Ngoxa uRashiya noSariya , uGeny , uGeny , wathetha ukuba abantu abagxinwa , yaye babetha ukuba bafundise indlela yokuziphatha okubulumko .\n",
      "2020-01-31 18:39:34,130 Example #2\n",
      "2020-01-31 18:39:34,130 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 18:39:34,131 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 18:39:34,131 \tHypothesis: URata wacela ukuba : “ Kuba makhoboka elifunayo lathi : “ Kuba luyayiphathi . ” — IZenzo 5 : 5 ; ISityhilelo 5 : 5 .\n",
      "2020-01-31 18:39:34,131 Example #3\n",
      "2020-01-31 18:39:34,131 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 18:39:34,131 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 18:39:34,131 \tHypothesis: Kunokubonisa ukuba nantoni na imilinganiselo yokuziphatha okulungileyo nokubonakalisa abantu .\n",
      "2020-01-31 18:39:34,131 Validation result (greedy) at epoch   1, step     3000: bleu:   3.14, loss: 91546.4219, ppl:  26.5742, duration: 38.9357s\n",
      "2020-01-31 18:39:50,212 Epoch   1 Step:     3100 Batch Loss:     3.027941 Tokens per Sec:    15582, Lr: 0.000300\n",
      "2020-01-31 18:40:06,205 Epoch   1 Step:     3200 Batch Loss:     3.273610 Tokens per Sec:    15425, Lr: 0.000300\n",
      "2020-01-31 18:40:22,337 Epoch   1 Step:     3300 Batch Loss:     3.520874 Tokens per Sec:    15811, Lr: 0.000300\n",
      "2020-01-31 18:40:38,590 Epoch   1 Step:     3400 Batch Loss:     3.401852 Tokens per Sec:    15706, Lr: 0.000300\n",
      "2020-01-31 18:40:54,597 Epoch   1 Step:     3500 Batch Loss:     3.456470 Tokens per Sec:    15741, Lr: 0.000300\n",
      "2020-01-31 18:41:10,632 Epoch   1 Step:     3600 Batch Loss:     3.243973 Tokens per Sec:    15588, Lr: 0.000300\n",
      "2020-01-31 18:41:26,608 Epoch   1 Step:     3700 Batch Loss:     3.362392 Tokens per Sec:    15635, Lr: 0.000300\n",
      "2020-01-31 18:41:42,823 Epoch   1 Step:     3800 Batch Loss:     2.824328 Tokens per Sec:    15950, Lr: 0.000300\n",
      "2020-01-31 18:41:58,974 Epoch   1 Step:     3900 Batch Loss:     3.353892 Tokens per Sec:    16165, Lr: 0.000300\n",
      "2020-01-31 18:42:15,175 Epoch   1 Step:     4000 Batch Loss:     3.418256 Tokens per Sec:    15726, Lr: 0.000300\n",
      "2020-01-31 18:42:42,735 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 18:42:42,735 Saving new checkpoint.\n",
      "2020-01-31 18:42:44,267 Example #0\n",
      "2020-01-31 18:42:44,268 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 18:42:44,268 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 18:42:44,268 \tHypothesis: Ndaziva ndazibetha ukuba ndiye ndaqalisa ukufumana ixesha elizeleyo .\n",
      "2020-01-31 18:42:44,268 Example #1\n",
      "2020-01-31 18:42:44,269 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 18:42:44,269 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 18:42:44,269 \tHypothesis: Nangona uPolin noPolia , uLaro wathetha ukuba iSikhumbuzo , yaye kwakukho imithetho yokomoya , yaye kwakufuneka abefundisi bokomoya .\n",
      "2020-01-31 18:42:44,269 Example #2\n",
      "2020-01-31 18:42:44,269 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 18:42:44,269 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 18:42:44,269 \tHypothesis: IZibhalo zithi : “ Kukho umhlambi wobugcisa umhlambi wobukhoboka . ” — IMizekeliso 5 : 5 ; ISityhilelo 5 : 5 .\n",
      "2020-01-31 18:42:44,269 Example #3\n",
      "2020-01-31 18:42:44,270 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 18:42:44,270 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 18:42:44,270 \tHypothesis: Kukho izinto eziyingozi nokuzikhuthaza abantu bakaThixo .\n",
      "2020-01-31 18:42:44,270 Validation result (greedy) at epoch   1, step     4000: bleu:   5.65, loss: 83060.1562, ppl:  19.6071, duration: 29.0950s\n",
      "2020-01-31 18:43:00,596 Epoch   1 Step:     4100 Batch Loss:     3.225327 Tokens per Sec:    15657, Lr: 0.000300\n",
      "2020-01-31 18:43:16,628 Epoch   1 Step:     4200 Batch Loss:     2.993990 Tokens per Sec:    15860, Lr: 0.000300\n",
      "2020-01-31 18:43:32,741 Epoch   1 Step:     4300 Batch Loss:     3.013757 Tokens per Sec:    15662, Lr: 0.000300\n",
      "2020-01-31 18:43:48,717 Epoch   1 Step:     4400 Batch Loss:     2.878743 Tokens per Sec:    16142, Lr: 0.000300\n",
      "2020-01-31 18:44:04,894 Epoch   1 Step:     4500 Batch Loss:     3.684171 Tokens per Sec:    15792, Lr: 0.000300\n",
      "2020-01-31 18:44:20,829 Epoch   1 Step:     4600 Batch Loss:     3.038558 Tokens per Sec:    15874, Lr: 0.000300\n",
      "2020-01-31 18:44:36,794 Epoch   1 Step:     4700 Batch Loss:     2.980048 Tokens per Sec:    15625, Lr: 0.000300\n",
      "2020-01-31 18:44:53,135 Epoch   1 Step:     4800 Batch Loss:     2.836524 Tokens per Sec:    15756, Lr: 0.000300\n",
      "2020-01-31 18:45:09,088 Epoch   1 Step:     4900 Batch Loss:     3.476342 Tokens per Sec:    15591, Lr: 0.000300\n",
      "2020-01-31 18:45:25,191 Epoch   1 Step:     5000 Batch Loss:     2.958634 Tokens per Sec:    15909, Lr: 0.000300\n",
      "2020-01-31 18:45:51,893 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 18:45:51,893 Saving new checkpoint.\n",
      "2020-01-31 18:45:53,054 Example #0\n",
      "2020-01-31 18:45:53,055 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 18:45:53,055 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 18:45:53,055 \tHypothesis: Ndandisoloko ndikhonza ngokuvisisana nomsebenzi wokushumayela .\n",
      "2020-01-31 18:45:53,055 Example #1\n",
      "2020-01-31 18:45:53,056 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 18:45:53,056 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 18:45:53,056 \tHypothesis: Ngoxa uPolyn ’ uPoland , uLeva wathi , kukho imithetho yeBhabhiloni , yaye kwakukho izigqibo eziyingozi , yaye kwakungekho nto leyo eyahlukileyo .\n",
      "2020-01-31 18:45:53,057 Example #2\n",
      "2020-01-31 18:45:53,057 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 18:45:53,057 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 18:45:53,057 \tHypothesis: IZibhalo zithi : “ Kuba ningqondo yathi : “ Bonke abantu abangenakudala . ” — IMizekeliso 6 : 9 ; Eksodus 5 : 5 .\n",
      "2020-01-31 18:45:53,057 Example #3\n",
      "2020-01-31 18:45:53,058 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 18:45:53,058 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 18:45:53,058 \tHypothesis: Kukho imiphumo yokuziphatha nokuzikhusela abantu nothando noThixo .\n",
      "2020-01-31 18:45:53,058 Validation result (greedy) at epoch   1, step     5000: bleu:   7.10, loss: 77366.0391, ppl:  15.9887, duration: 27.8664s\n",
      "2020-01-31 18:46:09,423 Epoch   1 Step:     5100 Batch Loss:     3.241308 Tokens per Sec:    15418, Lr: 0.000300\n",
      "2020-01-31 18:46:25,602 Epoch   1 Step:     5200 Batch Loss:     3.326140 Tokens per Sec:    16077, Lr: 0.000300\n",
      "2020-01-31 18:46:41,692 Epoch   1 Step:     5300 Batch Loss:     2.957384 Tokens per Sec:    16002, Lr: 0.000300\n",
      "2020-01-31 18:46:57,695 Epoch   1 Step:     5400 Batch Loss:     3.033994 Tokens per Sec:    15867, Lr: 0.000300\n",
      "2020-01-31 18:47:13,790 Epoch   1 Step:     5500 Batch Loss:     3.103406 Tokens per Sec:    15862, Lr: 0.000300\n",
      "2020-01-31 18:47:29,891 Epoch   1 Step:     5600 Batch Loss:     2.761618 Tokens per Sec:    15740, Lr: 0.000300\n",
      "2020-01-31 18:47:46,042 Epoch   1 Step:     5700 Batch Loss:     3.002301 Tokens per Sec:    15933, Lr: 0.000300\n",
      "2020-01-31 18:48:02,278 Epoch   1 Step:     5800 Batch Loss:     2.721744 Tokens per Sec:    16034, Lr: 0.000300\n",
      "2020-01-31 18:48:18,361 Epoch   1 Step:     5900 Batch Loss:     2.671431 Tokens per Sec:    16134, Lr: 0.000300\n",
      "2020-01-31 18:48:34,268 Epoch   1 Step:     6000 Batch Loss:     2.369029 Tokens per Sec:    15732, Lr: 0.000300\n",
      "2020-01-31 18:48:59,806 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 18:48:59,806 Saving new checkpoint.\n",
      "2020-01-31 18:49:00,848 Example #0\n",
      "2020-01-31 18:49:00,849 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 18:49:00,849 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 18:49:00,850 \tHypothesis: Ndafudukela ukusebenzisa umsebenzi wokuqala kwixesha elizeleyo .\n",
      "2020-01-31 18:49:00,850 Example #1\n",
      "2020-01-31 18:49:00,850 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 18:49:00,850 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 18:49:00,850 \tHypothesis: Ngoxa uPoland Poland , uLicaro wabona ukuba iHolo yoMbutho , apho kwakukho imibingelelo yokomoya , yaye kwakukho imfundo yokomoya yokomoya yokomoya .\n",
      "2020-01-31 18:49:00,850 Example #2\n",
      "2020-01-31 18:49:00,850 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 18:49:00,850 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 18:49:00,851 \tHypothesis: ISpanish ithi : “ Kuba ningxamisani na ukutshata nokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 5 .\n",
      "2020-01-31 18:49:00,851 Example #3\n",
      "2020-01-31 18:49:00,851 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 18:49:00,851 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 18:49:00,851 \tHypothesis: Kuphando olukhuthaza ukulungisa abantu nokuzikhusela abantu .\n",
      "2020-01-31 18:49:00,851 Validation result (greedy) at epoch   1, step     6000: bleu:   8.93, loss: 72942.6016, ppl:  13.6454, duration: 26.5829s\n",
      "2020-01-31 18:49:17,173 Epoch   1 Step:     6100 Batch Loss:     3.272591 Tokens per Sec:    15821, Lr: 0.000300\n",
      "2020-01-31 18:49:33,331 Epoch   1 Step:     6200 Batch Loss:     2.796846 Tokens per Sec:    15746, Lr: 0.000300\n",
      "2020-01-31 18:49:49,355 Epoch   1 Step:     6300 Batch Loss:     2.783082 Tokens per Sec:    15825, Lr: 0.000300\n",
      "2020-01-31 18:50:05,594 Epoch   1 Step:     6400 Batch Loss:     2.722397 Tokens per Sec:    15702, Lr: 0.000300\n",
      "2020-01-31 18:50:21,600 Epoch   1 Step:     6500 Batch Loss:     2.685393 Tokens per Sec:    15726, Lr: 0.000300\n",
      "2020-01-31 18:50:37,479 Epoch   1 Step:     6600 Batch Loss:     2.957675 Tokens per Sec:    15552, Lr: 0.000300\n",
      "2020-01-31 18:50:53,601 Epoch   1 Step:     6700 Batch Loss:     2.831558 Tokens per Sec:    16139, Lr: 0.000300\n",
      "2020-01-31 18:51:09,662 Epoch   1 Step:     6800 Batch Loss:     2.769505 Tokens per Sec:    15595, Lr: 0.000300\n",
      "2020-01-31 18:51:25,823 Epoch   1 Step:     6900 Batch Loss:     2.446182 Tokens per Sec:    15803, Lr: 0.000300\n",
      "2020-01-31 18:51:41,801 Epoch   1 Step:     7000 Batch Loss:     2.500067 Tokens per Sec:    15797, Lr: 0.000300\n",
      "2020-01-31 18:52:12,778 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 18:52:12,779 Saving new checkpoint.\n",
      "2020-01-31 18:52:13,836 Example #0\n",
      "2020-01-31 18:52:13,836 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 18:52:13,836 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 18:52:13,837 \tHypothesis: Ndafudukela ukusebenzisa ixesha lokusebenza kwixesha elizeleyo .\n",
      "2020-01-31 18:52:13,837 Example #1\n",
      "2020-01-31 18:52:13,837 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 18:52:13,837 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 18:52:13,837 \tHypothesis: Ngoxa kwavuyela ePoland , uLevera , uLeviet , yaye kwakukho imfundiso yokomoya , yaye kwakukho imfundo yokomoya yokomoya yokomoya .\n",
      "2020-01-31 18:52:13,837 Example #2\n",
      "2020-01-31 18:52:13,837 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 18:52:13,837 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 18:52:13,837 \tHypothesis: ISpeyin ithi : “ Ngokuqinisekileyo uBhuda ukuba uhambe ngokuqhelekileyo kunokwahlukana nokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 .\n",
      "2020-01-31 18:52:13,838 Example #3\n",
      "2020-01-31 18:52:13,838 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 18:52:13,838 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 18:52:13,838 \tHypothesis: Kuphando lwabantu ngokukhusela ngokuvisisana neempawu zabanye .\n",
      "2020-01-31 18:52:13,838 Validation result (greedy) at epoch   1, step     7000: bleu:   9.45, loss: 69971.7344, ppl:  12.2676, duration: 32.0364s\n",
      "2020-01-31 18:52:30,263 Epoch   1 Step:     7100 Batch Loss:     2.722150 Tokens per Sec:    15641, Lr: 0.000300\n",
      "2020-01-31 18:52:46,462 Epoch   1 Step:     7200 Batch Loss:     2.473864 Tokens per Sec:    15767, Lr: 0.000300\n",
      "2020-01-31 18:53:02,608 Epoch   1 Step:     7300 Batch Loss:     2.828389 Tokens per Sec:    15806, Lr: 0.000300\n",
      "2020-01-31 18:53:18,755 Epoch   1 Step:     7400 Batch Loss:     2.707949 Tokens per Sec:    16007, Lr: 0.000300\n",
      "2020-01-31 18:53:34,765 Epoch   1 Step:     7500 Batch Loss:     2.658384 Tokens per Sec:    15824, Lr: 0.000300\n",
      "2020-01-31 18:53:50,697 Epoch   1 Step:     7600 Batch Loss:     2.574796 Tokens per Sec:    15733, Lr: 0.000300\n",
      "2020-01-31 18:54:06,632 Epoch   1 Step:     7700 Batch Loss:     2.455312 Tokens per Sec:    15588, Lr: 0.000300\n",
      "2020-01-31 18:54:22,653 Epoch   1 Step:     7800 Batch Loss:     2.654092 Tokens per Sec:    16023, Lr: 0.000300\n",
      "2020-01-31 18:54:38,584 Epoch   1 Step:     7900 Batch Loss:     2.918579 Tokens per Sec:    15853, Lr: 0.000300\n",
      "2020-01-31 18:54:54,279 Epoch   1 Step:     8000 Batch Loss:     2.409480 Tokens per Sec:    15839, Lr: 0.000300\n",
      "2020-01-31 18:55:24,482 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 18:55:24,483 Saving new checkpoint.\n",
      "2020-01-31 18:55:25,928 Example #0\n",
      "2020-01-31 18:55:25,928 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 18:55:25,928 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 18:55:25,928 \tHypothesis: Ndandisebenza nzima ukusebenzisa umsebenzi wokushumayela .\n",
      "2020-01-31 18:55:25,928 Example #1\n",
      "2020-01-31 18:55:25,928 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 18:55:25,928 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 18:55:25,929 \tHypothesis: Ngoxa wayelindele uPoland , uLucaro wabona ukuba iOthodoki , yaye kwakukho abefundisi , ababingeleli befuna ukufumana imfundo yokomoya , ababengomfuneko yokuziphatha okubi .\n",
      "2020-01-31 18:55:25,929 Example #2\n",
      "2020-01-31 18:55:25,929 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 18:55:25,929 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 18:55:25,929 \tHypothesis: ISpanish ithi : “ Kuba ninani lokuhamba , nathi : “ Ukunzima ukuhamba . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 18:55:25,929 Example #3\n",
      "2020-01-31 18:55:25,929 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 18:55:25,929 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 18:55:25,929 \tHypothesis: Kukho into yokuqeqesha nokuqeqesha abantu noThixo .\n",
      "2020-01-31 18:55:25,930 Validation result (greedy) at epoch   1, step     8000: bleu:  11.45, loss: 66382.7500, ppl:  10.7873, duration: 31.6504s\n",
      "2020-01-31 18:55:42,215 Epoch   1 Step:     8100 Batch Loss:     2.482053 Tokens per Sec:    15678, Lr: 0.000300\n",
      "2020-01-31 18:55:58,281 Epoch   1 Step:     8200 Batch Loss:     2.994364 Tokens per Sec:    16026, Lr: 0.000300\n",
      "2020-01-31 18:56:14,420 Epoch   1 Step:     8300 Batch Loss:     2.507364 Tokens per Sec:    15977, Lr: 0.000300\n",
      "2020-01-31 18:56:30,677 Epoch   1 Step:     8400 Batch Loss:     2.465361 Tokens per Sec:    15945, Lr: 0.000300\n",
      "2020-01-31 18:56:46,647 Epoch   1 Step:     8500 Batch Loss:     2.489060 Tokens per Sec:    15641, Lr: 0.000300\n",
      "2020-01-31 18:57:02,810 Epoch   1 Step:     8600 Batch Loss:     2.458732 Tokens per Sec:    15758, Lr: 0.000300\n",
      "2020-01-31 18:57:18,628 Epoch   1 Step:     8700 Batch Loss:     2.629307 Tokens per Sec:    15738, Lr: 0.000300\n",
      "2020-01-31 18:57:34,892 Epoch   1 Step:     8800 Batch Loss:     2.390210 Tokens per Sec:    15922, Lr: 0.000300\n",
      "2020-01-31 18:57:49,165 Epoch   1: total training loss 30551.58\n",
      "2020-01-31 18:57:49,165 EPOCH 2\n",
      "2020-01-31 18:57:51,995 Epoch   2 Step:     8900 Batch Loss:     2.710046 Tokens per Sec:     9920, Lr: 0.000300\n",
      "2020-01-31 18:58:08,081 Epoch   2 Step:     9000 Batch Loss:     2.549092 Tokens per Sec:    15738, Lr: 0.000300\n",
      "2020-01-31 18:58:37,227 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 18:58:37,228 Saving new checkpoint.\n",
      "2020-01-31 18:58:38,337 Example #0\n",
      "2020-01-31 18:58:38,337 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 18:58:38,337 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 18:58:38,337 \tHypothesis: Ndandisebenza nzima ukusebenzisa umsebenzi wokushumayela .\n",
      "2020-01-31 18:58:38,337 Example #1\n",
      "2020-01-31 18:58:38,337 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 18:58:38,337 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 18:58:38,338 \tHypothesis: Ngoxa wayelindele uPoland , uLucaris wabona ukuba iOthodoki , abefundisi abangamaOrododododoki , babefunda imfundo yokomoya yokomoya .\n",
      "2020-01-31 18:58:38,338 Example #2\n",
      "2020-01-31 18:58:38,338 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 18:58:38,338 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 18:58:38,338 \tHypothesis: ISpanish ithi : “ Yiba nomnye umdala ongatshatanga kunoko kunokwenzeka kunokwenzeka . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 18:58:38,338 Example #3\n",
      "2020-01-31 18:58:38,338 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 18:58:38,338 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 18:58:38,338 \tHypothesis: Kulutsha olukhulu yaye luzikhusela abantu abanyanisekileyo noThixo .\n",
      "2020-01-31 18:58:38,339 Validation result (greedy) at epoch   2, step     9000: bleu:  12.41, loss: 63937.7812, ppl:   9.8826, duration: 30.2571s\n",
      "2020-01-31 18:58:54,681 Epoch   2 Step:     9100 Batch Loss:     2.372067 Tokens per Sec:    15677, Lr: 0.000300\n",
      "2020-01-31 18:59:10,818 Epoch   2 Step:     9200 Batch Loss:     2.438389 Tokens per Sec:    15605, Lr: 0.000300\n",
      "2020-01-31 18:59:27,029 Epoch   2 Step:     9300 Batch Loss:     2.443753 Tokens per Sec:    15768, Lr: 0.000300\n",
      "2020-01-31 18:59:43,222 Epoch   2 Step:     9400 Batch Loss:     2.432405 Tokens per Sec:    15883, Lr: 0.000300\n",
      "2020-01-31 18:59:59,188 Epoch   2 Step:     9500 Batch Loss:     2.722638 Tokens per Sec:    15670, Lr: 0.000300\n",
      "2020-01-31 19:00:15,423 Epoch   2 Step:     9600 Batch Loss:     2.199669 Tokens per Sec:    15882, Lr: 0.000300\n",
      "2020-01-31 19:00:31,395 Epoch   2 Step:     9700 Batch Loss:     2.398737 Tokens per Sec:    15561, Lr: 0.000300\n",
      "2020-01-31 19:00:47,494 Epoch   2 Step:     9800 Batch Loss:     2.548895 Tokens per Sec:    16045, Lr: 0.000300\n",
      "2020-01-31 19:01:03,728 Epoch   2 Step:     9900 Batch Loss:     2.526432 Tokens per Sec:    15815, Lr: 0.000300\n",
      "2020-01-31 19:01:19,865 Epoch   2 Step:    10000 Batch Loss:     2.429618 Tokens per Sec:    15943, Lr: 0.000300\n",
      "2020-01-31 19:01:44,805 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:01:44,805 Saving new checkpoint.\n",
      "2020-01-31 19:01:46,005 Example #0\n",
      "2020-01-31 19:01:46,005 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:01:46,006 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:01:46,006 \tHypothesis: Ndandisebenza ndaza ndafudukela kwisikolo sexesha elizeleyo .\n",
      "2020-01-31 19:01:46,006 Example #1\n",
      "2020-01-31 19:01:46,006 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:01:46,006 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:01:46,007 \tHypothesis: Ngoxa wayelindele uPoland , uLucaris wabona ukuba iOthodoki , yaye abingeleli nabafazi , babefanele banyamezele ngokomoya , babenomdla wokufundisa .\n",
      "2020-01-31 19:01:46,007 Example #2\n",
      "2020-01-31 19:01:46,007 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:01:46,007 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:01:46,007 \tHypothesis: Iphephandaba lithi : “ Bhangela ukuba uhambe ngumntu ongatshatanga kunoko kunokwenzeka . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:01:46,008 Example #3\n",
      "2020-01-31 19:01:46,008 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:01:46,008 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:01:46,008 \tHypothesis: Kukho inkcazelo engakumbi yokuqeqesha nokuqeqesha abantu noThixo .\n",
      "2020-01-31 19:01:46,008 Validation result (greedy) at epoch   2, step    10000: bleu:  13.30, loss: 61299.7969, ppl:   8.9913, duration: 26.1432s\n",
      "2020-01-31 19:02:02,289 Epoch   2 Step:    10100 Batch Loss:     2.201668 Tokens per Sec:    15605, Lr: 0.000300\n",
      "2020-01-31 19:02:18,319 Epoch   2 Step:    10200 Batch Loss:     2.152504 Tokens per Sec:    15713, Lr: 0.000300\n",
      "2020-01-31 19:02:34,400 Epoch   2 Step:    10300 Batch Loss:     2.380329 Tokens per Sec:    15736, Lr: 0.000300\n",
      "2020-01-31 19:02:50,492 Epoch   2 Step:    10400 Batch Loss:     2.420297 Tokens per Sec:    15954, Lr: 0.000300\n",
      "2020-01-31 19:03:06,705 Epoch   2 Step:    10500 Batch Loss:     2.457934 Tokens per Sec:    15732, Lr: 0.000300\n",
      "2020-01-31 19:03:22,757 Epoch   2 Step:    10600 Batch Loss:     2.355008 Tokens per Sec:    15994, Lr: 0.000300\n",
      "2020-01-31 19:03:38,670 Epoch   2 Step:    10700 Batch Loss:     2.351007 Tokens per Sec:    15744, Lr: 0.000300\n",
      "2020-01-31 19:03:54,545 Epoch   2 Step:    10800 Batch Loss:     2.512789 Tokens per Sec:    16035, Lr: 0.000300\n",
      "2020-01-31 19:04:10,636 Epoch   2 Step:    10900 Batch Loss:     2.528640 Tokens per Sec:    15663, Lr: 0.000300\n",
      "2020-01-31 19:04:26,559 Epoch   2 Step:    11000 Batch Loss:     2.655377 Tokens per Sec:    15908, Lr: 0.000300\n",
      "2020-01-31 19:04:49,648 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:04:49,649 Saving new checkpoint.\n",
      "2020-01-31 19:04:50,849 Example #0\n",
      "2020-01-31 19:04:50,850 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:04:50,850 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:04:50,850 \tHypothesis: Ndandisafudukela ukusebenzisa umsebenzi wokushumayela .\n",
      "2020-01-31 19:04:50,850 Example #1\n",
      "2020-01-31 19:04:50,850 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:04:50,850 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:04:50,850 \tHypothesis: Ngoxa kwatyelela uPoland , uLucaris wabona ukuba abefundisi baseOthodoki , yaye babingeleli , babefana nabangenakukwazi ukufumana inkqubela yokomoya njengokumelana nemfundo yabo .\n",
      "2020-01-31 19:04:50,850 Example #2\n",
      "2020-01-31 19:04:50,850 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:04:50,851 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:04:50,851 \tHypothesis: IThe Spanish ithi : “ Ngokuqinisekileyo , uBetsare watshata kunangaphambili . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:04:50,851 Example #3\n",
      "2020-01-31 19:04:50,851 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:04:50,851 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:04:50,851 \tHypothesis: Kulutsha luyazisa yaye londoloza izenzo ezihle nabantu noThixo .\n",
      "2020-01-31 19:04:50,851 Validation result (greedy) at epoch   2, step    11000: bleu:  13.97, loss: 59267.0000, ppl:   8.3597, duration: 24.2917s\n",
      "2020-01-31 19:05:07,169 Epoch   2 Step:    11100 Batch Loss:     2.481309 Tokens per Sec:    15503, Lr: 0.000300\n",
      "2020-01-31 19:05:23,317 Epoch   2 Step:    11200 Batch Loss:     2.702048 Tokens per Sec:    15967, Lr: 0.000300\n",
      "2020-01-31 19:05:39,364 Epoch   2 Step:    11300 Batch Loss:     2.326753 Tokens per Sec:    15762, Lr: 0.000300\n",
      "2020-01-31 19:05:55,482 Epoch   2 Step:    11400 Batch Loss:     2.422019 Tokens per Sec:    16073, Lr: 0.000300\n",
      "2020-01-31 19:06:11,750 Epoch   2 Step:    11500 Batch Loss:     2.402837 Tokens per Sec:    15844, Lr: 0.000300\n",
      "2020-01-31 19:06:27,850 Epoch   2 Step:    11600 Batch Loss:     2.178888 Tokens per Sec:    15693, Lr: 0.000300\n",
      "2020-01-31 19:06:43,775 Epoch   2 Step:    11700 Batch Loss:     2.536305 Tokens per Sec:    15894, Lr: 0.000300\n",
      "2020-01-31 19:06:59,915 Epoch   2 Step:    11800 Batch Loss:     2.372548 Tokens per Sec:    15882, Lr: 0.000300\n",
      "2020-01-31 19:07:15,984 Epoch   2 Step:    11900 Batch Loss:     2.372258 Tokens per Sec:    15674, Lr: 0.000300\n",
      "2020-01-31 19:07:32,015 Epoch   2 Step:    12000 Batch Loss:     2.282680 Tokens per Sec:    15892, Lr: 0.000300\n",
      "2020-01-31 19:07:54,263 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:07:54,263 Saving new checkpoint.\n",
      "2020-01-31 19:07:55,384 Example #0\n",
      "2020-01-31 19:07:55,385 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:07:55,385 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:07:55,385 \tHypothesis: Ndandisebenza nzima ukuze ndibe nomsebenzi wokushumayela .\n",
      "2020-01-31 19:07:55,385 Example #1\n",
      "2020-01-31 19:07:55,386 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:07:55,386 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:07:55,386 \tHypothesis: Ngoxa wayetyelele ePoland , uLucaris wabona ukuba iOthodoki , ababingeleli , yaye babefanele banyaniseke , babenomdla wokuzonwabisa ngokomoya .\n",
      "2020-01-31 19:07:55,386 Example #2\n",
      "2020-01-31 19:07:55,386 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:07:55,386 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:07:55,386 \tHypothesis: Iphephandaba laseSpanish ithi : “ Ukubhekisela ekutshatweni kunokwahlukileyo . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:07:55,386 Example #3\n",
      "2020-01-31 19:07:55,387 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:07:55,387 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:07:55,387 \tHypothesis: Kuluphi na uphawu olukhulu nokuqeqesha abantu nokuqeqesha uThixo .\n",
      "2020-01-31 19:07:55,387 Validation result (greedy) at epoch   2, step    12000: bleu:  14.60, loss: 57648.9570, ppl:   7.8889, duration: 23.3713s\n",
      "2020-01-31 19:08:11,776 Epoch   2 Step:    12100 Batch Loss:     2.367202 Tokens per Sec:    15541, Lr: 0.000300\n",
      "2020-01-31 19:08:27,865 Epoch   2 Step:    12200 Batch Loss:     2.549181 Tokens per Sec:    15588, Lr: 0.000300\n",
      "2020-01-31 19:08:43,716 Epoch   2 Step:    12300 Batch Loss:     2.298818 Tokens per Sec:    15695, Lr: 0.000300\n",
      "2020-01-31 19:08:59,928 Epoch   2 Step:    12400 Batch Loss:     2.568420 Tokens per Sec:    15999, Lr: 0.000300\n",
      "2020-01-31 19:09:15,919 Epoch   2 Step:    12500 Batch Loss:     2.503845 Tokens per Sec:    15813, Lr: 0.000300\n",
      "2020-01-31 19:09:31,864 Epoch   2 Step:    12600 Batch Loss:     2.399755 Tokens per Sec:    15664, Lr: 0.000300\n",
      "2020-01-31 19:09:47,936 Epoch   2 Step:    12700 Batch Loss:     1.754016 Tokens per Sec:    16062, Lr: 0.000300\n",
      "2020-01-31 19:10:04,264 Epoch   2 Step:    12800 Batch Loss:     2.400566 Tokens per Sec:    15971, Lr: 0.000300\n",
      "2020-01-31 19:10:20,304 Epoch   2 Step:    12900 Batch Loss:     2.667967 Tokens per Sec:    16069, Lr: 0.000300\n",
      "2020-01-31 19:10:36,175 Epoch   2 Step:    13000 Batch Loss:     2.354126 Tokens per Sec:    15580, Lr: 0.000300\n",
      "2020-01-31 19:10:59,857 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:10:59,857 Saving new checkpoint.\n",
      "2020-01-31 19:11:00,994 Example #0\n",
      "2020-01-31 19:11:00,995 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:11:00,995 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:11:00,995 \tHypothesis: Ndandisandul ’ ukugxeka ukusebenzisa umsebenzi wokushumayela .\n",
      "2020-01-31 19:11:00,995 Example #1\n",
      "2020-01-31 19:11:00,995 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:11:00,995 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:11:00,995 \tHypothesis: Ngoxa wayelindele uPoland , uLucaris wabona ukuba amaOthodoki , ababingeleli nabafazi , babefanele benze inkqubela ngokomoya njengento yokomoya yokufundisa .\n",
      "2020-01-31 19:11:00,995 Example #2\n",
      "2020-01-31 19:11:00,996 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:11:00,996 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:11:00,996 \tHypothesis: Iphephandaba lithi : “ Yiba nomdla wokuhamba kunengonyama . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:11:00,996 Example #3\n",
      "2020-01-31 19:11:00,996 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:11:00,996 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:11:00,996 \tHypothesis: Kukufumana inkcazelo yokuzikhusela nokuqeqesha izenzo ezilungileyo nabantu noThixo .\n",
      "2020-01-31 19:11:00,996 Validation result (greedy) at epoch   2, step    13000: bleu:  15.44, loss: 56069.9844, ppl:   7.4550, duration: 24.8211s\n",
      "2020-01-31 19:11:17,169 Epoch   2 Step:    13100 Batch Loss:     2.329527 Tokens per Sec:    15768, Lr: 0.000300\n",
      "2020-01-31 19:11:33,136 Epoch   2 Step:    13200 Batch Loss:     2.236630 Tokens per Sec:    15928, Lr: 0.000300\n",
      "2020-01-31 19:11:49,079 Epoch   2 Step:    13300 Batch Loss:     2.103081 Tokens per Sec:    16087, Lr: 0.000300\n",
      "2020-01-31 19:12:05,203 Epoch   2 Step:    13400 Batch Loss:     2.340047 Tokens per Sec:    15647, Lr: 0.000300\n",
      "2020-01-31 19:12:21,124 Epoch   2 Step:    13500 Batch Loss:     1.976359 Tokens per Sec:    15824, Lr: 0.000300\n",
      "2020-01-31 19:12:37,048 Epoch   2 Step:    13600 Batch Loss:     2.642913 Tokens per Sec:    15972, Lr: 0.000300\n",
      "2020-01-31 19:12:53,083 Epoch   2 Step:    13700 Batch Loss:     2.192989 Tokens per Sec:    15938, Lr: 0.000300\n",
      "2020-01-31 19:13:09,294 Epoch   2 Step:    13800 Batch Loss:     2.297466 Tokens per Sec:    15589, Lr: 0.000300\n",
      "2020-01-31 19:13:25,373 Epoch   2 Step:    13900 Batch Loss:     2.387267 Tokens per Sec:    15733, Lr: 0.000300\n",
      "2020-01-31 19:13:41,293 Epoch   2 Step:    14000 Batch Loss:     2.053336 Tokens per Sec:    15878, Lr: 0.000300\n",
      "2020-01-31 19:14:06,682 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:14:06,683 Saving new checkpoint.\n",
      "2020-01-31 19:14:07,846 Example #0\n",
      "2020-01-31 19:14:07,847 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:14:07,847 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:14:07,847 \tHypothesis: Ndandisiya kufuna ukusebenzisa umsebenzi wokushumayela .\n",
      "2020-01-31 19:14:07,847 Example #1\n",
      "2020-01-31 19:14:07,848 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:14:07,848 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:14:07,848 \tHypothesis: Ngoxa wayetyelele uPoland , uLucaris wabona ukuba iOthodoki , ababingeleli nabafazi , babebonakala benobuchule bokomoya njengokumelana nemfundo yabo .\n",
      "2020-01-31 19:14:07,848 Example #2\n",
      "2020-01-31 19:14:07,849 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:14:07,849 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:14:07,849 \tHypothesis: Imbono yaseSpeyin ithi : “ Beta ukuhamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:14:07,849 Example #3\n",
      "2020-01-31 19:14:07,849 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:14:07,849 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:14:07,850 \tHypothesis: Kukho ukungxolela nokuqeqesha abantu nokuqeqesha kakuhle nabantu noThixo .\n",
      "2020-01-31 19:14:07,850 Validation result (greedy) at epoch   2, step    14000: bleu:  15.19, loss: 54974.9492, ppl:   7.1682, duration: 26.5564s\n",
      "2020-01-31 19:14:24,134 Epoch   2 Step:    14100 Batch Loss:     1.970557 Tokens per Sec:    15816, Lr: 0.000300\n",
      "2020-01-31 19:14:40,047 Epoch   2 Step:    14200 Batch Loss:     2.066210 Tokens per Sec:    15891, Lr: 0.000300\n",
      "2020-01-31 19:14:55,961 Epoch   2 Step:    14300 Batch Loss:     2.229270 Tokens per Sec:    15696, Lr: 0.000300\n",
      "2020-01-31 19:15:12,085 Epoch   2 Step:    14400 Batch Loss:     2.238399 Tokens per Sec:    15693, Lr: 0.000300\n",
      "2020-01-31 19:15:28,203 Epoch   2 Step:    14500 Batch Loss:     2.268014 Tokens per Sec:    15833, Lr: 0.000300\n",
      "2020-01-31 19:15:44,155 Epoch   2 Step:    14600 Batch Loss:     2.501756 Tokens per Sec:    15876, Lr: 0.000300\n",
      "2020-01-31 19:16:00,233 Epoch   2 Step:    14700 Batch Loss:     2.527726 Tokens per Sec:    15603, Lr: 0.000300\n",
      "2020-01-31 19:16:16,475 Epoch   2 Step:    14800 Batch Loss:     2.005878 Tokens per Sec:    15786, Lr: 0.000300\n",
      "2020-01-31 19:16:32,601 Epoch   2 Step:    14900 Batch Loss:     2.057127 Tokens per Sec:    15948, Lr: 0.000300\n",
      "2020-01-31 19:16:48,734 Epoch   2 Step:    15000 Batch Loss:     2.230197 Tokens per Sec:    16051, Lr: 0.000300\n",
      "2020-01-31 19:17:11,433 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:17:11,434 Saving new checkpoint.\n",
      "2020-01-31 19:17:12,535 Example #0\n",
      "2020-01-31 19:17:12,536 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:17:12,536 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:17:12,536 \tHypothesis: Ndandisuka ndafudukela kwisikolo sexesha elizeleyo .\n",
      "2020-01-31 19:17:12,536 Example #1\n",
      "2020-01-31 19:17:12,537 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:17:12,537 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:17:12,537 \tHypothesis: Ngoxa wayetyelele uPoland , uLucaris wabona ukuba abefundisi baseOthodoki , ababingeleli nabafazi , babebonakala benobuchule bokomoya njengokuphumelelayo imfundo yabo .\n",
      "2020-01-31 19:17:12,537 Example #2\n",
      "2020-01-31 19:17:12,537 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:17:12,537 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:17:12,537 \tHypothesis: Umzekeliso weSpanish uthi : “ UBeta ukuhamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:17:12,537 Example #3\n",
      "2020-01-31 19:17:12,538 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:17:12,538 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:17:12,538 \tHypothesis: Kuqikelelwa ukuba nenxaxheba , nokhuthazo oluhle nabantu noThixo .\n",
      "2020-01-31 19:17:12,538 Validation result (greedy) at epoch   2, step    15000: bleu:  15.89, loss: 53827.4023, ppl:   6.8794, duration: 23.8036s\n",
      "2020-01-31 19:17:29,147 Epoch   2 Step:    15100 Batch Loss:     2.345869 Tokens per Sec:    15232, Lr: 0.000300\n",
      "2020-01-31 19:17:45,120 Epoch   2 Step:    15200 Batch Loss:     2.159432 Tokens per Sec:    15960, Lr: 0.000300\n",
      "2020-01-31 19:18:01,237 Epoch   2 Step:    15300 Batch Loss:     2.075932 Tokens per Sec:    15834, Lr: 0.000300\n",
      "2020-01-31 19:18:17,380 Epoch   2 Step:    15400 Batch Loss:     2.784971 Tokens per Sec:    15743, Lr: 0.000300\n",
      "2020-01-31 19:18:33,397 Epoch   2 Step:    15500 Batch Loss:     1.966434 Tokens per Sec:    15750, Lr: 0.000300\n",
      "2020-01-31 19:18:49,387 Epoch   2 Step:    15600 Batch Loss:     1.899693 Tokens per Sec:    15938, Lr: 0.000300\n",
      "2020-01-31 19:19:05,629 Epoch   2 Step:    15700 Batch Loss:     2.166316 Tokens per Sec:    15987, Lr: 0.000300\n",
      "2020-01-31 19:19:21,813 Epoch   2 Step:    15800 Batch Loss:     2.289099 Tokens per Sec:    16203, Lr: 0.000300\n",
      "2020-01-31 19:19:37,630 Epoch   2 Step:    15900 Batch Loss:     1.911375 Tokens per Sec:    15855, Lr: 0.000300\n",
      "2020-01-31 19:19:53,647 Epoch   2 Step:    16000 Batch Loss:     2.141429 Tokens per Sec:    15946, Lr: 0.000300\n",
      "2020-01-31 19:20:15,662 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:20:15,662 Saving new checkpoint.\n",
      "2020-01-31 19:20:16,763 Example #0\n",
      "2020-01-31 19:20:16,763 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:20:16,763 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:20:16,763 \tHypothesis: Ndafudukela ukusebenzisa umsebenzi wokushumayela ngexesha elizeleyo .\n",
      "2020-01-31 19:20:16,763 Example #1\n",
      "2020-01-31 19:20:16,764 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:20:16,764 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:20:16,764 \tHypothesis: Ngoxa wayeqinisekile ukuba uPoland , uLucaris wabona ukuba abefundisi baseOthodoki , ababingeleli , babefana nababingeleli , babekhe babeneengxaki zokomoya njengokungafanelekanga .\n",
      "2020-01-31 19:20:16,764 Example #2\n",
      "2020-01-31 19:20:16,764 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:20:16,764 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:20:16,764 \tHypothesis: Umzekeliso weSpanish uthi : “ Better ukuhamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:20:16,764 Example #3\n",
      "2020-01-31 19:20:16,765 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:20:16,765 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:20:16,765 \tHypothesis: Kuqhelekile ukugxotha nokuqhubeka nokwahlula abantu noThixo .\n",
      "2020-01-31 19:20:16,765 Validation result (greedy) at epoch   2, step    16000: bleu:  16.42, loss: 52690.0898, ppl:   6.6048, duration: 23.1177s\n",
      "2020-01-31 19:20:33,005 Epoch   2 Step:    16100 Batch Loss:     2.507012 Tokens per Sec:    15512, Lr: 0.000300\n",
      "2020-01-31 19:20:48,974 Epoch   2 Step:    16200 Batch Loss:     2.087208 Tokens per Sec:    15845, Lr: 0.000300\n",
      "2020-01-31 19:21:05,366 Epoch   2 Step:    16300 Batch Loss:     2.058357 Tokens per Sec:    15734, Lr: 0.000300\n",
      "2020-01-31 19:21:21,609 Epoch   2 Step:    16400 Batch Loss:     1.950909 Tokens per Sec:    15944, Lr: 0.000300\n",
      "2020-01-31 19:21:37,673 Epoch   2 Step:    16500 Batch Loss:     2.181335 Tokens per Sec:    15960, Lr: 0.000300\n",
      "2020-01-31 19:21:53,734 Epoch   2 Step:    16600 Batch Loss:     2.146904 Tokens per Sec:    16039, Lr: 0.000300\n",
      "2020-01-31 19:22:09,995 Epoch   2 Step:    16700 Batch Loss:     2.047394 Tokens per Sec:    15728, Lr: 0.000300\n",
      "2020-01-31 19:22:25,894 Epoch   2 Step:    16800 Batch Loss:     2.076939 Tokens per Sec:    15694, Lr: 0.000300\n",
      "2020-01-31 19:22:41,936 Epoch   2 Step:    16900 Batch Loss:     2.171442 Tokens per Sec:    16029, Lr: 0.000300\n",
      "2020-01-31 19:22:57,813 Epoch   2 Step:    17000 Batch Loss:     2.276564 Tokens per Sec:    15983, Lr: 0.000300\n",
      "2020-01-31 19:23:19,693 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:23:19,694 Saving new checkpoint.\n",
      "2020-01-31 19:23:20,848 Example #0\n",
      "2020-01-31 19:23:20,850 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:23:20,850 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:23:20,850 \tHypothesis: Ndafudukela kwisikolo sexesha elizeleyo .\n",
      "2020-01-31 19:23:20,850 Example #1\n",
      "2020-01-31 19:23:20,851 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:23:20,851 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:23:20,851 \tHypothesis: Ngoxa wayetyelele uPoland , uLucaris wabona ukuba iOthodoki , ababingeleli nabafazi , babevaleka , babenomdla wokomoya njengoko bephelelwa lithemba .\n",
      "2020-01-31 19:23:20,851 Example #2\n",
      "2020-01-31 19:23:20,852 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:23:20,852 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:23:20,852 \tHypothesis: Iphephandaba laseSpanish ithi : “ Better bahamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:23:20,852 Example #3\n",
      "2020-01-31 19:23:20,853 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:23:20,853 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:23:20,854 \tHypothesis: Kuyinyaniso ukuba kuza kufuneka kuhlale unxulumano oluhle nabantu noThixo .\n",
      "2020-01-31 19:23:20,854 Validation result (greedy) at epoch   2, step    17000: bleu:  17.20, loss: 52571.7188, ppl:   6.5768, duration: 23.0402s\n",
      "2020-01-31 19:23:37,192 Epoch   2 Step:    17100 Batch Loss:     1.906890 Tokens per Sec:    15829, Lr: 0.000300\n",
      "2020-01-31 19:23:53,157 Epoch   2 Step:    17200 Batch Loss:     2.300030 Tokens per Sec:    15874, Lr: 0.000300\n",
      "2020-01-31 19:24:09,266 Epoch   2 Step:    17300 Batch Loss:     2.002616 Tokens per Sec:    15515, Lr: 0.000300\n",
      "2020-01-31 19:24:25,371 Epoch   2 Step:    17400 Batch Loss:     2.254629 Tokens per Sec:    15954, Lr: 0.000300\n",
      "2020-01-31 19:24:41,296 Epoch   2 Step:    17500 Batch Loss:     2.087426 Tokens per Sec:    15763, Lr: 0.000300\n",
      "2020-01-31 19:24:57,427 Epoch   2 Step:    17600 Batch Loss:     2.564584 Tokens per Sec:    16164, Lr: 0.000300\n",
      "2020-01-31 19:25:13,619 Epoch   2 Step:    17700 Batch Loss:     2.159960 Tokens per Sec:    15782, Lr: 0.000300\n",
      "2020-01-31 19:25:24,706 Epoch   2: total training loss 20487.91\n",
      "2020-01-31 19:25:24,706 EPOCH 3\n",
      "2020-01-31 19:25:30,621 Epoch   3 Step:    17800 Batch Loss:     2.080243 Tokens per Sec:    13198, Lr: 0.000300\n",
      "2020-01-31 19:25:46,647 Epoch   3 Step:    17900 Batch Loss:     2.154772 Tokens per Sec:    15931, Lr: 0.000300\n",
      "2020-01-31 19:26:02,681 Epoch   3 Step:    18000 Batch Loss:     2.379032 Tokens per Sec:    15646, Lr: 0.000300\n",
      "2020-01-31 19:26:25,815 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:26:25,815 Saving new checkpoint.\n",
      "2020-01-31 19:26:26,922 Example #0\n",
      "2020-01-31 19:26:26,923 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:26:26,923 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:26:26,923 \tHypothesis: Ndandisiya kufuna ukusebenzisa umsebenzi wokushumayela ngexesha elizeleyo .\n",
      "2020-01-31 19:26:26,923 Example #1\n",
      "2020-01-31 19:26:26,923 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:26:26,923 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:26:26,923 \tHypothesis: Ngoxa wayetyelele uPoland , uLucaris wabona ukuba abefundisi baseOthodoki , ababingeleli nabafazi , babefana nabangenamsebenzi wokomoya njengokuphumela imfundo yawo .\n",
      "2020-01-31 19:26:26,923 Example #2\n",
      "2020-01-31 19:26:26,924 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:26:26,924 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:26:26,924 \tHypothesis: Umzekeliso weSpanish uthi : “ Better bahamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:26:26,924 Example #3\n",
      "2020-01-31 19:26:26,924 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:26:26,924 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:26:26,924 \tHypothesis: Kuqhelekile ukucela nokuhlala abantu nokuhlala bethembekile noThixo .\n",
      "2020-01-31 19:26:26,924 Validation result (greedy) at epoch   3, step    18000: bleu:  17.92, loss: 50939.8203, ppl:   6.2033, duration: 24.2423s\n",
      "2020-01-31 19:26:43,286 Epoch   3 Step:    18100 Batch Loss:     2.231197 Tokens per Sec:    15768, Lr: 0.000300\n",
      "2020-01-31 19:26:59,210 Epoch   3 Step:    18200 Batch Loss:     2.142736 Tokens per Sec:    15934, Lr: 0.000300\n",
      "2020-01-31 19:27:15,237 Epoch   3 Step:    18300 Batch Loss:     2.051769 Tokens per Sec:    15823, Lr: 0.000300\n",
      "2020-01-31 19:27:31,414 Epoch   3 Step:    18400 Batch Loss:     1.882968 Tokens per Sec:    15924, Lr: 0.000300\n",
      "2020-01-31 19:27:47,176 Epoch   3 Step:    18500 Batch Loss:     2.158881 Tokens per Sec:    15784, Lr: 0.000300\n",
      "2020-01-31 19:28:03,200 Epoch   3 Step:    18600 Batch Loss:     2.056439 Tokens per Sec:    15899, Lr: 0.000300\n",
      "2020-01-31 19:28:19,282 Epoch   3 Step:    18700 Batch Loss:     2.078226 Tokens per Sec:    15920, Lr: 0.000300\n",
      "2020-01-31 19:28:35,591 Epoch   3 Step:    18800 Batch Loss:     2.008245 Tokens per Sec:    15521, Lr: 0.000300\n",
      "2020-01-31 19:28:51,432 Epoch   3 Step:    18900 Batch Loss:     1.947831 Tokens per Sec:    16068, Lr: 0.000300\n",
      "2020-01-31 19:29:07,611 Epoch   3 Step:    19000 Batch Loss:     2.013793 Tokens per Sec:    15845, Lr: 0.000300\n",
      "2020-01-31 19:29:30,733 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:29:30,733 Saving new checkpoint.\n",
      "2020-01-31 19:29:32,097 Example #0\n",
      "2020-01-31 19:29:32,097 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:29:32,097 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:29:32,098 \tHypothesis: Ndafudukela ukuba ndibe nesabelo sokushumayela kwixesha elizeleyo .\n",
      "2020-01-31 19:29:32,098 Example #1\n",
      "2020-01-31 19:29:32,098 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:29:32,099 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:29:32,099 \tHypothesis: Ngoxa wayetyelele uPoland , uLucaris wabona ukuba abefundisi baseOthodoki apho , ababingeleli nabafazi , babefana nabangenabutho , babenomdla wokomoya ngenxa yokungafi kwemfundo yabo .\n",
      "2020-01-31 19:29:32,099 Example #2\n",
      "2020-01-31 19:29:32,099 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:29:32,099 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:29:32,101 \tHypothesis: Umzekeliso weSpanish uthi : “ Beta ukuhamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:29:32,101 Example #3\n",
      "2020-01-31 19:29:32,101 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:29:32,101 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:29:32,102 \tHypothesis: Kukucela ukuhlala nokona kulungileyo nabantu noThixo .\n",
      "2020-01-31 19:29:32,102 Validation result (greedy) at epoch   3, step    19000: bleu:  18.11, loss: 50313.3789, ppl:   6.0656, duration: 24.4904s\n",
      "2020-01-31 19:29:48,369 Epoch   3 Step:    19100 Batch Loss:     2.414448 Tokens per Sec:    15719, Lr: 0.000300\n",
      "2020-01-31 19:30:04,721 Epoch   3 Step:    19200 Batch Loss:     2.365117 Tokens per Sec:    15833, Lr: 0.000300\n",
      "2020-01-31 19:30:20,920 Epoch   3 Step:    19300 Batch Loss:     2.160855 Tokens per Sec:    15927, Lr: 0.000300\n",
      "2020-01-31 19:30:37,015 Epoch   3 Step:    19400 Batch Loss:     2.007684 Tokens per Sec:    15954, Lr: 0.000300\n",
      "2020-01-31 19:30:53,040 Epoch   3 Step:    19500 Batch Loss:     2.147624 Tokens per Sec:    16053, Lr: 0.000300\n",
      "2020-01-31 19:31:09,298 Epoch   3 Step:    19600 Batch Loss:     1.987720 Tokens per Sec:    15560, Lr: 0.000300\n",
      "2020-01-31 19:31:25,449 Epoch   3 Step:    19700 Batch Loss:     2.070888 Tokens per Sec:    15798, Lr: 0.000300\n",
      "2020-01-31 19:31:41,515 Epoch   3 Step:    19800 Batch Loss:     2.032286 Tokens per Sec:    15821, Lr: 0.000300\n",
      "2020-01-31 19:31:57,418 Epoch   3 Step:    19900 Batch Loss:     2.181452 Tokens per Sec:    15985, Lr: 0.000300\n",
      "2020-01-31 19:32:13,565 Epoch   3 Step:    20000 Batch Loss:     1.926594 Tokens per Sec:    15713, Lr: 0.000300\n",
      "2020-01-31 19:32:35,093 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:32:35,093 Saving new checkpoint.\n",
      "2020-01-31 19:32:36,180 Example #0\n",
      "2020-01-31 19:32:36,180 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:32:36,180 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:32:36,180 \tHypothesis: Ndandishukunyiselwa ukuba ndiphinde ndibe nomsebenzi wokushumayela ngexesha elizeleyo .\n",
      "2020-01-31 19:32:36,180 Example #1\n",
      "2020-01-31 19:32:36,181 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:32:36,181 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:32:36,181 \tHypothesis: Ngoxa wayetyelele uPoland , uLucaris wabona ukuba iOthodoki , ababingeleli nabafazi , babekuvuyela ngokomoya njengemeko yokomoya njengemfundo .\n",
      "2020-01-31 19:32:36,181 Example #2\n",
      "2020-01-31 19:32:36,181 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:32:36,181 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:32:36,181 \tHypothesis: Umzekeliso weSpeyin uthi : “ Ukubhekisela ekuhambeni kwexesha kunokuba nomtshato . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:32:36,181 Example #3\n",
      "2020-01-31 19:32:36,181 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:32:36,181 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:32:36,182 \tHypothesis: Kuqhelekile ukuncokola nokuhlala nempembelelo entle nabantu noThixo .\n",
      "2020-01-31 19:32:36,182 Validation result (greedy) at epoch   3, step    20000: bleu:  18.20, loss: 49794.1797, ppl:   5.9538, duration: 22.6158s\n",
      "2020-01-31 19:32:52,380 Epoch   3 Step:    20100 Batch Loss:     1.764348 Tokens per Sec:    15914, Lr: 0.000300\n",
      "2020-01-31 19:33:08,454 Epoch   3 Step:    20200 Batch Loss:     1.939225 Tokens per Sec:    15768, Lr: 0.000300\n",
      "2020-01-31 19:33:24,418 Epoch   3 Step:    20300 Batch Loss:     2.202337 Tokens per Sec:    15964, Lr: 0.000300\n",
      "2020-01-31 19:33:40,254 Epoch   3 Step:    20400 Batch Loss:     2.252888 Tokens per Sec:    15988, Lr: 0.000300\n",
      "2020-01-31 19:33:56,216 Epoch   3 Step:    20500 Batch Loss:     2.004152 Tokens per Sec:    16080, Lr: 0.000300\n",
      "2020-01-31 19:34:12,368 Epoch   3 Step:    20600 Batch Loss:     2.222592 Tokens per Sec:    15659, Lr: 0.000300\n",
      "2020-01-31 19:34:28,504 Epoch   3 Step:    20700 Batch Loss:     2.224942 Tokens per Sec:    16008, Lr: 0.000300\n",
      "2020-01-31 19:34:44,346 Epoch   3 Step:    20800 Batch Loss:     1.907822 Tokens per Sec:    16006, Lr: 0.000300\n",
      "2020-01-31 19:35:00,335 Epoch   3 Step:    20900 Batch Loss:     2.145103 Tokens per Sec:    15819, Lr: 0.000300\n",
      "2020-01-31 19:35:16,358 Epoch   3 Step:    21000 Batch Loss:     1.965159 Tokens per Sec:    15952, Lr: 0.000300\n",
      "2020-01-31 19:35:38,406 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:35:38,406 Saving new checkpoint.\n",
      "2020-01-31 19:35:39,427 Example #0\n",
      "2020-01-31 19:35:39,427 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:35:39,428 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:35:39,428 \tHypothesis: Ndafudukela ukusebenza kwixesha elizeleyo .\n",
      "2020-01-31 19:35:39,428 Example #1\n",
      "2020-01-31 19:35:39,428 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:35:39,428 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:35:39,429 \tHypothesis: Ngoxa wayetyelele uPoland , uLucaris wabona ukuba iOthodoki , ababingeleli nabafazi , babebanjwa , babekwimeko yokomoya eyingozi ngenxa yokungabi namsebenzi wabo wokufundisa .\n",
      "2020-01-31 19:35:39,429 Example #2\n",
      "2020-01-31 19:35:39,429 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:35:39,429 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:35:39,429 \tHypothesis: Umzekeliso weSpanish uthi : “ Ukubona ukuhamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:35:39,430 Example #3\n",
      "2020-01-31 19:35:39,430 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:35:39,430 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:35:39,430 \tHypothesis: Kuqikelelwa ukuba nenxaxheba yaye kuhlala nanamhlanje nabantu noThixo .\n",
      "2020-01-31 19:35:39,430 Validation result (greedy) at epoch   3, step    21000: bleu:  18.65, loss: 48987.7539, ppl:   5.7843, duration: 23.0716s\n",
      "2020-01-31 19:35:55,527 Epoch   3 Step:    21100 Batch Loss:     2.230871 Tokens per Sec:    15593, Lr: 0.000300\n",
      "2020-01-31 19:36:11,653 Epoch   3 Step:    21200 Batch Loss:     2.083902 Tokens per Sec:    15661, Lr: 0.000300\n",
      "2020-01-31 19:36:27,482 Epoch   3 Step:    21300 Batch Loss:     2.146367 Tokens per Sec:    15590, Lr: 0.000300\n",
      "2020-01-31 19:36:43,439 Epoch   3 Step:    21400 Batch Loss:     1.990246 Tokens per Sec:    15917, Lr: 0.000300\n",
      "2020-01-31 19:36:59,439 Epoch   3 Step:    21500 Batch Loss:     2.039554 Tokens per Sec:    15675, Lr: 0.000300\n",
      "2020-01-31 19:37:15,558 Epoch   3 Step:    21600 Batch Loss:     1.660940 Tokens per Sec:    16098, Lr: 0.000300\n",
      "2020-01-31 19:37:31,261 Epoch   3 Step:    21700 Batch Loss:     2.071054 Tokens per Sec:    15833, Lr: 0.000300\n",
      "2020-01-31 19:37:47,254 Epoch   3 Step:    21800 Batch Loss:     2.145315 Tokens per Sec:    15892, Lr: 0.000300\n",
      "2020-01-31 19:38:03,444 Epoch   3 Step:    21900 Batch Loss:     2.064876 Tokens per Sec:    15785, Lr: 0.000300\n",
      "2020-01-31 19:38:19,448 Epoch   3 Step:    22000 Batch Loss:     1.814961 Tokens per Sec:    15721, Lr: 0.000300\n",
      "2020-01-31 19:38:42,042 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:38:42,042 Saving new checkpoint.\n",
      "2020-01-31 19:38:43,150 Example #0\n",
      "2020-01-31 19:38:43,151 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:38:43,151 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:38:43,151 \tHypothesis: Ndandishukunyiselwa ukuba ndiphume umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:38:43,151 Example #1\n",
      "2020-01-31 19:38:43,151 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:38:43,151 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:38:43,151 \tHypothesis: Ngoxa wayetyelele uPoland , uLucaris wabona ukuba iOthodoki apho , ababingeleli nabefundisi , babebeka , babenomdla wokomoya njengokungabi namnye wabo wokufundisa .\n",
      "2020-01-31 19:38:43,151 Example #2\n",
      "2020-01-31 19:38:43,152 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:38:43,152 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:38:43,152 \tHypothesis: Umzekeliso weSpanish uthi : “ Better bahamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:38:43,152 Example #3\n",
      "2020-01-31 19:38:43,152 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:38:43,152 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:38:43,152 \tHypothesis: Kuqhelekile ukugcina izikhundla ezintle nabantu noThixo .\n",
      "2020-01-31 19:38:43,152 Validation result (greedy) at epoch   3, step    22000: bleu:  18.84, loss: 48507.9180, ppl:   5.6857, duration: 23.7037s\n",
      "2020-01-31 19:38:59,556 Epoch   3 Step:    22100 Batch Loss:     1.676952 Tokens per Sec:    15437, Lr: 0.000300\n",
      "2020-01-31 19:39:15,652 Epoch   3 Step:    22200 Batch Loss:     2.090927 Tokens per Sec:    15837, Lr: 0.000300\n",
      "2020-01-31 19:39:31,434 Epoch   3 Step:    22300 Batch Loss:     1.839975 Tokens per Sec:    15699, Lr: 0.000300\n",
      "2020-01-31 19:39:47,457 Epoch   3 Step:    22400 Batch Loss:     1.816359 Tokens per Sec:    16142, Lr: 0.000300\n",
      "2020-01-31 19:40:03,525 Epoch   3 Step:    22500 Batch Loss:     1.874738 Tokens per Sec:    15803, Lr: 0.000300\n",
      "2020-01-31 19:40:19,479 Epoch   3 Step:    22600 Batch Loss:     2.088058 Tokens per Sec:    15898, Lr: 0.000300\n",
      "2020-01-31 19:40:35,533 Epoch   3 Step:    22700 Batch Loss:     1.895125 Tokens per Sec:    16083, Lr: 0.000300\n",
      "2020-01-31 19:40:51,395 Epoch   3 Step:    22800 Batch Loss:     2.104737 Tokens per Sec:    15865, Lr: 0.000300\n",
      "2020-01-31 19:41:07,249 Epoch   3 Step:    22900 Batch Loss:     1.738716 Tokens per Sec:    15519, Lr: 0.000300\n",
      "2020-01-31 19:41:23,204 Epoch   3 Step:    23000 Batch Loss:     1.926125 Tokens per Sec:    15290, Lr: 0.000300\n",
      "2020-01-31 19:41:45,779 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:41:45,781 Saving new checkpoint.\n",
      "2020-01-31 19:41:46,900 Example #0\n",
      "2020-01-31 19:41:46,900 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:41:46,900 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:41:46,900 \tHypothesis: Ndafudukela ukusebenza kwixesha elizeleyo .\n",
      "2020-01-31 19:41:46,900 Example #1\n",
      "2020-01-31 19:41:46,901 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:41:46,901 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:41:46,901 \tHypothesis: Ngoxa wayetyelele uPoland , uLucaris wabona ukuba iOthodoki apho , ababingeleli nababingeleli , babebulawa , babenomdla wokomoya ngenxa yokungabi naluso lwabo .\n",
      "2020-01-31 19:41:46,901 Example #2\n",
      "2020-01-31 19:41:46,901 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:41:46,901 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:41:46,901 \tHypothesis: Umzekeliso weSpanish uthi : “ Better bahamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:41:46,901 Example #3\n",
      "2020-01-31 19:41:46,902 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:41:46,902 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:41:46,902 \tHypothesis: Kuqhelekile ukuhlala unxulumene nabantu noThixo .\n",
      "2020-01-31 19:41:46,902 Validation result (greedy) at epoch   3, step    23000: bleu:  19.42, loss: 47846.7305, ppl:   5.5526, duration: 23.6974s\n",
      "2020-01-31 19:42:03,055 Epoch   3 Step:    23100 Batch Loss:     2.109970 Tokens per Sec:    15760, Lr: 0.000300\n",
      "2020-01-31 19:42:19,085 Epoch   3 Step:    23200 Batch Loss:     1.734149 Tokens per Sec:    16024, Lr: 0.000300\n",
      "2020-01-31 19:42:35,248 Epoch   3 Step:    23300 Batch Loss:     2.115801 Tokens per Sec:    15996, Lr: 0.000300\n",
      "2020-01-31 19:42:51,074 Epoch   3 Step:    23400 Batch Loss:     1.978383 Tokens per Sec:    15822, Lr: 0.000300\n",
      "2020-01-31 19:43:07,267 Epoch   3 Step:    23500 Batch Loss:     1.964594 Tokens per Sec:    15975, Lr: 0.000300\n",
      "2020-01-31 19:43:23,288 Epoch   3 Step:    23600 Batch Loss:     1.810507 Tokens per Sec:    15673, Lr: 0.000300\n",
      "2020-01-31 19:43:39,316 Epoch   3 Step:    23700 Batch Loss:     1.867815 Tokens per Sec:    15992, Lr: 0.000300\n",
      "2020-01-31 19:43:55,318 Epoch   3 Step:    23800 Batch Loss:     1.921024 Tokens per Sec:    15914, Lr: 0.000300\n",
      "2020-01-31 19:44:11,150 Epoch   3 Step:    23900 Batch Loss:     1.794699 Tokens per Sec:    15657, Lr: 0.000300\n",
      "2020-01-31 19:44:27,228 Epoch   3 Step:    24000 Batch Loss:     1.890642 Tokens per Sec:    16050, Lr: 0.000300\n",
      "2020-01-31 19:44:50,807 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:44:50,808 Saving new checkpoint.\n",
      "2020-01-31 19:44:51,900 Example #0\n",
      "2020-01-31 19:44:51,900 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:44:51,901 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:44:51,901 \tHypothesis: Ndafudukela ekushumayeleni kwixesha elizeleyo .\n",
      "2020-01-31 19:44:51,901 Example #1\n",
      "2020-01-31 19:44:51,901 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:44:51,901 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:44:51,901 \tHypothesis: Ngoxa wayetyelele uPoland , uLucaris wabona ukuba iOthodoki , ababingeleli nababingeleli , babebandezeleka , babenomdla wokomoya ngenxa yokungabi nako ukufundiswa kwabo .\n",
      "2020-01-31 19:44:51,901 Example #2\n",
      "2020-01-31 19:44:51,901 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:44:51,902 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:44:51,902 \tHypothesis: Umzekeliso weSpanish uthi : “ Ukuncokola kunokwahlukana okubi kunokuba kutshate . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:44:51,902 Example #3\n",
      "2020-01-31 19:44:51,902 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:44:51,902 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:44:51,902 \tHypothesis: Kuqikelelwa ukuba nenxaxheba yaye nihlale nentembelo entle nabantu noThixo .\n",
      "2020-01-31 19:44:51,902 Validation result (greedy) at epoch   3, step    24000: bleu:  19.81, loss: 47432.7109, ppl:   5.4708, duration: 24.6734s\n",
      "2020-01-31 19:45:08,143 Epoch   3 Step:    24100 Batch Loss:     1.955562 Tokens per Sec:    15439, Lr: 0.000300\n",
      "2020-01-31 19:45:24,173 Epoch   3 Step:    24200 Batch Loss:     1.876623 Tokens per Sec:    15800, Lr: 0.000300\n",
      "2020-01-31 19:45:40,104 Epoch   3 Step:    24300 Batch Loss:     1.754569 Tokens per Sec:    15885, Lr: 0.000300\n",
      "2020-01-31 19:45:56,129 Epoch   3 Step:    24400 Batch Loss:     2.058750 Tokens per Sec:    15998, Lr: 0.000300\n",
      "2020-01-31 19:46:12,206 Epoch   3 Step:    24500 Batch Loss:     1.950211 Tokens per Sec:    15721, Lr: 0.000300\n",
      "2020-01-31 19:46:28,423 Epoch   3 Step:    24600 Batch Loss:     2.038509 Tokens per Sec:    16098, Lr: 0.000300\n",
      "2020-01-31 19:46:44,426 Epoch   3 Step:    24700 Batch Loss:     1.664691 Tokens per Sec:    15913, Lr: 0.000300\n",
      "2020-01-31 19:47:00,516 Epoch   3 Step:    24800 Batch Loss:     1.767862 Tokens per Sec:    15996, Lr: 0.000300\n",
      "2020-01-31 19:47:16,559 Epoch   3 Step:    24900 Batch Loss:     1.912610 Tokens per Sec:    15804, Lr: 0.000300\n",
      "2020-01-31 19:47:32,806 Epoch   3 Step:    25000 Batch Loss:     1.848682 Tokens per Sec:    16049, Lr: 0.000300\n",
      "2020-01-31 19:47:57,386 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:47:57,386 Saving new checkpoint.\n",
      "2020-01-31 19:48:00,349 Example #0\n",
      "2020-01-31 19:48:00,350 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:48:00,351 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:48:00,351 \tHypothesis: Ndandishukunyiselwa ukuba ndibe nesabelo kumsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:48:00,351 Example #1\n",
      "2020-01-31 19:48:00,351 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:48:00,351 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:48:00,351 \tHypothesis: Ngoxa wayetyelele uPoland , uLucaris wabona ukuba iOthodoki apho , ababingeleli nabahlulukana nabo , babekade banyamekela imeko yokomoya ngenxa yokungafuni kwabo imfundo .\n",
      "2020-01-31 19:48:00,351 Example #2\n",
      "2020-01-31 19:48:00,352 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:48:00,352 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:48:00,352 \tHypothesis: Umzekeliso weSpanish uthi : “ Khangela ukuhamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:48:00,352 Example #3\n",
      "2020-01-31 19:48:00,352 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:48:00,352 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:48:00,352 \tHypothesis: Kuqhelekile ukucela nokugcina abantu nokuhlala behlonela uThixo .\n",
      "2020-01-31 19:48:00,353 Validation result (greedy) at epoch   3, step    25000: bleu:  19.77, loss: 46872.3086, ppl:   5.3621, duration: 27.5464s\n",
      "2020-01-31 19:48:16,636 Epoch   3 Step:    25100 Batch Loss:     2.017351 Tokens per Sec:    15629, Lr: 0.000300\n",
      "2020-01-31 19:48:32,894 Epoch   3 Step:    25200 Batch Loss:     2.334259 Tokens per Sec:    16041, Lr: 0.000300\n",
      "2020-01-31 19:48:49,068 Epoch   3 Step:    25300 Batch Loss:     2.008636 Tokens per Sec:    15973, Lr: 0.000300\n",
      "2020-01-31 19:49:04,995 Epoch   3 Step:    25400 Batch Loss:     1.944842 Tokens per Sec:    15689, Lr: 0.000300\n",
      "2020-01-31 19:49:20,955 Epoch   3 Step:    25500 Batch Loss:     2.046381 Tokens per Sec:    15890, Lr: 0.000300\n",
      "2020-01-31 19:49:36,866 Epoch   3 Step:    25600 Batch Loss:     1.991149 Tokens per Sec:    15965, Lr: 0.000300\n",
      "2020-01-31 19:49:52,625 Epoch   3 Step:    25700 Batch Loss:     1.902037 Tokens per Sec:    15816, Lr: 0.000300\n",
      "2020-01-31 19:50:08,694 Epoch   3 Step:    25800 Batch Loss:     2.172020 Tokens per Sec:    16015, Lr: 0.000300\n",
      "2020-01-31 19:50:24,732 Epoch   3 Step:    25900 Batch Loss:     1.865381 Tokens per Sec:    15713, Lr: 0.000300\n",
      "2020-01-31 19:50:40,677 Epoch   3 Step:    26000 Batch Loss:     1.810811 Tokens per Sec:    15996, Lr: 0.000300\n",
      "2020-01-31 19:51:00,568 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:51:00,568 Saving new checkpoint.\n",
      "2020-01-31 19:51:01,964 Example #0\n",
      "2020-01-31 19:51:01,965 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:51:01,965 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:51:01,966 \tHypothesis: Ndafudukela ukuba ndilahle umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:51:01,966 Example #1\n",
      "2020-01-31 19:51:01,967 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:51:01,967 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:51:01,967 \tHypothesis: Ngoxa wayetyelele uPoland , uLucaris wabona ukuba iOthodoki , ababingeleli nababingeleli , babebulawa , babeneengxaki ezinzulu zokomoya ngenxa yokungabi namsebenzi .\n",
      "2020-01-31 19:51:01,967 Example #2\n",
      "2020-01-31 19:51:01,968 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:51:01,968 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:51:01,968 \tHypothesis: Umzekeliso waseSpeyin uthi : “ Beth ukuhamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:51:01,969 Example #3\n",
      "2020-01-31 19:51:01,969 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:51:01,969 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:51:01,969 \tHypothesis: Kuqhelekile ukucela nokugcina abantu noThixo .\n",
      "2020-01-31 19:51:01,969 Validation result (greedy) at epoch   3, step    26000: bleu:  20.69, loss: 46487.9492, ppl:   5.2887, duration: 21.2915s\n",
      "2020-01-31 19:51:18,007 Epoch   3 Step:    26100 Batch Loss:     1.984425 Tokens per Sec:    15602, Lr: 0.000300\n",
      "2020-01-31 19:51:34,354 Epoch   3 Step:    26200 Batch Loss:     2.159876 Tokens per Sec:    15656, Lr: 0.000300\n",
      "2020-01-31 19:51:50,437 Epoch   3 Step:    26300 Batch Loss:     2.287200 Tokens per Sec:    16169, Lr: 0.000300\n",
      "2020-01-31 19:52:06,686 Epoch   3 Step:    26400 Batch Loss:     1.978302 Tokens per Sec:    15807, Lr: 0.000300\n",
      "2020-01-31 19:52:22,587 Epoch   3 Step:    26500 Batch Loss:     1.995272 Tokens per Sec:    15937, Lr: 0.000300\n",
      "2020-01-31 19:52:38,497 Epoch   3 Step:    26600 Batch Loss:     2.198090 Tokens per Sec:    16006, Lr: 0.000300\n",
      "2020-01-31 19:52:47,662 Epoch   3: total training loss 17875.30\n",
      "2020-01-31 19:52:47,663 EPOCH 4\n",
      "2020-01-31 19:52:55,588 Epoch   4 Step:    26700 Batch Loss:     2.101232 Tokens per Sec:    13858, Lr: 0.000300\n",
      "2020-01-31 19:53:11,769 Epoch   4 Step:    26800 Batch Loss:     1.871485 Tokens per Sec:    15853, Lr: 0.000300\n",
      "2020-01-31 19:53:27,901 Epoch   4 Step:    26900 Batch Loss:     1.849780 Tokens per Sec:    16020, Lr: 0.000300\n",
      "2020-01-31 19:53:44,097 Epoch   4 Step:    27000 Batch Loss:     1.948030 Tokens per Sec:    16066, Lr: 0.000300\n",
      "2020-01-31 19:54:08,619 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:54:08,619 Saving new checkpoint.\n",
      "2020-01-31 19:54:09,755 Example #0\n",
      "2020-01-31 19:54:09,756 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:54:09,756 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:54:09,756 \tHypothesis: Ndafudukela ukuba ndibe nesabelo kumsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:54:09,756 Example #1\n",
      "2020-01-31 19:54:09,757 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:54:09,757 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:54:09,757 \tHypothesis: Ngoxa wayetyelele ePoland , uLucaris wabona ukuba iOthodoki apho , ababingeleli nabahlulahlulayo , babeyimeko eyingozi yokomoya eyingozi ngenxa yokungabi namfundo .\n",
      "2020-01-31 19:54:09,757 Example #2\n",
      "2020-01-31 19:54:09,758 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:54:09,758 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:54:09,758 \tHypothesis: Umzekeliso weSpanish uthi : “ Beta ukuhamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:54:09,758 Example #3\n",
      "2020-01-31 19:54:09,758 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:54:09,759 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:54:09,759 \tHypothesis: Kuyinto esisiseko ekuxoleleni nokuhlala sineempawu ezintle nabantu noThixo .\n",
      "2020-01-31 19:54:09,759 Validation result (greedy) at epoch   4, step    27000: bleu:  20.64, loss: 45977.6719, ppl:   5.1929, duration: 25.6613s\n",
      "2020-01-31 19:54:26,083 Epoch   4 Step:    27100 Batch Loss:     1.933636 Tokens per Sec:    15302, Lr: 0.000300\n",
      "2020-01-31 19:54:41,938 Epoch   4 Step:    27200 Batch Loss:     1.694029 Tokens per Sec:    15762, Lr: 0.000300\n",
      "2020-01-31 19:54:58,007 Epoch   4 Step:    27300 Batch Loss:     1.597217 Tokens per Sec:    15919, Lr: 0.000300\n",
      "2020-01-31 19:55:13,971 Epoch   4 Step:    27400 Batch Loss:     2.012005 Tokens per Sec:    15797, Lr: 0.000300\n",
      "2020-01-31 19:55:29,962 Epoch   4 Step:    27500 Batch Loss:     1.948907 Tokens per Sec:    15889, Lr: 0.000300\n",
      "2020-01-31 19:55:46,067 Epoch   4 Step:    27600 Batch Loss:     1.815176 Tokens per Sec:    15865, Lr: 0.000300\n",
      "2020-01-31 19:56:01,946 Epoch   4 Step:    27700 Batch Loss:     1.808047 Tokens per Sec:    15806, Lr: 0.000300\n",
      "2020-01-31 19:56:18,055 Epoch   4 Step:    27800 Batch Loss:     1.828019 Tokens per Sec:    16031, Lr: 0.000300\n",
      "2020-01-31 19:56:34,164 Epoch   4 Step:    27900 Batch Loss:     1.651852 Tokens per Sec:    15817, Lr: 0.000300\n",
      "2020-01-31 19:56:50,114 Epoch   4 Step:    28000 Batch Loss:     1.984577 Tokens per Sec:    15866, Lr: 0.000300\n",
      "2020-01-31 19:57:13,436 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 19:57:13,437 Saving new checkpoint.\n",
      "2020-01-31 19:57:14,638 Example #0\n",
      "2020-01-31 19:57:14,639 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 19:57:14,639 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 19:57:14,639 \tHypothesis: Ndafudukela ukuba ndiphumelele umsebenzi wokushumayela ixesha elizeleyo .\n",
      "2020-01-31 19:57:14,639 Example #1\n",
      "2020-01-31 19:57:14,640 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 19:57:14,640 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 19:57:14,640 \tHypothesis: Ngoxa wayetyelela uPoland , uLucaris wabona ukuba iOthodoki apho , ababingeleli nabahlulayo , babenomdla wokomoya njengoko babephumela kwimfundo yabo .\n",
      "2020-01-31 19:57:14,640 Example #2\n",
      "2020-01-31 19:57:14,641 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 19:57:14,641 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:57:14,641 \tHypothesis: Umzekeliso waseSpeyin uthi : “ Better bahamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 19:57:14,641 Example #3\n",
      "2020-01-31 19:57:14,641 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 19:57:14,641 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 19:57:14,642 \tHypothesis: Kuyinto ebalulekileyo ekunyamezeleni nokugcina izibakala ezilungileyo nabantu noThixo .\n",
      "2020-01-31 19:57:14,642 Validation result (greedy) at epoch   4, step    28000: bleu:  20.76, loss: 45603.0117, ppl:   5.1237, duration: 24.5269s\n",
      "2020-01-31 19:57:30,930 Epoch   4 Step:    28100 Batch Loss:     1.701148 Tokens per Sec:    15581, Lr: 0.000300\n",
      "2020-01-31 19:57:46,848 Epoch   4 Step:    28200 Batch Loss:     1.512524 Tokens per Sec:    15913, Lr: 0.000300\n",
      "2020-01-31 19:58:02,938 Epoch   4 Step:    28300 Batch Loss:     1.914443 Tokens per Sec:    15919, Lr: 0.000300\n",
      "2020-01-31 19:58:18,794 Epoch   4 Step:    28400 Batch Loss:     1.847424 Tokens per Sec:    15880, Lr: 0.000300\n",
      "2020-01-31 19:58:34,899 Epoch   4 Step:    28500 Batch Loss:     1.649415 Tokens per Sec:    15851, Lr: 0.000300\n",
      "2020-01-31 19:58:50,851 Epoch   4 Step:    28600 Batch Loss:     1.828344 Tokens per Sec:    15905, Lr: 0.000300\n",
      "2020-01-31 19:59:06,843 Epoch   4 Step:    28700 Batch Loss:     1.995473 Tokens per Sec:    15842, Lr: 0.000300\n",
      "2020-01-31 19:59:22,870 Epoch   4 Step:    28800 Batch Loss:     1.828884 Tokens per Sec:    15994, Lr: 0.000300\n",
      "2020-01-31 19:59:38,819 Epoch   4 Step:    28900 Batch Loss:     1.944134 Tokens per Sec:    15893, Lr: 0.000300\n",
      "2020-01-31 19:59:54,695 Epoch   4 Step:    29000 Batch Loss:     1.663180 Tokens per Sec:    15830, Lr: 0.000300\n",
      "2020-01-31 20:00:19,507 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 20:00:19,508 Saving new checkpoint.\n",
      "2020-01-31 20:00:20,613 Example #0\n",
      "2020-01-31 20:00:20,613 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 20:00:20,614 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 20:00:20,614 \tHypothesis: Ndandishukunyiselwa ukuba ndiphumelele umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 20:00:20,614 Example #1\n",
      "2020-01-31 20:00:20,614 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 20:00:20,614 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 20:00:20,614 \tHypothesis: Ngoxa wayetyelela uPoland , uLucaris wabona ukuba iOthodoki apho , ababingeleli nabahlulukahlukeneyo , babekwimeko ebalulekileyo yokomoya ngenxa yokungabi namfundo .\n",
      "2020-01-31 20:00:20,615 Example #2\n",
      "2020-01-31 20:00:20,615 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 20:00:20,615 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 20:00:20,615 \tHypothesis: Umzekeliso weSpeyin uthi : “ Ukubona ukuhamba ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 20:00:20,616 Example #3\n",
      "2020-01-31 20:00:20,616 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 20:00:20,616 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 20:00:20,616 \tHypothesis: Kuluncedo ekuxoleleni nokuhlala uhlakulele abantu nabantu noThixo .\n",
      "2020-01-31 20:00:20,616 Validation result (greedy) at epoch   4, step    29000: bleu:  21.11, loss: 45219.0039, ppl:   5.0537, duration: 25.9205s\n",
      "2020-01-31 20:00:36,802 Epoch   4 Step:    29100 Batch Loss:     1.851156 Tokens per Sec:    15644, Lr: 0.000300\n",
      "2020-01-31 20:00:52,569 Epoch   4 Step:    29200 Batch Loss:     1.930954 Tokens per Sec:    16059, Lr: 0.000300\n",
      "2020-01-31 20:01:08,652 Epoch   4 Step:    29300 Batch Loss:     1.707275 Tokens per Sec:    15995, Lr: 0.000300\n",
      "2020-01-31 20:01:24,576 Epoch   4 Step:    29400 Batch Loss:     1.938883 Tokens per Sec:    15935, Lr: 0.000300\n",
      "2020-01-31 20:01:40,475 Epoch   4 Step:    29500 Batch Loss:     2.056416 Tokens per Sec:    15907, Lr: 0.000300\n",
      "2020-01-31 20:01:56,354 Epoch   4 Step:    29600 Batch Loss:     1.911711 Tokens per Sec:    16018, Lr: 0.000300\n",
      "2020-01-31 20:02:12,665 Epoch   4 Step:    29700 Batch Loss:     1.961162 Tokens per Sec:    16166, Lr: 0.000300\n",
      "2020-01-31 20:02:28,949 Epoch   4 Step:    29800 Batch Loss:     1.645589 Tokens per Sec:    15358, Lr: 0.000300\n",
      "2020-01-31 20:02:45,007 Epoch   4 Step:    29900 Batch Loss:     1.797100 Tokens per Sec:    15955, Lr: 0.000300\n",
      "2020-01-31 20:03:00,875 Epoch   4 Step:    30000 Batch Loss:     1.723886 Tokens per Sec:    15692, Lr: 0.000300\n",
      "2020-01-31 20:03:25,184 Hooray! New best validation result [ppl]!\n",
      "2020-01-31 20:03:25,185 Saving new checkpoint.\n",
      "2020-01-31 20:03:26,333 Example #0\n",
      "2020-01-31 20:03:26,334 \tSource:     I was moved to fill out an application for the full - time preaching work .\n",
      "2020-01-31 20:03:26,335 \tReference:  Ndashukunyiselwa ekubeni ndifake isicelo sokwenza umsebenzi wokushumayela wexesha elizeleyo .\n",
      "2020-01-31 20:03:26,335 \tHypothesis: Ndafudukela ukuba ndilahle umsebenzi wokushumayela ixesha elizeleyo .\n",
      "2020-01-31 20:03:26,335 Example #1\n",
      "2020-01-31 20:03:26,336 \tSource:     While visiting Poland , Lucaris saw that the Orthodox there , priests and laity alike , were in a deplorable spiritual condition as a result of their lack of education .\n",
      "2020-01-31 20:03:26,336 \tReference:  Xa wayetyelele ePoland , uLucaris waphawula ukuba amaOthodoki alapho , abefundisi kunye namarhamente ngokufanayo , ayekwimeko engentle ngokomoya ngenxa yokuba engafundanga .\n",
      "2020-01-31 20:03:26,336 \tHypothesis: Ngoxa wayetyelele ePoland , uLucaris wabona ukuba iOthodoki apho , ababingeleli nababingeleli , babebeka kwimeko yokomoya eyingozi ngenxa yokungabi namfundo .\n",
      "2020-01-31 20:03:26,336 Example #2\n",
      "2020-01-31 20:03:26,337 \tSource:     A Spanish proverb states it plainly : “ Better to walk single than be badly married . ” ​ — Proverbs 21 : 9 ; Ecclesiastes 5 : 2 .\n",
      "2020-01-31 20:03:26,337 \tReference:  Iqhalo leSpanish likwenza kucace oku ngokuthi : “ Kulunge ngakumbi ukuhamba wedwa kunokutshata kakubi . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 20:03:26,337 \tHypothesis: Umzekeliso waseSpeyin uthi : “ Better bahambe ngaphandle kokutshata . ” — IMizekeliso 21 : 9 ; INtshumayeli 5 : 2 .\n",
      "2020-01-31 20:03:26,338 Example #3\n",
      "2020-01-31 20:03:26,338 \tSource:     It is fundamental in forging and maintaining good relations with people and with God .\n",
      "2020-01-31 20:03:26,338 \tReference:  Oko kubalulekile ukuba sifuna ukuba nolwalamano oluhle nabantu kunye noThixo .\n",
      "2020-01-31 20:03:26,339 \tHypothesis: Kuqhelekile ukuba nenxaxheba nokuhlala sineentlobano zesini nabantu noThixo .\n",
      "2020-01-31 20:03:26,339 Validation result (greedy) at epoch   4, step    30000: bleu:  20.88, loss: 44814.5547, ppl:   4.9810, duration: 25.4635s\n",
      "2020-01-31 20:03:42,457 Epoch   4 Step:    30100 Batch Loss:     1.848635 Tokens per Sec:    15545, Lr: 0.000300\n",
      "2020-01-31 20:03:58,565 Epoch   4 Step:    30200 Batch Loss:     1.924325 Tokens per Sec:    15865, Lr: 0.000300\n",
      "2020-01-31 20:04:14,616 Epoch   4 Step:    30300 Batch Loss:     2.065831 Tokens per Sec:    15819, Lr: 0.000300\n",
      "2020-01-31 20:04:30,740 Epoch   4 Step:    30400 Batch Loss:     1.750277 Tokens per Sec:    15793, Lr: 0.000300\n",
      "2020-01-31 20:04:46,844 Epoch   4 Step:    30500 Batch Loss:     1.977096 Tokens per Sec:    15769, Lr: 0.000300\n",
      "2020-01-31 20:05:02,804 Epoch   4 Step:    30600 Batch Loss:     1.836526 Tokens per Sec:    15811, Lr: 0.000300\n",
      "2020-01-31 20:05:19,002 Epoch   4 Step:    30700 Batch Loss:     1.702420 Tokens per Sec:    15934, Lr: 0.000300\n",
      "2020-01-31 20:05:35,276 Epoch   4 Step:    30800 Batch Loss:     2.236043 Tokens per Sec:    15826, Lr: 0.000300\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
    "!cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MBoDS09JM807"
   },
   "outputs": [],
   "source": [
    "# Copy the created models from the temporary storage to main storage on google drive for persistant storage \n",
    "!cp -r \"/content/drive/My Drive/masakhane/model-temp/\"* \"$gdrive_path/models/${src}${tgt}_transformer/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "n94wlrCjVc17",
    "outputId": "805b1f09-0563-49d4-ebee-174a43686fb5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Steps: 31000\tLoss: 44567.95312\tPPL: 4.93715\tbleu: 21.41492\tLR: 0.00030000\t*\n",
      "Steps: 32000\tLoss: 44391.17578\tPPL: 4.90598\tbleu: 22.20231\tLR: 0.00030000\t*\n"
     ]
    }
   ],
   "source": [
    "# Output our validation accuracy\n",
    "! cat \"$gdrive_path/models/${src}${tgt}_transformer/validations.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "66WhRE9lIhoD",
    "outputId": "6680cb8f-0cf8-414a-cf36-26eee05ee0f6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-01-31 22:20:14,857 Hello! This is Joey-NMT.\n",
      "2020-01-31 22:21:08,861  dev bleu:  22.67 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
      "2020-01-31 22:21:56,797 test bleu:  31.51 [Beam search decoding with beam size = 5 and alpha = 1.0]\n"
     ]
    }
   ],
   "source": [
    "# Test our model\n",
    "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${src}${tgt}_transformer/config.yaml\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KaXDFfm-zgjK"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "starter_notebook.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}