Language Classifier-Copy1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spoken Language Classifier\n",
    "\n",
    "This notebook trains a convolutional neural network to classfify audio files of voice recordings into the languages that were spoken. The dataset I used contained 65.000 files across 176 languages. I found it on TopCoder (https://goo.gl/G5XBJl). I liked the idea behind this problem, because it's very hard for humans to do. It's intersting to see that CNNs perform well on problems where intuition doesn't get you anywhere."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 Imports, Variables and Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from utils import *\n",
    "import vgg16\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import scipy as sp\n",
    "import os\n",
    "import librosa as lr\n",
    "import shutil\n",
    "#import dask.array as da\n",
    "import h5py\n",
    "import glob\n",
    "from glob import glob\n",
    "import bcolz\n",
    "\n",
    "\n",
    "from keras.models import Model, load_model\n",
    "from keras.layers import Conv2D, MaxPooling2D, Dense, Flatten\n",
    "from keras.layers import Dropout, Input, BatchNormalization\n",
    "from keras.optimizers import Nadam\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from keras.utils.np_utils import to_categorical\n",
    "from keras.utils import np_utils\n",
    "from keras import backend as K\n",
    "from keras.utils.data_utils import get_file\n",
    "from keras.utils import np_utils\n",
    "from keras.utils.np_utils import to_categorical\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Input, Embedding, Reshape, merge, LSTM, Bidirectional\n",
    "from keras.layers import TimeDistributed, Activation, SimpleRNN, GRU\n",
    "from keras.layers.core import Flatten, Dense, Dropout, Lambda\n",
    "from keras.regularizers import l2, l1\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "from keras.optimizers import SGD, RMSprop, Adam\n",
    "from keras.metrics import categorical_crossentropy, categorical_accuracy\n",
    "from keras.layers.convolutional import *\n",
    "from keras.preprocessing import image, sequence\n",
    "from keras.preprocessing.text import Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "in_dim = (192,192,1)\n",
    "out_dim = 176\n",
    "batch_size = 32\n",
    "mp3_path = 'data/roller/'\n",
    "tr_path = 'data/train_wav/'\n",
    "va_path = 'data/valid_wav/'\n",
    "te_path = 'data/test_wav/'\n",
    "data_size = 66176\n",
    "tr_size = 52800\n",
    "va_size = 4576\n",
    "te_size = 8800"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cd /home/aind2/workspace/language-recognition/data/roller"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "g = glob('*/*.wav')\n",
    "fold=[d.split('/')[0] for d in g]\n",
    "file_name=[d.split('/')[1] for d in g]\n",
    "test= list(zip (fold, file_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pf=pd.Series (zip(fold, file_name))\n",
    "pf.to_csv('train_list_new.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will convert a single mp3 file to a spectrogram and return the image. The mel-spectrogram is used to get more information in the lower frequencies similar to human hearing. The intensities and the frequencies are then scaled logarithmically. This function will also cut away 5% of the beginning and end of the file. This is to get rid of silence and ensure the same dimensions of each file it outputs. The conversion takes roughly 1sec per minute of audio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def mp3_to_img(path, height=192, width=192):\n",
    "    signal, sr = lr.load(path, res_type='kaiser_fast')\n",
    "    hl = signal.shape[0]//(width*1.1) #this will cut away 5% from start and end\n",
    "    spec = lr.feature.melspectrogram(signal, n_mels=height, hop_length=int(hl))\n",
    "    img = lr.logamplitude(spec)**2\n",
    "    start = (img.shape[1] - width) // 2\n",
    "    return img[:, start:start+width]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def wav_to_img(path, height=192, width=192):\n",
    "    signal, sr = lr.load(path, res_type='kaiser_fast')\n",
    "    hl = signal.shape[0]//(width*1.1) #this will cut away 5% from start and end\n",
    "    spec = lr.feature.melspectrogram(signal, n_mels=height, hop_length=int(hl))\n",
    "    img = lr.logamplitude(spec)**2\n",
    "    start = (img.shape[1] - width) // 2\n",
    "    return img[:, start:start+width]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Batch convert all mp3-files to spectrogram jpgs. process_audio_with_classes() will use the labels to sort all jpgs in coresponding subfolders. This is useful for the flow_from_directory function in Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def process_audio(in_folder, out_folder):\n",
    "    os.makedirs(out_folder, exist_ok=True)\n",
    "    files = glob.glob(in_folder+'*.wav')\n",
    "    start = len(in_folder)\n",
    "    for file in files:\n",
    "        img = mp3_to_img(file)\n",
    "        sp.misc.imsave(out_folder + file[start:] + '.jpg', img)\n",
    "        \n",
    "def process_audio_with_classes(in_folder, out_folder, labels):\n",
    "    os.makedirs(out_folder, exist_ok=True)\n",
    "    for i in range(len(labels['Sample Filename'])):\n",
    "        file = labels['Sample Filename'][i]\n",
    "        lang = labels['Language'][i]\n",
    "        os.makedirs(out_folder + lang, exist_ok=True)\n",
    "        img = mp3_to_img(in_folder+file)\n",
    "        sp.misc.imsave(out_folder + lang + '/' + file + '.jpg', img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert a directory of images to a HDF5 file storing the images in an array. The shape of the array will be (img_num, height, width[, channels])."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def jpgs_to_h5(source, target, name):\n",
    "    da.image.imread(source + '*.jpg').to_hdf5(target, name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 Preparing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The raw data consists of 66176 44,1kHz stereo mp3 file with a length of 10 seconds each. The dataset is perfectly balanced with 376 files per language."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize this file by converting it to a log-mel-spectrogram.\n",
    "\n",
    "- the y-axis shows the frequency\n",
    "- the x-axis shows the time\n",
    "- the color shows the intensity of a frequency at a given time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The spectrograms gave me headaches at first. Although it's easy to read a spectrogram, it's hard to \"intuitively\" judge the content of an audio file. My first try of converting them looked totally fine, but I wasn't able to train my network at all. I used a regular spectrogram which I then converted to log scale frequencies. The trouble was that by squeezing the higher frequencies I was pulling apart the lower frequencies. That's the general idea of a log scale and all, but I didn't take into account that the resolution in the lower frequencies would suffer badly. To my luck I wasn't the first person with this problem, so in 1980 a couple of guys came up with the mel-spectrogram which gives more resolution to lower frequencies. After that I also log scaled the intensities of my data points which seemed to help as well.\n",
    "\n",
    "I started out with a resolution of 224x448 pixels but this took forever on my computer. I applied some asymetrical resizing and noticed that my assumption of reserving more space for the time axis was wrong. Square images seemed to perform best. So, I went ahead and converted everything to 192x192, which didn't hurt the performance all that much.\n",
    "\n",
    "The \"sanity check\" of the data turned out to be difficult with this dataset. Apparently you can have 176 different languages without including english, german or french. But all the dutch samples sounded like what dutch people sound like, so I figured it couldn't be that wrong."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the mp3 files to jpgs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "process_audio('data/roller/fast', 'data/wav2jpg/fast')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Covert the folder of images to a compressed container file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "jpgs_to_h5('data/jpg/', 'data/data.h5', 'data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shuffle the data and split it into train, valid and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = pd.read_csv('data/train_list.csv')['Language']\n",
    "y = pd.get_dummies(y)\n",
    "y = y.reindex_axis(sorted(y.columns), axis=1)\n",
    "y = y.values\n",
    "y = da.from_array(y, chunks=1000)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "x = h5py.File('data/data.h5')['data']\n",
    "x = da.from_array(x, chunks=1000)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "shfl = np.random.permutation(data_size)\n",
    "\n",
    "tr_idx = shfl[:tr_size]\n",
    "va_idx = shfl[tr_size:tr_size+va_size]\n",
    "te_idx = shfl[tr_size+va_size:]\n",
    "\n",
    "x[tr_idx].to_hdf5('data/x_tr.h5', 'x_tr')\n",
    "y[tr_idx].to_hdf5('data/y_tr.h5', 'y_tr')\n",
    "x[va_idx].to_hdf5('data/x_va.h5', 'x_va')\n",
    "y[va_idx].to_hdf5('data/y_va.h5', 'y_va')\n",
    "x[te_idx].to_hdf5('data/x_te.h5', 'x_te')\n",
    "y[te_idx].to_hdf5('data/y_te.h5', 'y_te')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# USE BATCHES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path = \"data/\"\n",
    "batch_size=64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_batches(dirname, gen=image.ImageDataGenerator(), shuffle=True, batch_size=4, class_mode='categorical',\n",
    "                target_size=(192,192)):\n",
    "    return gen.flow_from_directory(dirname, target_size=target_size, color_mode='greyscale',\n",
    "            class_mode=class_mode, shuffle=shuffle, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def onehot(x):\n",
    "    return to_categorical(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_classes(path):\n",
    "    batches = get_batches(path+'train_wav', shuffle=False, batch_size=1)\n",
    "    val_batches = get_batches(path+'valid_wav', shuffle=False, batch_size=1)\n",
    "    test_batches = get_batches(path+'test_wav', shuffle=False, batch_size=1)\n",
    "    return (val_batches.classes, batches.classes, onehot(val_batches.classes), onehot(batches.classes),\n",
    "        val_batches.filenames, batches.filenames, test_batches.filenames)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "('Invalid color mode:', 'greyscale', '; expected \"rgb\" or \"grayscale\".')",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-9-f9a6fa3782fd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mbatches\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_batches\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m'train_wav'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mval_batches\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_batches\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m'valid_wav'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mshuffle\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m (val_classes, trn_classes, val_labels, trn_labels, \n\u001b[1;32m      5\u001b[0m     val_filenames, filenames, test_filenames) = get_classes(path)\n",
      "\u001b[0;32m<ipython-input-8-a156d8e67954>\u001b[0m in \u001b[0;36mget_batches\u001b[0;34m(dirname, gen, shuffle, batch_size, class_mode, target_size)\u001b[0m\n\u001b[1;32m      2\u001b[0m                 target_size=(192,192)):\n\u001b[1;32m      3\u001b[0m     return gen.flow_from_directory(dirname, target_size=target_size, color_mode='greyscale',\n\u001b[0;32m----> 4\u001b[0;31m             class_mode=class_mode, shuffle=shuffle, batch_size=batch_size)\n\u001b[0m",
      "\u001b[0;32m/home/aind2/anaconda3/envs/aind2/lib/python3.6/site-packages/keras/preprocessing/image.py\u001b[0m in \u001b[0;36mflow_from_directory\u001b[0;34m(self, directory, target_size, color_mode, classes, class_mode, batch_size, shuffle, seed, save_to_dir, save_prefix, save_format, follow_links)\u001b[0m\n\u001b[1;32m    472\u001b[0m             \u001b[0msave_prefix\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msave_prefix\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    473\u001b[0m             \u001b[0msave_format\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msave_format\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 474\u001b[0;31m             follow_links=follow_links)\n\u001b[0m\u001b[1;32m    475\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    476\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mstandardize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/aind2/anaconda3/envs/aind2/lib/python3.6/site-packages/keras/preprocessing/image.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, directory, image_data_generator, target_size, color_mode, classes, class_mode, batch_size, shuffle, seed, data_format, save_to_dir, save_prefix, save_format, follow_links)\u001b[0m\n\u001b[1;32m    868\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcolor_mode\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m'rgb'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'grayscale'\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    869\u001b[0m             raise ValueError('Invalid color mode:', color_mode,\n\u001b[0;32m--> 870\u001b[0;31m                              '; expected \"rgb\" or \"grayscale\".')\n\u001b[0m\u001b[1;32m    871\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcolor_mode\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcolor_mode\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    872\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata_format\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdata_format\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: ('Invalid color mode:', 'greyscale', '; expected \"rgb\" or \"grayscale\".')"
     ]
    }
   ],
   "source": [
    "batches = get_batches(path+'train_wav', batch_size=batch_size)\n",
    "val_batches = get_batches(path+'valid_wav', batch_size=batch_size*2, shuffle=False)\n",
    "\n",
    "(val_classes, trn_classes, val_labels, trn_labels, \n",
    "    val_filenames, filenames, test_filenames) = get_classes(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_data(path, target_size=(192,192)):\n",
    "    \n",
    "    batches = get_batches(path, shuffle=False, batch_size=1, class_mode=None, target_size=target_size)\n",
    "    print (batches.batch_size)\n",
    "    return np.concatenate([batches.next() for i in range(batches.batch_size)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn = get_data(path+'train_wav')\n",
    "val = get_data(path+'valid_wav')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = get_data(path+'test_wav')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import bcolz as bcolz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "save_array(path+'results/trn.dat', trn)\n",
    "save_array(path+'results/val.dat', val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "save_array(path+'results/test.dat', test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gen = image.ImageDataGenerator()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 Load and process the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the data we've prepared and check its dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_tr = da.from_array(h5py.File('data/x_tr.h5')['x_tr'], chunks=1000)\n",
    "y_tr = da.from_array(h5py.File('data/y_tr.h5')['y_tr'], chunks=1000)\n",
    "print(x_tr.shape, y_tr.shape)\n",
    "\n",
    "x_va = da.from_array(h5py.File('data/x_va.h5')['x_va'], chunks=1000)\n",
    "y_va = da.from_array(h5py.File('data/y_va.h5')['y_va'], chunks=1000)\n",
    "print(x_va.shape, y_va.shape)\n",
    "\n",
    "x_te = da.from_array(h5py.File('data/x_te.h5')['x_te'], chunks=1000)\n",
    "y_te = da.from_array(h5py.File('data/y_te.h5')['y_te'], chunks=1000)\n",
    "print(x_te.shape, y_te.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "trn /= 255.\n",
    "val /= 255.\n",
    "test /= 255."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check a sample just to be sure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_img = x_tr[0, :, :, 0]\n",
    "plt.imshow(test_img)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I tried roughly 30 different models with focus on newer architectures like residual networks, networks in networks, squeezing and expanding convolutions, but in the end a 5x-Conv-MaxPool worked best. I really wanted to replace the last Dense layers with AveragePooling. They give a little more insight to what's happening in comparison to the \"black box\"-model that results from dense layers. However it didn't work out as well. I'm guessing this is because spectrograms show a different abstraction of information in comparison to a regular photo showing one object.\n",
    "\n",
    "In my tests the use of Elu replaced the need for Batch Normalization, which usually improves any model. I didn't include them for performance reasons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = Input(shape=in_dim)\n",
    "m = Conv2D(16, (3, 3), activation='elu', padding='same')(i)\n",
    "m = MaxPooling2D()(m)\n",
    "m = Conv2D(32, (3, 3), activation='elu', padding='same')(m)\n",
    "m = MaxPooling2D()(m)\n",
    "m = Conv2D(64, (3, 3), activation='elu', padding='same')(m)\n",
    "m = MaxPooling2D()(m)\n",
    "m = Conv2D(128, (3, 3), activation='elu', padding='same')(m)\n",
    "m = MaxPooling2D()(m)\n",
    "m = Conv2D(256, (3, 3), activation='elu', padding='same')(m)\n",
    "m = MaxPooling2D()(m)\n",
    "m = Flatten()(m)\n",
    "m = Dense(512, activation='elu')(m)\n",
    "m = Dropout(0.5)(m)\n",
    "o = Dense(out_dim, activation='softmax')(m)\n",
    "\n",
    "model = Model(inputs=i, outputs=o)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.fit(trn, trn_labels, batch_size=batch_size, nb_epoch=3, validation_data=(val, val_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer=Nadam(lr=1e-3), metrics=['accuracy'])\n",
    "model.fit(x_tr, y_tr, epochs=2, verbose=1, validation_data=(x_va, y_va))\n",
    "model.compile(loss='categorical_crossentropy', optimizer=Nadam(lr=1e-4), metrics=['accuracy'])\n",
    "model.fit(x_tr, y_tr, epochs=3, verbose=1, validation_data=(x_va, y_va))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = load_model('speech_v9.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#model.evaluate(x_te, y_te)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#model.predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layers = model.layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "last_conv_idx = [index for index,layer in enumerate(layers) \n",
    "                     if type(layer) is Convolution2D][-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_conv_idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layers[last_conv_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "conv_layers = layers[:last_conv_idx+1]\n",
    "conv_model = Sequential(conv_layers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for layer in model.layers: layer.trainable = False\n",
    "# Look how easy it is to connect two models together!\n",
    "#conv_model.add(fc_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer=Nadam(lr=1e-3), metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn.shape="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(trn, trn_labels, batch_size=batch_size, nb_epoch=3, validation_data=(val, val_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}