Scientific_Python-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float: right;\" src=\"http://www2.le.ac.uk/liscb1.jpg\">\n",
    "\n",
    "# Analysing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By: TJ Ragan  \n",
    "Data: Software Carpentry"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python's real power lies in it's libraries.  Implementing new data analysis algorithms or strategies can take hours, or months.  However, remember that you're probably not the first person to try to do most things, and if anyone else has tried it in python, they've probably made a library so you can do it too.  The most common libraries for data analysis in python are *numpy*, *pandas* and *matplotlib*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have some data from an inflamation study stored in `.csv` files in the `data` directory.  Each row represents one patient, and each column represents their inflamation score as the study progressed.  Each file is from a different group of patients.\n",
    "\n",
    "Lets try to analyse the data a few different ways:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Analysis using just python and *matplotlib*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by getting a list of the files, using python's *glob* library, which contains only one funciton, `glob`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "glob.glob('data/*.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a list of files, lets look a the first file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_filenames = glob.glob('data/*.csv')\n",
    "first_filename = data_filenames[0]\n",
    "print(first_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at the data using an IPython command to list the file contents, just like we would on the command line:  \n",
    "*Note that you can click on the area to the left of the output to shrink it down.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%cat data/inflammation-01.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we see what the data look like, we can formulate a stragegy for analysing it:\n",
    "1. Open the file\n",
    "2. Read each line\n",
    "3. Split the values at the commas\n",
    "4. Convert each value into an integer\n",
    "5. Add that patient's data to your study"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Files are funny things.  If you open a file and forget to close it, bad things happen.  If your program crashes half-way through, bad things happen.  If you try to open it more than once, bad things happen.  Python has a trick that takes care of all of this for you:  `with open( ) as f:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "study_participants = []\n",
    "\n",
    "# Open the file\n",
    "with open(first_filename) as file:\n",
    "    # Read each line\n",
    "    for line in file:\n",
    "        # Split the values at the commas\n",
    "        split_line = line.split(',')\n",
    "        inflamation_scores = []\n",
    "        for inflamation_score in split_line:\n",
    "            inflamation_scores.append(int(inflamation_score))\n",
    "        study_participants.append(inflamation_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can look at the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for participant in study_participants:\n",
    "    print(participant)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also ask basic questions, like what's the minimum, average, and maximum inflamation value for each participant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for participant in study_participants:\n",
    "    minimum_inflamation = min(participant)\n",
    "    average_inflamation = sum(participant) /len(participant)\n",
    "    maximum_inflamation = max(participant)\n",
    "    print('min:', minimum_inflamation, 'avg:', average_inflamation, 'max:', maximum_inflamation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What?  everyone's got a minimum score of 0!**\n",
    "\n",
    "Now we can plot each participant using *matplotlib* to see if we can see what's going on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for participant in study_participants:\n",
    "    plt.plot(participant);  # The semicolon keeps the notebook from printing out extra information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.  Looks very busy.\n",
    "2. Looks very triangular!  \n",
    "\n",
    "Now we could try to re-orient our list of lists to look at things along the other axis.  But we're really talking about 2D data here, so why not use a library meant to work with 2D (or nD) data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Analysis using numpy and matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "study_participants_array = np.array(study_participants)\n",
    "study_participants_array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy arrays have all sorts of nice features.  For example, we can easily find out what the shape of the array is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "study_participants_array.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we have 60 participants with 40 observations each.  It turns out that reading these files into Numpy is common enough that we don't need those nested *for* loops to do it - someone's already done it for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.loadtxt(fname=first_filename, delimiter=',')\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Slicing\n",
    "One of the nice features of numpy arrays is that we can easliy select subsets of the data.  A 60 x 40 array of data is too big to look at easliy, which is why we had all those `...` above, so we'll make a smaller one for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "array = np.array([[1,2,3], [4,5,6], [7,8,9], [10,11,12]])\n",
    "print(array)\n",
    "print(array.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that arrays are oriented rows x columns.  This is the standard way of representing matrices in linear algebra - one of the primary uses of Numpy, but can get a little confusing.  In the same way we could slice lists or tuples, we can slice nD arrays.  The only difference here is that we can work directly in nD.  \n",
    "\n",
    "It's worth noting here that you must specify the rows to take.  If you leave out the columns, numpy assumes you want all of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Both of these slices do the same thing\n",
    "print(array[0:2])\n",
    "print()\n",
    "print(array[0:2, :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want all the rows, but a subset of columns, you have to be specific:\n",
    "print(array[:, 0:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plotting  \n",
    "Because numpy is designed to work with arrays of values, we can easily remake the plot above, without the loop.  \n",
    "\n",
    "As you can see, the default orientation plots one line per observation - meaning we see 40 lines (one for each observation,) each with 60 data points on the x-axis for the 60 participants."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(data);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What a mess.  What we wanted was a plot across the observations, not participants.  Fortunatelly, we can just swap the axes of the array using a 'Transpose'.  In the same way nD arrays carry around their shape in the `.shape` attribute, they carry around their transpose in the `.T` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(data.T);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### EXERCISE 1 - Min, mean, max\n",
    "\n",
    "1. Extract the data for the second patient\n",
    "2. Calculate the minimum, mean, and maximum inflamation scores for that patient\n",
    "3. Using the `axis=` parameter, calculate the minimum, mean, and maximum inflamation scores for each observation  \n",
    "    *tip: since there are 60 patients with 40 observations each, you can check that you're working observation-wise and not patient wise* \n",
    "4. Plot the minimum, average, and maximum inflamation scores per observation\n",
    "\n",
    "__BONUS__\n",
    "\n",
    "Ask google how to add a figure legend to your plot.\n",
    "1. Search google for \"add figure legend to matplotlib\"\n",
    "2. Choose the first link to `stackoverflow.com`\n",
    "3. Look at the top answer, which generally should have a green check mark "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, so it's time to call the IRB and report someone for faking (badly) the data.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### EXERCISE 2 - multiple files\n",
    "Re-create the plot above for the first three `.csv` files.  In order to get a new figure, use: `plt.figure()`\n",
    "\n",
    "__BONUS__\n",
    "\n",
    "Ask google how to add a figure title to each plot, so that you can tell which file it comes from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, so it's time to call the IRB and report everyone for faking (badly) the data.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Analysis using pandas and matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy is designed and built for doing array manipulations.  It's good at doing the kinds of table-like operations we've been doing so far, but it's really meant for doing math.  The Pandas library, on the other hand, is built from the ground up for doing this type of work.  \n",
    "\n",
    "The most common feature of Pandas you're likely to use is called a DataFrame (if you're familiar with the R programming languate, these are the same as data frames in that language.)  These are 2D tables of data, that can behave like both Excel spreadsheets and database tables.  More on Excel later,...\n",
    "\n",
    "Like Numpy and Matplotlib, Pandas is a large, powerful library, and we're only going to look at a small portion in this workshop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pd.read_csv('data/inflammation-01.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops!  By default, Pandas assumes that the first row is the column names.  As our data has no header, we need to tell Pandas that the header is `None`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.read_csv('data/inflammation-01.csv', header=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While Numpy arrays are designed to do math, Pandas dataframes are designed to hold data, so in general you should try to treat them as immutable.  They also tend to be column focused, so while you think you may be doing a Numpy type slice, Pandas may think you're asking for either a column or some rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inflammation_01 = pd.read_csv('data/inflammation-01.csv', header=None)\n",
    "inflammation_01[0]  # Column 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inflammation_01[0:3]  # but this is the first three rows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Ouch!__  \n",
    "\n",
    "To simplify matters, Pandas provides a location indexer that behaves exactly like Numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inflammation_01.loc[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may have noticed that the slice we've just taken brings it's own index with it (the numbers 0 through 39).  Remember that Pandas behaves as a spreadsheet.  While using `.loc[]` looks like it uses 'positional' slicing, what you're actually doing is slicing based on the index.  \n",
    "\n",
    "If we create a dataframe with labeled rows and columns, the behaviour becomes more clear"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame([[1,2,3], [4,5,6], [7,8,9]], index=['one', 'two', 'three'], columns = ['a', 'b', 'c'])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[['a','b']]  # We can ask for a list of columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can use the `.loc[]` indexing to do the same kind of slice we do with Numpy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc['one':'two']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But we can also provide a list of indices, and Pandas will give us those back in the order we ask for them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[['three','one'], ['c', 'a', 'b']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what do we get for this increased complexity?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### EXERCISE 3 - General information on our csv file with Pandas\n",
    "1. Load the second csv file into a Pandas DataFrame called inflammation_02\n",
    "2. Create a variable called inflammation_02_description using the .describe() method of the DataFrame\n",
    "3. Examine the contents of the inflammation_02_description variable  \n",
    "\n",
    "4. Using the `.loc[]` indexer, extract the max, min, and average values and conver them to lists\n",
    "5. Plot the max, min, and average values using plt.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the .loc indexer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make the plots\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because Pandas dataframes behave like Excel spreadsheets, the people who created pandas decided that you should be able to work with Excel spreadsheets.  \n",
    "\n",
    "Load a all the sheets from the `inflamation.xlsx` Excel file by telling the `read_excel` function not to take a specific one (note that by default it takes the first one).  This will give us a Dictionary of sheets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inflammation_workbook = pd.read_excel('data/inflammation.xlsx', sheet_name=None, header=None)\n",
    "inflammation_workbook.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inflammation_workbook['inflammation-01']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One final bit of Pandas.  It plots, too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inflammation_02_description.loc['min'].plot()\n",
    "inflammation_02_description.loc['mean'].plot()\n",
    "inflammation_02_description.loc['max'].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### EXERCISE 4 - Plot all the things!\n",
    "\n",
    "Plot min, average, and max values for all the sheets in your excel file.  \n",
    "\n",
    "__bonus__:  \n",
    "Plot them in order.  \n",
    "*tip: you can order the keys in a dictionary using `sorted(dictionary.keys())`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmm,...  That last one looks funny."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One final plot...  \n",
    "As our data is 2-dimensional, one final way we can plot it is to show it as an image.  Because Pandas and Matplotlib work together well, it's easy.  We just use the `imshow` method of the `pyplot` library, and then add a colorbar to get a range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(inflammation_workbook['inflammation-01'], cmap='jet')\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MathWorks, the company who makes Matlab, used to use the Jet colormap (and so did Matplotlib before version 2.0.)  This colormap was so popular (and pretty,) it became the default in most packages.  Unfortunatelly, Jet is a terrible colormap. Recently, there has been a lot of research about the effects of different colormaps on our perception of data that has shown how truly awful the Jet colormap is.  In [one recent study](http://www.eecs.harvard.edu/~kgajos/papers/2011/borkin11-infoviz.pdf), physicians who were switched from Jet to a perceptually 'appropriate' map showed a 47% increase in the ability to detect potential sites of coronary artery disease.\n",
    "\n",
    "Because Jet was so bad, Mathworks changed their default map to one called *parula*, which is much better, but still not ideal.  Matplotlib has gone for a map called *viridis* which is perceptually uniform, looks the same if you have red-green colour blindness, and prints nicely in black and white.\n",
    "\n",
    "As an example, we'll plot the same data as above using both a greyscale map and viridis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(6,6))  # Notice that you can change the figure size by poviding an (x,y) tuple\n",
    "\n",
    "plt.imshow(inflammation_workbook['inflammation-01'], cmap='Greys_r')\n",
    "plt.colorbar()\n",
    "\n",
    "plt.figure(figsize=(6,6))\n",
    "\n",
    "plt.imshow(inflammation_workbook['inflammation-01'], cmap='viridis')\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### EXERCISE 5 - Try some other colormaps\n",
    "\n",
    "Repeat the plot above using *at least* three different maps.\n",
    "\n",
    "__tip: Google 'matplotlib colormap'__\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### EXERCISE 6 - I promise Jet is bad\n",
    "\n",
    "Plot all the spreadsheets in the viridis, Greys_r, and jet colormaps.  \n",
    "\n",
    "Do you see anything interesting???"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}