Data-Extraction--NASA-Text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data extraction - NASA climate plaintext data \n",
    "\n",
    "A set of examples on how to extract machine-readable data from the raw, official sources. No pandas needed, just requests and regex and xlrd (for Excel spreadsheets)\n",
    "\n",
    "(in progress)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## File system setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from os import makedirs\n",
    "from os.path import dirname, join\n",
    "DATA_DIR = join('data', 'climate')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "import re\n",
    "import requests\n",
    "from xlrd import open_workbook\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global Surface Air Temperature Anomaly, via NASA\n",
    "\n",
    "![NASA GISS](images/climate/source/nasa-gistemp-globaltemp-fig-a.gif)\n",
    "\n",
    "Source: [NASA Goddard Institute for Space Studies (GISS) Surface Temperature Analysis](http://data.giss.nasa.gov/gistemp/).\n",
    "\n",
    "> Our traditional analysis using only meteorological station data is a line plot of global annual-mean surface air temperature change, with the base period 1951-1980, derived from the meteorological station network [This is an update of Plate 6(b) in Hansen et al. (2001).] Uncertainty bars (95% confidence limits) are shown for both the annual and five-year means, account only for incomplete spatial sampling of data.\n",
    "\n",
    "\n",
    "### The data file\n",
    "\n",
    "- Direct link to the source data file: [http://data.giss.nasa.gov/gistemp/graphs_v3/Fig.A.txt](http://data.giss.nasa.gov/gistemp/graphs_v3/Fig.A.txt)\n",
    "- Mirror: [data/climate/raw/nasa-gistemp-figA.txt](data/climate/raw/nasa-gistemp-figA.txt)\n",
    "\n",
    "The contents: From 1880 to 2015, the change in global average surface air temperature change, compared to the average global temperature measured in the period __1951 to 1980__.\n",
    "\n",
    "An excerpt of the [file](data/climate/raw/nasa-gistemp-figA.txt):\n",
    "\n",
    "```\n",
    "Global Surface Air Temperature Anomaly (C) (Base: 1951-1980)\n",
    "------------------------------------------------------------\n",
    " Year  Annual_Mean 5-year_Mean\n",
    "--------------------------------\n",
    " 1880     -0.49         *\n",
    " 1881     -0.47         *\n",
    " 1882     -0.38     -0.48\n",
    " 1883     -0.39     -0.48\n",
    " 1884     -0.67     -0.52\n",
    " 1885     -0.51     -0.58\n",
    "...\n",
    " 2012      0.80      0.86\n",
    " 2013      0.84      0.87\n",
    " 2014      0.90         *\n",
    " 2015      1.01         *\n",
    "```\n",
    "\n",
    "The years 1885 and 2015 are said to have a global average temperature of **-0.51** and **+1.01** degrees Celsius, respectively, from the average temperature as measured in the period of 1951-1980.\n",
    "\n",
    "The __5-year mean__ of 1882 -- __-0.48__ -- is the rolling average of the annual means for 1880 through 1884.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parsing and wrangling the temperature text file\n",
    "\n",
    "This can be done with using regular expressions and `re.findall()`. In the snippet below, I write __two__ files, since there aren't 5-year mean values for every annual mean value:\n",
    "\n",
    "- [data/climate/cleaned/nasa-gistemp-annual-mean.csv](data/climate/cleaned/nasa-gistemp-annual-mean.csv)\n",
    "- [data/climate/cleaned/nasa-gistemp-5year-mean.csv](data/climate/cleaned/nasa-gistemp-5year-mean.csv)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# writing the annual means to file\n",
    "source_data_url = 'http://data.giss.nasa.gov/gistemp/graphs_v3/Fig.A.txt'\n",
    "resp = requests.get(source_data_url)\n",
    "pattern = re.compile(r'^ +(\\d{4}) +(-?\\d+\\.\\d+)', re.MULTILINE)\n",
    "data = re.findall(pattern, resp.text)\n",
    "\n",
    "# write to file\n",
    "destfilename = join(DATA_DIR, 'extracted', 'nasa-gistemp-annual-mean.csv')\n",
    "makedirs(dirname(destfilename), exist_ok=True)\n",
    "with open(destfilename, 'w') as f:\n",
    "    c = csv.writer(f)\n",
    "    c.writerow(['year', 'annual_mean'])\n",
    "    c.writerows(data)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# writing the 5-year means to file\n",
    "# assume that `resp` still contains the download\n",
    "pattern = re.compile(r'^ +(\\d{4}) .+?(-?\\d+\\.\\d+) *$', re.MULTILINE)\n",
    "data = re.findall(pattern, resp.text)\n",
    "\n",
    "# write to file\n",
    "destfilename = join(DATA_DIR,  'extracted', 'nasa-gistemp-5year-mean.csv')\n",
    "makedirs(dirname(destfilename), exist_ok=True)\n",
    "with open(destfilename, 'w') as f:\n",
    "    c = csv.writer(f)\n",
    "    c.writerow(['year', 'fiveyear_mean'])\n",
    "    c.writerows(data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NASA CO2 gases\n",
    "\n",
    "Source...?: [NASA GISS: Forcings in GISS Climate Model](http://data.giss.nasa.gov/modelforce/ghgases/).\n",
    "\n",
    "I'm not really sure what the landing page for the following data set comes from. It seems to consist of data from:\n",
    "\n",
    "- [NOAA's World Data Center for Paleoclimatology's ice core research](https://www.ncdc.noaa.gov/data-access/paleoclimatology-data/datasets/ice-core).\n",
    "- NASA measurements from Mauna Loa\n",
    "- NOAA/ESRL measurements\n",
    "\n",
    "### The data file\n",
    "\n",
    "- Direct link to the source file: [http://data.giss.nasa.gov/modelforce/ghgases/Fig1A.ext.txt](http://data.giss.nasa.gov/modelforce/ghgases/Fig1A.ext.txt)\n",
    "- Mirror: [data/climate/raw/nasa-ghgases-fig1A.ext.txt](data/climate/raw/nasa-ghgases-fig1A.ext.txt)\n",
    "\n",
    "The observed global average of carbon dioxide gas in parts-per million.\n",
    "\n",
    "An excerpt of the file:\n",
    "\n",
    "```\n",
    "               Global Mean CO2 Mixing Ratios (ppm): Observations\n",
    "----------------------------------------------------------------------------------\n",
    "Data                                     Data\n",
    "Source  Year  MixR          Yar   MixR   Source Year  MixR          Year  MixR\n",
    "----------------------------------------------------------------------------------\n",
    "Ice-    1850  285.2         1900  295.7         1950  311.3         2000  369.64\n",
    "Core    1851  285.1         1901  296.2         1951  311.8         2001  371.15\n",
    "Data    1852  285.0         1902  296.6         1952  312.2         2002  373.15\n",
    "Adjus-  1853  285.0         1903  297.0         1953  312.6         2003  375.64\n",
    "ted     1854  284.9         1904  297.5         1954  313.2   NOAA/ 2004  377.44\n",
    "for     1855  285.1         1905  298.0         1955  313.7   ESRL/ 2005  379.46\n",
    "Global  1856  285.4         1906  298.4         1956  314.3  trends 2006  381.59\n",
    "Mean    1857  285.6         1907  298.8         1957  314.8  change 2007  383.37\n",
    "        1858  285.9         1908  299.3  SIO    1958  315.34  added 2008  385.46\n",
    "        1859  286.1         1909  299.7  Mauna  1959  316.18     to 2009  386.95\n",
    "        1860  286.4         1910  300.1  Loa    1960  317.07   2003 2010  389.21\n",
    "        1861  286.6         1911  300.6    &    1961  317.73   data 2011  391.15\n",
    "        1862  286.7         1912  301.0  South  1962  318.43\n",
    "        1863  286.8         1913  301.3  Pole   1963  319.08\n",
    "        1864  286.9         1914  301.4  Adjus- 1964  319.65\n",
    "        1865  287.1         1915  301.6  ted    1965  320.23\n",
    "        1866  287.2         1916  302.0  for    1966  321.59\n",
    "        1867  287.3         1917  302.4  Global 1967  322.31\n",
    "        1868  287.4         1918  302.8  Mean   1968  323.04\n",
    "        1869  287.5         1919  303.0         1969  324.23\n",
    "\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parsing and wrangling the global gases file\n",
    "\n",
    "As in the temperatures-file example, just adroit use of regular expressions. However, there's one wrinkle: [the data file](data/climate/raw/nasa-ghgases-fig1A.ext.txt) contains two sections; observations, as excerpted above, and \"Future Scenarios\":\n",
    "\n",
    "\n",
    "```\n",
    "             Global Mean CO2 Mixing Ratio (ppm): Future Scenarios\n",
    "----------------------------------------------------------------------------------\n",
    "           Alternative Scenario                 2 Degree C Scenario\n",
    "        Year  MixR      Year  MixR          Year  MixR      Year  MixR\n",
    "----------------------------------------------------------------------------------\n",
    "        2000  370.0     2050  445.0         2000  370.0     2050  486.2\n",
    "        2001  371.7     2051  446.2         2001  371.7     2051  489.2\n",
    "        2002  373.4     2052  447.4         2002  373.4     2052  492.1\n",
    "        2003  375.1     2053  448.5         2003  375.1     2053  494.9\n",
    "```\n",
    "\n",
    "We want to wrangle only the data _before_ the future scenarios section:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# writing the gases data to file\n",
    "source_data_url = 'http://data.giss.nasa.gov/modelforce/ghgases/Fig1A.ext.txt'\n",
    "resp = requests.get(source_data_url)\n",
    "txt = resp.text\n",
    "stop_line_num = [x.strip() for x in txt.splitlines()].index('Global Mean CO2 Mixing Ratio (ppm): Future Scenarios')\n",
    "# just get the lines before the unwanted Future Scenarios section\n",
    "excerpt = \"\\n\".join(txt.splitlines()[:stop_line_num])\n",
    "pattern = re.compile(r'(\\b\\d{4})  (\\d{3}\\.\\d{1,2}\\b)')\n",
    "data = re.findall(pattern, excerpt)\n",
    "\n",
    "# I like non-mutating operations\n",
    "sorteddata = sorted(data, key=lambda d: int(d[0]))\n",
    "# write to file\n",
    "destfilename = join(DATA_DIR, 'extracted', 'nasa-ghgases-co2-mean.csv')\n",
    "makedirs(dirname(destfilename), exist_ok=True)\n",
    "with open(destfilename, 'w') as f:\n",
    "    c = csv.writer(f)\n",
    "    c.writerow(['year', 'co2_ppm_mean'])\n",
    "    c.writerows(sorteddata)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}