data-retrieval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retreiving NBA Salary and Win Data \n",
    "\n",
    "This section retrieves the data from external sources, saves them locally, then loads them to build a JSON object. \n",
    "\n",
    "Re-running these steps takes considerable time because I save 900+ html files to avoid having to revist them in consideration of the site owner. These files are used to capture the teams for each year dating back to the 1990-1991 season as well as the players and salaries for each team. These files have been provided in a zip file that are current as of 1/4/2021, which is the beginning of the 2021-2022 season. \n",
    "\n",
    "We will also fetch standings data from the unofficial NBA API that are current as of a similar time.\n",
    "\n",
    "The final output is `nbaSalaryData.json` which contains the salary and win data necessary for analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='top'></a>\n",
    "# Table of Contents\n",
    "- [Import](#import)\n",
    "\n",
    "- [Retrieve base HTML which contains all the years](#retrieve-base-html)\n",
    "\n",
    "- [Get the years from base HTML, save to JSON, and save soup and data](#get-the-years)\n",
    "\n",
    "- [Crawl for each past year and save HTML](#crawl-for-each)\n",
    "\n",
    "- [Read the teams and team salary from each saved year soup and add to JSON object](#read-the-teams)\n",
    "\n",
    "- [Crawl each team in each year and save HTML](#crawl-each-team)\n",
    "\n",
    "- [Get the players' salaries from each team from each year and add to JSON object](#get-the-players)\n",
    "\n",
    "- [Adding win data](#adding-win-data)\n",
    "  - [Fetch the raw standings data from unofficial nba api](#fetch-the-raw)\n",
    "  - [Extracting and normalizing the win data for each team](#extracting-and-normalizing)\n",
    "  - [Add win data to JSON](#add-win-data)\n",
    "\n",
    "- [Appendix](#appendix)\n",
    "- [Data sources](#data-sources)\n",
    "- [Git history](#git-history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='import'></a>\n",
    "# Import [top](#top)\n",
    "All imports for the notebook are handled here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data retrieval \n",
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "import json\n",
    "import os\n",
    "from time import sleep\n",
    "from nba_api.stats.endpoints import leaguestandings\n",
    "import datetime as dt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='retrieve-base-html'></a>\n",
    "## Retrieve base HTML which contains all the years [top](#top)\n",
    "\n",
    "I am getting all of my salary data from https://hoopshype.com/salaries/. I was tempted to use https://www.basketball-reference.com which is the gold-standard for nba data but they [explicitly prohibit scraping](https://www.sports-reference.com/data_use.html).\n",
    "\n",
    "By saving the soup to file we will avoid having to re-scrape the site to get the page into memory. Putting `sleep()` in `getSoup()` ensures we make requests too quickly later when we are scraping lots of different urls and retrying requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getSoup(url):\n",
    "    page = requests.get(url)\n",
    "    sleep(0.6)\n",
    "    return BeautifulSoup(page.content, 'html.parser')\n",
    "\n",
    "def saveSoup(soup, filePath):\n",
    "    path = filePath.rpartition('/')[0] # == 'parent/child' for 'parent/child/file.txt'\n",
    "    \n",
    "    if not os.path.exists(path):\n",
    "        os.makedirs(path)\n",
    "\n",
    "    with open(filePath, 'w', encoding='utf-8') as fp:\n",
    "            fp.write(str(soup))\n",
    "\n",
    "def loadSoup(filePath):\n",
    "    with open(filePath, 'rb') as html:\n",
    "        return BeautifulSoup(html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Requests if html file does not exist locally, or if forced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File already exists. Set force to 'True' if you want to re-request.\n"
     ]
    }
   ],
   "source": [
    "baseDir = 'data/raw/soups/base.html'\n",
    "force = False\n",
    "\n",
    "if not os.path.exists(baseDir) or force:\n",
    "    print(f'{baseDir} doesn\\'t exist.\\nRequesting and saving. ')\n",
    "    soup = getSoup('https://hoopshype.com/salaries')\n",
    "    saveSoup(soup, baseDir)\n",
    "else:\n",
    "    print(\"File already exists. Set force to 'True' if you want to re-request.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The website displays the year in a format like 2019/20 but the URL uses the format 2019-2020. So before saving the date we will transform it to this new format. We will also need to reformat it to the long version later as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def toShortYear(longYear):\n",
    "    '''converts from format like 2019-2020 to format like 2019-20'''\n",
    "    return f'{longYear[:5]}{longYear[7:]}'\n",
    "    \n",
    "def toLongYear(shortYear):\n",
    "    '''converts from format like 2019/20 to format like 2019-2020'''\n",
    "    longYear = shortYear.replace(\"/\",\"-\")\n",
    "\n",
    "    firstTwo = shortYear[:2] #20 from 2019\n",
    "    lastTwo = shortYear[2:4] #19 from 2019\n",
    "\n",
    "    indexToInsert = longYear.find(\"-\") + 1\n",
    "\n",
    "    if lastTwo != '99':\n",
    "        longYear = longYear[:indexToInsert] + firstTwo  + longYear[indexToInsert:]\n",
    "    else:\n",
    "        newFirstTwo = str(int(firstTwo) + 1)\n",
    "        longYear = longYear[:indexToInsert] + newFirstTwo  + longYear[indexToInsert:]\n",
    "\n",
    "    return longYear"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='get-the-years'></a>\n",
    "## Get the years from base HTML, save to JSON, and save soup and data [top](#top)\n",
    "As we parse data at the different levels (years, teams of years, players of teams) we will save to our main data object. I will save to a work-in-progress JSON object until it is finalized to avoid saving over our complete data object later by executing an earlier cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def saveDataWIP(data):\n",
    "    '''Use this function for saving to JSON while building the object'''\n",
    "    with open('data/nbaSalaryData-WIP.json', 'w') as fp:\n",
    "        json.dump(data, fp, indent=4)\n",
    "\n",
    "def loadDataWIP():\n",
    "    with open('data/nbaSalaryData-WIP.json', 'r') as fp:\n",
    "        return json.load(fp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we find all the year strings from the base html. We use this to start our JSON object and will refer back to it to continue scraping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 31 years\n"
     ]
    }
   ],
   "source": [
    "#first two links are duplicate\n",
    "soup = loadSoup('data/raw/soups/base.html')\n",
    "yearLinks = soup.find('li', class_='all').find_all('a')[1:]\n",
    "yearStrings = []\n",
    "\n",
    "# strip\n",
    "for year in yearLinks:\n",
    "    txt = toLongYear(year.text.strip())\n",
    "    yearStrings.append(txt)\n",
    "print(f\"Found {len(yearStrings)} years\")\n",
    "        \n",
    "#sort from low year to high year, and fill dict\n",
    "yearStrings.sort()\n",
    "currentYear = yearStrings[-1]\n",
    "\n",
    "#start data object and save\n",
    "nbaSalaryData = dict.fromkeys(yearStrings)\n",
    "saveDataWIP(nbaSalaryData)\n",
    "\n",
    "saveSoup(soup, f'data/raw/soups/{currentYear}/{currentYear}.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='crawl-for-each'></a>\n",
    "## Crawl for each past year and save HTML [top](#top)\n",
    "I chose to save the HTML pages and wait between requests because we will be making a request for each year. I do not want to make many requests in quick succession nor make the same requests again in the future unless I expect that the data has actually changed.\n",
    "\n",
    "Occasionally scraping a year works but the teams don't populate in the table. To deal with this, I check the number of teams in the table. When the table of teams doens't populate I retry the request a limited number of times, and print the year that has failed with the number of retry attempts. If all retries fail they have to be manually scraped.\n",
    "\n",
    "I first check if the soup exists before making any new requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def soupIsValid(soup):\n",
    "    # detects if table is not populated correctly\n",
    "    return len(soup.find_all('td', class_='name')) > 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Did not make requests for 30 years because they were found locally: \n",
      "['1990-1991', '1991-1992', '1992-1993', '1993-1994', '1994-1995', '1995-1996', '1996-1997', '1997-1998', '1998-1999', '1999-2000', '2000-2001', '2001-2002', '2002-2003', '2003-2004', '2004-2005', '2005-2006', '2006-2007', '2007-2008', '2008-2009', '2009-2010', '2010-2011', '2011-2012', '2012-2013', '2013-2014', '2014-2015', '2015-2016', '2016-2017', '2017-2018', '2018-2019', '2019-2020']\n"
     ]
    }
   ],
   "source": [
    "nbaSalaryData = loadDataWIP()\n",
    "force = False\n",
    "existLocally = []\n",
    "\n",
    "# past years only because current year was saved earlier\n",
    "pastYears = list(nbaSalaryData.keys())[:-1]\n",
    "\n",
    "for year in pastYears:\n",
    "    soupDir = f'data/raw/soups/{year}/{year}.html'\n",
    "    \n",
    "    if not os.path.exists(soupDir) or force:\n",
    "        print(f\"Getting soup for {year}\")\n",
    "        soup = getSoup(f'https://hoopshype.com/salaries/{year}/')\n",
    "    \n",
    "        if not soupIsValid(soup):\n",
    "            print(f'soup retreived for year {year} is not valid')\n",
    "            for n in range(5):\n",
    "                print(f'retrying {n}')\n",
    "                soup = getSoup(url)\n",
    "        #save soup to .html file\n",
    "        saveSoup(soup, soupDir)\n",
    "    else:\n",
    "        existLocally.append(year)\n",
    "\n",
    "if len(existLocally) > 0:\n",
    "    print(f\"Did not make requests for {len(existLocally)} \" \\\n",
    "        f\"years because they were found locally: \\n{existLocally}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='read-the-teams'></a>\n",
    "## Read the teams and team salary from each saved year soup and add to JSON object [top](#top)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Team information added to WIP JSON\n"
     ]
    }
   ],
   "source": [
    "# load data and soup files\n",
    "nbaSalaryData = loadDataWIP()\n",
    "yearFiles = ([f'{x[0]}/{x[2][0]}' for x in list(os.walk(\"data/raw/soups\"))[1:]])\n",
    "yearFiles\n",
    "# add teams to year data for each year\n",
    "for f in yearFiles:\n",
    "    soup = loadSoup(f)\n",
    "    year = f.split(\"/\")[-1].split('.')[0]\n",
    "    \n",
    "    # assign empty object so we can assign object with 'nbaSalaryData[year][name]' later\n",
    "    # otherwise 'nbaSalaryData[year][name]' results in NoneType error (because nbaSalaryData[year]=None)\n",
    "    nbaSalaryData[year] = {}\n",
    "\n",
    "    #filter for elements containing the team names\n",
    "    options = soup.find_all('td', class_='name')\n",
    "    options = options[1:] # get rid of first elemented which does't contain a team\n",
    "        \n",
    "    #parse team data\n",
    "    for o in options:\n",
    "        #find team name and salary\n",
    "        teamTags = o.find_all('a')\n",
    "        \n",
    "        for t in teamTags:\n",
    "            name = t.text.strip()\n",
    "            salary = t.parent.find_next_sibling('td').text.strip()\n",
    "            url = t.get('href')\n",
    "            nbaSalaryData[year][name] = {\"salary\": salary, \"players\": {}, \"url\": url}\n",
    "\n",
    "# save\n",
    "saveDataWIP(nbaSalaryData)\n",
    "print(f\"Team information added to WIP JSON\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='crawl-each-team'></a>\n",
    "## Crawl each team in each year and save HTML [top](#top)\n",
    "These are the last soup objects we will need to collect.\n",
    "\n",
    "This step will take a while (about 20 minutes in my experience) because there are 31 years and at least 28 teams pear year and `getSoup()` waits 0.6 seconds per request. We could theoretically shorten this by reducing the pause but this might negatively impact the website and our ability to scrape from it. Alternative, the repository contains [a zip file current as of 1/4/2020](https://github.com/BlairCurrey/nba-salary-distribution/tree/main/data/raw). If the pages exist locally then they are not rescraped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "nbaSalaryData = loadDataWIP()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Did not make requests for 906/906 teams\n"
     ]
    }
   ],
   "source": [
    "force = False\n",
    "existLocally = 0\n",
    "total = 0\n",
    "\n",
    "for year in nbaSalaryData.keys():\n",
    "    for team in nbaSalaryData[year]:\n",
    "        soupDir = f'data/raw/soups/{year}/{team}.html'\n",
    "        \n",
    "        if not os.path.exists(soupDir) or force:\n",
    "            print(f\"Saving soup for {year}: {team}\")\n",
    "            soup = getSoup(nbaSalaryData[year][team]['url'])\n",
    "            saveSoup(soup, f'data/raw/soups/{year}/{team}.html')\n",
    "        else:\n",
    "            existLocally += 1\n",
    "        total += 1\n",
    "\n",
    "if existLocally > 0:\n",
    "    print(f\"Did not make requests for {existLocally}/{total} teams\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The soup object file structure now looks something like this:\n",
    "```\n",
    "data\n",
    " |_raw\n",
    " |  |_soups\n",
    " |     |_base.html           #homepage. used to find years\n",
    " |     |_1990-1991\n",
    " |     |   |_1990-1991.html  #page containing teams for 1990-1991\n",
    " |     |   |_Atlanta.html    #page containing Atlana\n",
    " |     |   |_Boston.html\n",
    " |     |   ...\n",
    " |     |   |_Washington\n",
    " |     |_1991-1992\n",
    " |     ...\n",
    " |     |_2020-2021\n",
    " |_nbaSalaryData-WIP.json        \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='get-the-players'></a>\n",
    "## Get the players' salaries from each team from each year and add to JSON object [top](#top)\n",
    "Checks if players are already loaded into JSON before saving. This can be overriden by setting `force` to `True` to save over existing player salary information. This step is slow, taking roughly 30 seconds to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "nbaSalaryData = loadDataWIP()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Skipped 906 teams because salary information was already found.\n"
     ]
    }
   ],
   "source": [
    "force = False\n",
    "existJson = []\n",
    "addedTo = 0\n",
    "\n",
    "# list of paths to each year\n",
    "yearPaths = [x[0] for x in os.walk(\"data/raw/soups\")][1:]\n",
    "\n",
    "for y in yearPaths:\n",
    "    year = y.split(\"\\\\\")[1]\n",
    "    #list of team files in each year\n",
    "    teamFiles = list(os.walk(y))[0][2][1:] \n",
    "    \n",
    "    for tf in teamFiles:\n",
    "        team = tf.split(\".\")[0]\n",
    "        \n",
    "        if not nbaSalaryData[year][team][\"players\"] or force:\n",
    "            soup = loadSoup(f\"{y}\\\\{tf}\")\n",
    "\n",
    "            #filter for elements containing the team names\n",
    "            options = soup.find('table', class_='hh-salaries-team-table').find_all('td', class_=\"name\")\n",
    "\n",
    "            #parse player data\n",
    "            for o in options:\n",
    "                #find player name and salary\n",
    "                playerTags = o.find_all('a')\n",
    "\n",
    "                for p in playerTags:\n",
    "                    name = p.text.strip()\n",
    "                    salary = p.parent.find_next_sibling('td').text.strip()\n",
    "                    nbaSalaryData[year][team]['players'][name] = salary\n",
    "            addedTo += 1\n",
    "        else:\n",
    "            existJson.append(f\"{year}: {team}\")\n",
    "# save\n",
    "saveDataWIP(nbaSalaryData)\n",
    "\n",
    "if len(existJson) > 0:\n",
    "    print(f\"Skipped {len(existJson)} teams because salary information was already found.\")\n",
    "else:\n",
    "    print(f\"{addedTo} team's player salary information added to WIP JSON \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This salary data can be accessed like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Channing Frye': '$8,579,088',\n",
       " 'Al Harrington': '$7,609,800',\n",
       " 'Glen Davis': '$6,600,000',\n",
       " 'Victor Oladipo': '$4,978,200',\n",
       " 'Ben Gordon': '$4,500,000',\n",
       " 'Aaron Gordon': '$3,992,040',\n",
       " 'Nikola Vucevic': '$2,902,757',\n",
       " 'Luke Ridnour': '$2,750,000',\n",
       " 'Tobias Harris': '$2,511,432',\n",
       " 'Elfrid Payton': '$2,397,840',\n",
       " 'Jameer Nelson': '$2,000,000',\n",
       " 'Moe Harkless': '$1,887,840',\n",
       " 'Anthony Randolph': '$1,825,359',\n",
       " 'Andrew Nicholson': '$1,545,840',\n",
       " 'Evan Fournier': '$1,483,920',\n",
       " 'Willie Green': '$1,448,490',\n",
       " \"Kyle O'Quinn\": '$915,243',\n",
       " 'Devyn Marble': '$884,879',\n",
       " 'Dewayne Dedmon': '$816,482'}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nbaSalaryData[\"2014-2015\"][\"Orlando\"][\"players\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='adding-win-data'></a>\n",
    "## Adding win data [top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='fetch-the-raw'></a>\n",
    "### Fetch the raw standings data from unofficial nba api [top](#top)\n",
    "Checks if it exists locally before requesting from the [unofficial nba api](https://github.com/swar/nba_api). Each request sleeps for 0.6 seconds but there are only 31 requests so the overall time is not that long. Saves after retrieving. Can be forced by setting `force` to `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw standings data already exists locally.\n"
     ]
    }
   ],
   "source": [
    "force = False\n",
    "rsDir = 'data/raw/standings.json'\n",
    "seasons = [toShortYear(k) for k in nbaSalaryData.keys()]\n",
    "\n",
    "# gets win data from nba.com api unless found locally\n",
    "if not os.path.isfile(rsDir) or force:\n",
    "    print(f'Requesting data starting in {seasons[0]} and ending in {seasons[-1]}')\n",
    "    \n",
    "    rawStandingsData = {\"resourceSets\": []}\n",
    "    \n",
    "    # get data for all seasons\n",
    "    for s in seasons:\n",
    "        print(f'Requesting data for {s}')\n",
    "        standing = json.loads(leaguestandings.LeagueStandings(season=s).get_json())\n",
    "        rawStandingsData[\"resourceSets\"].append(standing)\n",
    "        sleep(0.6)\n",
    "\n",
    "    with open(rsDir, 'w') as fp:\n",
    "        json.dump(rawStandingsData, fp, indent=4)\n",
    "    \n",
    "else:\n",
    "    print(\"Raw standings data already exists locally.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='extracting-and-normalizing'></a>\n",
    "### Extracting and normalizing the win data for each team [top](#top)\n",
    "The teams in `rawStandingsData` are organized differently than our JSON data. In order to combine this with our salary data we need to normalize the cities. For example, this data source includes `Seattle` and `Oklahoma City,` which our previous data source simplifies to `Oklahoma City` (the Seattle team moved to Oklahoma City in 2008). We don't care about preserving these distinctions so the simplified version that we already have works better. Other normalizations include converted the two Los Angeles teams (Lakers, Clippers) to `LA Lakers` and `LA Clippers` to match the `nbaSalaryData` format. This transformation happens after we access the team name and city data from `rawStandingsData`.\n",
    "\n",
    "We can validate that the team names have been normalized by printing all the teams in from our win data and comparing it to the teams in our JSON object. This is defined in `printInvalidTeams()` and utilized called later to print out the number of teams (if any) that differ. They should be 0 if they have all been converted. The following functions are used in normalizing the data from the new source:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalizeCity(originalCity, teamName=None):\n",
    "    cityMap = {\n",
    "        \"New Jersey\": \"Brooklyn\",\n",
    "        \"Vancouver\": \"Memphis\",\n",
    "        \"New Orleans/Oklahoma City\": \"New Orleans\",\n",
    "        \"Seattle\": \"Oklahoma City\",\n",
    "        \"LA\": \"Los Angeles\"\n",
    "}\n",
    "    \n",
    "    c = originalCity\n",
    "    if c in cityMap:\n",
    "        c = cityMap[originalCity]\n",
    "    if c == \"Los Angeles\":\n",
    "        c = whichLA(teamName)\n",
    "    return c\n",
    "\n",
    "def whichLA(teamName):\n",
    "    if teamName == \"Lakers\":\n",
    "        return \"LA Lakers\"\n",
    "    elif teamName == \"Clippers\":\n",
    "        return \"LA Clippers\"\n",
    "    else:\n",
    "        raise Exception('teamName did not match expected values')\n",
    "\n",
    "def printInvalidTeams(winData, nbaSalaryData):\n",
    "    teams = set()\n",
    "\n",
    "    for year in list(winData.keys()):\n",
    "        list1 = list(nbaSalaryData[year].keys())\n",
    "        list2 = list(winData[year].keys())\n",
    "        diff = list(set(list1) - set(list2))\n",
    "        for d in diff:\n",
    "            teams.add(d)\n",
    "\n",
    "    if len(teams)>0:\n",
    "        print(f\"{len(teams)} team(s) in requested data but not in nbaSalaryData:\")\n",
    "        print(teams)\n",
    "    else:\n",
    "        print(\"No teams in requested data that don't exist in nbaSalaryData\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the win data from our new source we will use `getWinData()`. This traverses the `rawStandingsData` and returns a winData object like so:\n",
    "   ```\n",
    "   {\n",
    "       '1990-1991': \n",
    "           {\n",
    "               'Portland': 0.768, \n",
    "               'Chicago': 0.744, \n",
    "               ...\n",
    "           }, \n",
    "        ...\n",
    "   }\n",
    "   ```\n",
    " This is also where we use `normalizeCity()` to map this data source's naming scheme to our json object's naming scheme."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getWinData(rawStandingsData):\n",
    "    winData = {}\n",
    "\n",
    "    #get indexes for categories we need\n",
    "    rawStandingsDataHeaders = rawStandingsData[\"resourceSets\"][0][\"resultSets\"][0][\"headers\"]\n",
    "    iCity = rawStandingsDataHeaders.index(\"TeamCity\")\n",
    "    iName = rawStandingsDataHeaders.index(\"TeamName\")\n",
    "    iWinPct = rawStandingsDataHeaders.index(\"WinPCT\")\n",
    "\n",
    "    for year in rawStandingsData[\"resourceSets\"]:\n",
    "        y = toLongYear(year[\"parameters\"][\"SeasonYear\"])\n",
    "        winData[y] = {}\n",
    "        for team in year[\"resultSets\"][0][\"rowSet\"]:\n",
    "            t = normalizeCity(team[iCity], team[iName])\n",
    "            winData[y][t] = team[iWinPct]\n",
    "\n",
    "    return winData"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='add-win-data'></a>\n",
    "### Add win data to JSON [top](#top)\n",
    "The following functions are used to add the data to our json object and to save it. I also created a new `loadData()` function to accompany `saveData()` because `saveData()` saves our now finalized JSON to a different location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def addWinDataToJson(winData, nbaSalaryData):\n",
    "    for year in winData.keys():\n",
    "        for team in winData[year].keys():\n",
    "            nbaSalaryData[year][team][\"winPct\"] = winData[year][team]\n",
    "    \n",
    "    return nbaSalaryData\n",
    "\n",
    "def saveData(nbaSalaryData):\n",
    "    with open('data/nbaSalaryData.json', 'w') as fp:\n",
    "        json.dump(nbaSalaryData, fp, indent=4)\n",
    "        \n",
    "def loadData():\n",
    "    with open('data/nbaSalaryData.json', 'r') as fp:\n",
    "        return json.load(fp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell utilizes the above functions to go through all the years in our raw data set, normalizes the team city names, print if the cities match, and adds it to our JSON unless it has already been added. This can be overriden by setting `force` to `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "winPct already added to nbaSalaryData.json\n"
     ]
    }
   ],
   "source": [
    "force = False\n",
    "\n",
    "# open raw standings data\n",
    "with open(rsDir, 'r') as fp:\n",
    "    rawStandingsData = json.load(fp)\n",
    "\n",
    "# get win pct for each team and add to json object   \n",
    "if not os.path.exists('data/nbaSalaryData.json') or force:\n",
    "    nbaSalaryData = loadDataWIP()\n",
    "    winData = getWinData(rawStandingsData)\n",
    "    printInvalidTeams(winData, nbaSalaryData)\n",
    "    nbaSalaryData = addWinDataToJson(winData, nbaSalaryData)\n",
    "    saveData(nbaSalaryData)\n",
    "    print(\"Saved winPct to nbaSalary.json\")\n",
    "else:\n",
    "    print(\"winPct already added to nbaSalaryData.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we are done building our WIP json object we can delete the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "removed json\n"
     ]
    }
   ],
   "source": [
    "if os.path.exists('data/nbaSalaryData-WIP.json'):\n",
    "    os.remove('data/nbaSalaryData-WIP.json')\n",
    "    print(\"removed json\")\n",
    "else:\n",
    "    print(\"json not found\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can access a team's win percentage for a year like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.697"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nbaSalaryData = loadData()\n",
    "nbaSalaryData[\"2011-2012\"][\"Miami\"][\"winPct\"] # returns 0.697"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This concludes are data retrieval and storage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='appendix'></a>\n",
    "# Appendix [top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='data-sources'></a>\n",
    "## Data sources [top](#top)\n",
    "- https://hoopshype.com/salaries/ for team, player, and salary records\n",
    "- An unofficial API for https://www.nba.com/stats/, maintained here https://github.com/swar/nba_api"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='git-history'></a>\n",
    "## Git history [top](#top)\n",
    "\n",
    "I used git and github for version control of this project. The git history can be seen here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* 0cf956e (HEAD -> make-presentable, origin/main, main) removed, didnt work\n",
      "* 079439c moved load function to part2\n",
      "* f4aafa8 (refs/original/refs/heads/main) added note about soups\n",
      "* 48a3403 removed unzipped soups from git repo\n",
      "* bea35e1 small tweaks\n",
      "* 985194b linguist-detectable=false for html\n",
      "* 29bb867 added conclusion and appendix\n",
      "* 3123431 requirements added\n",
      "* d888703 updated data\n",
      "* 2c49acf directory structure changes and refactor\n",
      "* 06d0508 no longer holds rel std dev\n",
      "* 35087a3 removed dirs during refactor\n",
      "* c68259f major refactor\n",
      "* bf5025d added soup backup\n",
      "* b111794 added relStdDev and winPct\n",
      "* 5b043b4 added details on sources\n",
      "* 1a670f1 added file\n",
      "* d1cbd24 changed file structure\n",
      "* 0be867a added backup\n",
      "* c38081a modularized and added more data and analysis\n",
      "* f80ef0e init commit\n"
     ]
    }
   ],
   "source": [
    "!git log --oneline --decorate --graph --all"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nba-salaries",
   "language": "python",
   "name": "nba-salaries"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}