04_caselaw.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# JSON and Case Law\n",
    "\n",
    "How can we extract data from APIs (machine-readable online data sources)?\n",
    "\n",
    "In this lesson, we look at how we can use data from the web.\n",
    "We will use real-world data from Harvard’s [Caselaw Access Project](https://case.law/) (\"CAP\").\n",
    "CAP aims to make all published US courts decisions freely available in a standard, machine-readable format.\n",
    "CAP and the data format is [documented here](https://case.law/api/).\n",
    "\n",
    "## JSON\n",
    "\n",
    "JSON is a machine-readable data format.\n",
    "Machine-readable data makes it easy to read and process the information with a computer.\n",
    "JSON data is usually tree structured, with multiple levels containing information.\n",
    "This is kind of like a directory tree containing files.\n",
    "\n",
    "## Fetching Data\n",
    "To fetch data from the web, we can use a library called requests that makes this task quite easy. Since we are are fetching data in the JSON format, we will also import a library to decode JSON data. Libraries are collections of code written by others that we can utilize instead of writing everything from scratch ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to specify the URL to the data we want to fetch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "URL = \"https://api.case.law/v1/cases/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We include some parameters that specifies which cases we want to load:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = {'jurisdiction': 'ill',\n",
    "              'full_case': 'true',\n",
    "              'decision_date_min': '2011-01-01',\n",
    "              'page_size': 3}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `jurisdiction` is Illinois in this example\n",
    "- `full_case` include the full text of each case\n",
    "- `decision_date_min` is the minimum date, we only want decisions later than this date\n",
    "- `page_size` is the number of items\n",
    "\n",
    "More parameters are listed in the CAP documentation linked above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's fetch the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = requests.get(URL, params=parameters).json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This step both fetches and decodes the json data in one line. We can also do this step-by-step, to see how the process works.\n",
    "If you don't want to get into the details at this point, you can skip ahead to the section \"Using the data\". \n",
    "The server response also contains metadata, but we want the content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(URL, params=parameters)\n",
    "content = response.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at the first 100 characters from the raw data. We can see the same data if we open the URL in a web browser:\n",
    "https://api.case.law/v1/cases/?jurisdiction=ill&full_case=true&decision_date_min=2011-01-01&page_size=3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(content[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the data, we must decode them. We must specify the character set, which is often UTF-8. Then we decode the json format into a Python dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = content.decode(\"utf-8\")\n",
    "data = json.loads(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print the data, however this is a lot of text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead, we can print only the keys using `list()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keys = list(data)\n",
    "print(keys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The field `count` contains the number of hits in the database. This is usually different from the number of items we requested.\n",
    "If the `count` is zero, we don't have any results and need to check the query in the URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data[\"count\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks good. Let's fetch the list of cases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cases = data[\"results\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can inspect each case. Let's loop over the cases and get some of the information.\n",
    "The data contains various metadata about each case, such as the case name and the abbreviated case name.\n",
    "\n",
    "It's often useful to look at the data in a web browser to get an overview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for case in cases:\n",
    "    print(\"Case name:\", case[\"name_abbreviation\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:green\"> Exercise: Docket Number </span>\n",
    "\n",
    "Complete the code below to print the docket number and decision date of each case.\n",
    "You will need to browse the [data](https://api.case.law/v1/cases/?jurisdiction=ill&full_case=true&decision_date_min=2011-01-01&page_size=1\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "URL = \"https://api.case.law/v1/cases/?jurisdiction=ill&full_case=true&decision_date_min=2011-01-01&page_size=3\"\n",
    "data = requests.get(URL).json()\n",
    "\n",
    "cases = data[\"results\"]\n",
    "for case in cases:\n",
    "    print(\"Case name:\", case[\"name_abbreviation\"])\n",
    "    #your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Following the Path\n",
    "\n",
    "As mentioned, JSON data is a tree structure. It can contain many nested levels. In that case, we need to follow the path to find the entry we're looking for.\n",
    "It's usually advisable to follow the path one step at a time. This makes it easier to find errors in our programs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cases = data[\"results\"]\n",
    "for case in cases:\n",
    "    print(\"Case name:\", case[\"name_abbreviation\"])\n",
    "\n",
    "    # Step-by-step:\n",
    "    court = case[\"court\"]\n",
    "    court_name = court[\"name\"]\n",
    "    print(\"Court name:\", court_name)\n",
    "\n",
    "    # We start a new path from the root:\n",
    "    casebody = case[\"casebody\"]\n",
    "    case_data = casebody[\"data\"] #we have already used the variable name data\n",
    "    attorneys = case_data[\"attorneys\"]\n",
    "    print(\"Attorneys:\", attorneys)\n",
    "\n",
    "    # Extra linebreak:\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  <span style=\"color:green\"> Exercise: Attorneys and Head Matter </span>\n",
    "\n",
    "Complete the code below to print the attorneys and head matter of each case.\n",
    "You will need to browse the [data](https://api.case.law/v1/cases/?jurisdiction=ill&full_case=true&decision_date_min=2011-01-01&page_size=1\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "URL = \"https://api.case.law/v1/cases/?jurisdiction=ill&full_case=true&decision_date_min=2011-01-01&page_size=3\"\n",
    "data = requests.get(URL).json()\n",
    "\n",
    "cases = data[\"results\"]\n",
    "for case in cases:\n",
    "    print(\"Case name:\", case[\"name_abbreviation\"])\n",
    "    # Your solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Lists\n",
    "\n",
    "Each case in the data set contains a list of one or more opinions. These lists are located quite deep in the data structure, in ` case[\"casebody\"][\"data\"][\"opinions\"]`. These levels are somewhat like directories in a file tree, and can be seen when browsing the [web interface](https://api.case.law/v1/cases/?jurisdiction=ill&full_case=true&decision_date_min=2011-01-01&page_size=3\").\n",
    "\n",
    "We can look at some of the data for each opinion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for case in cases:\n",
    "    print(\"Case name:\", case[\"name_abbreviation\"])\n",
    "    court = case[\"court\"]\n",
    "    print(\"Court name:\", court[\"name\"])\n",
    "    casebody = case[\"casebody\"]\n",
    "    case_data = casebody[\"data\"]\n",
    "    opinions = case_data[\"opinions\"]\n",
    "    \n",
    "    for opinion in opinions:\n",
    "        print(\"  Opinion author:\", opinion[\"author\"])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  <span style=\"color:blue\">Key Points</span>\n",
    "\n",
    "- The `requests` library can be used to fetch data from the web.\n",
    "- Many data providers provide an API from which we can fetch data programatically.\n",
    "- Parameters can be used to control what data we get from an API.\n",
    "- Most APIs provide data in the JSON format, and JSON is well supported in Python.\n",
    "- Additional filtering and processing of the retrieved data can be done using loops and conditions (`if`-statements, next chapter)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}