Data_exploration_template.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Template to get started with data exploration\n",
    "\n",
    "The other notebooks show the results of existing analysis. Most of the resulting code has been moved from the notebooks to the associated python modules, in order to showcase the results. But that makes it harder to experiment with the data and come up with new analyses. This is particularly true because the current data structures that store the data are a little complicated. Maybe after we switch to xarrays in the future, we will no longer need this!\n",
    "\n",
    "But for now, users can use this exploration template and plug in their code/analyses here. And finally, they can put the code into a module for re-use elsewhere"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up the dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for reading and validating data\n",
    "import emeval.input.spec_details as eisd\n",
    "import emeval.input.phone_view as eipv\n",
    "import emeval.input.eval_view as eiev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualization helpers\n",
    "import emeval.viz.phone_view as ezpv\n",
    "import emeval.viz.eval_view as ezev\n",
    "import emeval.viz.geojson as ezgj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For plots\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For maps\n",
    "import folium\n",
    "import branca.element as bre"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For easier debugging while working on modules\n",
    "import importlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import arrow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Not used in this notebook, but let's make sure that the imports work properly\n",
    "import geopandas as gpd\n",
    "import shapely as shp\n",
    "import folium"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The spec\n",
    "\n",
    "The spec defines what experiments were done, and over which time ranges. Once the experiment is complete, most of the structure is read back from the data, but we use the spec to validate that it all worked correctly. The spec also contains the ground truth for the legs. Here, we read the spec for the trip to UC Berkeley."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "AUTHOR_EMAIL = \"shankari@eecs.berkeley.edu\"\n",
    "\n",
    "# If using ServerSpecDetails, data can alternatively be retrieved as such:\n",
    "# DATASTORE_LOC = \"http://localhost:8080\"\n",
    "# sd = eisd.ServerSpecDetails(DATASTORE_LOC, AUTHOR_EMAIL, \"train_bus_ebike_mtv_ucb\")\n",
    "\n",
    "# You must run `cd bin/ && python dump_data_to_file.py --spec-id train_bus_ebike_mtv_ucb`\n",
    "# before using this notebook!\n",
    "\n",
    "DATASTORE_LOC = \"bin/data/\"\n",
    "sd = eisd.FileSpecDetails(DATASTORE_LOC, AUTHOR_EMAIL, \"train_bus_ebike_mtv_ucb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The views\n",
    "\n",
    "There are two main views for the data - the phone view and the evaluation view. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phone view\n",
    "\n",
    "In the phone view, the phone is primary, and then there is a tree that you can traverse to get the data that you want. Traversing that tree typically involves nested for loops; here's an example of loading the phone view and traversing it. You can replace the print statements with real code. When you are ready to check this in, please move the function to one of the python modules so that we can invoke it more generally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pv = eipv.PhoneView(sd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for phone_os, phone_map in pv.map().items():\n",
    "    print(15 * \"=*\")\n",
    "    print(phone_os, phone_map.keys())\n",
    "    for phone_label, phone_detail_map in phone_map.items():\n",
    "        print(4 * ' ', 15 * \"-*\")\n",
    "        print(4 * ' ', phone_label, phone_detail_map.keys())\n",
    "        # this spec does not have any calibration ranges, but evaluation ranges are actually cooler\n",
    "        for r in phone_detail_map[\"evaluation_ranges\"]:\n",
    "            print(8 * ' ', 30 * \"=\")\n",
    "            print(8 * ' ',r.keys())\n",
    "            print(8 * ' ',r[\"trip_id\"], r[\"eval_common_trip_id\"], r[\"eval_role\"], len(r[\"evaluation_trip_ranges\"]), arrow.get(r[\"start_ts\"]))\n",
    "            for tr in r[\"evaluation_trip_ranges\"]:\n",
    "                print(12 * ' ', 30 * \"-\")\n",
    "                print(12 * ' ',tr[\"trip_id\"], tr.keys())\n",
    "                for sr in tr[\"evaluation_section_ranges\"]:\n",
    "                    print(16 * ' ', 30 * \"~\")\n",
    "                    print(16 * ' ',sr[\"trip_id\"], sr.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Eval view\n",
    "\n",
    "In the eval view, the experiment is primary, and then there is a similar tree that you can traverse to get the data that you want. Traversing that tree typically involves nested for loops; here's an example of manipulating the eval view and traversing it. You can replace the print statements with real code. When you are ready to check this in, please move the function to one of the python modules so that we can invoke it more generally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ev = eiev.EvaluationView()\n",
    "ev.from_view_eval_trips(pv, \"\", \"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see these evaluation trips have labels such as `HAHFDC_0`, or `HAMFDC_0`. What do these mean?\n",
    "\n",
    "`HA` or `MA` refer to high accuracy or medium accuracy, respectively. Trips with `HA` will tend to favor GPS utilization and result in high power consumption.\n",
    "\n",
    "`HF` or `MF` refer to high frequency or medium accuracy, respectively. Trips with `HF` will sense and process more often, and are likely to have higher spatiotemporal accuracy (e.g. will hug corners more accurately), albeit with higher power consumption."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for phone_os, eval_map in ev.map(\"evaluation\").items():\n",
    "    print(15 * \"=*\")\n",
    "    print(phone_os, eval_map.keys())\n",
    "    for (curr_calibrate, curr_calibrate_trip_map) in eval_map.items():\n",
    "        print(4 * ' ', 15 * \"-*\")\n",
    "        print(4 * ' ', curr_calibrate, curr_calibrate_trip_map.keys())\n",
    "        for trip_id, trip_map in curr_calibrate_trip_map.items():\n",
    "            print(8 * ' ', 30 * \"=\")\n",
    "            print(8 * ' ', trip_id, trip_map.keys())\n",
    "            for run_label, tr in trip_map.items():\n",
    "                print(12 * ' ', 30 * \"=\")\n",
    "                print(12 * ' ', run_label, tr.keys())\n",
    "                for sr in tr[\"evaluation_section_ranges\"]:\n",
    "                    print(16 * ' ', 30 * \"~\")\n",
    "                    print(16 * ' ',sr[\"trip_id\"], sr.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ground truth\n",
    "\n",
    "The ground truth is stored in the spec, and we can retrieve it from there. Once we have retrieved the trip, there are many possible analyses using them. Please see `get_concat_trajectories` for an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For trips\n",
    "\n",
    "Using the phone view as an example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for phone_os, phone_map in pv.map().items():\n",
    "    print(15 * \"=*\")\n",
    "    print(phone_os, phone_map.keys())\n",
    "    for phone_label, phone_detail_map in phone_map.items():\n",
    "        print(4 * ' ', 15 * \"-*\")\n",
    "        print(4 * ' ', phone_label, phone_detail_map[\"role\"], phone_detail_map.keys())\n",
    "        # this spec does not have any calibration ranges, but evaluation ranges are actually cooler\n",
    "        for r in phone_detail_map[\"evaluation_ranges\"]:\n",
    "            print(8 * ' ', 30 * \"=\")\n",
    "            print(8 * ' ',r.keys())\n",
    "            print(8 * ' ',r[\"trip_id\"], r[\"eval_common_trip_id\"], r[\"eval_role\"], len(r[\"evaluation_trip_ranges\"]))\n",
    "            for tr in r[\"evaluation_trip_ranges\"]:\n",
    "                print(12 * ' ', 30 * \"-\")\n",
    "                print(12 * ' ',tr[\"trip_id\"], tr.keys())\n",
    "                # I am not printing the actual trajectories since that would be too long, only displaying modes\n",
    "                print(tr)\n",
    "                gt_trip = sd.get_ground_truth_for_trip(tr[\"trip_id_base\"], tr[\"start_ts\"], tr[\"end_ts\"])\n",
    "                print(12 * ' ', eisd.SpecDetails.get_concat_trajectories(gt_trip)[\"properties\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For sections\n",
    "\n",
    "Using the eval view as an example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for phone_os, eval_map in ev.map(\"evaluation\").items():\n",
    "    print(15 * \"=*\")\n",
    "    print(phone_os, eval_map.keys())\n",
    "    for (curr_calibrate, curr_calibrate_trip_map) in phone_map.items():\n",
    "        print(4 * ' ', 15 * \"-*\")\n",
    "        print(4 * ' ', curr_calibrate, curr_calibrate_trip_map.keys())\n",
    "        for r in curr_calibrate_trip_map[\"evaluation_ranges\"]:\n",
    "            print(8 * ' ', 30 * \"=\")\n",
    "            print(8 * ' ',r.keys())\n",
    "            print(8 * ' ',r[\"trip_id\"], r[\"eval_common_trip_id\"], r[\"eval_role\"], len(r[\"evaluation_trip_ranges\"]))\n",
    "            for tr in r[\"evaluation_trip_ranges\"]:\n",
    "                print(12 * ' ', 30 * \"-\")\n",
    "                print(12 * ' ',tr[\"trip_id\"], tr.keys())\n",
    "                for sr in tr[\"evaluation_section_ranges\"]:\n",
    "                    print(16 * ' ', 30 * \"-\")\n",
    "                    gt_leg = sd.get_ground_truth_for_leg(tr[\"trip_id_base\"], sr[\"trip_id_base\"], sr[\"start_ts\"], sr[\"end_ts\"])\n",
    "                    print(16 * ' ', sr[\"trip_id\"], gt_leg[\"mode\"], sr.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Work with a single trip\n",
    "\n",
    "You can also work with the details of a single trip - here, we look at the battery drain across phones for the third repetition. Code inspired by `plot_all_power_drain`, located in `emeval.viz.phone_view`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ifig, ax = plt.subplots(ncols=1, nrows=1, figsize=(10,5))\n",
    "for phone_os, phone_map in pv.map().items():\n",
    "    print(15 * \"=*\")\n",
    "    print(phone_os, phone_map.keys())\n",
    "    for phone_label, phone_detail_map in phone_map.items():\n",
    "        print(4 * ' ', 15 * \"-*\")\n",
    "        print(4 * ' ', phone_label, phone_detail_map[\"role\"], phone_detail_map.keys())\n",
    "        curr_range = phone_detail_map[\"evaluation_ranges\"][2]\n",
    "        print(8 * ' ',r[\"trip_id\"], r[\"eval_common_trip_id\"], r[\"eval_role\"], len(r[\"evaluation_trip_ranges\"]))\n",
    "        battery_df = curr_range[\"battery_df\"]\n",
    "        battery_df.plot(x=\"hr\", y=\"battery_level_pct\", ax=ax,\n",
    "                        label=\"%s (%s)\" % (phone_label, phone_detail_map[\"role\"]), ylim=(0,100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Work with a single leg\n",
    "\n",
    "You can also work with the details of a single leg. This is not likely to be useful for power estimates because there are so few points, but it is going to be easier to work with trajectory estimates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "third_repetition = pv.map()[\"ios\"][\"ucb-sdb-ios-1\"][\"evaluation_ranges\"][2]; third_repetition[\"trip_id\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_ucb_trip = third_repetition[\"evaluation_trip_ranges\"][0]; print(to_ucb_trip[\"trip_id\"])\n",
    "bart_leg = to_ucb_trip[\"evaluation_section_ranges\"][5]; bart_leg[\"trip_id\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gt_leg = sd.get_ground_truth_for_leg(to_ucb_trip[\"trip_id_base\"], bart_leg[\"trip_id_base\"], bart_leg[\"start_ts\"], bart_leg[\"end_ts\"]); gt_leg[\"id\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display the leg\n",
    "\n",
    "Note the layer control on the map that allows you to toggle the lines separately"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curr_map = folium.Map()\n",
    "gt_leg_gj = sd.get_geojson_for_leg(gt_leg)\n",
    "sensed_section_gj = ezgj.get_geojson_for_section(bart_leg)\n",
    "gt_leg_gj_feature = folium.GeoJson(gt_leg_gj, name=\"ground_truth\")\n",
    "sensed_leg_gj_feature = folium.GeoJson(sensed_section_gj, name=\"sensed_values\")\n",
    "curr_map.add_child(gt_leg_gj_feature)\n",
    "curr_map.add_child(sensed_leg_gj_feature)\n",
    "curr_map.fit_bounds(sensed_leg_gj_feature.get_bounds())\n",
    "folium.LayerControl().add_to(curr_map)\n",
    "curr_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display the leg with points\n",
    "\n",
    "In this case, the points are in a separate layer so they can be toggled indepdendently of the underlying lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curr_map = folium.Map()\n",
    "gt_leg_gj = sd.get_geojson_for_leg(gt_leg)\n",
    "sensed_section_gj = ezgj.get_geojson_for_section(bart_leg)\n",
    "gt_leg_gj_feature = folium.GeoJson(gt_leg_gj, name=\"ground_truth\")\n",
    "gt_leg_gj_points = ezgj.get_point_markers(gt_leg_gj[2], name=\"ground_truth_points\", color=\"green\")\n",
    "sensed_leg_gj_feature = folium.GeoJson(sensed_section_gj, name=\"sensed_values\")\n",
    "sensed_leg_gj_points = ezgj.get_point_markers(sensed_section_gj, name=\"sensed_points\", color=\"red\")\n",
    "curr_map.add_child(gt_leg_gj_feature)\n",
    "curr_map.add_child(gt_leg_gj_points)\n",
    "curr_map.add_child(sensed_leg_gj_feature)\n",
    "curr_map.add_child(sensed_leg_gj_points)\n",
    "curr_map.fit_bounds(sensed_leg_gj_feature.get_bounds())\n",
    "folium.LayerControl().add_to(curr_map)\n",
    "curr_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}