Exercise-2-problem-3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem 3: How long distance individuals have travelled? (8 points)\n",
    "\n",
    "In this problem the aim is to calculate the distance in meters that the individuals have travelled according the social media posts (Euclidean distances between points). In this problem, we will need the `userid` -column an the points created in the previous problem. You will need the shapefile `Kruger_posts.shp` generated in Problem 2 as input file.\n",
    "\n",
    "Our goal is to answer these questions based on the input data:\n",
    "\n",
    " - What was the shortest distance travelled in meters?\n",
    " - What was the mean distance travelled in meters?\n",
    " - What was the maximum distance travelled in meters?\n",
    "\n",
    "**In your code, you should first:**\n",
    " - Import required modules\n",
    " - Read in the shapefile as a geodataframe called `data`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "988845921e1467790d136a8aafbdcfb1",
     "grade": false,
     "grade_id": "read_data",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epsg:4326\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE HERE\n",
    "import geopandas as gpd\n",
    "import math \n",
    "import os\n",
    "\n",
    "\n",
    "try: \n",
    "    outfolder = r\"C:\\Users\\paul-\\Desktop\\AutoGIS\\Lesson2\\Exercise\\data\"\n",
    "    filename = \"Kruger_posts.shp\"\n",
    "    fp = os.path.join(outfolder, filename)\n",
    "    data = gpd.read_file(fp)\n",
    "    print(data.crs)\n",
    "    \n",
    "except:\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - Check the crs of the input data. If this information is missing, set it as epsg:4326 (WGS84).\n",
    " - Reproject the data from WGS84 to `EPSG:32735` -projection which stands for UTM Zone 35S (UTM zone for South Africa) to transform the data into metric system. (don't create a new variable, update the existing variable `data`!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2f08d863ddefcbdd535aaae138bde86c",
     "grade": false,
     "grade_id": "reproject",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epsg:32735\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE HERE\n",
    "from pyproj import CRS\n",
    "\n",
    "try:\n",
    "    data_wgs84 = data.copy()\n",
    "    \n",
    "    data_utm35s = data.to_crs(epsg=32735)\n",
    "    data = data_utm35s\n",
    "    print(data.crs)\n",
    "    \n",
    "except:\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "af7945d4a1fac737bc0d7da2d248bddb",
     "grade": true,
     "grade_id": "problem_3_gdp_test",
     "locked": true,
     "points": 0.5,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         lat        lon         timestamp    userid  \\\n",
      "0 -24.980792  31.484633  2015-07-07 03:02  66487960   \n",
      "1 -25.499225  31.508906  2015-07-07 03:18  65281761   \n",
      "2 -24.342578  30.930866  2015-03-07 03:38  90916112   \n",
      "3 -24.854614  31.519718  2015-10-07 05:04  37959089   \n",
      "4 -24.921069  31.520836  2015-10-07 05:19  27793716   \n",
      "\n",
      "                            geometry  \n",
      "0  POINT (-4695752.719 14973674.275)  \n",
      "1  POINT (-4748939.258 15014098.837)  \n",
      "2  POINT (-4672729.591 14859391.193)  \n",
      "3  POINT (-4679391.656 14969037.444)  \n",
      "4  POINT (-4686373.982 14973910.589)  \n"
     ]
    }
   ],
   "source": [
    "# NON-EDITABLE CODE CELL FOR TESTING YOUR SOLUTION\n",
    "print(data.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "498786348a23b92458293547750fbad9",
     "grade": true,
     "grade_id": "problem_3_CRS_test",
     "locked": true,
     "points": 1.5,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epsg:32735\n"
     ]
    }
   ],
   "source": [
    "# NON-EDITABLE CODE CELL FOR TESTING YOUR SOLUTION\n",
    "# Check that the crs is correct after re-projecting (should be epsg:32735)\n",
    "print(data.crs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - Group the data by userid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6192e1b2610e7991dc92ac6665886bb3",
     "grade": false,
     "grade_id": "group_data",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             lat        lon         timestamp    userid  \\\n",
      "11543 -24.920770  31.522062  2015-01-04 13:18  37959089   \n",
      "19774 -24.921250  31.520842  2015-01-09 03:37  37959089   \n",
      "12343 -24.921069  31.520836  2015-01-12 03:11  37959089   \n",
      "12639 -24.920897  31.521201  2015-01-15 02:20  37959089   \n",
      "26746 -24.946092  31.469872  2015-01-15 14:54  37959089   \n",
      "...          ...        ...               ...       ...   \n",
      "31763 -24.897742  31.509882  2015-12-14 10:59  37959089   \n",
      "12583 -24.920902  31.521211  2015-12-14 12:08  37959089   \n",
      "21069 -24.921016  31.521449  2015-12-17 13:13  37959089   \n",
      "27548 -24.918002  31.523565  2015-12-21 05:01  37959089   \n",
      "14373 -25.210852  31.035177  2015-12-28 14:11  37959089   \n",
      "\n",
      "                                geometry  \n",
      "11543  POINT (-4686240.069 14974041.654)  \n",
      "19774  POINT (-4686392.690 14973924.281)  \n",
      "12343  POINT (-4686373.982 14973910.589)  \n",
      "12639  POINT (-4686325.270 14973943.802)  \n",
      "26746  POINT (-4693284.175 14969359.364)  \n",
      "...                                  ...  \n",
      "31763  POINT (-4684801.695 14970885.928)  \n",
      "12583  POINT (-4686324.999 14973945.374)  \n",
      "21069  POINT (-4686317.298 14973983.064)  \n",
      "27548  POINT (-4685820.241 14974031.039)  \n",
      "14373  POINT (-4757982.753 14934070.178)  \n",
      "\n",
      "[88 rows x 5 columns]\n",
      "(-4757982.753418185, 14755992.995801535, -4439933.009703052, 14986225.851437792)\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE HERE\n",
    "from shapely.geometry import Point, LineString\n",
    "try:\n",
    "    grouped = data.groupby(\"userid\")\n",
    "    group1 = grouped.get_group(37959089 )\n",
    "    group1_sort = group1.sort_values(by=['timestamp'])\n",
    "    print(group1_sort)\n",
    "    line = LineString(list(group1_sort['geometry']))\n",
    "    print(line.bounds)\n",
    "except:\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ec8a3931cb867faa117f893813b4b205",
     "grade": true,
     "grade_id": "problem_3_grouped_test",
     "locked": true,
     "points": 0.5,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# NON-EDITABLE CODE CELL FOR TESTING YOUR SOLUTION\n",
    "assert len(grouped.groups) == data[\"userid\"].nunique(), \"Number of groups should match number of unique users!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Then:**\n",
    "- Create an empty GeoDataFrame called `movements`\n",
    "- Create a for-loop where you iterate over the grouped object. For each user's data: \n",
    "    - [sort](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sort_values.html) the rows by timestamp \n",
    "    - create a LineString object based on the user's points\n",
    "    - add the geometry and the userid into the `movements` dataframe (one userid per row). You can achieve this either by using the `.at` indexer, or the `append` method. See hints for more help.\n",
    "- Set the CRS of the ``movements`` GeoDataFrame as ``EPSG:32735`` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "474c18c655461eeae34816f4a8c79b95",
     "grade": false,
     "grade_id": "create_lines",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "from shapely.geometry import Point, LineString\n",
    "\n",
    "d = {'geometry':[], 'userid':[]}\n",
    "movements = gpd.GeoDataFrame(data=d)\n",
    "\n",
    "for key, group in grouped:\n",
    "    if len(group) >= 2:\n",
    "        group.sort_values(by=['timestamp'])\n",
    "        line = LineString(list(group['geometry']))\n",
    "        movements.at[key,'userid'] = int(key)\n",
    "        movements.at[key,'geometry'] = line\n",
    "\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "93faad3176849406a2f5eee31d3a8e91",
     "grade": true,
     "grade_id": "problem_3_movements_tests",
     "locked": true,
     "points": 3,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>geometry</th>\n",
       "      <th>userid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16301</th>\n",
       "      <td>LINESTRING (-4684246.015 14939886.378, -468155...</td>\n",
       "      <td>16301.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45136</th>\n",
       "      <td>LINESTRING (-4770692.230 14940874.449, -477069...</td>\n",
       "      <td>45136.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50136</th>\n",
       "      <td>LINESTRING (-4687987.866 14987928.782, -468073...</td>\n",
       "      <td>50136.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88775</th>\n",
       "      <td>LINESTRING (-4773713.345 14938272.132, -477371...</td>\n",
       "      <td>88775.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88918</th>\n",
       "      <td>LINESTRING (-4699374.159 14988142.858, -468798...</td>\n",
       "      <td>88918.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                geometry   userid\n",
       "16301  LINESTRING (-4684246.015 14939886.378, -468155...  16301.0\n",
       "45136  LINESTRING (-4770692.230 14940874.449, -477069...  45136.0\n",
       "50136  LINESTRING (-4687987.866 14987928.782, -468073...  50136.0\n",
       "88775  LINESTRING (-4773713.345 14938272.132, -477371...  88775.0\n",
       "88918  LINESTRING (-4699374.159 14988142.858, -468798...  88918.0"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# NON-EDITABLE CODE CELL FOR TESTING YOUR SOLUTION\n",
    "movements.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Finally:**\n",
    "- Check once the crs definition of your dataframe (should be epsg:32735, define the correct crs if this information is missing)\n",
    "- Calculate the lenghts of the lines into a new column called ``distance`` in ``movements`` GeoDataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "62a930ec9c24c5520f9a28db361e8090",
     "grade": false,
     "grade_id": "calculate_distance",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epsg:32735\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE HERE\n",
    "from pyproj import CRS\n",
    "\n",
    "try:\n",
    "    movements.set_crs(epsg=32735, inplace=True)\n",
    "    print(movements.crs)\n",
    "except:\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4cc148908d9a6ab5b7e83b51dcc3bf6a",
     "grade": true,
     "grade_id": "problem_3_distances_test",
     "locked": true,
     "points": 2,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>geometry</th>\n",
       "      <th>userid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16301</th>\n",
       "      <td>LINESTRING (-4684246.015 14939886.378, -468155...</td>\n",
       "      <td>16301.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45136</th>\n",
       "      <td>LINESTRING (-4770692.230 14940874.449, -477069...</td>\n",
       "      <td>45136.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50136</th>\n",
       "      <td>LINESTRING (-4687987.866 14987928.782, -468073...</td>\n",
       "      <td>50136.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88775</th>\n",
       "      <td>LINESTRING (-4773713.345 14938272.132, -477371...</td>\n",
       "      <td>88775.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88918</th>\n",
       "      <td>LINESTRING (-4699374.159 14988142.858, -468798...</td>\n",
       "      <td>88918.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                geometry   userid\n",
       "16301  LINESTRING (-4684246.015 14939886.378, -468155...  16301.0\n",
       "45136  LINESTRING (-4770692.230 14940874.449, -477069...  45136.0\n",
       "50136  LINESTRING (-4687987.866 14987928.782, -468073...  50136.0\n",
       "88775  LINESTRING (-4773713.345 14938272.132, -477371...  88775.0\n",
       "88918  LINESTRING (-4699374.159 14988142.858, -468798...  88918.0"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# NON-EDITABLE CODE CELL FOR TESTING YOUR SOLUTION\n",
    "movements.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "318c1e1c021eda696ae482bbd26ef9da",
     "grade": false,
     "grade_id": "questions",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "You should now be able to print answers to the following questions: \n",
    "\n",
    " - What was the shortest distance travelled in meters?\n",
    " - What was the mean distance travelled in meters?\n",
    " - What was the maximum distance travelled in meters?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Max area: 5486426.0 meters\n",
      "Min area: 0.0 meters\n",
      "Mean area: 90257.0 meters\n"
     ]
    }
   ],
   "source": [
    "movements[\"line_length\"] = movements['geometry'].length\n",
    "\n",
    "max_length = movements['line_length'].max()\n",
    "mean_length = movements['line_length'].mean()\n",
    "min_length = movements['line_length'].min()\n",
    "\n",
    "print(\"Max area: {maximum} meters\".format(maximum=round(max_length, 0)))\n",
    "print(\"Min area: {minimum} meters\".format(minimum=round(min_length, 0)))\n",
    "print(\"Mean area: {mean} meters\".format(mean=round(mean_length, 0)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Finally, save the movements of into a Shapefile called ``some_movements.shp``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "157bee0141aa073963a58a4362ccc50d",
     "grade": false,
     "grade_id": "save_file",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "\n",
    "try:\n",
    "    out_folder = r\"C:\\Users\\paul-\\Desktop\\AutoGIS\\Lesson2\\Exercise\\data\"\n",
    "    fn = \"some_movements.shp\"\n",
    "    output_fp = os.path.join(out_folder, fn)\n",
    "    movements.to_file(output_fp)\n",
    "except:\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ae12a4596eb26865cecdcbfd160ae72f",
     "grade": true,
     "grade_id": "problem_3_result_tests",
     "locked": true,
     "points": 0.5,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# NON-EDITABLE CODE CELL FOR TESTING YOUR SOLUTION\n",
    "import os\n",
    "assert os.path.isfile(fp), \"output shapefile does not exits\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all for this week!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}