Lec17b.MRsystems.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Map/Reduce Semantics and Systems\n",
    "\n",
    "### Types and transformations\n",
    "* Map is a transformation\n",
    "  * Input domain to output domain\n",
    "* Reduce is a collection\n",
    "  * No domain change\n",
    "  \n",
    "$$\n",
    "map (k1, v1) \\rightarrow list(k2,v2) \\\\\n",
    "reduce (k2, list(v2)) \\rightarrow list(v2)\n",
    "$$\n",
    "\n",
    "* Google C++ implementation is based all on strings\n",
    "  * User code must convert to structured types\n",
    "* Hadoop! has type wrappers\n",
    "\n",
    "### Parallelism in Map/Reduce\n",
    "\n",
    "* How much potential paralleism in mappers? in reducers?\n",
    "\n",
    "\n",
    ".\n",
    "\n",
    ".\n",
    "\n",
    "......spoiler alert.......\n",
    "  \n",
    ".  \n",
    "\n",
    ".\n",
    "* Mappers: up to a parallel process for each input (typically a file)\n",
    "* Reducers: up to a parallel process for each key\n",
    "* So for the WordCount example\n",
    "  * two files = two mappers\n",
    "  * 5 different words = five reducers\n",
    "  * but this is scalable with input\n",
    "\n",
    "\n",
    "Differentiating betweens mapper/reducer and map/reduce processes\n",
    "* A cluster will typically configure number of available phyical processes\n",
    "  * this number is typically much smaller than potential parallelism\n",
    "  * we refer to `mappers` as potential parallelism and `map processes` as the number of phyical processes running map functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Map/Reduce Runtimes\n",
    "\n",
    "From the Google paper https://www.usenix.org/legacy/events/osdi04/tech/dean.html\n",
    "\n",
    "<img src=\"images/mr.png\" width=512 />\n",
    "\n",
    "* Automatically partition input data\n",
    "  * 16-64 MB chunks for Google\n",
    "* Create M map tasks: one for each chunk\n",
    "  * Assign available workers (up to M) to tasks\n",
    "* Write intermediate pairs to local (to worker) disk\n",
    "* R reduce tasks (defined by user) read and process intermediate results\n",
    "* Output is up to R files available on shared file system\n",
    "* Master tracks state\n",
    "  * Asssignment of M map tasks and R reduce tasks to workers\n",
    "  * State and liveliness of the above\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Systems Issues\n",
    "\n",
    "The map/reduce runtime must deal with:\n",
    "* Master failure\n",
    "  * Checkpoint/restart, classic distributed systems/replication problem\n",
    "* Failed worker\n",
    "  * Heartbeat liveness detection, restart\n",
    "* Slow worker\n",
    "  * Backup tasks\n",
    "* Locality of processing to data\n",
    "  * Big deal, they don’t really solve\n",
    "  * But, much subsequent research does\n",
    "* Task granularity\n",
    "  * Metadata size and protocol scaling (not inherent parallelism) limit the size of M and R\n",
    "  \n",
    "### Google File System: The Data Service\n",
    "\n",
    "* Goals\n",
    "  * Wide-distribution\n",
    "  * Commodity hardware\n",
    "  * High (aggregate) performance\n",
    "* Different assumptions than traditional file systems\n",
    "  * Component failures are normal behavior\n",
    "  * Files are huge (new to Google environment ca. 2004)\n",
    "* Most files have append-only writing, \n",
    "  * Mandate append-only writing to realize good I/O properties\n",
    "  * Why append only?\n",
    "      * reduce contention -- logical locking of tail rather than physcial locking of offset\n",
    "      * no data reorganization for writes\n",
    "\n",
    "GFS architecture (from https://www.cs.rutgers.edu/~pxk/417/notes/16-dfs.html)\n",
    "    \n",
    "<img src=\"images/gfs.png\" width=512 />\n",
    "\n",
    "<img src=\"images/gfs2.png\" width=512 />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*  Reliable services\n",
    "  * Master, scheduler, lock services fault tolerant\n",
    "* Data are triple replicated\n",
    "  * On nodes that have independent failure properties (different racks, power supplies, networks)\n",
    "  * This became standard practice in cloud key/value stores (for a decade, now supersedes by distributed error coding)\n",
    "  * Tolerates two failures\n",
    "  * Rereplicated on failure detection\n",
    "  \n",
    "<!--\n",
    "Here is the originial diagram from https://www.usenix.org/legacy/events/osdi04/tech/dean.html\n",
    "\n",
    "<img src=\"images/gfs.orig.png\" width=700 />\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How GFS changed the world\n",
    "\n",
    "* Atomic checkpoint and append\n",
    "  * Major mode for writing\n",
    "  * Great semantics for limited usage\n",
    "* Abandon POSIX file system semantics\n",
    "* In-memory metadata at Master\n",
    "  * Gotta keep it small\n",
    "  * Even for big data (scale metadata memory in proportion to aggregate storage)\n",
    "* Re-replication\n",
    "  * Keep three, detect missing on read/write\n",
    "  * Forget reliable storage, forget RAID\n",
    "* Design for failures (not recovery)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}