Lec17a.mapreduce.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Map/Reduce\n",
    "\n",
    "* A structured programming model for data parallelism\n",
    "  * Geared toward data intensive applications, not compute.\n",
    "  * Data-parallel, distributed-memory, data-intensive  \n",
    "* Map/Reduce is co-deployed with The **Google File System**\n",
    "  * Large scale parallel file system for large objects\n",
    "  * Failures are the normal case\n",
    "  * M/R requires such a global data service\n",
    "* Open-source re-implementation\n",
    "  * HADOOP: Map/Reduce programming\n",
    "  * HDFS: Haddop Distributed File System\n",
    "\n",
    "### Some important properties\n",
    "\n",
    "* Automated parallelism\n",
    "  * programmer has no interaction with number of processes workers.\n",
    "  * same program runs on any M/R cluster\n",
    "* Functional programming concepts\n",
    "  * break progam into a data parallel portion (`map`) and a reduction (`reduce`)\n",
    "  * user-defined (procedural) functions for each task executed in a functional framework\n",
    "  * No *working memory* in either mappers or reducers. The functions are created and applied on each data element.\n",
    "* Sequential/streaming data access\n",
    "  * data are accessed from disk to mappers\n",
    "  * mapper outputs a stream (of keys and values)\n",
    "  * stream shuffled to reducers\n",
    "  * reducers output to file system\n",
    "\n",
    "  \n",
    "<img src=\"https://cdn-images-1.medium.com/max/1600/1*KKm4roOpsum147kKk5qp7A.jpeg\" width=768 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Key/Value\n",
    "\n",
    "* All M/R data are key/value pairs\n",
    "    * keys are interpreted by the M/R system. Must be sortable.\n",
    "    * values are not interpreted by M/R system. Only used in functions. \n",
    "* Colors in this figure represent keys\n",
    "  * that are not changed by mapper (in this example)\n",
    "  * shuffled by the engine and delivered to reducer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Map/Reduce Example\n",
    "\n",
    "Wordcount example from the original Google paper. Produce a count of the occurrence of each word in a set of documents.\n",
    "\n",
    "```python\n",
    "    map ( String key, String value ):\n",
    "        // key: document name (file name)\n",
    "        // value: document contents\n",
    "        for each word w in value:\n",
    "            EmitIntermediate ( word, \"1\" );\n",
    "```\n",
    "\n",
    "Mapper outputs `key=word, value=\"1\"` for each word. Note that the output key and input key are different.\n",
    "  * map is a **transformation** of the key/value schema\n",
    "  * the only effect of the function is the emtited key, value pair.\n",
    "\n",
    "```python\n",
    "    reduce ( String key, Iterator values ):\n",
    "        // key: a word\n",
    "        // value: a list of counts\n",
    "        int result = 0;\n",
    "        for each word v in values:\n",
    "            result += ParsseInt ( v );\n",
    "        EmitAsString ( result );\n",
    "```\n",
    "Reducer sums the counts of words.  Some properties:\n",
    "  * reducer gets a list of values at a key\n",
    "  * reduce cannot change the key, emits a value, that is reduced from list\n",
    "  * user defined function\n",
    "  \n",
    "### WordCount Visualized\n",
    "\n",
    "<img src=\"https://www.researchgate.net/profile/Oscar_Pereira3/publication/270448794/figure/fig6/AS:295098651824130@1447368409317/Word-count-program-flow-executed-with-MapReduce-5.png\" width=768 title=\"from Oscar Perreira @ ResearchGate\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What can you do?\n",
    "\n",
    "It seems like a limited programming framework.  But, many string processing programs fit naturally in this model.\n",
    "  * Grep\n",
    "  * Web-log processing\n",
    "  * Reverse web-link graph\n",
    "  * Term vectors per host\n",
    "  * Inverted index\n",
    "  * Distributed sort"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Java",
   "language": "java",
   "name": "java"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "Java",
   "pygments_lexer": "java",
   "version": "11.0.6+8-b765.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}