cs120_lab1a_math_review.ipynb

{"cells":[{"cell_type":"markdown","source":["<a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-nd/4.0/\"><img alt=\"Creative Commons License\" style=\"border-width:0\"src=\"https://i.creativecommons.org/l/by-nc-nd/4.0/88x31.png\"/></a><br/>This work is licensed under a<a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-nd/4.0/\">Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 InternationalLicense</a>."],"metadata":{}},{"cell_type":"markdown","source":["![ML Logo](http://spark-mooc.github.io/web-assets/images/CS190.1x_Banner_300.png)\n# Math and Python review\n\nThis notebook reviews vector and matrix math, the [NumPy](http://www.numpy.org/) Python package, and Python lambda expressions.  Part 1 covers vector and matrix math, and you'll do a few exercises by hand.  In Part 2, you'll learn about NumPy and use `ndarray` objects to solve the math exercises.   Part 3 provides additional information about NumPy and how it relates to array usage in Spark's [MLlib](https://spark.apache.org/mllib/).  Part 4 provides an overview of lambda expressions.\n\nTo move through the notebook just run each of the cells.  You can run a cell by pressing \"shift-enter\", which will compute the current cell and advance to the next cell, or by clicking in a cell and pressing \"control-enter\", which will compute the current cell and remain in that cell.  You should move through the notebook from top to bottom and run all of the cells.  If you skip some cells, later cells might not work as expected.\nNote that there are several exercises within this notebook.  You will need to provide solutions for cells that start with: `# TODO: Replace <FILL IN> with appropriate code`.\n\n** This notebook covers: **\n* *Part 1:* Math review\n* *Part 2:* NumPy\n* *Part 3:* Additional NumPy and Spark linear algebra\n* *Part 4:* Python lambda expressions\n* *Appendix A:* Submitting your exercises to the Autograder"],"metadata":{}},{"cell_type":"code","source":["labVersion = 'cs120x-lab1a-1.0.0'"],"metadata":{},"outputs":[],"execution_count":3},{"cell_type":"markdown","source":["## Part 1: Math review"],"metadata":{}},{"cell_type":"markdown","source":["### (1a) Scalar multiplication: vectors\n\nIn this exercise, you will calculate the product of a scalar and a vector by hand and enter the result in the code cell below.  Scalar multiplication is straightforward.  The resulting vector equals the product of the scalar, which is a single value, and each item in the original vector.\nIn the example below, \\\\( a \\\\) is the scalar (constant) and \\\\( \\mathbf{v} \\\\) is the vector.  \\\\[ a \\mathbf{v} = \\begin{bmatrix} a v_1 \\\\\\ a v_2 \\\\\\ \\vdots \\\\\\ a v_n \\end{bmatrix} \\\\]\n\nCalculate the value of \\\\( \\mathbf{x} \\\\): \\\\[ \\mathbf{x} = 3 \\begin{bmatrix} 1 \\\\\\ -2 \\\\\\ 0 \\end{bmatrix} \\\\]\nCalculate the value of \\\\( \\mathbf{y} \\\\): \\\\[ \\mathbf{y} = 2 \\begin{bmatrix} 2 \\\\\\ 4 \\\\\\ 8 \\end{bmatrix} \\\\]"],"metadata":{}},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\n# Manually calculate your answer and represent the vector as a list of integers.\n# For example, [2, 4, 8].\nvectorX = [3, -6, 0]\nvectorY = [4, 8, 16]"],"metadata":{},"outputs":[],"execution_count":6},{"cell_type":"code","source":["# TEST Scalar multiplication: vectors (1a)\n# Import test library\nfrom databricks_test_helper import Test\n\nTest.assertEqualsHashed(vectorX, 'e460f5b87531a2b60e0f55c31b2e49914f779981',\n                        'incorrect value for vectorX')\nTest.assertEqualsHashed(vectorY, 'e2d37ff11427dbac7f833a5a7039c0de5a740b1e',\n                        'incorrect value for vectorY')"],"metadata":{},"outputs":[],"execution_count":7},{"cell_type":"markdown","source":["### (1b) Element-wise multiplication: vectors\n\nIn this exercise, you will calculate the element-wise multiplication of two vectors by hand and enter the result in the code cell below.  You'll later see that element-wise multiplication is the default method when two NumPy arrays are multiplied together.  Note we won't be performing element-wise multiplication in future labs, but we are introducing it here to distinguish it from other vector operators. It is also a common operation in NumPy, as we will discuss in Part (2b).\n\nThe element-wise calculation is as follows: \\\\[ \\mathbf{x} \\odot \\mathbf{y} =  \\begin{bmatrix} x_1 y_1 \\\\\\  x_2 y_2 \\\\\\ \\vdots \\\\\\ x_n y_n \\end{bmatrix} \\\\]\n\nCalculate the value of \\\\( \\mathbf{z} \\\\): \\\\[ \\mathbf{z} = \\begin{bmatrix} 1 \\\\\\  2 \\\\\\ 3 \\end{bmatrix} \\odot \\begin{bmatrix} 4 \\\\\\  5 \\\\\\ 6 \\end{bmatrix} \\\\]"],"metadata":{}},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\n# Manually calculate your answer and represent the vector as a list of integers.\nz = [4, 10, 18]"],"metadata":{},"outputs":[],"execution_count":9},{"cell_type":"code","source":["# TEST Element-wise multiplication: vectors (1b)\nTest.assertEqualsHashed(z, '4b5fe28ee2d274d7e0378bf993e28400f66205c2',\n                        'incorrect value for z')"],"metadata":{},"outputs":[],"execution_count":10},{"cell_type":"markdown","source":["### (1c) Dot product\n\nIn this exercise, you will calculate the dot product of two vectors by hand and enter the result in the code cell below.  Note that the dot product is equivalent to performing element-wise multiplication and then summing the result.\n\nBelow, you'll find the calculation for the dot product of two vectors, where each vector has length \\\\( n \\\\): \\\\[ \\mathbf{w} \\cdot \\mathbf{x} = \\sum_{i=1}^n w_i x_i \\\\]\n\nNote that you may also see \\\\( \\mathbf{w} \\cdot \\mathbf{x} \\\\) represented as \\\\( \\mathbf{w}^\\top \\mathbf{x} \\\\)\n\nCalculate the value for \\\\( c_1 \\\\) based on the dot product of the following two vectors:\n\\\\[ c_1 = \\begin{bmatrix} 1 & -3 \\end{bmatrix} \\cdot \\begin{bmatrix} 4 \\\\\\ 5 \\end{bmatrix}\\\\]\n\nCalculate the value for \\\\( c_2 \\\\) based on the dot product of the following two vectors:\n\\\\[ c_2 = \\begin{bmatrix} 3 & 4 & 5 \\end{bmatrix} \\cdot \\begin{bmatrix} 1 \\\\\\ 2 \\\\\\ 3 \\end{bmatrix}\\\\]"],"metadata":{}},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\n# Manually calculate your answer and set the variables to their appropriate integer values.\nc1 = -11\nc2 = 26"],"metadata":{},"outputs":[],"execution_count":12},{"cell_type":"code","source":["# TEST Dot product (1c)\nTest.assertEqualsHashed(c1, '8d7a9046b6a6e21d66409ad0849d6ab8aa51007c', 'incorrect value for c1')\nTest.assertEqualsHashed(c2, '887309d048beef83ad3eabf2a79a64a389ab1c9f', 'incorrect value for c2')"],"metadata":{},"outputs":[],"execution_count":13},{"cell_type":"markdown","source":["### (1d) Matrix multiplication\n\nIn this exercise, you will calculate the result of multiplying two matrices together by hand and enter the result in the code cell below.\nRefer to the slides for the formula for multiplying two matrices together.\n\nFirst, you'll calculate the value for \\\\( \\mathbf{X} \\\\).\n\\\\[ \\mathbf{X} = \\begin{bmatrix} 1 & 2 & 3 \\\\\\ 4 & 5 & 6 \\end{bmatrix} \\begin{bmatrix} 1 & 2 \\\\\\ 3 & 4 \\\\\\ 5 & 6 \\end{bmatrix} \\\\]\n\nNext, you'll perform an outer product and calculate the value for \\\\( \\mathbf{Y} \\\\).\n\n\\\\[ \\mathbf{Y} = \\begin{bmatrix} 1 \\\\\\ 2 \\\\\\ 3 \\end{bmatrix} \\begin{bmatrix} 1 & 2 & 3 \\end{bmatrix} \\\\]\n\nThe resulting matrices should be stored row-wise (see [row-major order](https://en.wikipedia.org/wiki/Row-major_order)). This means that the matrix is organized by rows. For instance, a 2x2 row-wise matrix would be represented as: \\\\( [[r_1c_1, r_1c_2], [r_2c_1, r_2c_2]] \\\\) where r stands for row and c stands for column.\n\nNote that outer product is just a special case of general matrix multiplication and follows the same rules as normal matrix multiplication."],"metadata":{}},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\n# Represent matrices as lists within lists. For example, [[1,2,3], [4,5,6]] represents a matrix with\n# two rows and three columns. Use integer values.\nmatrixX = [[22, 28], [49, 64]]\nmatrixY = [[1, 2, 3], [2, 4, 6], [3, 6, 9]]"],"metadata":{},"outputs":[],"execution_count":15},{"cell_type":"code","source":["# TEST Matrix multiplication (1d)\nTest.assertEqualsHashed(matrixX, 'c2ada2598d8a499e5dfb66f27a24f444483cba13',\n                        'incorrect value for matrixX')\nTest.assertEqualsHashed(matrixY, 'f985daf651531b7d776523836f3068d4c12e4519',\n                        'incorrect value for matrixY')"],"metadata":{},"outputs":[],"execution_count":16},{"cell_type":"markdown","source":["##  Part 2: NumPy"],"metadata":{}},{"cell_type":"markdown","source":["### (2a) Scalar multiplication\n\n[NumPy](http://docs.scipy.org/doc/numpy/reference/) is a Python library for working with arrays.  NumPy provides abstractions that make it easy to treat these underlying arrays as vectors and matrices.  The library is optimized to be fast and memory efficient, and we'll be using it throughout the course.  The building block for NumPy is the [ndarray](http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.html), which is a multidimensional array of fixed-size that contains elements of one type (e.g. array of floats).\n\nFor this exercise, you'll create a `ndarray` consisting of the elements \\[1, 2, 3\\] and multiply this array by 5.  Use [np.array()](http://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html) to create the array.  Note that you can pass a Python list into `np.array()`.  To perform scalar multiplication with an `ndarray` just use `*`.\n\nNote that if you create an array from a Python list of integers you will obtain a one-dimensional array, *which is equivalent to a vector for our purposes*."],"metadata":{}},{"cell_type":"code","source":["# It is convention to import NumPy with the alias np\nimport numpy as np"],"metadata":{},"outputs":[],"execution_count":19},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\n# Create a numpy array with the values 1, 2, 3\nsimpleArray = np.array([1, 2, 3])\n# Perform the scalar product of 5 and the numpy array\ntimesFive = 5 * simpleArray\nprint 'simpleArray\\n{0}'.format(simpleArray)\nprint '\\ntimesFive\\n{0}'.format(timesFive)"],"metadata":{},"outputs":[],"execution_count":20},{"cell_type":"code","source":["# TEST Scalar multiplication (2a)\nTest.assertTrue(np.all(timesFive == [5, 10, 15]), 'incorrect value for timesFive')"],"metadata":{},"outputs":[],"execution_count":21},{"cell_type":"markdown","source":["### (2b) Element-wise multiplication and dot product\n\nNumPy arrays support both element-wise multiplication and dot product.  Element-wise multiplication occurs automatically when you use the `*` operator to multiply two `ndarray` objects of the same length.\n\nTo perform the dot product you can use either [np.dot()](http://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html#numpy.dot) or [np.ndarray.dot()](http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.dot.html).  For example, if you had NumPy arrays `x` and `y`, you could compute their dot product four ways: `np.dot(x, y)`, `np.dot(y, x)`, `x.dot(y)`, or `y.dot(x)`.\n\nFor this exercise, multiply the arrays `u` and `v` element-wise and compute their dot product."],"metadata":{}},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\n# Create a ndarray based on a range and step size.\nu = np.arange(0, 5, .5)\nv = np.arange(5, 10, .5)\n\nelementWise = u * v\ndotProduct = u.dot(v)\nprint 'u: {0}'.format(u)\nprint 'v: {0}'.format(v)\nprint '\\nelementWise\\n{0}'.format(elementWise)\nprint '\\ndotProduct\\n{0}'.format(dotProduct)"],"metadata":{},"outputs":[],"execution_count":23},{"cell_type":"code","source":["# TEST Element-wise multiplication and dot product (2b)\nTest.assertTrue(np.all(elementWise == [ 0., 2.75, 6., 9.75, 14., 18.75, 24., 29.75, 36., 42.75]),\n                'incorrect value for elementWise')\nTest.assertEquals(dotProduct, 183.75, 'incorrect value for dotProduct')"],"metadata":{},"outputs":[],"execution_count":24},{"cell_type":"markdown","source":["### (2c) Matrix math\nWith NumPy it is very easy to perform matrix math.  You can use [np.matrix()](http://docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.html) to generate a NumPy matrix.  Just pass a two-dimensional `ndarray` or a list of lists to the function.  You can perform matrix math on NumPy matrices using `*`.\n\nYou can transpose a matrix by calling [numpy.matrix.transpose()](http://docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.transpose.html) or by using `.T` on the matrix object (e.g. `myMatrix.T`).  Transposing a matrix produces a matrix where the new rows are the columns from the old matrix. For example: \\\\[  \\begin{bmatrix} 1 & 2 & 3 \\\\\\ 4 & 5 & 6 \\end{bmatrix}^\\top = \\begin{bmatrix} 1 & 4 \\\\\\ 2 & 5 \\\\\\ 3 & 6 \\end{bmatrix} \\\\]\n\nInverting a matrix can be done using [numpy.linalg.inv()](http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.inv.html).  Note that only square matrices can be inverted, and square matrices are not guaranteed to have an inverse.  If the inverse exists, then multiplying a matrix by its inverse will produce the identity matrix.  \\\\( \\scriptsize ( A^{-1} A = I_n ) \\\\)  The identity matrix \\\\( \\scriptsize I_n \\\\) has ones along its diagonal and zeros elsewhere. \\\\[ I_n = \\begin{bmatrix} 1 & 0 & 0 & ... & 0 \\\\\\ 0 & 1 & 0 & ... & 0 \\\\\\ 0 & 0 & 1 & ... & 0 \\\\\\ ... & ... & ... & ... & ... \\\\\\ 0 & 0 & 0 & ... & 1 \\end{bmatrix} \\\\]\n\nFor this exercise, multiply \\\\( A \\\\) times its transpose \\\\( ( A^\\top ) \\\\) and then calculate the inverse of the result \\\\( (  [ A A^\\top ]^{-1}  ) \\\\)."],"metadata":{}},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\nfrom numpy.linalg import inv\n\nA = np.matrix([[1,2,3,4],[5,6,7,8]])\nprint 'A:\\n{0}'.format(A)\n# Print A transpose\nprint '\\nA transpose:\\n{0}'.format(A.T)\n\n# Multiply A by A transpose\nAAt = A * A.T\nprint '\\nAAt:\\n{0}'.format(AAt)\n\n# Invert AAt with np.linalg.inv()\nAAtInv = inv(AAt)\nprint '\\nAAtInv:\\n{0}'.format(AAtInv)\n\n# Show inverse times matrix equals identity\n# We round due to numerical precision\nprint '\\nAAtInv * AAt:\\n{0}'.format((AAtInv * AAt).round(4))"],"metadata":{},"outputs":[],"execution_count":26},{"cell_type":"code","source":["# TEST Matrix math (2c)\nTest.assertTrue(np.all(AAt == np.matrix([[30, 70], [70, 174]])), 'incorrect value for AAt')\nTest.assertTrue(np.allclose(AAtInv, np.matrix([[0.54375, -0.21875], [-0.21875, 0.09375]])),\n                'incorrect value for AAtInv')"],"metadata":{},"outputs":[],"execution_count":27},{"cell_type":"markdown","source":["### Part 3: Additional NumPy and Spark linear algebra"],"metadata":{}},{"cell_type":"markdown","source":["### (3a) Slices\n\nYou can select a subset of a one-dimensional NumPy `ndarray`'s elements by using slices.  These slices operate the same way as slices for Python lists.  For example, `[0, 1, 2, 3][:2]` returns the first two elements `[0, 1]`.  NumPy, additionally, has more sophisticated slicing that allows slicing across multiple dimensions; however, you'll only need to use basic slices in future labs for this course.\n\nNote that if no index is placed to the left of a `:`, it is equivalent to starting at 0, and hence `[0, 1, 2, 3][:2]` and `[0, 1, 2, 3][0:2]` yield the same result.  Similarly, if no index is placed to the right of a `:`, it is equivalent to slicing to the end of the object.  Also, you can use negative indices to index relative to the end of the object, so `[-2:]` would return the last two elements of the object.\n\nFor this exercise, return the last 3 elements of the array `features`."],"metadata":{}},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\nfeatures = np.array([1, 2, 3, 4])\nprint 'features:\\n{0}'.format(features)\n\n# The last three elements of features\nlastThree = features[-3:]\n\nprint '\\nlastThree:\\n{0}'.format(lastThree)"],"metadata":{},"outputs":[],"execution_count":30},{"cell_type":"code","source":["# TEST Slices (3a)\nTest.assertTrue(np.all(lastThree == [2, 3, 4]), 'incorrect value for lastThree')"],"metadata":{},"outputs":[],"execution_count":31},{"cell_type":"markdown","source":["### (3b) Combining `ndarray` objects\n\nNumPy provides many functions for creating new arrays from existing arrays.  We'll explore two functions: [np.hstack()](http://docs.scipy.org/doc/numpy/reference/generated/numpy.hstack.html), which allows you to combine arrays column-wise, and [np.vstack()](http://docs.scipy.org/doc/numpy/reference/generated/numpy.vstack.html), which allows you to combine arrays row-wise.  Note that both `np.hstack()` and `np.vstack()` take in a tuple of arrays as their first argument.  To horizontally combine three arrays `a`, `b`, and `c`, you would run `np.hstack((a, b, c))`.\nIf we had two arrays: `a = [1, 2, 3, 4]` and `b = [5, 6, 7, 8]`, we could use `np.vstack((a, b))` to produce the two-dimensional array: \\\\[  \\begin{bmatrix} 1 & 2 & 3 & 4 \\\\\\ 5 & 6 & 7 & 8 \\end{bmatrix} \\\\]\n\nFor this exercise, you'll combine the `zeros` and `ones` arrays both horizontally (column-wise) and vertically (row-wise).\nNote that the result of stacking two arrays is an `ndarray`.  If you need the result to be a matrix, you can call `np.matrix()` on the result, which will return a NumPy matrix."],"metadata":{}},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\nzeros = np.zeros(8)\nones = np.ones(8)\nprint 'zeros:\\n{0}'.format(zeros)\nprint '\\nones:\\n{0}'.format(ones)\n\nzerosThenOnes = np.hstack((zeros, ones))   # A 1 by 16 array\nzerosAboveOnes = np.vstack((zeros, ones))  # A 2 by 8 array\n\nprint '\\nzerosThenOnes:\\n{0}'.format(zerosThenOnes)\nprint '\\nzerosAboveOnes:\\n{0}'.format(zerosAboveOnes)"],"metadata":{},"outputs":[],"execution_count":33},{"cell_type":"code","source":["# TEST Combining ndarray objects (3b)\nTest.assertTrue(np.all(zerosThenOnes == [0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1]),\n                'incorrect value for zerosThenOnes')\nTest.assertTrue(np.all(zerosAboveOnes == [[0,0,0,0,0,0,0,0],[1,1,1,1,1,1,1,1]]),\n                'incorrect value for zerosAboveOnes')"],"metadata":{},"outputs":[],"execution_count":34},{"cell_type":"markdown","source":["### (3c) PySpark's DenseVector\n\nIn frequent ML scenarios, you may end up with very long vectors, possibly 100k's to millions, where most of the values are zeroes.  PySpark provides a [DenseVector](https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.linalg.DenseVector) class (in module the module [pyspark.mllib.linalg](https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#module-pyspark.mllib.linalg)), which allows you to more efficiently operate and store these sparse vectors.\n\n`DenseVector` is used to store arrays of values for use in PySpark.  `DenseVector` actually stores values in a NumPy array and delegates calculations to that object.  You can create a new `DenseVector` using `DenseVector()` and passing in a NumPy array or a Python list.\n\n`DenseVector` implements several functions.  The only function needed for this course is `DenseVector.dot()`, which operates just like `np.ndarray.dot()`.\nNote that `DenseVector` stores all values as `np.float64`, so even if you pass in an NumPy array of integers, the resulting `DenseVector` will contain floating-point numbers. Also, `DenseVector` objects exist locally and are not inherently distributed.  `DenseVector` objects can be used in the distributed setting by either passing functions that contain them to resilient distributed dataset (RDD) transformations or by distributing them directly as RDDs.  You'll learn more about RDDs in the spark tutorial.\n\nFor this exercise, create a `DenseVector` consisting of the values `[3.0, 4.0, 5.0]` and compute the dot product of this vector with `numpyVector`."],"metadata":{}},{"cell_type":"code","source":["from pyspark.mllib.linalg import DenseVector"],"metadata":{},"outputs":[],"execution_count":36},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\nnumpyVector = np.array([-3, -4, 5])\nprint '\\nnumpyVector:\\n{0}'.format(numpyVector)\n\n# Create a DenseVector consisting of the values [3.0, 4.0, 5.0]\nmyDenseVector = DenseVector([3.0, 4.0, 5.0])\n# Calculate the dot product between the two vectors.\ndenseDotProduct =myDenseVector.dot(numpyVector)\n\nprint 'myDenseVector:\\n{0}'.format(myDenseVector)\nprint '\\ndenseDotProduct:\\n{0}'.format(denseDotProduct)"],"metadata":{},"outputs":[],"execution_count":37},{"cell_type":"code","source":["# TEST PySpark's DenseVector (3c)\nTest.assertTrue(isinstance(myDenseVector, DenseVector), 'myDenseVector is not a DenseVector')\nTest.assertTrue(np.allclose(myDenseVector, np.array([3., 4., 5.])),\n                'incorrect value for myDenseVector')\nTest.assertTrue(np.allclose(denseDotProduct, 0.0), 'incorrect value for denseDotProduct')"],"metadata":{},"outputs":[],"execution_count":38},{"cell_type":"markdown","source":["## Part 4: Python lambda expressions"],"metadata":{}},{"cell_type":"markdown","source":["### (4a) Lambda is an anonymous function\n\nWe can use a lambda expression to create a function.  To do this, you type `lambda` followed by the names of the function's parameters separated by commas, followed by a `:`, and then the expression statement that the function will evaluate.  For example, `lambda x, y: x + y` is an anonymous function that computes the sum of its two inputs.\n\nLambda expressions return a function when evaluated.  The function is not bound to any variable, which is why lambdas are associated with anonymous functions.  However, it is possible to assign the function to a variable.  Lambda expressions are particularly useful when you need to pass a simple function into another function.  In that case, the lambda expression generates a function that is bound to the parameter being passed into the function.\n\nBelow, we'll see an example of how we can bind the function returned by a lambda expression to a variable named `addSLambda`.  From this example, we can see that `lambda` provides a shortcut for creating a simple function.  Note that the behavior of the function created using `def` and the function created using `lambda` is equivalent.  Both functions have the same type and return the same results.  The only differences are the names and the way they were created.\nFor this exercise, first run the two cells below to compare a function created using `def` with a corresponding anonymous function.  Next, write your own lambda expression that creates a function that multiplies its input (a single parameter) by 10.\n\nHere are some additional references that explain lambdas: [Lambda Functions](http://www.secnetix.de/olli/Python/lambda_functions.hawk), [Lambda Tutorial](https://pythonconquerstheuniverse.wordpress.com/2011/08/29/lambda_tutorial/), and [Python Functions](http://www.bogotobogo.com/python/python_functions_lambda.php)."],"metadata":{}},{"cell_type":"code","source":["# Example function\ndef addS(x):\n    return x + 's'\nprint type(addS)\nprint addS\nprint addS('cat')"],"metadata":{},"outputs":[],"execution_count":41},{"cell_type":"code","source":["# As a lambda\naddSLambda = lambda x: x + 's'\nprint type(addSLambda)\nprint addSLambda\nprint addSLambda('cat')"],"metadata":{},"outputs":[],"execution_count":42},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\n# Recall that: \"lambda x, y: x + y\" creates a function that adds together two numbers\nmultiplyByTen = lambda x: x * 10\nprint multiplyByTen(5)\n\n# Note that the function still shows its name as <lambda>\nprint '\\n', multiplyByTen"],"metadata":{},"outputs":[],"execution_count":43},{"cell_type":"code","source":["# TEST Python lambda expressions (4a)\nTest.assertEquals(multiplyByTen(10), 100, 'incorrect definition for multiplyByTen')"],"metadata":{},"outputs":[],"execution_count":44},{"cell_type":"markdown","source":["### (4b) `lambda` fewer steps than `def`\n\n`lambda` generates a function and returns it, while `def` generates a function and assigns it to a name.  The function returned by `lambda` also automatically returns the value of its expression statement, which reduces the amount of code that needs to be written.\n\nFor this exercise, recreate the `def` behavior using `lambda`.  Note that since a lambda expression returns a function, it can be used anywhere an object is expected. For example, you can create a list of functions where each function in the list was generated by a lambda expression."],"metadata":{}},{"cell_type":"code","source":["# Code using def that we will recreate with lambdas\ndef plus(x, y):\n    return x + y\n\ndef minus(x, y):\n    return x - y\n\nfunctions = [plus, minus]\nprint functions[0](4, 5)\nprint functions[1](4, 5)"],"metadata":{},"outputs":[],"execution_count":46},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\n# The first function should add two values, while the second function should subtract the second\n# value from the first value.\nlambdaFunctions = [lambda x, y: x + y,  lambda x, y: x - y]\nprint lambdaFunctions[0](4, 5)\nprint lambdaFunctions[1](4, 5)"],"metadata":{},"outputs":[],"execution_count":47},{"cell_type":"code","source":["# TEST lambda fewer steps than def (4b)\nTest.assertEquals(lambdaFunctions[0](10, 10), 20, 'incorrect first lambdaFunction')\nTest.assertEquals(lambdaFunctions[1](10, 10), 0, 'incorrect second lambdaFunction')"],"metadata":{},"outputs":[],"execution_count":48},{"cell_type":"markdown","source":["### (4c) Lambda expression arguments\n\nLambda expressions can be used to generate functions that take in zero or more parameters.  The syntax for `lambda` allows for multiple ways to define the same function.  For example, we might want to create a function that takes in a single parameter, where the parameter is a tuple consisting of two values, and the function adds the two values together.  The syntax could be either: `lambda x: x[0] + x[1]` or `lambda (x0, x1): x0 + x1`.  If we called either function on the tuple `(3, 4)` it would return `7`.  Note that the second `lambda` relies on the tuple `(3, 4)` being unpacked automatically, which means that `x0` is assigned the value `3` and `x1` is assigned the value `4`.\n\nAs an other example, consider the following parameter lambda expressions: `lambda x, y: (x[0] + y[0], x[1] + y[1])` and `lambda (x0, x1), (y0, y1): (x0 + y0, x1 + y1)`.  The result of applying either of these functions to tuples  `(1, 2)` and `(3, 4)` would be the tuple `(4, 6)`.\n\nFor this exercise: you'll create one-parameter functions `swap1` and `swap2` that swap the order of a tuple; a one-parameter function `swapOrder` that takes in a tuple with three values and changes the order to: second element, third element, first element; and finally, a three-parameter function `sumThree` that takes in three tuples, each with two values, and returns a tuple containing two values: the sum of the first element of each tuple and the sum of second element of each tuple."],"metadata":{}},{"cell_type":"code","source":["# Examples.  Note that the spacing has been modified to distinguish parameters from tuples.\n\n# One-parameter function\na1 = lambda x: x[0] + x[1]\na2 = lambda (x0, x1): x0 + x1\nprint 'a1( (3,4) ) = {0}'.format( a1( (3,4) ) )\nprint 'a2( (3,4) ) = {0}'.format( a2( (3,4) ) )\n\n# Two-parameter function\nb1 = lambda x, y: (x[0] + y[0], x[1] + y[1])\nb2 = lambda (x0, x1), (y0, y1): (x0 + y0, x1 + y1)\nprint '\\nb1( (1,2), (3,4) ) = {0}'.format( b1( (1,2), (3,4) ) )\nprint 'b2( (1,2), (3,4) ) = {0}'.format( b2( (1,2), (3,4) ) )"],"metadata":{},"outputs":[],"execution_count":50},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\n# Use both syntaxes to create a function that takes in a tuple of two values and swaps their order\n# E.g. (1, 2) => (2, 1)\nswap1 = lambda x: (x[1], x[0])\nswap2 = lambda (x0, x1): (x1, x0)\nprint 'swap1((1, 2)) = {0}'.format(swap1((1, 2)))\nprint 'swap2((1, 2)) = {0}'.format(swap2((1, 2)))\n\n# Using either syntax, create a function that takes in a tuple with three values and returns a tuple\n# of (2nd value, 3rd value, 1st value).  E.g. (1, 2, 3) => (2, 3, 1)\nswapOrder = lambda x: (x[1], x[2], x[0])\nprint 'swapOrder((1, 2, 3)) = {0}'.format(swapOrder((1, 2, 3)))\n\n# Using either syntax, create a function that takes in three tuples each with two values.  The\n# function should return a tuple with the values in the first position summed and the values in the\n# second position summed. E.g. (1, 2), (3, 4), (5, 6) => (1 + 3 + 5, 2 + 4 + 6) => (9, 12)\nsumThree = lambda x, y, z: (x[0] + y[0] + z[0], x[1] + y[1] + z[1])\nprint 'sumThree((1, 2), (3, 4), (5, 6)) = {0}'.format(sumThree((1, 2), (3, 4), (5, 6)))"],"metadata":{},"outputs":[],"execution_count":51},{"cell_type":"code","source":["# TEST Lambda expression arguments (4c)\nTest.assertEquals(swap1((1, 2)), (2, 1), 'incorrect definition for swap1')\nTest.assertEquals(swap2((1, 2)), (2, 1), 'incorrect definition for swap2')\nTest.assertEquals(swapOrder((1, 2, 3)), (2, 3, 1), 'incorrect definition for swapOrder')\nTest.assertEquals(sumThree((1, 2), (3, 4), (5, 6)), (9, 12), 'incorrect definition for sumThree')"],"metadata":{},"outputs":[],"execution_count":52},{"cell_type":"markdown","source":["### (4d) Restrictions on lambda expressions\n\n[Lambda expressions](https://docs.python.org/2/reference/expressions.html#lambda) consist of a single [expression statement](https://docs.python.org/2/reference/simple_stmts.html#expression-statements) and cannot contain other [simple statements](https://docs.python.org/2/reference/simple_stmts.html).  In short, this means that the lambda expression needs to evaluate to a value and exist on a single logical line.  If more complex logic is necessary, use `def` in place of `lambda`.\n\nExpression statements evaluate to a value (sometimes that value is None).  Lambda expressions automatically return the value of their expression statement.  In fact, a `return` statement in a `lambda` would raise a `SyntaxError`.\n\n The following Python keywords refer to simple statements that cannot be used in a lambda expression: `assert`, `pass`, `del`, `print`, `return`, `yield`, `raise`, `break`, `continue`, `import`, `global`, and `exec`.  Also, note that assignment statements (`=`) and augmented assignment statements (e.g. `+=`) cannot be used either."],"metadata":{}},{"cell_type":"code","source":["# Just run this code\n# This code will fail with a syntax error, as we can't use print in a lambda expression\nimport traceback\ntry:\n    exec \"lambda x: print x\"\nexcept:\n    traceback.print_exc()"],"metadata":{},"outputs":[],"execution_count":54},{"cell_type":"markdown","source":["### (4e) Functional programming\n\nThe `lambda` examples we have shown so far have been somewhat contrived.  This is because they were created to demonstrate the differences and similarities between `lambda` and `def`.  An excellent use case for lambda expressions is functional programming.  In functional programming, you will often pass functions to other functions as parameters, and `lambda` can be used to reduce the amount of code necessary and to make the code more readable.\nSome commonly used functions in functional programming are map, filter, and reduce.  Map transforms a series of elements by applying a function individually to each element in the series.  It then returns the series of transformed elements.  Filter also applies a function individually to each element in a series; however, with filter, this function evaluates to `True` or `False` and only elements that evaluate to `True` are retained.  Finally, reduce operates on pairs of elements in a series.  It applies a function that takes in two values and returns a single value.  Using this function, reduce is able to, iteratively, \"reduce\" a series to a single value.\n\nFor this exercise, you'll create three simple `lambda` functions, one each for use in map, filter, and reduce.  The map `lambda` will multiply its input by 5, the filter `lambda` will evaluate to `True` for even numbers, and the reduce `lambda` will add two numbers.\n\n> Note:\n> * We have created a class called `FunctionalWrapper` so that the syntax for this exercise matches the syntax you'll see in PySpark.\n> * Map requires a one parameter function that returns a new value, filter requires a one parameter function that returns `True` or `False`, and reduce requires a two parameter function that combines the two parameters and returns a new value."],"metadata":{}},{"cell_type":"code","source":["# Create a class to give our examples the same syntax as PySpark\nclass FunctionalWrapper(object):\n    def __init__(self, data):\n        self.data = data\n    def map(self, function):\n        \"\"\"Call `map` on the items in `data` using the provided `function`\"\"\"\n        return FunctionalWrapper(map(function, self.data))\n    def reduce(self, function):\n        \"\"\"Call `reduce` on the items in `data` using the provided `function`\"\"\"\n        return reduce(function, self.data)\n    def filter(self, function):\n        \"\"\"Call `filter` on the items in `data` using the provided `function`\"\"\"\n        return FunctionalWrapper(filter(function, self.data))\n    def __eq__(self, other):\n        return (isinstance(other, self.__class__)\n            and self.__dict__ == other.__dict__)\n    def __getattr__(self, name):  return getattr(self.data, name)\n    def __getitem__(self, k):  return self.data.__getitem__(k)\n    def __repr__(self):  return 'FunctionalWrapper({0})'.format(repr(self.data))\n    def __str__(self):  return 'FunctionalWrapper({0})'.format(str(self.data))"],"metadata":{},"outputs":[],"execution_count":56},{"cell_type":"code","source":["# Map example\n\n# Create some data\nmapData = FunctionalWrapper(range(5))\n\n# Define a function to be applied to each element\nf = lambda x: x + 3\n\n# Imperative programming: loop through and create a new object by applying f\nmapResult = FunctionalWrapper([])  # Initialize the result\nfor element in mapData:\n    mapResult.append(f(element))  # Apply f and save the new value\nprint 'Result from for loop: {0}'.format(mapResult)\n\n# Functional programming: use map rather than a for loop\nprint 'Result from map call: {0}'.format(mapData.map(f))\n\n# Note that the results are the same but that the map function abstracts away the implementation\n# and requires less code"],"metadata":{},"outputs":[],"execution_count":57},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\ndataset = FunctionalWrapper(range(10))\n\n# Multiply each element by 5\nmapResult = dataset.map(lambda x: x*5)\n# Keep the even elements\n# Note that \"x % 2\" evaluates to the remainder of x divided by 2\nfilterResult = dataset.filter(lambda x: not x%2)\n# Sum the elements\nreduceResult = dataset.reduce(lambda x, y: x + y)\n\nprint 'mapResult: {0}'.format(mapResult)\nprint '\\nfilterResult: {0}'.format(filterResult)\nprint '\\nreduceResult: {0}'.format(reduceResult)"],"metadata":{},"outputs":[],"execution_count":58},{"cell_type":"code","source":["# TEST Functional programming (4e)\nTest.assertEquals(mapResult, FunctionalWrapper([0, 5, 10, 15, 20, 25, 30, 35, 40, 45]),\n                  'incorrect value for mapResult')\nTest.assertEquals(filterResult, FunctionalWrapper([0, 2, 4, 6, 8]),\n                  'incorrect value for filterResult')\nTest.assertEquals(reduceResult, 45, 'incorrect value for reduceResult')"],"metadata":{},"outputs":[],"execution_count":59},{"cell_type":"markdown","source":["### (4f) Composability\n\nSince our methods for map and filter in the `FunctionalWrapper` class return `FunctionalWrapper` objects, we can compose (or chain) together our function calls.  For example, `dataset.map(f1).filter(f2).reduce(f3)`, where `f1`, `f2`, and `f3` are functions or lambda expressions, first applies a map operation to `dataset`, then filters the result from map, and finally reduces the result from the first two operations.\n\n Note that when we compose (chain) an operation, the output of one operation becomes the input for the next operation, and operations are applied from left to right.  It's likely you've seen chaining used with Python strings.  For example, `'Split this'.lower().split(' ')` first returns a new string object `'split this'` and then `split(' ')` is called on that string to produce `['split', 'this']`.\n\nFor this exercise, reuse your lambda expressions from (4e) but apply them to `dataset` in the sequence: map, filter, reduce.\n\n> Note:\n> * Since we are composing the operations our result will be different than in (4e).\n> * We can write our operations on separate lines to improve readability."],"metadata":{}},{"cell_type":"code","source":["# Example of a multi-line expression statement\n# Note that placing parentheses around the expression allows it to exist on multiple lines without\n# causing a syntax error.\n(dataset\n .map(lambda x: x + 2)\n .reduce(lambda x, y: x * y))"],"metadata":{},"outputs":[],"execution_count":61},{"cell_type":"code","source":["# TODO: Replace <FILL IN> with appropriate code\n# Multiply the elements in dataset by five, keep just the even values, and sum those values\nfinalSum = dataset.map(lambda x: x*5).filter(lambda x: not x%2).reduce(lambda x,y: x+y)\nprint finalSum"],"metadata":{},"outputs":[],"execution_count":62},{"cell_type":"code","source":["# TEST Composability (4f)\nTest.assertEquals(finalSum, 100, 'incorrect value for finalSum')"],"metadata":{},"outputs":[],"execution_count":63},{"cell_type":"markdown","source":["## Appendix A: Submitting Your Exercises to the Autograder\n\nThis section guides you through Step 2 of the grading process (\"Submit to Autograder\").\n\nOnce you confirm that your lab notebook is passing all tests, you can submit it first to the course autograder and then second to the edX website to receive a grade.\n\n** Note that you can only submit to the course autograder once every 1 minute. **"],"metadata":{}},{"cell_type":"markdown","source":["### Step 2(a): Restart your cluster by clicking on the dropdown next to your cluster name and selecting \"Restart Cluster\".\n\nYou can do this step in either notebook, since there is one cluster for your notebooks.\n\n<img src=\"http://spark-mooc.github.io/web-assets/images/submit_restart.png\" alt=\"Drawing\" />"],"metadata":{}},{"cell_type":"markdown","source":["### Step 2(b): _IN THIS NOTEBOOK_, click on \"Run All\" to run all of the cells.\n\n<img src=\"http://spark-mooc.github.io/web-assets/images/submit_runall.png\" alt=\"Drawing\" style=\"height: 80px\"/>\n\nThis step will take some time.\n\nWait for your cluster to finish running the cells in your lab notebook before proceeding."],"metadata":{}},{"cell_type":"markdown","source":["### Step 2(c): Publish this notebook\n\nPublish _this_ notebook by clicking on the \"Publish\" button at the top.\n\n<img src=\"http://spark-mooc.github.io/web-assets/images/Lab0_Publish0.png\" alt=\"Drawing\" style=\"height: 150px\"/>\n\nWhen you click on the button, you will see the following popup.\n\n<img src=\"http://spark-mooc.github.io/web-assets/images/Lab0_Publish1.png\" alt=\"Drawing\" />\n\nWhen you click on \"Publish\", you will see a popup with your notebook's public link. **Copy the link and set the `notebook_URL` variable in the AUTOGRADER notebook (not this notebook).**\n\n<img src=\"http://spark-mooc.github.io/web-assets/images/Lab0_Publish2.png\" alt=\"Drawing\" />"],"metadata":{}},{"cell_type":"markdown","source":["### Step 2(d): Set the notebook URL and Lab ID in the Autograder notebook, and run it\n\nGo to the Autograder notebook and paste the link you just copied into it, so that it is assigned to the `notebook_url` variable.\n\n```\nnotebook_url = \"...\" # put your URL here\n```\n\nThen, find the line that looks like this:\n\n```\nlab = <FILL IN>\n```\nand change `<FILL IN>` to \"CS120x-lab1a\":\n\n```\nlab = \"CS120x-lab1a\"\n```\n\nThen, run the Autograder notebook to submit your lab."],"metadata":{}},{"cell_type":"markdown","source":["### <img src=\"http://spark-mooc.github.io/web-assets/images/oops.png\" style=\"height: 200px\"/> If things go wrong\n\nIt's possible that your notebook looks fine to you, but fails in the autograder. (This can happen when you run cells out of order, as you're working on your notebook.) If that happens, just try again, starting at the top of Appendix A."],"metadata":{}}],"metadata":{"name":"cs120_lab1a_math_review","notebookId":3354834026783850},"nbformat":4,"nbformat_minor":0}