18 - Python 10min a day - A bit more on generators and how to create them.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"assets/jeremy-lapak-CVvFVQ_-oUg-unsplash.png\" alt=\"Python Envs\" style=\"display: block; margin: 0 auto\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning Python 10 minutes a day #18\n",
    "## A bit more on generators and how to create them\n",
    "[Medium article link](https://towardsdatascience.com/learning-python-10-minutes-a-day-18-4718eb73758c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a [series](https://python-10-minutes-a-day.rocks) of short 10 minute Python articles helping you to get started with Python. I try to post an article each day (no promises), starting from the very basics, going up to more complex idioms. Feel free to contact me on [LinkedIn](https://www.linkedin.com/in/dennisbakhuis/) for questions or requests on particular subjects of Python, you want to know about."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have already come across generators, an iterable that lazily computes the value when it is needed. This can have benefits in speed, but the biggest benefit is the reduction in memory. Regular list need to be allocated all at once and a list of one million values can have a tremendous memory foot print. When we process the list one (or a couple of) value at a time, it is much smarter to use generators. We have already seen the range() function, this is however officially not a generator (but still a bit similar). One thing about generators is that it can only iterate over the iterable once. Range() can have repeated full iterations. This is of course only a subtle difference but good to keep in mind. In the last session, we discussed comprehensions, but we did not mention one special type: the generator comprehension (or generator expression). This type of comprehension looks identical to a list comprehension but uses parenthesis instead of brackets and as the name suggests, returns a generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a generator comprehension\n",
    "generator = (x for x in range(10))\n",
    "print('type:', type(generator))\n",
    "\n",
    "# we can iterate over it\n",
    "for number in generator:\n",
    "    print(number, end=' ')\n",
    "else:\n",
    "    print()\n",
    "    \n",
    "# but only once\n",
    "for number in generator:\n",
    "    print('this will not print as it is finished')\n",
    "\n",
    "# recreate it\n",
    "generator = (x for x in range(3))\n",
    "print('first value :', next(generator))\n",
    "print('second value:', next(generator))\n",
    "print('third value :', next(generator))\n",
    "print('this will give an \"StopIteration\" exception', next(generator))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generators do not have a length and therefore, you cannot use the length() function. In this example, we are aware of the length because we have defined it (range(10)). However, if we use a generator to iterate through a large file, line by line, the generator by definition does not have a length. Actually, to get the length, i.e. the amount of lines to read, you need to read the complete file and count the ‘\\n’ characters. This would already cost a full iteration cycle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to get the next value from a generator is by using the next() function. You have to keep in mind that if the generator is empty, i.e. the full iterations has finished, the next() function will raise an StopIteration exception. This might make you wonder how the for-loop works as you cannot check if there is still an item remaining. It turns out that the for-loop catches the StopIteration exception: [It is better to ask for forgiveness then permission](https://towardsdatascience.com/learning-python-10-minutes-a-day-12-44806ea44852)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We learned that generators are great for reducing the memory footprint of your Python program or even make it possible to read through an 100GB file. To do many of these specialized tasks, you need to define your own generator functions. A generator function looks like a regular function and calculates the current value. This value is then returned, but instead of using the return keyword, we have to use yield. Normally, when using return, the state of the function is garbage-collected. This means that all variables inside the function are removed. Using yield, the state of the function is maintained, keeping all variables in tact. When a for-loop or next() call asks for the next value, the generator function is resumed to generate the next value. Lets create a simple generator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define our generator function\n",
    "def sequential_numbers(max_number):\n",
    "    \"\"\"\n",
    "    Simple generator example\n",
    "    \"\"\"\n",
    "    current_number = 0\n",
    "    while current_number <= max_number:\n",
    "        yield current_number\n",
    "        current_number += 1\n",
    "\n",
    "# create our new generator\n",
    "generator = sequential_numbers(5)\n",
    "\n",
    "# it is really a generator\n",
    "print('type:', type(generator))\n",
    "\n",
    "# which we can loop through\n",
    "for number in generator:\n",
    "    print(number, end=' ')\n",
    "else:\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the generator function itself, it is already clear why the generator is empty after a full iteration. After it reaches max_number, the function is finished and it cannot generate a new value. To iterate over the generator again, we need to re-initiate the generator again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function itself, can be anything. Examples we have already mentioned is a line-by-line iteration of a larger-than-memory file, but there can be many other usages. A generator can combine one or more database calls, can download information from the internet, and can be defined such that they can run indefinitely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generators are an advanced feature in Python and they can be useful in certain situations. As I already mentioned, when memory is an issue, generators are a possible solution. Values are calculated on the fly. If you only need a couple of values (lets say the first ten of a total of 100), you can omit calculations that are not required, resulting in speed-ups. More advanced things you can do with generators are for example stacked generators. Combining generators can mimic pipe-lines, an efficient technique known for example in the Unix world."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice for today:\n",
    "When discussing more advanced topics, the assignments can also get more interesting. As I like numbers, today we are going to generate [Fibonacci numbers](https://en.wikipedia.org/wiki/Fibonacci_number). The Fibonacci numbers (or the Fibonacci sequence) are quite magical as they return very frequently in nature. The sequence is defines that the current Fibonacci numbers is the sum of the two previous numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "F = [0, 1, 1, 2, 3, 5, 8, 13, ....... ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assignment:\n",
    "Create a infinite generator to generate Fibonacci numbers. Test you function with a while loop (and a break). As the generator is infinite, you can continue with a second loop."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hints:\n",
    "1. an infinite loop can be created with a while-loop that is True forever.\n",
    "2. the generator function needs to keep track of the last two values and yield the sum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "actual_pi = math.pin = 100  # amount of numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have posted a [solution](https://gist.github.com/dennisbakhuis/ad7ceab8238e1eac2df6e98d06c859d2) on my Github."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have any questions, feel free to contact me through [LinkedIn](https://www.linkedin.com/in/dennisbakhuis/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}