Downloading of images scraped from google image.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we are going to download a list of images we previously scraped on google images. \n",
    "\n",
    "The urls of the images are stored in a .txt file, separated by '\\n' characters.\n",
    "\n",
    "You can learn how to do this by following this notebook here : \n",
    "- https://github.com/fastai/course-v3/blob/master/nbs/dl1/lesson2-download.ipynb\n",
    "\n",
    "Or by following this article in French there:\n",
    "- https://medium.com/france-school-of-ai/scraper-un-dataset-depuis-google-images-342f670b9bad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing the .txt files containing the urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import os\n",
    "\n",
    "# Get current directory\n",
    "p = Path('.')\n",
    "\n",
    "# Provide path to the .txt files with the url inside\n",
    "tomato_urls_txt = p/\"tomato_urls.txt\"\n",
    "pepper_urls_txt = p/\"pepper_urls.txt\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Downloading images with fast.ai library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the directories to store the images ..\n",
    "# .. we split images into two folders\n",
    "# .. because there is two classes.\n",
    "os.makedirs(p/\"tomatoes\", exist_ok=True)\n",
    "os.makedirs(p/\"peppers\", exist_ok=True)\n",
    "\n",
    "# Import of the helper function which will help us download the images.\n",
    "from fastai.vision import download_images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading the tomatoes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  - Downloading Tomato Images -   \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='200' class='' max='200', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [200/200 00:19<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error https://greenstalkgarden.com/wp-content/uploads/2016/03/tomato-garden-tower.jpg HTTPSConnectionPool(host='greenstalkgarden.com', port=443): Max retries exceeded with url: /wp-content/uploads/2016/03/tomato-garden-tower.jpg (Caused by SSLError(SSLError(\"bad handshake: SysCallError(104, 'ECONNRESET')\")))\n",
      "Error http://www.tomatodirt.com/images/mulching-tomatoes-straw-gardenweb-forum.jpg HTTPConnectionPool(host='www.tomatodirt.com', port=80): Max retries exceeded with url: /images/mulching-tomatoes-straw-gardenweb-forum.jpg (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f96e0e4d3c8>, 'Connection to www.tomatodirt.com timed out. (connect timeout=4)'))\n",
      "Error https://ottan.me/wp-content/uploads/in-tomato-gardening-ideas-5aef2b58caecf.jpg HTTPSConnectionPool(host='ottan.me', port=443): Max retries exceeded with url: /wp-content/uploads/in-tomato-gardening-ideas-5aef2b58caecf.jpg (Caused by SSLError(SSLCertVerificationError(\"hostname 'ottan.me' doesn't match either of '*.web-hosting.com', 'web-hosting.com'\")))\n",
      "Error https://www.veggiegardener.com/wp-content/uploads/sites/3/2010/05/Growing-tomatoes-620x264.jpg HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f96e0b9f080>: Failed to establish a new connection: [Errno 111] Connection refused'))\n"
     ]
    }
   ],
   "source": [
    "print(\"  - Downloading Tomato Images -   \")\n",
    "download_images(urls=p/\"tomato_urls.txt\",\n",
    "                dest=p/'tomatoes')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading the peppers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  - Downloading Pepper Images -   \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='100' class='' max='100', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [100/100 00:11<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"  - Downloading Pepper Images -   \")\n",
    "download_images(urls=p/\"pepper_urls.txt\",\n",
    "                dest=p/'peppers')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Verifying each images \n",
    "We want images we can read and are in RGB format, so we verify each image one by one and discard the unsuitable ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tomatoes\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='185' class='' max='185', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [185/185 00:01<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "peppers\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='95' class='' max='95', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [95/95 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from fastai.vision import verify_images\n",
    "\n",
    "classes = ['tomatoes', 'peppers']\n",
    "for c in classes:\n",
    "    print(c)  \n",
    "    path_to_class_folder = p/c\n",
    "    \n",
    "    # verify images have correct properties for training\n",
    "    verify_images(path_to_class_folder,\n",
    "                  delete=True, img_format=f'{c} %d')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bravo, here are your images stored into two folders\n",
    "\n",
    "Now, you are ready to do whatever you want with it.\n",
    "\n",
    "Like preprocessing them and feeding them in a neural network like done here:\n",
    "\n",
    "- https://github.com/fastai/course-v3/blob/master/nbs/dl1/lesson2-download.ipynb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fastai",
   "language": "python",
   "name": "fastai"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}