Lecture1_FeatureProcessing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.2"
    },
    "colab": {
      "name": "Lecture1_FeatureProcessing.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2SFVvbTdQHl4"
      },
      "source": [
        "# **How to Tackle a Machine Learning Competition**\n",
        "\n",
        "## **Lecture 1: Feature Processing**\n",
        "(Author: Simone Azeglio, simone.azeglio@edu.unito.it)\n",
        "\n",
        "---\n",
        "---\n",
        "\n",
        "## **Overview**\n",
        "\n",
        "* [Data Spectrometry](#section1)\n",
        "    * [Macro, Meso, Micro-scale in Science](#section1.1)\n",
        "    * [Scaling in Data Science](#section1.2)\n",
        "    * [Preprocessing Data](#section1.3)\n",
        "      * [Standardization](#section1.3.1)\n",
        "      * [Scaling Features to a Range](#section1.3.2)\n",
        "      * [Non-linear Transformations](#section1.3.3)\n",
        "      * [Normalization](#section1.3.4)\n",
        "      * [Encoding Categorical Features](#section1.3.5)\n",
        "\n",
        "* [Feature Engineering](#section2)\n",
        "    * [Feature Importance](#section2.0)\n",
        "      * [Coefficients as Feature Importance](#section2.0.1)\n",
        "      * [Decision Tree Feature Importance](#section2.0.2)\n",
        "      * [Random Forest Feature Importance](#section2.0.3)\n",
        "      * [XGBoost Feature Importance](#section2.0.4)\n",
        "      * [Permutation Feature Importance](#section2.0.5)\n",
        "\n",
        "    * [Feature Selection](#section2.1)\n",
        "      * [Removing Features with Low Variance](#section2.1.0)\n",
        "      * [Univariate Feature Selection](#section2.1.1)\n",
        "      * [Recursive Feature Elimination](#section2.1.2)\n",
        "\n",
        "    * [Feature Extraction (Bonus)](#section2.2)\n",
        "      * [Loading Features from Dicts](#section2.2.1)\n",
        "      * [Feature Hashing](#section2.2.2)\n",
        "      \n",
        "    * [Feature Construction](#section2.3)\n",
        "\n",
        "* [Handling missing values](#section3)\n",
        "  * [Univariate Feature Imputation](#section3.1)\n",
        "  * [Multivariate Feature Imputation](#section3.2)\n",
        "  * [Nearest Neighbors Imputation](#section3.3)\n",
        "\n",
        "* [References & Additional Material](#section4)\n",
        "\n",
        "---\n",
        "---\n",
        "\n",
        "<a id='section1'></a>\n",
        "# **Data Spectrometry**\n",
        "In this first part, we'd like to tell you about some practical tricks for making **gradient descent** work well, in particular, we're going to delve into feature scaling. As an introductory view, it seems reasonable to try to depict an intuition of the concept of *scale*. \n",
        "\n",
        "<a id='section1.1'></a>\n",
        "## **Macro, Meso, Micro-scale in Science**\n",
        "\n",
        "As scientists, we are well aware of the effects of using a specific measurement tool in order to characterize some quantity and describe reality. As an ideal example we consider the **length scale**. \n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/MachineLearningJournalClub/HowToTackleAMLCompetition/master/img/Lecture1/1.0.png\" width=\"500\" height=\"300\">\n",
        "\n",
        "We can identify three different points of view: *microscopic*, *mesoscopic* and *macroscopic*; which are intimately related to the adopted lenght scale. \n",
        "\n",
        "We usually deal with the *macroscopic scale* when the observer is in such a position (pretty far, in terms of distance), with respect to the object, that she/he can describe its global characteristics. Instead, we do refer to the *microscopic scale* when the observer is so close to the object that she/he can describe its atomistic details or elementary parts (e.g. molecules, atoms, quarks). Last but not least, we talk about *mesoscopic scale* everytime we are in between micro and macro. \n",
        "\n",
        "These definitions are deliberately vague, since delineating a precise and neat explanation would be higly difficult and complex, and it's actually far from our purposes. \n",
        "\n",
        "On the other side, this kind of introduction is quite useful, we should take a few minutes to think about the \"active\" role of the observer and about the fact that, to be honest, for every length scale, there's some specific theory, i.e. there's no global theory for a multi-scale description of some phenomenon. \n",
        "\n",
        "<a id='section1.2'></a>\n",
        "## **Scaling in Data Science**\n",
        "\n",
        "If our beloved observer (i.e. the scientist) has some kind of \"privilege\", i.e. choosing the right measurement tool, which is nothing but choosing the right scale in the description of some phenomenon, we can't really say the same for a data scientist. \n",
        "\n",
        "It's a sort of paradox, but a data scientist can't really deal with data retrieval most of the times. Because of that, a data scientist is often left alone in front of data, without even knowing from which measurement tool they're coming from. There's no way to interact with the length scale for example. \n",
        "\n",
        "Is there something that we can do about it? The only thing we can do is assuming that features are independent and scale these features in order to have something compatible from one to the other. This procedure is called **feature scaling**, and soon we'll understand why it is useful even for ML algorithms, such as gradient descent. \n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/MachineLearningJournalClub/HowToTackleAMLCompetition/master/img/Lecture1/1.1.png\" width=\"500\" height=\"300\">\n",
        "\n",
        "If you make sure that features are on similar scales, i.e. features take on similar range of values, then gradient descent can converge more quickly. \n",
        "\n",
        "More concretely, let's say we have a problem with two features where $x_1$ is the length of a football field and take values between $90$ (meters) and $115$ (meters) and $x_2$ is the radius of a ball which takes values between $10.5* 10^{-2}$ (meters) to $11.5* 10^{-2}$ (meters). If you plot the countours of the cost function $J(\\omega)$ then you might get something similar to the *left plot*, and because of these very skewed elliptical shape, if we run gradient descent on this cost function, it may end up taking a long time and oscillating back and forth before reaching the global minimum. \n",
        "\n",
        "In these settings, as stated previously, a useful thing to do is to scale the features. Generally, the idea is to get every feature into approximately a $-1$ to $+1$ range. By doing this, we get the *right plot*. In this way, you can find a much more direct path to the global minimum rather than taking a much more convoluted path where you're sort of trying to follow a very complicated trajectory. \n",
        "\n",
        "<a id='section1.3'></a>\n",
        "## **Preprocessing Data**\n",
        "\n",
        "> In any Machine Learning process, Data Preprocessing is that step in which the data gets transformed, or *encoded*, to bring it to such a state that now the machine can easily parse it. In other words, the features of the data can now be easily interpreted by the algorithm.\n",
        "\n",
        "We're going to dive into Scikit-Learn for this section and exploit its powerful *processing* package. \n",
        "\n",
        "We've been talking about scaling our data, now it's time to understand how to put our hands on code and try to do that. Usually, as previously stated, learning algorithms benefit from standardization of the data set. If some outliers are present in the set, robust scalers or transformers might be more appropriate. (Take a look at [Compare the effect of different scalers on data with outliers](https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py), you'll see the behaviors of the different scalers, transformers and normalizers with outliers). \n",
        "\n",
        "<a id='section1.3.1'></a>\n",
        "### Standardization \n",
        "\n",
        "Many Machine Learning estimators require *standardization* of datasets, elseways they might behave badly because data are far from a Gaussian (with zero mean and unit variance) distribution. \n",
        "\n",
        "Most of the times, we ignore the shape of the distribution and just transform the data by subtracting the mean value of each feature, then scale by dividing features by their standard deviation. \n",
        "\n",
        "---\n",
        "\n",
        "*Do you have in mind some models that assume that all features are centered around zero and have variance in the same order of magnitude? Can you think about possible issues related to the objective function in these cases?* \n",
        "\n",
        "<ins>Answer</ins>:  many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the l1 and l2 regularizers of linear models) assume that all features are centered around zero and have variance in the same order. If a feature has a variance that is orders of magnitude larger than others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.\n",
        "\n",
        "---\n",
        "\n",
        "There's a fast way to do that on a single array, by means of the *scale* function\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X08yunwD8w6F",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 156
        },
        "outputId": "edbbf08d-9a16-48d8-fb36-be711850c773"
      },
      "source": [
        "from sklearn import preprocessing\n",
        "import numpy as np\n",
        "X_train = np.random.randint(5, size = (3,3))\n",
        "X_scaled = preprocessing.scale(X_train)\n",
        "\n",
        "print(X_train)\n",
        "print(X_scaled)\n",
        "print(X_scaled.mean(axis=0))\n",
        "print(X_scaled.std(axis=0))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[1 2 0]\n",
            " [2 3 4]\n",
            " [1 4 3]]\n",
            "[[-0.70710678 -1.22474487 -1.37281295]\n",
            " [ 1.41421356  0.          0.98058068]\n",
            " [-0.70710678  1.22474487  0.39223227]]\n",
            "[ 1.48029737e-16  0.00000000e+00 -7.40148683e-17]\n",
            "[1. 1. 1.]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Une2CkAg-JKZ"
      },
      "source": [
        "The *preprocessing* module provides a utility class [*StandardScaler*](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler) that compute the mean and std on a training set so as to be able to later reapply the same transform on the test set. \n",
        "\n",
        "(You should be well aware of what [*sklearn.pipeline.Pipeline*](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html#sklearn.pipeline.Pipeline) is, it's crucial for strategies' deployment.)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BoDix2zbws3r",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        },
        "outputId": "1832a082-566a-4565-b7b7-38060a9cbad8"
      },
      "source": [
        "scaler = preprocessing.StandardScaler().fit(X_train)\n",
        "print(scaler.mean_)\n",
        "print(scaler.scale_)\n",
        "print(scaler.transform(X_train))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[1.33333333 3.         2.33333333]\n",
            "[0.47140452 0.81649658 1.69967317]\n",
            "[[-0.70710678 -1.22474487 -1.37281295]\n",
            " [ 1.41421356  0.          0.98058068]\n",
            " [-0.70710678  1.22474487  0.39223227]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cOgsh6rX68m1"
      },
      "source": [
        "Now we can use the scaler instance on new data, to transform them in the same way we did previously."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cV02i1QGDiRy",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "d261b072-3d08-4fc4-a466-34c581596c3b"
      },
      "source": [
        "X_test = [[-1., 1., 0.]]\n",
        "scaler.transform(X_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[-4.94974747, -2.44948974, -1.37281295]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9vM1kqrnDn6J"
      },
      "source": [
        "It is possible to disable centering or scaling by passing *with_mean = False* or *with_std = False*. The first one might be particularly useful if applied to sparse CSR or CSC matrices to avoid breaking the sparsity structure of the data. \n",
        "\n",
        "<a id='section1.3.2'></a>\n",
        "**<ins>Scaling Features to a Range</ins>**\n",
        "\n",
        "Another standardization is scaling features to lie between a given minimum and maximum value, or so that the maximum absolute value of each future is scaled to unit size. This can be achieved with [*MinMaxScaler*](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html#sklearn.preprocessing.MinMaxScaler) or [*MaxAbsScaler*](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MaxAbsScaler.html#sklearn.preprocessing.MaxAbsScaler). \n",
        "\n",
        "Here you can see how to scale a toy data matrix to the $[0,1]$ range: \n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TBV03PztFNL5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "645905d7-40b8-40b0-847d-cd7dd94fa4f0"
      },
      "source": [
        "X_train = np.random.randint(10, size = (3,3))\n",
        "\n",
        "min_max_scaler = preprocessing.MinMaxScaler().fit(X_train)\n",
        "X_train_minmax = min_max_scaler.transform(X_train)\n",
        "X_train_minmax"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1.        , 0.        , 1.        ],\n",
              "       [0.        , 0.75      , 0.85714286],\n",
              "       [0.25      , 1.        , 0.        ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kCo1MJ-VI5DG"
      },
      "source": [
        "In the same way as above, the same instance of the transformer can be applied to some new test data: same scaling and shifting will be applied to be consistent. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dcvX_h1QFNJT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 201
        },
        "outputId": "1e6b4715-a321-4a50-e3da-0ca25a3cc51b"
      },
      "source": [
        "X_test = np.array([[3., 1.,  4.]])\n",
        "X_test_minmax = min_max_scaler.transform(X_test)\n",
        "X_test_minmax"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-f17357233e8f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mX_test\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m3.\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1.\u001b[0m\u001b[0;34m,\u001b[0m  \u001b[0;36m4.\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mX_test_minmax\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmin_max_scaler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtransform\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mX_test_minmax\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'np' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0005DUh5JGM9"
      },
      "source": [
        "It's pretty useful to let the scaler reveal some details about the transformation learned on the training data: \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "awJKnoBuJVfx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "79940e51-7fdb-48e7-8230-734925edf599"
      },
      "source": [
        "print(min_max_scaler.scale_)\n",
        "print(min_max_scaler.min_)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[0.25       0.33333333 1.        ]\n",
            "[-0.5        -0.33333333  0.        ]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mOeXk00qLYlY"
      },
      "source": [
        "---\n",
        "\n",
        "Can you retrieve the explicit formula for *MinMaxScaler*? \n",
        "\n",
        "X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0))\n",
        "\n",
        "X_scaled = X_std * (max - min) + min\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fqajX9xkLif_"
      },
      "source": [
        "*MaxAbsScaler* works in a similar fashion, the data will lie in the range $[-1,1]$. It is meant for data that is already centered at zero or sparse data. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a6AB8r8wL2WS",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "3a9ca2aa-01e2-4b23-bf13-cc17e53f9779"
      },
      "source": [
        "max_abs_scaler = preprocessing.MaxAbsScaler()\n",
        "X_train_maxabs = max_abs_scaler.fit_transform(X_train)\n",
        "X_train_maxabs"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1.        , 1.        , 1.        ],\n",
              "       [0.33333333, 0.75      , 0.        ],\n",
              "       [0.83333333, 0.25      , 0.        ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1waiboCTL_Sh",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "10a1af8a-be76-48ca-b036-2fc523801533"
      },
      "source": [
        "X_test_maxabs = max_abs_scaler.transform(X_test)\n",
        "X_test_maxabs"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[-0.5 , -0.25,  4.  ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3wWp9tRJMCXU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "0bb1db41-859b-4695-b647-28284ca3aefb"
      },
      "source": [
        "max_abs_scaler.scale_"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([6., 4., 1.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F4_4i1WlL2Gd"
      },
      "source": [
        "**<ins>Scaling Data with Outliers</ins>**\n",
        "\n",
        "If our data contain many outliers, scaling using the mean and variance of the data is not likely to work well. In this case, we can use [*RobustScaler*](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html#sklearn.preprocessing.RobustScaler).\n",
        "\n",
        "This scaler removes the [median](https://en.wikipedia.org/wiki/Median) and scales data according to the [IQR](https://en.wikipedia.org/wiki/Interquartile_range) (InterQuartile Range). \n",
        "\n",
        "Centering and scaling happen independently on each feature by computing the relevant statistics on the samples in the training set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W3Qc07nMTXxo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "c5d465bd-1eab-441c-ee4f-019171973b04"
      },
      "source": [
        "from sklearn.preprocessing import RobustScaler\n",
        "X_train = [[ 1., -2.,  2.],\n",
        "     [ -2.,  1.,  3.],\n",
        "     [ 40.,  1., -2.]]\n",
        "transformer = RobustScaler().fit(X_train)\n",
        "transformer.transform(X_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 0.        , -2.        ,  0.        ],\n",
              "       [-0.14285714,  0.        ,  0.4       ],\n",
              "       [ 1.85714286,  0.        , -1.6       ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q0qK3kMoTxlo"
      },
      "source": [
        "Median and interquartile range are then stored to be used on later data using the *transform* method.\n",
        "\n",
        "<a id='section1.3.3'></a>\n",
        "### Non-linear transformations\n",
        "\n",
        "It's possible to generalize to non-linear transformations. We are going to talk about two types of transformations: *quantile transforms* and *power transforms*. The main take-home message is that we need *monotonic* transformations to preserve the rank of the values along each feature. \n",
        "\n",
        "Quantile transforms smooth out unusual distributions and are less influenced by outliers than scaling methods. It does distort correlations and distances within and across features. \n",
        "\n",
        "Power transforms are, indeed, a family of parametric transformations that aim to map data from any distribution to as close to a Gaussian distribution. \n",
        "\n",
        "**<ins>Mapping to a Uniform Distribution</ins>**\n",
        "\n",
        "[*QuantileTransformer*](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.QuantileTransformer.html#sklearn.preprocessing.QuantileTransformer) provides a non-parametric transformation to map the data to a uniform distribution with values between 0 and 1: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n_6MPt62oL6n",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "d686ae4c-0194-4cb7-8653-e185a03eaf9b"
      },
      "source": [
        "from sklearn.datasets import load_iris\n",
        "from sklearn.model_selection import train_test_split\n",
        "# load dataset\n",
        "X, y = load_iris(return_X_y=True)\n",
        "# train test splitting\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
        "# Quantile transform\n",
        "quantile_transformer = preprocessing.QuantileTransformer(random_state=0)\n",
        "X_train_trans = quantile_transformer.fit_transform(X_train)\n",
        "X_test_trans = quantile_transformer.transform(X_test)\n",
        "np.percentile(X_train[:, 0], [0, 25, 50, 75, 100]) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/preprocessing/_data.py:2357: UserWarning: n_quantiles (1000) is greater than the total number of samples (112). n_quantiles is set to n_samples.\n",
            "  % (self.n_quantiles, n_samples))\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([4.3, 5.1, 5.8, 6.5, 7.9])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eoIqlgXPo_0V"
      },
      "source": [
        "This feature corresponds to the sepal length in cm. Once the quantile transform is applied, those landmarks approach closely the percentiles previously defined"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ws1wquUQpNhd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "ac51a2ad-4a77-44c2-beae-290e2a9f758c"
      },
      "source": [
        "np.percentile(X_train_trans[:, 0], [0, 25, 50, 75, 100])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.        , 0.23873874, 0.50900901, 0.74324324, 1.        ])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YPPJtbfeqwSH"
      },
      "source": [
        "Some more applications [here](https://machinelearningmastery.com/quantile-transforms-for-machine-learning/)\n",
        "\n",
        "**<ins>Mapping to a Gaussian Distribution</ins>**\n",
        "\n",
        "Many machine learning algorithms prefer or perform better when numerical input variables and even output variables in the case of regression have a Gaussian distribution. Power transforms are a family of parametric, monotonic transforms that aim to map data from any distribution to as close to a Gaussian distribution as possible, in order to stabilize variance and minimize skewness. \n",
        "\n",
        "[*PowerTransformer*](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PowerTransformer.html#sklearn.preprocessing.PowerTransformer) provides two transformations, the *Yeo-Johnson* transform: \n",
        "\n",
        "$\\begin{split}x_i^{(\\lambda)} =\n",
        "\\begin{cases}\n",
        " [(x_i + 1)^\\lambda - 1] / \\lambda & \\text{if } \\lambda \\neq 0, x_i \\geq 0, \\\\[8pt]\n",
        "\\ln{(x_i + 1)} & \\text{if } \\lambda = 0, x_i \\geq 0 \\\\[8pt]\n",
        "-[(-x_i + 1)^{2 - \\lambda} - 1] / (2 - \\lambda) & \\text{if } \\lambda \\neq 2, x_i < 0, \\\\[8pt]\n",
        " - \\ln (- x_i + 1) & \\text{if } \\lambda = 2, x_i < 0\n",
        "\\end{cases}\\end{split}$\n",
        "\n",
        "\n",
        "and the *Box-Cox* transform:\n",
        "\n",
        "$\\begin{split}x_i^{(\\lambda)} =\n",
        "\\begin{cases}\n",
        "\\dfrac{x_i^\\lambda - 1}{\\lambda} & \\text{if } \\lambda \\neq 0, \\\\[8pt]\n",
        "\\ln{(x_i)} & \\text{if } \\lambda = 0,\n",
        "\\end{cases}\\end{split}$\n",
        "\n",
        "Box-Cox can only be applied to strictly positive data. In both methods, the transformation is parametrized by $\\lambda$, which is determined trough maximum-likelihood estimation. Here an example of using Box-Cox to map samples drawn from a lognormal distribution to a normal distribution: \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kz5m37muTwzv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "70f5207c-9d5b-401f-dfee-daf471e0136c"
      },
      "source": [
        "pt = preprocessing.PowerTransformer(method='box-cox', standardize=False)\n",
        "X_lognormal = np.random.RandomState(616).lognormal(size=(3, 3))\n",
        "pt.fit_transform(X_lognormal)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 0.49024349,  0.17881995, -0.1563781 ],\n",
              "       [-0.05102892,  0.58863195, -0.57612415],\n",
              "       [ 0.69420009, -0.84857822,  0.10051454]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a0QLFYmFwRLl"
      },
      "source": [
        "(Some more applications [here](https://machinelearningmastery.com/power-transforms-with-scikit-learn/))\n",
        "\n",
        "Below some examples of the two transforms applied to various probability distributions, *any comment*? \n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/MachineLearningJournalClub/HowToTackleAMLCompetition/master/img/Lecture1/1.3.png\" width=\"400\" height=\"800\">\n",
        "\n",
        "<a id='section1.3.4'></a>\n",
        "### Normalization\n",
        "\n",
        "As scientists, we feel much more comfortable with Vector Space Models. *Normalization* is the process of scaling individual samples to have unit norm. This process might be useful if we plan to use a dot-product or some kernel to quantify similarities of pairs of samples.\n",
        "\n",
        "The function [*normalize*](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.normalize.html#sklearn.preprocessing.normalize) provides a quick and easy way to perform this operation on a single array, using [L1 or L2 norms](https://medium.com/@montjoile/l0-norm-l1-norm-l2-norm-l-infinity-norm-7a7d18a4f40c): \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5TsXpkJd2ung",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "dd38e0f8-d18f-4539-ad33-96788ce270d5"
      },
      "source": [
        "X = [[ 1., -1.,  2.],\n",
        "     [ 2.,  0.,  0.],\n",
        "     [ 0.,  1., -1.]]\n",
        "X_normalized = preprocessing.normalize(X, norm='l2')\n",
        "\n",
        "X_normalized"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 0.40824829, -0.40824829,  0.81649658],\n",
              "       [ 1.        ,  0.        ,  0.        ],\n",
              "       [ 0.        ,  0.70710678, -0.70710678]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vnLcKPZV7aQP"
      },
      "source": [
        "The *preprocessing* module provides a utility class *Normalizer* that implements the same operation using the *Transformer* API. This class is suitable for *sklearn.pipeline.Pipeline*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f2bz0iIqxJc7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "9e2d5127-9e0d-466a-c9dd-c6b4808f2312"
      },
      "source": [
        "normalizer = preprocessing.Normalizer().fit(X)  # fit does nothing\n",
        "normalizer.transform(X)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 0.40824829, -0.40824829,  0.81649658],\n",
              "       [ 1.        ,  0.        ,  0.        ],\n",
              "       [ 0.        ,  0.70710678, -0.70710678]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qLUCWMXZxJaE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "367e1b6c-0c43-44c0-bc85-f2cdc2a28f9c"
      },
      "source": [
        "normalizer.transform([[-1.,1.,0.]])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[-0.70710678,  0.70710678,  0.        ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FGouGHyT8dFn"
      },
      "source": [
        "<a id='section1.3.5'></a>\n",
        "### Encoding Categorical Features\n",
        "\n",
        "In many cases, features are not continous values but categorical. E.g. a person could have some features: ``[\"from Italy\", \"from France\", \"from Germany\"]``, ``[\"play sports\", \"doesn't play sports\"]``, ``[\"uses Firefox\", \"uses Opera\", \"uses Chrome\", \"uses Safari\", \"uses Internet Explorer\"]``. \n",
        "\n",
        "Such features can be efficiently coded as integers, for instance ``[\"from France\", \"play sports\", \"uses Chrome\"]`` could be ``[1,0,2]``. \n",
        "\n",
        "To convert categorical features to such integer codes, we can use the [*OrdinalEncoder*](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html#sklearn.preprocessing.OrdinalEncoder). In this way, we transform each feature to one new feature of integers *(0* to *n_categories-1)*:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dSdeho3-ET9T",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "77e9374b-dffd-4af8-b0c5-5631542fdddd"
      },
      "source": [
        "enc = preprocessing.OrdinalEncoder()\n",
        "X = [['from Italy', 'play sports', 'uses Safari'], ['from Germany', \"doesn't play sports\", 'uses Firefox']]\n",
        "enc.fit(X)\n",
        "\n",
        "print(enc.transform([['from Germany', \"doesn't play sports\", 'uses Firefox']]))\n",
        "print(enc.transform([['from Italy', 'play sports', 'uses Safari']]))\n",
        "\n",
        "print(enc.transform([['from Germany', 'play sports', 'uses Safari']]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0. 0. 0.]]\n",
            "[[1. 1. 1.]]\n",
            "[[0. 1. 1.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ArLv6bUPHUxq"
      },
      "source": [
        "Some scikit-learn estimators expect continuous input, and would interpret categories as ordered, which is usually not desired. \n",
        "\n",
        "There's another way to convert categorical features to features that can be used with scikit-learn estimators: *one-hot encoding*. It can be obtained with the [*OneHotEncoder*](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder), which transforms each categorical feature with ``n_categories``possible values into ``n_categories`` binary features, with one of them 1, and all others 0. \n",
        "\n",
        "Let's continue with the example above:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gJlDmOCH_BQn",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "9caa1028-3637-4b24-cc4a-c568dfac557b"
      },
      "source": [
        "enc = preprocessing.OneHotEncoder()\n",
        "X = [['from Italy', 'play sports', 'uses Safari'], ['from Germany', \"doesn't play sports\", 'uses Firefox']]\n",
        "enc.fit(X)\n",
        "\n",
        "print(enc.transform([['from Germany', \"doesn't play sports\", 'uses Firefox']]).toarray())\n",
        "print(enc.transform([['from Italy', 'play sports', 'uses Safari']]).toarray())\n",
        "\n",
        "print(enc.transform([['from Germany', 'play sports', 'uses Safari']]).toarray())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[1. 0. 1. 0. 1. 0.]]\n",
            "[[0. 1. 0. 1. 0. 1.]]\n",
            "[[1. 0. 0. 1. 0. 1.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6KvO-m9VCvFn"
      },
      "source": [
        "The values each feature can take is inferred automatically from the dataset and can be found in the ``categories_``attribute: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "biELVS_ICunJ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "49335fe7-5d94-41d6-b599-6e0ada3014ee"
      },
      "source": [
        "enc.categories_"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[array(['from Germany', 'from Italy'], dtype=object),\n",
              " array([\"doesn't play sports\", 'play sports'], dtype=object),\n",
              " array(['uses Firefox', 'uses Safari'], dtype=object)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cw7c41QfwqPq"
      },
      "source": [
        "\n",
        "---\n",
        "---\n",
        "\n",
        "<a id='section2'></a>\n",
        "# **Feature Engineering**\n",
        "\n",
        "The real deal is that nobody explicitly tells you what **feature engineering** is, in some way, you are expected to understand for yourself what are good features.\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/MachineLearningJournalClub/HowToTackleAMLCompetition/master/img/Lecture1/1.2.jpg\" width=\"800\" height=\"250\">\n",
        "\n",
        "> Feature engineering is another topic which doesn’t seem to merit any review papers or books, or even chapters in books, but it is absolutely vital to ML success. […] Much of the success of machine learning is actually success in engineering features that a learner can understand. \n",
        "\n",
        ">*(Scott Locklin, in “Neglected machine learning ideas”)* \n",
        "\n",
        "Let's try to figure out what feature engineering is. \n",
        "\n",
        "In solving such problems, our goal is to get the best possible result from a model. In order to achieve that, we need to extract useful information and get the most from what we have. On one side, this includes getting the best possible result from the algorithms we are employing. On the other side, it also involves getting the most out of the available data. \n",
        "\n",
        "*How do we get the most out of our data for predictive modeling?* \n",
        "\n",
        "Feature engineering tries to find an answer to this question. \n",
        "\n",
        "> Actually, the success of all Machine Learning algorithms depends on how you present the data. \n",
        "\n",
        "> (*Mohammad Pezeshki, answer to “What are some general tips on feature selection and engineering that every data scientist should know?\"*)\n",
        "\n",
        "\n",
        "\n",
        "<a id='section2.0'></a>\n",
        "## **Feature Importance**\n",
        "Feature importance refers to a bunch of techniques that assign a score to input features based on how useful they are at predicting a target variable. These scores play an important role in predictive modeling, they usually provide useful insights into the dataset and the basis for dimensionality reduction and feature selection. \n",
        "\n",
        "Feature importance scores can be calculated both for regression and classification problems. \n",
        "\n",
        "These scores can be used in a range of situations, such as:\n",
        "* *Better understanding the data*: the relative scorse can highlight which features may be most relevant to the target, and on the other side, which are least relevant. This could be a useful notion for a domain expert and could be used as a basis for gathering more or different data. \n",
        "\n",
        "\n",
        "* *Better understanding a model*: inspecting the importance score provides insights into the specific model we're using and which features are the most important to the model when elaborating a prediction. \n",
        "\n",
        "\n",
        "* *Reducing the number of input features*: we can use the importance scores to select those features to delete (lowest score) and those to keep (highest scores). \n",
        "\n",
        "Now let's jot down a few lines of code in order to grasp this topic in a better way.In order to explore feature importance scores, we'll import a few test datasets directly from sklearn. \n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xFmSLA3SRUhn"
      },
      "source": [
        "\n",
        "**<ins>Classification Dataset</ins>**\n",
        "\n",
        "Easy peasy, we can use the [*make_classification()*](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html) function to create a test binary classification dataset. \n",
        "\n",
        "We can specify the number of samples and the number of features, some of them are going to be informative and the remaining redundant. (*Tip*: you should fix the *random seed*, in this way you'll get a reproducible result)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CchIfeKMSO_g",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "b65f05b2-a86e-48e0-fc36-5c33a1553f12"
      },
      "source": [
        "# classification dataset\n",
        "from sklearn.datasets import make_classification\n",
        "# define dataset\n",
        "X_clf, y_clf = make_classification(n_samples=1000, n_features=8, n_informative=3, n_redundant=5, random_state=1)\n",
        "# summarize the dataset\n",
        "print(X_clf.shape, y_clf.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(1000, 8) (1000,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p9ZoNTa1SOVV"
      },
      "source": [
        "**<ins>Regression Dataset</ins>**\n",
        "\n",
        "In a parallel fashion, we'll use the [*make_regression()*](https://machinelearningmastery.com/calculate-feature-importance-with-python/#:~:text=Feature%20importance%20refers%20to%20techniques,at%20predicting%20a%20target%20variable.) function to create a regression dataset. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JlC9R4MOTkYD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "49916a77-e7b9-4add-e99f-4ae8bdc48857"
      },
      "source": [
        "# test regression dataset\n",
        "from sklearn.datasets import make_regression\n",
        "# define dataset\n",
        "X_reg, y_reg = make_regression(n_samples=1000, n_features=8, n_informative=3, random_state=1)\n",
        "# summarize the dataset\n",
        "print(X_reg.shape, y_reg.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(1000, 8) (1000,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nZ9hLi-7VZim"
      },
      "source": [
        "<a id='section2.0.1'></a>\n",
        "### Coefficients as Feature Importance \n",
        "\n",
        "When we think about linear machine learning algorithms, we always fit a model where the prediction is the weighted sum of the input values (e.g. linear regression, logistic regression, ridge regression etc..) \n",
        "\n",
        "These coefficients can be used directly as naive feature importance scores. Firstly we'll fit a model on the dataset to find the coefficients, then summarize the importance scores for each input feature and create a bar chart to get an idea of the relative importance. \n",
        "\n",
        "**<ins>Linear Regression Feature Importance</ins>**\n",
        "\n",
        "It's time to fit a [*LinearRegression()*](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) model on the regression dataset and get the *coef_* property that conatins the coefficients. The only assumption is that the input variables have the same scale or have been scaled prior to fitting the model. \n",
        "\n",
        "This same approach can be used with regularized linear models, such as Ridge and ElasticNet. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ko_JB14wXLrC",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 512
        },
        "outputId": "e9aefb38-9baf-4d4d-d193-e140fa5d1ef1"
      },
      "source": [
        "# linear regression feature importance\n",
        "from sklearn.linear_model import LinearRegression\n",
        "from matplotlib import pyplot as plt \n",
        "# define the model\n",
        "model = LinearRegression()\n",
        "# fit the model\n",
        "model.fit(X_reg, y_reg)\n",
        "# get importance\n",
        "importance = model.coef_\n",
        "# summarize feature importance\n",
        "for i,v in enumerate(importance):\n",
        "\tprint('Feature: %0d, Score: %.5f' % (i,v))\n",
        "# plot feature importance\n",
        "plt.figure(figsize = (10,6))\n",
        "plt.bar([x for x in range(len(importance))], importance)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Feature: 0, Score: -0.00000\n",
            "Feature: 1, Score: 41.28219\n",
            "Feature: 2, Score: 0.00000\n",
            "Feature: 3, Score: 41.81266\n",
            "Feature: 4, Score: 45.15258\n",
            "Feature: 5, Score: 0.00000\n",
            "Feature: 6, Score: 0.00000\n",
            "Feature: 7, Score: 0.00000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAAFlCAYAAAAkvdbGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAO5klEQVR4nO3dcchvBX3H8c83r1Grbda8iHhjVygaMljGxRWOGDqHYZR/xEi2kCG4P9owGjTrnxHsD/un2h9jINp2x1rWrDAqtokZFWzW1Wyl1jIxUqx7o6TcHwvruz/uaVjo7vO993me8/M+rxc8PL9zfr/H8+V4r76fc87v/Kq7AwDA1j1n7QEAAJ5tBBQAwJCAAgAYElAAAEMCCgBgSEABAAzt282NnX322X3w4MHd3CQAwEm5++67v9fd+5/uuV0NqIMHD+bIkSO7uUkAgJNSVd96puecwgMAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIChfWsPALBTDl7/ybVH2FYP33DF2iMAC0egAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADO1bewBgZxy8/pNrj7CtHr7hirVHAPg/jkABAAwJKACAIQEFADDkGqjTkGtfAGBnOQIFADAkoAAAhgQUAMCQgAIAGNpyQFXVGVX1par6xLJ8flXdVVUPVtWHquq5OzcmAMDmmByBui7JA09ZfneS93b3S5P8IMk12zkYAMCm2lJAVdWBJFckuWlZriSXJLl1ecnhJFfuxIAAAJtmq0eg3pfk7Ul+uiz/WpLHu/vJZfmRJOdt82wAABvphAFVVa9LcrS77z6ZDVTVtVV1pKqOHDt27GT+EQAAG2UrR6AuTvL6qno4yS05furur5OcVVU/u5P5gSSPPt0Pd/eN3X2ouw/t379/G0YGAFjXCQOqu9/R3Qe6+2CSNyX5dHf/YZI7k7xxednVSW7bsSkBADbIqdwH6i+SvK2qHszxa6Ju3p6RAAA22+jDhLv7M0k+szx+KMlF2z8SAMBmcydyAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMDQCQOqqp5XVV+oqi9X1X1V9a5l/flVdVdVPVhVH6qq5+78uAAA69vKEaj/SXJJd/9WklckubyqXpXk3Une290vTfKDJNfs3JgAAJvjhAHVxz2xLJ65fHWSS5Lcuqw/nOTKHZkQAGDDbOkaqKo6o6ruTXI0ye1Jvpnk8e5+cnnJI0nOe4afvbaqjlTVkWPHjm3HzAAAq9pSQHX3T7r7FUkOJLkoyW9sdQPdfWN3H+ruQ/v37z/JMQEANsfoXXjd/XiSO5O8OslZVbVveepAkke3eTYAgI20lXfh7a+qs5bHz09yWZIHcjyk3ri87Ookt+3UkAAAm2TfiV+Sc5Mcrqozcjy4Ptzdn6iq+5PcUlV/leRLSW7ewTkBADbGCQOqu/8zyYVPs/6hHL8eCgBgT3EncgCAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADA0AkDqqpeUlV3VtX9VXVfVV23rH9xVd1eVd9Yvr9o58cFAFjfVo5APZnkz7v7giSvSvKWqrogyfVJ7ujulyW5Y1kGADjtnTCguvux7r5nefyjJA8kOS/JG5IcXl52OMmVOzUkAMAmGV0DVVUHk1yY5K4k53T3Y8tT30lyzjP8zLVVdaSqjhw7duwURgUA2AxbDqiqemGSjyR5a3f/8KnPdXcn6af7ue6+sbsPdfeh/fv3n9KwAACbYEsBVVVn5ng8faC7P7qs/m5Vnbs8f26SozszIgDAZtnKu/Aqyc1JHuju9zzlqY8nuXp5fHWS27Z/PACAzbNvC6+5OMmbk3ylqu5d1r0zyQ1JPlxV1yT5VpI/2JkRAQA2ywkDqrs/n6Se4elLt3ccAIDN507kAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIChEwZUVb2/qo5W1Vefsu7FVXV7VX1j+f6inR0TAGBzbOUI1N8nufwX1l2f5I7uflmSO5ZlAIA94YQB1d2fTfL9X1j9hiSHl8eHk1y5zXMBAGysk70G6pzufmx5/J0k5zzTC6vq2qo6UlVHjh07dpKbAwDYHKd8EXl3d5L+f56/sbsPdfeh/fv3n+rmAABWd7IB9d2qOjdJlu9Ht28kAIDNdrIB9fEkVy+Pr05y2/aMAwCw+bZyG4MPJvn3JC+vqkeq6pokNyS5rKq+keT3lmUAgD1h34le0N1XPcNTl27zLAAAzwruRA4AMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMDQKQVUVV1eVV+vqger6vrtGgoAYJOddEBV1RlJ/ibJa5NckOSqqrpguwYDANhUp3IE6qIkD3b3Q9394yS3JHnD9owFALC5TiWgzkvy7acsP7KsAwA4re3b6Q1U1bVJrl0Wn6iqr+/0NnfJ2Um+t/YQK9uVfVDv3uktnBJ/Dvw5SOyDxN+FxD5ITr998OvP9MSpBNSjSV7ylOUDy7qf0903JrnxFLazkarqSHcfWnuONdkH9kFiHyT2QWIfJPZBsrf2wamcwvtikpdV1flV9dwkb0ry8e0ZCwBgc530EajufrKq/jTJvyY5I8n7u/u+bZsMAGBDndI1UN39qSSf2qZZnm1Ou9OSJ8E+sA8S+yCxDxL7ILEPkj20D6q7154BAOBZxUe5AAAMCaiTsNc/wqaq3l9VR6vqq2vPspaqeklV3VlV91fVfVV13doz7baqel5VfaGqvrzsg3etPdNaquqMqvpSVX1i7VnWUFUPV9VXqureqjqy9jxrqKqzqurWqvpaVT1QVa9ee6bdVFUvX/79/+zrh1X11rXn2klO4Q0tH2HzX0kuy/Gbh34xyVXdff+qg+2iqnpNkieS/EN3/+ba86yhqs5Ncm5331NVv5zk7iRX7rE/B5XkBd39RFWdmeTzSa7r7v9YebRdV1VvS3Ioya909+vWnme3VdXDSQ519+l0/5+Rqjqc5HPdfdPyzvRf6u7H155rDcv/Jx9N8tvd/a2159kpjkDN7fmPsOnuzyb5/tpzrKm7H+vue5bHP0ryQPbYnfj7uCeWxTOXrz33G1lVHUhyRZKb1p6FdVTVryZ5TZKbk6S7f7xX42lxaZJvns7xlAiok+EjbPg5VXUwyYVJ7lp3kt23nLq6N8nRJLd3957bB0nel+TtSX669iAr6iT/VlV3L58+sdecn+RYkr9bTuXeVFUvWHuoFb0pyQfXHmKnCSg4BVX1wiQfSfLW7v7h2vPstu7+SXe/Isc/ieCiqtpTp3Sr6nVJjnb33WvPsrLf6e5XJnltkrcsp/n3kn1JXpnkb7v7wiT/nWTPXR+bJMvpy9cn+ee1Z9lpAmpuSx9hw+lvue7nI0k+0N0fXXueNS2nK+5Mcvnas+yyi5O8frkG6JYkl1TVP6470u7r7keX70eTfCzHL3XYSx5J8shTjsDemuNBtRe9Nsk93f3dtQfZaQJqzkfY8LMLqG9O8kB3v2ftedZQVfur6qzl8fNz/I0VX1t3qt3V3e/o7gPdfTDH/1vw6e7+o5XH2lVV9YLljRRZTlv9fpI99Q7d7v5Okm9X1cuXVZcm2TNvKPkFV2UPnL5LTvFO5HuRj7BJquqDSX43ydlV9UiSv+zum9edatddnOTNSb6yXAOUJO9c7s6/V5yb5PDyjpvnJPlwd+/Jt/Hvceck+djx3ymyL8k/dfe/rDvSKv4syQeWX6wfSvLHK8+z65aAvizJn6w9y25wGwMAgCGn8AAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAw9L8+S7nXKpCqOwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 720x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "di-9BNGJY7mU"
      },
      "source": [
        "**<ins>Logistic Regression Feature Importance</ins>**\n",
        "\n",
        "In a similar fashion, we can do the same to fit a [*LogisticRegression()*](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) model. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LS4mefGrZgei",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 403
        },
        "outputId": "df416700-6170-4623-8173-bed076d747f4"
      },
      "source": [
        "# logistic regression for feature importance\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from matplotlib import pyplot as plt\n",
        "# define the model\n",
        "model = LogisticRegression()\n",
        "# fit the model\n",
        "model.fit(X_clf, y_clf)\n",
        "# get importance\n",
        "importance = model.coef_[0]\n",
        "# summarize feature importance\n",
        "for i,v in enumerate(importance):\n",
        "\tprint('Feature: %0d, Score: %.5f' % (i,v))\n",
        "# plot feature importance\n",
        "plt.bar([x for x in range(len(importance))], importance)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Feature: 0, Score: -1.08328\n",
            "Feature: 1, Score: 0.35669\n",
            "Feature: 2, Score: -0.13472\n",
            "Feature: 3, Score: 0.58331\n",
            "Feature: 4, Score: -0.40560\n",
            "Feature: 5, Score: -0.38912\n",
            "Feature: 6, Score: 0.31175\n",
            "Feature: 7, Score: -0.88263\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAARf0lEQVR4nO3df6zddX3H8efLYnHTKWAb7Fq0GBsV51LcFWNc3AZFyzCUbaiQuFWD6bLIpjE6ykg0Y5LULRn+45Y1gFZlguII3ahDRJxbHMpFKz9Faq2jFe0V/DGHwoD3/rjf4uHu3vbe+z3tufXzfCQn5/v9fD+f7/fNpTmv8/l+v+ecVBWSpHY9ZdQFSJJGyyCQpMYZBJLUOINAkhpnEEhS444YdQHzsWTJklq5cuWoy5Ckw8qtt976/apaOrX9sAyClStXMj4+PuoyJOmwkuTb07V7akiSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUuMPyA2XSoJUbrxvZsXdtOn1kx5aGxRmBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJatxQgiDJ2iT3JNmRZOM029+cZCLJ9u7x1oFt65Pc2z3WD6MeSdLs9f4cQZJFwAeBU4HdwC1JtlbVXVO6XlVV500ZewzwXmAMKODWbuwP+tYlSZqdYcwITgJ2VNXOqnoEuBJYN8uxrwVuqKoHuxf/G4C1Q6hJkjRLwwiC5cB9A+u7u7ap/iDJbUmuTnLcHMeSZEOS8STjExMTQyhbkgSH7mLxPwMrq+rXmXzXv2WuO6iqzVU1VlVjS5f+v99eliTN0zCCYA9w3MD6iq7tCVX1QFU93K1eCvzGbMdKkg6uYQTBLcCqJMcnWQycDWwd7JBk2cDqGcDd3fL1wGuSHJ3kaOA1XZsk6RDpfddQVT2a5DwmX8AXAZdX1Z1JLgLGq2or8GdJzgAeBR4E3tyNfTDJXzEZJgAXVdWDfWuSJM3eUL6Guqq2AdumtL1nYPkC4IIZxl4OXD6MOiRJc+cniyWpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYN5buGNBwrN143smPv2nT6yI4tabScEUhS4wwCSWqcQSBJjTMIJKlxXiyWtOB448Sh5YxAkhpnEEhS4wwCSWqcQSBJjRtKECRZm+SeJDuSbJxm+zuT3JXktiQ3JnnewLbHkmzvHluHUY8kafZ63zWUZBHwQeBUYDdwS5KtVXXXQLevAmNV9VCSPwH+Gnhjt+2nVbW6bx2SpPkZxozgJGBHVe2sqkeAK4F1gx2q6qaqeqhbvRlYMYTjSpKGYBhBsBy4b2B9d9c2k3OBTw+sPy3JeJKbk5w506AkG7p+4xMTE/0qliQ94ZB+oCzJm4Ax4LcGmp9XVXuSPB/4XJLbq+qbU8dW1WZgM8DY2FgdkoIlqQHDmBHsAY4bWF/RtT1JkjXAhcAZVfXwvvaq2tM97wQ+D5w4hJokSbM0jCC4BViV5Pgki4GzgSfd/ZPkROAfmAyBvQPtRyc5slteArwKGLzILEk6yHqfGqqqR5OcB1wPLAIur6o7k1wEjFfVVuBvgGcAn0wC8F9VdQbwYuAfkjzOZChtmnK3kSTpIBvKNYKq2gZsm9L2noHlNTOM+yLw0mHUIEmaHz9ZLEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxg0lCJKsTXJPkh1JNk6z/cgkV3Xbv5Rk5cC2C7r2e5K8dhj1SJJmr3cQJFkEfBA4DTgBOCfJCVO6nQv8oKpeAFwCvL8bewJwNvASYC3wd93+JEmHyDBmBCcBO6pqZ1U9AlwJrJvSZx2wpVu+GjglSbr2K6vq4ar6FrCj258k6RA5Ygj7WA7cN7C+G3jFTH2q6tEkPwKe3bXfPGXs8ukOkmQDsAHguc997ryLXbnxunmP7WvXptN7bR8l/27zs5D/bgu5Nv+fTu9g/V0Om4vFVbW5qsaqamzp0qWjLkeSfmEMIwj2AMcNrK/o2qbtk+QI4FnAA7McK0k6iIYRBLcAq5Icn2Qxkxd/t07psxVY3y2fBXyuqqprP7u7q+h4YBXw5SHUJEmapd7XCLpz/ucB1wOLgMur6s4kFwHjVbUVuAz4aJIdwINMhgVdv08AdwGPAm+rqsf61iRJmr1hXCymqrYB26a0vWdg+WfA62cYezFw8TDqkCTN3WFzsViSdHAYBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1LihfMWEfvEt5O+Hl9SPMwJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjesVBEmOSXJDknu756On6bM6yX8muTPJbUneOLDtw0m+lWR791jdpx5J0tz1nRFsBG6sqlXAjd36VA8Bf1RVLwHWAh9IctTA9ndX1erusb1nPZKkOeobBOuALd3yFuDMqR2q6htVdW+3/B1gL7C053ElSUPSNwiOrar7u+XvAsfur3OSk4DFwDcHmi/uThldkuTI/YzdkGQ8yfjExETPsiVJ+xwwCJJ8Nskd0zzWDfarqgJqP/tZBnwUeEtVPd41XwC8CHg5cAxw/kzjq2pzVY1V1djSpU4oJGlYDvh7BFW1ZqZtSb6XZFlV3d+90O+dod8zgeuAC6vq5oF975tNPJzkQ8C75lS9JKm3vqeGtgLru+X1wLVTOyRZDFwDfKSqrp6ybVn3HCavL9zRsx5J0hz1DYJNwKlJ7gXWdOskGUtyadfnDcCrgTdPc5voFUluB24HlgDv61mPJGmOev1UZVU9AJwyTfs48NZu+WPAx2YYf3Kf40uS+vOTxZLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuN6fdeQpMPXrk2nj7oELRDOCCSpcc4IpIPId906HDgjkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcb2CIMkxSW5Icm/3fPQM/R5Lsr17bB1oPz7Jl5LsSHJVksV96pEkzV3fGcFG4MaqWgXc2K1P56dVtbp7nDHQ/n7gkqp6AfAD4Nye9UiS5qhvEKwDtnTLW4AzZzswSYCTgavnM16SNBx9g+DYqrq/W/4ucOwM/Z6WZDzJzUn2vdg/G/hhVT3are8Gls90oCQbun2MT0xM9CxbkrTPAb9rKMlngedMs+nCwZWqqiQ1w26eV1V7kjwf+FyS24EfzaXQqtoMbAYYGxub6TiSpDk6YBBU1ZqZtiX5XpJlVXV/kmXA3hn2sad73pnk88CJwKeAo5Ic0c0KVgB75vHfIEnqoe+poa3A+m55PXDt1A5Jjk5yZLe8BHgVcFdVFXATcNb+xkuSDq6+QbAJODXJvcCabp0kY0ku7fq8GBhP8jUmX/g3VdVd3bbzgXcm2cHkNYPLetYjSZqjXr9HUFUPAKdM0z4OvLVb/iLw0hnG7wRO6lODJKkfP1ksSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTG9QqCJMckuSHJvd3z0dP0+Z0k2wceP0tyZrftw0m+NbBtdZ96JElz13dGsBG4sapWATd2609SVTdV1eqqWg2cDDwEfGagy7v3ba+q7T3rkSTNUd8gWAds6Za3AGceoP9ZwKer6qGex5UkDUnfIDi2qu7vlr8LHHuA/mcDH5/SdnGS25JckuTImQYm2ZBkPMn4xMREj5IlSYMOGARJPpvkjmke6wb7VVUBtZ/9LANeClw/0HwB8CLg5cAxwPkzja+qzVU1VlVjS5cuPVDZkqRZOuJAHapqzUzbknwvybKqur97od+7n129Abimqv53YN/7ZhMPJ/kQ8K5Z1i1JGpK+p4a2Auu75fXAtfvpew5TTgt14UGSMHl94Y6e9UiS5qhvEGwCTk1yL7CmWyfJWJJL93VKshI4Dvi3KeOvSHI7cDuwBHhfz3okSXN0wFND+1NVDwCnTNM+Drx1YH0XsHyafif3Ob4kqT8/WSxJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcb2+YkKSWrNr0+mjLmHonBFIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJalyvIEjy+iR3Jnk8ydh++q1Nck+SHUk2DrQfn+RLXftVSRb3qUeSNHd9ZwR3AL8PfGGmDkkWAR8ETgNOAM5JckK3+f3AJVX1AuAHwLk965EkzVGvIKiqu6vqngN0OwnYUVU7q+oR4EpgXZIAJwNXd/22AGf2qUeSNHeH4hrBcuC+gfXdXduzgR9W1aNT2qeVZEOS8STjExMTB61YSWrNAb+GOslngedMs+nCqrp2+CVNr6o2A5sBxsbG6lAdV5J+0R0wCKpqTc9j7AGOG1hf0bU9AByV5IhuVrCvXZJ0CB2KU0O3AKu6O4QWA2cDW6uqgJuAs7p+64FDNsOQJE3qe/vo7yXZDbwSuC7J9V37rybZBtC92z8PuB64G/hEVd3Z7eJ84J1JdjB5zeCyPvVIkuau109VVtU1wDXTtH8H+N2B9W3Atmn67WTyriJJ0oj4yWJJapxBIEmNMwgkqXEGgSQ1rtfF4sPRrk2nj7oESVpQnBFIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjMvn7MIeXJBPAt0d0+CXA90d07AOxtvmxtvmxtvkZZW3Pq6qlUxsPyyAYpSTjVTU26jqmY23zY23zY23zsxBr89SQJDXOIJCkxhkEc7d51AXsh7XNj7XNj7XNz4KrzWsEktQ4ZwSS1DiDQJIaZxDMUpK1Se5JsiPJxlHXMyjJ5Un2Jrlj1LUMSnJckpuS3JXkziRvH3VN+yR5WpIvJ/laV9tfjrqmqZIsSvLVJP8y6loGJdmV5PYk25OMj7qeQUmOSnJ1kq8nuTvJK0ddE0CSF3Z/r32PHyd5x6jr2sdrBLOQZBHwDeBUYDdwC3BOVd010sI6SV4N/AT4SFX92qjr2SfJMmBZVX0lya8AtwJnLoS/W5IAT6+qnyR5KvAfwNur6uYRl/aEJO8ExoBnVtXrRl3PPkl2AWNVteA+sJVkC/DvVXVpksXAL1fVD0dd16Du9WQP8IqqGtUHY5/EGcHsnATsqKqdVfUIcCWwbsQ1PaGqvgA8OOo6pqqq+6vqK93yfwN3A8tHW9WkmvSTbvWp3WPBvCtKsgI4Hbh01LUcLpI8C3g1cBlAVT2y0EKgcwrwzYUSAmAQzNZy4L6B9d0skBe0w0WSlcCJwJdGW8nPdadetgN7gRuqasHUBnwA+HPg8VEXMo0CPpPk1iQbRl3MgOOBCeBD3Sm1S5M8fdRFTeNs4OOjLmKQQaCDLskzgE8B76iqH4+6nn2q6rGqWg2sAE5KsiBOqyV5HbC3qm4ddS0z+M2qehlwGvC27tTkQnAE8DLg76vqROB/gIV2PW8xcAbwyVHXMsggmJ09wHED6yu6Nh1Ad/79U8AVVfVPo65nOt3pg5uAtaOupfMq4IzuXPyVwMlJPjbakn6uqvZ0z3uBa5g8dboQ7AZ2D8zsrmYyGBaS04CvVNX3Rl3IIINgdm4BViU5vkv0s4GtI65pwesuyF4G3F1VfzvqegYlWZrkqG75l5i8EeDro61qUlVdUFUrqmolk//WPldVbxpxWQAkeXp34Z/utMtrgAVxt1pVfRe4L8kLu6ZTgJHfmDDFOSyw00IwOZXSAVTVo0nOA64HFgGXV9WdIy7rCUk+Dvw2sCTJbuC9VXXZaKsCJt/Z/iFwe3cuHuAvqmrbCGvaZxmwpbuD4ynAJ6pqQd2muUAdC1wzmfEcAfxjVf3raEt6kj8FrujesO0E3jLiep7QBeepwB+PupapvH1UkhrnqSFJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhr3f8N7EGVePpd1AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b32qQh04axsK"
      },
      "source": [
        "Recall that this is a classification problem with classes 0 and 1 (binary). Notice that the coefficients are both positive and negative, positive scores indicate a feature that predicts class 1 while negative scores indicate a feature that predicts class 0. \n",
        "\n",
        "*Why can't we analyze a regression problem with Logistic Regression?* (A pretty naive question, try to answer tho)\n",
        "\n",
        "<a id='section2.0.2'></a>\n",
        "### Decision Tree Feature Importance\n",
        "\n",
        "Decision Tree algorithms like **C**lassification **A**nd **R**egression **T**rees (**CART**) offer importance scores based on the reduction in the criterion used to select split points, like Gini or Entropy. This approach can be also used for ensembles of decision trees, such as Random Forest and Gradient Boositng algorithms. \n",
        "\n",
        "We can directly use the CART algorithm for feature importance implemented in Scikit-Learn as the [*DecisionTreeRegressor*](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html) and [*DecisionTreeClassifier*](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html). \n",
        "\n",
        "The model provides a ``feature_importances_``\n",
        " property that tells us the relative importance scores for each feature. \n",
        "\n",
        "**<ins>CART Regression Feature Importance</ins>**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yEI54ffFft7n",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 403
        },
        "outputId": "9e57cec9-d06f-47de-a94b-c7c1548d4840"
      },
      "source": [
        "# decision tree for feature importance on a regression problem\n",
        "from sklearn.tree import DecisionTreeRegressor\n",
        "from matplotlib import pyplot as plt\n",
        "# define the model\n",
        "model = DecisionTreeRegressor()\n",
        "# fit the model\n",
        "model.fit(X_reg, y_reg)\n",
        "# get importance\n",
        "importance = model.feature_importances_\n",
        "# summarize feature importance\n",
        "for i,v in enumerate(importance):\n",
        "\tprint('Feature: %0d, Score: %.5f' % (i,v))\n",
        "# plot feature importance\n",
        "plt.bar([x for x in range(len(importance))], importance)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Feature: 0, Score: 0.00394\n",
            "Feature: 1, Score: 0.27784\n",
            "Feature: 2, Score: 0.00367\n",
            "Feature: 3, Score: 0.33327\n",
            "Feature: 4, Score: 0.37304\n",
            "Feature: 5, Score: 0.00377\n",
            "Feature: 6, Score: 0.00273\n",
            "Feature: 7, Score: 0.00173\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAR+klEQVR4nO3db4xd913n8fcHB6eQ0hLICBXbiV1wEYauEjQ4WhWyiCapoyA7D1rVkboKq0qmq3gpihC4C0pYI6S0SBUP1tBYrVddIJg0AWlEzWazJLBbsWk9+UOzdjCdmFCPVTamztItLUmdfHkwx+HkMs6czNzJnfz6fklXc87vz73fO4k+9/h3zj2TqkKS1K5vmXQBkqTVZdBLUuMMeklqnEEvSY0z6CWpcRdNuoBRl112WW3evHnSZUjS68ojjzzyd1U1tVjfmgv6zZs3Mzs7O+kyJOl1JcnfXKjPpRtJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWrcmvtmrPR6snnfpyf22k/feePEXluvLx7RS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekho3KOiT7EhyIslckn2L9H8gyRNJHk/ymSTbuvbNSb7etT+e5GPjfgOSpFe25Ddjk6wDDgDXAfPA0SQzVXW8N+zuqvpYN34n8FFgR9f3VFVdOd6yJUlDDTmi3w7MVdXJqnoeOAzs6g+oqq/0di8BanwlSpJWYkjQbwBO9fbnu7aXSXJrkqeAjwA/2+vakuSxJH+W5MdXVK0k6VUb28nYqjpQVd8H/CLwy13zl4DLq+oq4Dbg7iRvGp2bZE+S2SSzZ86cGVdJkiSGBf1pYFNvf2PXdiGHgZsAquq5qvpyt/0I8BTwttEJVXWwqqaranpqampo7ZKkAYYE/VFga5ItSdYDu4GZ/oAkW3u7NwJf6NqnupO5JHkrsBU4OY7CJUnDLHnVTVWdS7IXuB9YBxyqqmNJ9gOzVTUD7E1yLfAN4Fnglm76NcD+JN8AXgQ+UFVnV+ONSJIWN+gPj1TVEeDISNvtve0PXmDefcB9KylQkrQyfjNWkhpn0EtS4wx6SWqcQS9JjRt0MlaapM37Pj2x1376zhsn9trSuHhEL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaNyjok+xIciLJXJJ9i/R/IMkTSR5P8pkk23p9H+rmnUjyrnEWL0la2pJBn2QdcAC4AdgG3NwP8s7dVfX2qroS+Ajw0W7uNmA38EPADuA3u+eTJL1GhhzRbwfmqupkVT0PHAZ29QdU1Vd6u5cA1W3vAg5X1XNV9dfAXPd8kqTXyJC/MLUBONXbnweuHh2U5FbgNmA98JO9uQ+PzN2wyNw9wB6Ayy+/fEjdkqSBxnYytqoOVNX3Ab8I/PKrnHuwqqaranpqampcJUmSGBb0p4FNvf2NXduFHAZuWuZcSdKYDQn6o8DWJFuSrGfh5OpMf0CSrb3dG4EvdNszwO4kFyfZAmwFPrfysiVJQy25Rl9V55LsBe4H1gGHqupYkv3AbFXNAHuTXAt8A3gWuKWbeyzJPcBx4Bxwa1W9sErvRZK0iCEnY6mqI8CRkbbbe9sffIW5vwb82nILlCStjN+MlaTGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcYOuo9d4bN736Ym99tN33jix15Y0WR7RS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjRsU9El2JDmRZC7JvkX6b0tyPMnnk/xJkit6fS8kebx7zIyzeEnS0pa8qVmSdcAB4DpgHjiaZKaqjveGPQZMV9XXkvx74CPAe7u+r1fVlWOuW5I00JAj+u3AXFWdrKrngcPArv6Aqnqoqr7W7T4MbBxvmZKk5RoS9BuAU739+a7tQt4P/HFv/w1JZpM8nOSmxSYk2dONmT1z5syAkiRJQ431fvRJ3gdMA/+m13xFVZ1O8lbgwSRPVNVT/XlVdRA4CDA9PV3jrEmSvtkNOaI/DWzq7W/s2l4mybXALwE7q+q58+1Vdbr7eRL4U+CqFdQrSXqVhgT9UWBrki1J1gO7gZddPZPkKuAuFkL+mV77pUku7rYvA94B9E/iSpJW2ZJLN1V1Lsle4H5gHXCoqo4l2Q/MVtUM8OvAG4FPJQH4YlXtBH4QuCvJiyx8qNw5crWOJGmVDVqjr6ojwJGRttt729deYN6fA29fSYGSpJXxm7GS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wYFfZIdSU4kmUuyb5H+25IcT/L5JH+S5Ipe3y1JvtA9bhln8ZKkpS0Z9EnWAQeAG4BtwM1Jto0MewyYrqp/BdwLfKSb+13AHcDVwHbgjiSXjq98SdJShhzRbwfmqupkVT0PHAZ29QdU1UNV9bVu92FgY7f9LuCBqjpbVc8CDwA7xlO6JGmIIUG/ATjV25/v2i7k/cAfv5q5SfYkmU0ye+bMmQElSZKGGuvJ2CTvA6aBX38186rqYFVNV9X01NTUOEuSpG96Q4L+NLCpt7+xa3uZJNcCvwTsrKrnXs1cSdLqGRL0R4GtSbYkWQ/sBmb6A5JcBdzFQsg/0+u6H7g+yaXdSdjruzZJ0mvkoqUGVNW5JHtZCOh1wKGqOpZkPzBbVTMsLNW8EfhUEoAvVtXOqjqb5FdZ+LAA2F9VZ1flnUiSFrVk0ANU1RHgyEjb7b3ta19h7iHg0HILlCStjN+MlaTGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUuEFBn2RHkhNJ5pLsW6T/miSPJjmX5N0jfS8kebx7zIzOlSStriX/ZmySdcAB4DpgHjiaZKaqjveGfRH4aeDnF3mKr1fVlWOoVZK0DEP+OPh2YK6qTgIkOQzsAl4K+qp6uut7cRVqlCStwJClmw3Aqd7+fNc21BuSzCZ5OMlNr6o6SdKKDTmiX6krqup0krcCDyZ5oqqe6g9IsgfYA3D55Ze/BiVJ0jePIUf0p4FNvf2NXdsgVXW6+3kS+FPgqkXGHKyq6aqanpqaGvrUkqQBhgT9UWBrki1J1gO7gUFXzyS5NMnF3fZlwDvore1LklbfkkFfVeeAvcD9wJPAPVV1LMn+JDsBkvxoknngPcBdSY51038QmE3yF8BDwJ0jV+tIklbZoDX6qjoCHBlpu723fZSFJZ3ReX8OvH2FNUqSVsBvxkpS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNGxT0SXYkOZFkLsm+RfqvSfJoknNJ3j3Sd0uSL3SPW8ZVuCRpmCWDPsk64ABwA7ANuDnJtpFhXwR+Grh7ZO53AXcAVwPbgTuSXLrysiVJQw05ot8OzFXVyap6HjgM7OoPqKqnq+rzwIsjc98FPFBVZ6vqWeABYMcY6pYkDTQk6DcAp3r7813bEIPmJtmTZDbJ7JkzZwY+tSRpiDVxMraqDlbVdFVNT01NTbocSWrKkKA/DWzq7W/s2oZYyVxJ0hgMCfqjwNYkW5KsB3YDMwOf/37g+iSXdidhr+/aJEmvkSWDvqrOAXtZCOgngXuq6liS/Ul2AiT50STzwHuAu5Ic6+aeBX6VhQ+Lo8D+rk2S9Bq5aMigqjoCHBlpu723fZSFZZnF5h4CDq2gRknSCqyJk7GSpNVj0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJatygoE+yI8mJJHNJ9i3Sf3GS3+/6P5tkc9e+OcnXkzzePT423vIlSUtZ8o+DJ1kHHACuA+aBo0lmqup4b9j7gWer6vuT7AY+DLy363uqqq4cc92SpIGGHNFvB+aq6mRVPQ8cBnaNjNkFfLLbvhd4Z5KMr0xJ0nINCfoNwKne/nzXtuiYqjoH/D3w3V3fliSPJfmzJD++2Ask2ZNkNsnsmTNnXtUbkCS9stU+Gfsl4PKqugq4Dbg7yZtGB1XVwaqarqrpqampVS5Jkr65DAn608Cm3v7Grm3RMUkuAt4MfLmqnquqLwNU1SPAU8DbVlq0JGm4IUF/FNiaZEuS9cBuYGZkzAxwS7f9buDBqqokU93JXJK8FdgKnBxP6ZKkIZa86qaqziXZC9wPrAMOVdWxJPuB2aqaAT4B/HaSOeAsCx8GANcA+5N8A3gR+EBVnV2NNyJJWtySQQ9QVUeAIyNtt/e2/xF4zyLz7gPuW2GNkqQV8JuxktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMGBX2SHUlOJJlLsm+R/ouT/H7X/9kkm3t9H+raTyR51/hKlyQNseQfB0+yDjgAXAfMA0eTzFTV8d6w9wPPVtX3J9kNfBh4b5JtwG7gh4DvBf5HkrdV1QvjfiPnbd736dV66iU9feeNE3vtlfL31p61/N90LdfWoiWDHtgOzFXVSYAkh4FdQD/odwG/0m3fC/znJOnaD1fVc8BfJ5nrnu9/j6d8SRqvFj+EhgT9BuBUb38euPpCY6rqXJK/B767a394ZO6G0RdIsgfY0+1+NcmJQdWP32XA3y13cj48xkr+JWtbHmtbHmtbnknWdsWFOoYE/aqrqoPAwUnXkWS2qqYnXcdirG15rG15rG151mptQ07GngY29fY3dm2LjklyEfBm4MsD50qSVtGQoD8KbE2yJcl6Fk6uzoyMmQFu6bbfDTxYVdW17+6uytkCbAU+N57SJUlDLLl006257wXuB9YBh6rqWJL9wGxVzQCfAH67O9l6loUPA7px97Bw4vYccOtqXnEzBhNfPnoF1rY81rY81rY8a7K2LBx4S5Ja5TdjJalxBr0kNc6g7yx1m4dJSXIoyTNJ/s+kaxmVZFOSh5IcT3IsyQcnXdN5Sd6Q5HNJ/qKr7T9NuqZRSdYleSzJH026lr4kTyd5IsnjSWYnXU9fku9Mcm+Sv0zyZJJ/PemaAJL8QPf7Ov/4SpKfm3Rd57lGz0u3efgrerd5AG4euc3DRCS5Bvgq8F+r6ocnXU9fkrcAb6mqR5N8B/AIcNMa+b0FuKSqvprkW4HPAB+sqoeXmPqaSXIbMA28qap+atL1nJfkaWC6qpb9xZ/VkuSTwP+qqo93VwF+e1X9v0nX1dflyWng6qr6m0nXAx7Rn/fSbR6q6nng/G0eJq6q/icLVzKtOVX1pap6tNv+/8CTLPLN50moBV/tdr+1e6yZo5okG4EbgY9PupbXiyRvBq5h4So/qur5tRbynXcCT62VkAeD/rzFbvOwJgLr9aK7Y+lVwGcnW8k/65ZGHgeeAR6oqjVTG/AbwC8AL066kEUU8N+TPNLdnmSt2AKcAf5Lt+T18SSXTLqoRewGfm/SRfQZ9FqxJG8E7gN+rqq+Mul6zquqF6rqSha+kb09yZpY+kryU8AzVfXIpGu5gB+rqh8BbgBu7ZYP14KLgB8BfquqrgL+AVgz59MAuuWkncCnJl1Ln0G/wFs1LFO3/n0f8LtV9QeTrmcx3T/vHwJ2TLqWzjuAnd1a+GHgJ5P8zmRL+mdVdbr7+Qzwhywsba4F88B8719m97IQ/GvJDcCjVfV/J11In0G/YMhtHjSiO+H5CeDJqvropOvpSzKV5Du77W9j4UT7X062qgVV9aGq2lhVm1n4f+3BqnrfhMsCIMkl3Yl1umWR64E1ccVXVf0tcCrJD3RN7+Tlt0tfC25mjS3bwBq5e+WkXeg2DxMuC4Akvwf8BHBZknngjqr6xGSresk7gH8LPNGthQP8x6o6MsGaznsL8MnuCohvAe6pqjV1GeMa9T3AHy58hnMRcHdV/bfJlvQy/wH43e6A7CTw7yZcz0u6D8brgJ+ZdC2jvLxSkhrn0o0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY37Jxu5ShWvlhMuAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BADfg2mfhEFX"
      },
      "source": [
        "\n",
        "**<ins>CART Classification Feature Importance</ins>**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3BWjhfOhi17W",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 403
        },
        "outputId": "78ae85dc-f4fd-492d-d582-bec579cc7421"
      },
      "source": [
        "# decision tree for feature importance on a classification problem\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from matplotlib import pyplot as plt\n",
        "# define the model\n",
        "model = DecisionTreeClassifier()\n",
        "# fit the model\n",
        "model.fit(X_clf, y_clf)\n",
        "# get importance\n",
        "importance = model.feature_importances_\n",
        "# summarize feature importance\n",
        "for i,v in enumerate(importance):\n",
        "\tprint('Feature: %0d, Score: %.5f' % (i,v))\n",
        "# plot feature importance\n",
        "plt.bar([x for x in range(len(importance))], importance)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Feature: 0, Score: 0.61084\n",
            "Feature: 1, Score: 0.06682\n",
            "Feature: 2, Score: 0.00798\n",
            "Feature: 3, Score: 0.08570\n",
            "Feature: 4, Score: 0.07629\n",
            "Feature: 5, Score: 0.03331\n",
            "Feature: 6, Score: 0.01803\n",
            "Feature: 7, Score: 0.10103\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAOpElEQVR4nO3df6zdd13H8eeLlvpjIPzRq1najttoIWmQ8ONaNBgkwEyXkZYE1NZAwIDVhOrIjNqpmVr/GZCg/NEY6pgZyqhjiLlKtRKZUUyGvRsDbEvxUou9jbrLmCIaKZW3f9xv59ndae+3t+f23H76fCQ3O9/v+eSc95rlue/9fr/nNFWFJOna94xxDyBJGg2DLkmNMOiS1AiDLkmNMOiS1Ii143rj9evX1+Tk5LjeXpKuSQ8//PBXqmpi2HNjC/rk5CQzMzPjentJuiYl+fLFnvOUiyQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1otcnRZNsB94HrAHurqq7hqz5ceA3gAI+W1U/OcI5n2Jy38dX6qV7OX3XrWN9f0kaZsmgJ1kDHABuBuaAo0mmq+r4wJotwB3AK6rqiSTfvVIDS5KG63PKZRswW1WnquoccAjYuWjNTwMHquoJgKp6bLRjSpKW0ifoG4AzA9tz3b5Bzween+TvkjzUnaJ5miR7kswkmZmfn1/exJKkoUZ1UXQtsAV4FbAb+L0kz128qKoOVtVUVU1NTAz99kdJ0jL1CfpZYNPA9sZu36A5YLqqvllV/wR8kYXAS5Kukj5BPwpsSbI5yTpgFzC9aM2fsHB0TpL1LJyCOTXCOSVJS1gy6FV1HtgLHAFOAPdX1bEk+5Ps6JYdAR5Pchx4EPjFqnp8pYaWJD1dr/vQq+owcHjRvjsHHhdwe/cjSRoDPykqSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY3oFfQk25OcTDKbZN+Q59+aZD7Jo93P20c/qiTpUtYutSDJGuAAcDMwBxxNMl1Vxxct/aOq2rsCM0qSeuhzhL4NmK2qU1V1DjgE7FzZsSRJl6tP0DcAZwa257p9i70hyeeSPJBk07AXSrInyUySmfn5+WWMK0m6mFFdFP1TYLKqXgR8Arh32KKqOlhVU1U1NTExMaK3liRBv6CfBQaPuDd2+55UVY9X1Te6zbuBl41mPElSX32CfhTYkmRzknXALmB6cEGSGwc2dwAnRjeiJKmPJe9yqarzSfYCR4A1wD1VdSzJfmCmqqaBn0+yAzgPfBV46wrOLEkaYsmgA1TVYeDwon13Djy+A7hjtKNJki6HnxSVpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEb0CnqS7UlOJplNsu8S696QpJJMjW5ESVIfSwY9yRrgAHALsBXYnWTrkHXPBm4DPj3qISVJS+tzhL4NmK2qU1V1DjgE7Byy7reAdwH/M8L5JEk99Qn6BuDMwPZct+9JSV4KbKqqj1/qhZLsSTKTZGZ+fv6yh5UkXdwVXxRN8gzgvcAvLLW2qg5W1VRVTU1MTFzpW0uSBvQJ+llg08D2xm7fBc8GXgj8dZLTwA8C014YlaSrq0/QjwJbkmxOsg7YBUxfeLKq/qOq1lfVZFVNAg8BO6pqZkUmliQNtWTQq+o8sBc4ApwA7q+qY0n2J9mx0gNKkvpZ22dRVR0GDi/ad+dF1r7qyseSJF0uPykqSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY3oFfQk25OcTDKbZN+Q5382yeeTPJrkU0m2jn5USdKlLBn0JGuAA8AtwFZg95Bg31dV319VLwbeDbx35JNKki6pzxH6NmC2qk5V1TngELBzcEFVfW1g8wagRjeiJKmPtT3WbADODGzPAS9fvCjJO4DbgXXAq4e9UJI9wB6Am2666XJnlSRdwsguilbVgar6XuCXgV+7yJqDVTVVVVMTExOjemtJEv2CfhbYNLC9sdt3MYeA11/JUJKky9cn6EeBLUk2J1kH7AKmBxck2TKweSvwj6MbUZLUx5Ln0KvqfJK9wBFgDXBPVR1Lsh+YqappYG+S1wLfBJ4A3rKSQ0uSnq7PRVGq6jBweNG+Owce3zbiuSRJl8lPikpSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSI3oFPcn2JCeTzCbZN+T525McT/K5JH+V5HmjH1WSdClLBj3JGuAAcAuwFdidZOuiZZ8BpqrqRcADwLtHPagk6dL6HKFvA2ar6lRVnQMOATsHF1TVg1X1393mQ8DG0Y4pSVpKn6BvAM4MbM91+y7mbcCfD3siyZ4kM0lm5ufn+08pSVrSSC+KJnkTMAW8Z9jzVXWwqqaqampiYmKUby1J1721PdacBTYNbG/s9j1FktcCvwr8SFV9YzTjSZL66nOEfhTYkmRzknXALmB6cEGSlwDvB3ZU1WOjH1OStJQlg15V54G9wBHgBHB/VR1Lsj/Jjm7Ze4BnAR9J8miS6Yu8nCRphfQ55UJVHQYOL9p358Dj1454LknSZfKTopLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUiF5BT7I9yckks0n2DXn+lUkeSXI+yRtHP6YkaSlLBj3JGuAAcAuwFdidZOuiZf8MvBW4b9QDSpL6WdtjzTZgtqpOASQ5BOwEjl9YUFWnu+e+tQIzSpJ66BP0DcCZge054OXLebMke4A9ADfddNNyXkKSrtjkvo+P9f1P33XrirzuVb0oWlUHq2qqqqYmJiau5ltLUvP6BP0ssGlge2O3T5K0ivQJ+lFgS5LNSdYBu4DplR1LknS5lgx6VZ0H9gJHgBPA/VV1LMn+JDsAkvxAkjngx4D3Jzm2kkNLkp6uz0VRquowcHjRvjsHHh9l4VSMtCytXqSSriY/KSpJjTDoktSIXqdcpOuZp4N0rfAIXZIaYdAlqRGechkxfz2XNC4eoUtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSI/wuF+ka5ncHaZBH6JLUCIMuSY0w6JLUCIMuSY3woqikFeEF26vPI3RJaoRBl6RG9DrlkmQ78D5gDXB3Vd216PlvAz4IvAx4HPiJqjo92lF1pfwVWGrbkkfoSdYAB4BbgK3A7iRbFy17G/BEVX0f8NvAu0Y9qCTp0vqcctkGzFbVqao6BxwCdi5asxO4t3v8APCaJBndmJKkpaSqLr0geSOwvare3m2/GXh5Ve0dWPMP3Zq5bvtL3ZqvLHqtPcCebvMFwMlR/YtcpvXAV5ZcNR7OtjzOtjzOtjzjnO15VTUx7ImrettiVR0EDl7N9xwmyUxVTY17jmGcbXmcbXmcbXlW62x9TrmcBTYNbG/s9g1dk2Qt8BwWLo5Kkq6SPkE/CmxJsjnJOmAXML1ozTTwlu7xG4FP1lLnciRJI7XkKZeqOp9kL3CEhdsW76mqY0n2AzNVNQ18APiDJLPAV1mI/mo29tM+l+Bsy+Nsy+Nsy7MqZ1vyoqgk6drgJ0UlqREGXZIacV0FPcn2JCeTzCbZN+55BiW5J8lj3T39q0qSTUkeTHI8ybEkt417pguSfHuSv0/y2W623xz3TIOSrEnymSR/Nu5ZFktyOsnnkzyaZGbc8wxK8twkDyT5QpITSX5o3DMBJHlB9+d14edrSd457rkuuG7OoXdfYfBF4GZgjoW7d3ZX1fGxDtZJ8krg68AHq+qF455nUJIbgRur6pEkzwYeBl6/Gv7suk8k31BVX0/yTOBTwG1V9dCYRwMgye3AFPBdVfW6cc8zKMlpYGrxBwBXgyT3An9bVXd3d9d9Z1X9+7jnGtQ15SwLH6L88rjngevrCL3PVxiMTVX9DQt3CK06VfUvVfVI9/g/gRPAhvFOtaAWfL3bfGb3syqOUpJsBG4F7h73LNeSJM8BXsnC3XNU1bnVFvPOa4AvrZaYw/UV9A3AmYHtOVZJlK4lSSaBlwCfHu8k/687rfEo8BjwiapaLbP9DvBLwLfGPchFFPCXSR7uvpZjtdgMzAO/352uujvJDeMeaohdwIfHPcSg6ynoukJJngV8FHhnVX1t3PNcUFX/W1UvZuFTzNuSjP2UVZLXAY9V1cPjnuUSfriqXsrCN6m+ozvttxqsBV4K/G5VvQT4L2C1XfNaB+wAPjLuWQZdT0Hv8xUGuoju/PRHgQ9V1R+Pe55hul/LHwS2j3sW4BXAju489SHg1Un+cLwjPVVVne3++RjwMRZOS64Gc8DcwG9aD7AQ+NXkFuCRqvq3cQ8y6HoKep+vMNAQ3YXHDwAnquq9455nUJKJJM/tHn8HCxe9vzDeqaCq7qiqjVU1ycJ/a5+sqjeNeawnJbmhu8BNdzrjR4FVcYdVVf0rcCbJC7pdrwHGfgF+kd2sstMtcB39JdEX+wqDMY/1pCQfBl4FrE8yB/x6VX1gvFM96RXAm4HPd+eqAX6lqg6PcaYLbgTu7e44eAZwf1WtulsEV6HvAT7W/bUFa4H7quovxjvSU/wc8KHu4OsU8FNjnudJ3f8AbwZ+ZtyzLHbd3LYoSa27nk65SFLTDLokNcKgS1IjDLokNcKgS1IjDLokNcKgS1Ij/g9cFeJFYCtBhwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DStIbMZgjLa_"
      },
      "source": [
        "<a id='section2.0.3'></a>\n",
        "### Random Forest Feature Importance\n",
        "\n",
        "Analogously, we can use the RandomForest algorithm for feature importance implemented in scikit-learn as the [*RandomForestRegressor*](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html) and [*RandomForestClassifier*](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html). \n",
        "\n",
        "As above, the model provides a *feature_importances_* property. \n",
        "\n",
        "**<ins>Random Forest Regression Feature Importance</ins>**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HPdY8MDzkBqu",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 403
        },
        "outputId": "482796c1-89be-46e5-fdda-b3d7b5974eb4"
      },
      "source": [
        "# random forest for feature importance on a regression problem\n",
        "from sklearn.ensemble import RandomForestRegressor\n",
        "from matplotlib import pyplot as plt\n",
        "# define the model\n",
        "model = RandomForestRegressor()\n",
        "# fit the model\n",
        "model.fit(X_reg, y_reg)\n",
        "# get importance\n",
        "importance = model.feature_importances_\n",
        "# summarize feature importance\n",
        "for i,v in enumerate(importance):\n",
        "\tprint('Feature: %0d, Score: %.5f' % (i,v))\n",
        "# plot feature importance\n",
        "plt.bar([x for x in range(len(importance))], importance)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Feature: 0, Score: 0.00488\n",
            "Feature: 1, Score: 0.27663\n",
            "Feature: 2, Score: 0.00440\n",
            "Feature: 3, Score: 0.33057\n",
            "Feature: 4, Score: 0.36924\n",
            "Feature: 5, Score: 0.00467\n",
            "Feature: 6, Score: 0.00460\n",
            "Feature: 7, Score: 0.00501\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAR8ElEQVR4nO3df6zdd13H8efLjg4dgoPdGNJ2a8FqKGI2vXYx6DSyjS4z7f6A0BnMNEsqZlXMYrQo2WIJycAE9Y8qa6AGkFnHpsmNVOsiEyVk0rsfMttZuauT3gZdpVNEcKPb2z/ut3h2vN399t5ze+4+PB/JSb+fz/fzOfd9uu11vvt8f9xUFZKkdn3buAuQJC0vg16SGmfQS1LjDHpJapxBL0mNu2DcBQy75JJLav369eMuQ5JeVB588MF/r6qJ+fatuKBfv34909PT4y5Dkl5UkvzL2fa5dCNJjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY1bcXfGSi8m63d9cmw/+4k7rh/bz9aLi0f0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXG9gj7JliRHk8wk2TXP/nckeTTJI0k+k2RT178+yde7/keSfHDUH0CS9MIWvI4+ySpgD3ANMAscSjJVVUcGht1VVR/sxm8FPgBs6fY9XlWXj7ZsSVJffY7oNwMzVXWsqp4B9gPbBgdU1VcGmhcBNboSJUlL0Sfo1wDHB9qzXd/zJLklyePA+4FfGti1IcnDST6d5MeWVK0k6ZyN7GRsVe2pqtcCvwa8u+v+EnBpVV0B3ArcleTlw3OT7EgynWT65MmToypJkkS/oD8BrBtor+36zmY/cANAVT1dVV/uth8EHge+d3hCVe2tqsmqmpyYmOhbuySphz5BfwjYmGRDktXAdmBqcECSjQPN64EvdP0T3clckrwG2AgcG0XhkqR+FrzqpqpOJ9kJHARWAfuq6nCS3cB0VU0BO5NcDXwDeAq4qZt+FbA7yTeA54B3VNWp5fggkqT59XpMcVUdAA4M9d02sP3Os8y7F7h3KQVKkpbGO2MlqXEGvSQ1zqCXpMYZ9JLUOINekhrnLwfXiucv4JaWxiN6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxvUK+iRbkhxNMpNk1zz735Hk0SSPJPlMkk0D+97VzTua5M2jLF6StLAFgz7JKmAPcB2wCbhxMMg7d1XVG6rqcuD9wAe6uZuA7cDrgS3A73XvJ0k6T/oc0W8GZqrqWFU9A+wHtg0OqKqvDDQvAqrb3gbsr6qnq+qfgZnu/SRJ50mfXzyyBjg+0J4FrhwelOQW4FZgNfCTA3MfGJq7ZlGVSpIWZWQnY6tqT1W9Fvg14N3nMjfJjiTTSaZPnjw5qpIkSfQL+hPAuoH22q7vbPYDN5zL3KraW1WTVTU5MTHRoyRJUl99gv4QsDHJhiSrmTu5OjU4IMnGgeb1wBe67Slge5ILk2wANgKfW3rZkqS+Flyjr6rTSXYCB4FVwL6qOpxkNzBdVVPAziRXA98AngJu6uYeTnI3cAQ4DdxSVc8u02eRJM2jz8lYquoAcGCo77aB7Xe+wNz3Au9dbIGSpKXxzlhJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS43rdMKXRWL/rk2P72U/ccf3Yfrak8fKIXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjegV9ki1JjiaZSbJrnv23JjmS5PNJ/irJZQP7nk3ySPeaGmXxkqSFLfismySrgD3ANcAscCjJVFUdGRj2MDBZVV9L8gvA+4G3dfu+XlWXj7huSVJPfY7oNwMzVXWsqp4B9gPbBgdU1f1V9bWu+QCwdrRlSpIWq0/QrwGOD7Rnu76zuRn484H2S5NMJ3kgyQ2LqFGStAQjfUxxkrcDk8CPD3RfVlUnkrwG+FSSR6vq8aF5O4AdAJdeeukoS5Kkb3l9juhPAOsG2mu7vudJcjXwG8DWqnr6TH9Vnej+PAb8NXDF8Nyq2ltVk1U1OTExcU4fQJL0wvoE/SFgY5INSVYD24HnXT2T5ArgTuZC/smB/ouTXNhtXwK8ERg8iStJWmYLLt1U1ekkO4GDwCpgX1UdTrIbmK6qKeC3gJcBn0gC8MWq2gq8DrgzyXPMfancMXS1jiRpmfVao6+qA8CBob7bBravPsu8zwJvWEqBkqSl8c5YSWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXG9gj7JliRHk8wk2TXP/luTHEny+SR/leSygX03JflC97pplMVLkha2YNAnWQXsAa4DNgE3Jtk0NOxhYLKqfgC4B3h/N/eVwO3AlcBm4PYkF4+ufEnSQvoc0W8GZqrqWFU9A+wHtg0OqKr7q+prXfMBYG23/Wbgvqo6VVVPAfcBW0ZTuiSpjz5BvwY4PtCe7frO5mbgzxc5V5I0YheM8s2SvB2YBH78HOftAHYAXHrppaMsSZK+5fU5oj8BrBtor+36nifJ1cBvAFur6ulzmVtVe6tqsqomJyYm+tYuSeqhT9AfAjYm2ZBkNbAdmBockOQK4E7mQv7JgV0HgWuTXNydhL2265MknScLLt1U1ekkO5kL6FXAvqo6nGQ3MF1VU8BvAS8DPpEE4ItVtbWqTiV5D3NfFgC7q+rUsnwSSdK8eq3RV9UB4MBQ320D21e/wNx9wL7FFihJWhrvjJWkxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuN6BX2SLUmOJplJsmue/VcleSjJ6SRvGdr3bJJHutfUqAqXJPVzwUIDkqwC9gDXALPAoSRTVXVkYNgXgZ8FfmWet/h6VV0+glolSYuwYNADm4GZqjoGkGQ/sA34ZtBX1RPdvueWoUZJ0hL0WbpZAxwfaM92fX29NMl0kgeS3DDfgCQ7ujHTJ0+ePIe3liQt5HycjL2sqiaBnwZ+J8lrhwdU1d6qmqyqyYmJifNQkiR96+gT9CeAdQPttV1fL1V1ovvzGPDXwBXnUJ8kaYn6BP0hYGOSDUlWA9uBXlfPJLk4yYXd9iXAGxlY25ckLb8Fg76qTgM7gYPAY8DdVXU4ye4kWwGS/HCSWeCtwJ1JDnfTXwdMJ/l74H7gjqGrdSRJy6zPVTdU1QHgwFDfbQPbh5hb0hme91ngDUusUZK0BN4ZK0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDWuV9An2ZLkaJKZJLvm2X9VkoeSnE7ylqF9NyX5Qve6aVSFS5L6WTDok6wC9gDXAZuAG5NsGhr2ReBngbuG5r4SuB24EtgM3J7k4qWXLUnqq88R/WZgpqqOVdUzwH5g2+CAqnqiqj4PPDc0983AfVV1qqqeAu4DtoygbklST32Cfg1wfKA92/X10Wtukh1JppNMnzx5sudbS5L6WBEnY6tqb1VNVtXkxMTEuMuRpKb0CfoTwLqB9tqur4+lzJUkjUCfoD8EbEyyIclqYDsw1fP9DwLXJrm4Owl7bdcnSTpPFgz6qjoN7GQuoB8D7q6qw0l2J9kKkOSHk8wCbwXuTHK4m3sKeA9zXxaHgN1dnyTpPLmgz6CqOgAcGOq7bWD7EHPLMvPN3QfsW0KNkqQlWBEnYyVJy8egl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUuF5Bn2RLkqNJZpLsmmf/hUn+uNv/d0nWd/3rk3w9ySPd64OjLV+StJAFfzl4klXAHuAaYBY4lGSqqo4MDLsZeKqqvifJduB9wNu6fY9X1eUjrluS1FOfI/rNwExVHauqZ4D9wLahMduAj3Tb9wBvSpLRlSlJWqw+Qb8GOD7Qnu365h1TVaeB/wRe1e3bkOThJJ9O8mPz/YAkO5JMJ5k+efLkOX0ASdILW+6TsV8CLq2qK4BbgbuSvHx4UFXtrarJqpqcmJhY5pIk6VtLn6A/AawbaK/t+uYdk+QC4BXAl6vq6ar6MkBVPQg8DnzvUouWJPXXJ+gPARuTbEiyGtgOTA2NmQJu6rbfAnyqqirJRHcylySvATYCx0ZTuiSpjwWvuqmq00l2AgeBVcC+qjqcZDcwXVVTwIeBjyWZAU4x92UAcBWwO8k3gOeAd1TVqeX4IJKk+S0Y9ABVdQA4MNR328D2/wBvnWfevcC9S6xRkrQE3hkrSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNa7X74xNsgX4XeZ+OfiHquqOof0XAh8Ffgj4MvC2qnqi2/cu4GbgWeCXqurgyKqfx/pdn1zOt39BT9xx/dh+9lL599aelfzP1Nrmt1z/LSx4RJ9kFbAHuA7YBNyYZNPQsJuBp6rqe4DfBt7Xzd0EbAdeD2wBfq97P0nSedJn6WYzMFNVx6rqGWA/sG1ozDbgI932PcCbkqTr319VT1fVPwMz3ftJks6TPks3a4DjA+1Z4Mqzjamq00n+E3hV1//A0Nw1wz8gyQ5gR9f8apKjvaofvUuAf1/s5LxvhJX8f9a2ONa2ONa2OOOs7bKz7ei1Rr/cqmovsHfcdSSZrqrJcdcxH2tbHGtbHGtbnJVaW5+lmxPAuoH22q5v3jFJLgBewdxJ2T5zJUnLqE/QHwI2JtmQZDVzJ1enhsZMATd1228BPlVV1fVvT3Jhkg3ARuBzoyldktTHgks33Zr7TuAgc5dX7quqw0l2A9NVNQV8GPhYkhngFHNfBnTj7gaOAKeBW6rq2WX6LKMw9uWjF2Bti2Nti2Nti7Mia8vcgbckqVXeGStJjTPoJalxBn0nyZYkR5PMJNk17nrOSLIvyZNJ/mHctQxLsi7J/UmOJDmc5J3jrumMJC9N8rkkf9/V9pvjrmlYklVJHk7yZ+OuZVCSJ5I8muSRJNPjrmdQku9Kck+Sf0zyWJIfGXdNAEm+r/v7OvP6SpJfHnddZ7hGzzcf8/BPwDXM3dR1CLixqo6MtTAgyVXAV4GPVtX3j7ueQUleDby6qh5K8p3Ag8ANK+TvLcBFVfXVJC8BPgO8s6oeWGDqeZPkVmASeHlV/dS46zkjyRPAZFUt+saf5ZLkI8DfVtWHuqsAv6Oq/mPcdQ3q8uQEcGVV/cu46wGP6M/o85iHsaiqv2HuSqYVp6q+VFUPddv/BTzGPHc+j0PN+WrXfEn3WjFHNUnWAtcDHxp3LS8WSV4BXMXcVX5U1TMrLeQ7bwIeXykhDwb9GfM95mFFBNaLRZL1wBXA3423kv/TLY08AjwJ3FdVK6Y24HeAXwWeG3ch8yjgL5M82D2eZKXYAJwE/qBb8vpQkovGXdQ8tgN/NO4iBhn0WrIkLwPuBX65qr4y7nrOqKpnq+py5u7I3pxkRSx9Jfkp4MmqenDctZzFj1bVDzL3xNpbuuXDleAC4AeB36+qK4D/BlbM+TSAbjlpK/CJcdcyyKCf46MaFqlb/74X+HhV/cm465lP97/39zP3qOyV4I3A1m4tfD/wk0n+cLwl/Z+qOtH9+STwp6ycJ87OArMD/2d2D3PBv5JcBzxUVf827kIGGfRz+jzmQUO6E54fBh6rqg+Mu55BSSaSfFe3/e3MnWj/x/FWNaeq3lVVa6tqPXP/rn2qqt4+5rIASHJRd2KdblnkWmBFXPFVVf8KHE/yfV3Xm5i7634luZEVtmwDK+TpleN2tsc8jLksAJL8EfATwCVJZoHbq+rD463qm94I/AzwaLcWDvDrVXVgjDWd8WrgI90VEN8G3F1VK+oyxhXqu4E/nfsO5wLgrqr6i/GW9Dy/CHy8OyA7BvzcmOv5pu6L8Rrg58ddyzAvr5Skxrl0I0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4/4Xuz1N3yzmAJYAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yeHsgcgelZRi"
      },
      "source": [
        "**<ins>Random Forest Classification Feature Importance</ins>**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cR0BTOLrlf7v",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 403
        },
        "outputId": "9ac3888f-8ad4-4169-d73f-abdcaafd8beb"
      },
      "source": [
        "# random forest for feature importance on a classification problem\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from matplotlib import pyplot as plt\n",
        "# define the model\n",
        "model = RandomForestClassifier()\n",
        "# fit the model\n",
        "model.fit(X_clf, y_clf)\n",
        "# get importance\n",
        "importance = model.feature_importances_\n",
        "# summarize feature importance\n",
        "for i,v in enumerate(importance):\n",
        "\tprint('Feature: %0d, Score: %.5f' % (i,v))\n",
        "# plot feature importance\n",
        "plt.bar([x for x in range(len(importance))], importance)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Feature: 0, Score: 0.27838\n",
            "Feature: 1, Score: 0.11923\n",
            "Feature: 2, Score: 0.07076\n",
            "Feature: 3, Score: 0.13927\n",
            "Feature: 4, Score: 0.10233\n",
            "Feature: 5, Score: 0.10529\n",
            "Feature: 6, Score: 0.04857\n",
            "Feature: 7, Score: 0.13616\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAPYUlEQVR4nO3dfayed13H8feHlg0FhUEbQ9puLVIJQ8yGhy5mOo17oMtIyx8jdAYzzJKqYQZCjCmSbLH8MyBR/pm6htVMhJWxiTmR4lzY8CFk0LMHwHZUzmpZT4OuUAQnuNnt6x/31XHveMq5eh563/vt/Uru9Lp+D/f9PU3zua/+roeTqkKS1K4XjboASdLyMuglqXEGvSQ1zqCXpMYZ9JLUuJWjLmC2VatW1fr160ddhiQ9rzzwwAPfrqrVc/WNXdCvX7+eqampUZchSc8rSb55qj6XbiSpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXFjd2fsYq3f8dmRffbhm64a2WdL0ql4RC9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mN6xX0STYnOZhkOsmOOfrfl+RAkq8m+XyS84b6nk7ycPeaXMriJUnzm/dXCSZZAdwMXA7MAPuSTFbVgaFhDwETVfWDJL8LfBh4R9f3w6q6YInrliT11OeIfhMwXVWHquopYA+wdXhAVd1XVT/odu8H1i5tmZKkheoT9GuAI0P7M13bqVwHfG5o/yVJppLcn+Rtc01Isr0bM3Xs2LEeJUmS+pp36eZ0JHknMAH86lDzeVV1NMlrgHuTfK2qHh2eV1W7gF0AExMTtZQ1SdILXZ8j+qPAuqH9tV3bcyS5DPgAsKWqnjzZXlVHuz8PAV8ALlxEvZKk09Qn6PcBG5NsSHIWsA14ztUzSS4EbmEQ8o8PtZ+T5OxuexVwMTB8EleStMzmXbqpqhNJrgfuBlYAu6tqf5KdwFRVTQIfAV4GfDoJwGNVtQV4PXBLkmcYfKncNOtqHUnSMuu1Rl9Ve4G9s9puGNq+7BTzvgi8cTEFSpIWxztjJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1LheQZ9kc5KDSaaT7Jij/31JDiT5apLPJzlvqO/aJN/oXtcuZfGSpPnNG/RJVgA3A1cC5wPXJDl/1rCHgImq+gXgTuDD3dxXAjcCFwGbgBuTnLN05UuS5tPniH4TMF1Vh6rqKWAPsHV4QFXdV1U/6HbvB9Z2228B7qmq41X1XeAeYPPSlC5J6qNP0K8Bjgztz3Rtp3Id8LnTmZtke5KpJFPHjh3rUZIkqa8lPRmb5J3ABPCR05lXVbuqaqKqJlavXr2UJUnSC16foD8KrBvaX9u1PUeSy4APAFuq6snTmStJWj59gn4fsDHJhiRnAduAyeEBSS4EbmEQ8o8Pdd0NXJHknO4k7BVdmyTpDFk534CqOpHkegYBvQLYXVX7k+wEpqpqksFSzcuATycBeKyqtlTV8SQfZPBlAbCzqo4vy08iSZrTvEEPUFV7gb2z2m4Y2r7sx8zdDexeaIGSpMXxzlhJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1rlfQJ9mc5GCS6SQ75ui/JMmDSU4kuXpW39NJHu5ek0tVuCSpn5XzDUiyArgZuByYAfYlmayqA0PDHgPeBfz+HG/xw6q6YAlqlSQtwLxBD2wCpqvqEECSPcBW4Nmgr6rDXd8zy1CjJGkR+izdrAGODO3PdG19vSTJVJL7k7xtrgFJtndjpo4dO3Yaby1Jms+ZOBl7XlVNAL8BfDTJz84eUFW7qmqiqiZWr159BkqSpBeOPkF/FFg3tL+2a+ulqo52fx4CvgBceBr1SZIWqU/Q7wM2JtmQ5CxgG9Dr6pkk5yQ5u9teBVzM0Nq+JGn5zRv0VXUCuB64G3gEuKOq9ifZmWQLQJI3J5kB3g7ckmR/N/31wFSSrwD3ATfNulpHkrTM+lx1Q1XtBfbOarthaHsfgyWd2fO+CLxxkTVKkhbBO2MlqXEGvSQ1rtfSjTRK63d8dmSfffimq0b22dJS8Yhekhpn0EtS41y6kaQhLS4VekQvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zmfdnEEtPkND0vjziF6SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhrnnbFSo7wTWyf1OqJPsjnJwSTTSXbM0X9JkgeTnEhy9ay+a5N8o3tdu1SFS5L6mTfok6wAbgauBM4Hrkly/qxhjwHvAj45a+4rgRuBi4BNwI1Jzll82ZKkvvos3WwCpqvqEECSPcBW4MDJAVV1uOt7ZtbctwD3VNXxrv8eYDNw+6Irl8aAyyN6PuizdLMGODK0P9O19dFrbpLtSaaSTB07dqznW0uS+hiLq26qaldVTVTVxOrVq0ddjiQ1pU/QHwXWDe2v7dr6WMxcSdIS6BP0+4CNSTYkOQvYBkz2fP+7gSuSnNOdhL2ia5MknSHzBn1VnQCuZxDQjwB3VNX+JDuTbAFI8uYkM8DbgVuS7O/mHgc+yODLYh+w8+SJWUnSmdHrhqmq2gvsndV2w9D2PgbLMnPN3Q3sXkSNkqRFGIuTsZKk5WPQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1Lhev3hE7Vu/47Mj++zDN101ss+WXgg8opekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhrXK+iTbE5yMMl0kh1z9J+d5FNd/5eSrO/a1yf5YZKHu9efL235kqT5zPtQsyQrgJuBy4EZYF+Syao6MDTsOuC7VfXaJNuADwHv6PoeraoLlrhuSVJPfY7oNwHTVXWoqp4C9gBbZ43ZCtzWbd8JXJokS1emJGmh+gT9GuDI0P5M1zbnmKo6AXwPeFXXtyHJQ0n+IcmvLLJeSdJpWu7n0X8LOLeqvpPkF4G/SfKGqvr+8KAk24HtAOeee+4ylyRJLyx9gv4osG5of23XNteYmSQrgZcD36mqAp4EqKoHkjwK/BwwNTy5qnYBuwAmJiZqAT+HpOcRf9HNmdVn6WYfsDHJhiRnAduAyVljJoFru+2rgXurqpKs7k7mkuQ1wEbg0NKULknqY94j+qo6keR64G5gBbC7qvYn2QlMVdUkcCvw8STTwHEGXwYAlwA7k/wv8AzwO1V1fDl+EEnS3Hqt0VfVXmDvrLYbhrb/B3j7HPPuAu5aZI2SpEXwzlhJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMb1Cvokm5McTDKdZMcc/Wcn+VTX/6Uk64f63t+1H0zylqUrXZLUx7xBn2QFcDNwJXA+cE2S82cNuw74blW9FvgT4EPd3POBbcAbgM3An3bvJ0k6Q/oc0W8CpqvqUFU9BewBts4asxW4rdu+E7g0Sbr2PVX1ZFX9GzDdvZ8k6QxZ2WPMGuDI0P4McNGpxlTViSTfA17Vtd8/a+6a2R+QZDuwvdt9IsnBXtUvvVXAtxc6OR9awkr+P2tbGGtbGGtbmFHWdt6pOvoE/bKrql3ArlHXkWSqqiZGXcdcrG1hrG1hrG1hxrW2Pks3R4F1Q/tru7Y5xyRZCbwc+E7PuZKkZdQn6PcBG5NsSHIWg5Ork7PGTALXdttXA/dWVXXt27qrcjYAG4EvL03pkqQ+5l266dbcrwfuBlYAu6tqf5KdwFRVTQK3Ah9PMg0cZ/BlQDfuDuAAcAJ4d1U9vUw/y1IY+fLRj2FtC2NtC2NtCzOWtWVw4C1JapV3xkpS4wx6SWqcQd+Z7zEPo5Jkd5LHk/zLqGuZLcm6JPclOZBkf5L3jLqmk5K8JMmXk3ylq+2PRl3TbElWJHkoyd+OupZhSQ4n+VqSh5NMjbqeYUlekeTOJF9P8kiSXxp1TQBJXtf9fZ18fT/Je0dd10mu0fPsYx7+FbicwU1d+4BrqurASAsDklwCPAH8ZVX9/KjrGZbk1cCrq+rBJD8FPAC8bUz+3gK8tKqeSPJi4J+B91TV/fNMPWOSvA+YAH66qt466npOSnIYmKiqBd/4s1yS3Ab8U1V9rLsK8Cer6j9HXdewLk+OAhdV1TdHXQ94RH9Sn8c8jERV/SODK5nGTlV9q6oe7Lb/C3iEOe58HoUaeKLbfXH3GpujmiRrgauAj426lueLJC8HLmFwlR9V9dS4hXznUuDRcQl5MOhPmusxD2MRWM8X3RNLLwS+NNpKfqRbGnkYeBy4p6rGpjbgo8AfAM+MupA5FPD3SR7oHk8yLjYAx4C/6Ja8PpbkpaMuag7bgNtHXcQwg16LluRlwF3Ae6vq+6Ou56SqerqqLmBwR/amJGOx9JXkrcDjVfXAqGs5hV+uqjcxeGLtu7vlw3GwEngT8GdVdSHw38DYnE8D6JaTtgCfHnUtwwz6AR/VsEDd+vddwCeq6q9HXc9cuv/e38fgUdnj4GJgS7cWvgf49SR/NdqSfqSqjnZ/Pg58hvF54uwMMDP0P7M7GQT/OLkSeLCq/mPUhQwz6Af6POZBs3QnPG8FHqmqPx51PcOSrE7yim77JxicaP/6aKsaqKr3V9XaqlrP4N/avVX1zhGXBUCSl3Yn1umWRa4AxuKKr6r6d+BIktd1TZcyuOt+nFzDmC3bwJg8vXLUTvWYhxGXBUCS24FfA1YlmQFurKpbR1vVsy4GfhP4WrcWDvCHVbV3hDWd9Grgtu4KiBcBd1TVWF3GOKZ+BvjM4DuclcAnq+rvRlvSc/we8InugOwQ8FsjrudZ3Rfj5cBvj7qW2by8UpIa59KNJDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mN+z/UtTOAwhlj+QAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OgWdtRnyl1GU"
      },
      "source": [
        "<a id='section2.0.4'></a>\n",
        "### XGBoost Feature Importance\n",
        "[XGBoost](https://xgboost.readthedocs.io/en/latest/index.html) is a Python library that provides an efficient implementation of the stochastic gradient boostig algorithm. (For an introduction to Boosted Trees, you can take a look [here](https://xgboost.readthedocs.io/en/latest/tutorials/model.html))\n",
        "\n",
        "This algorithm can be integrated with Scikit-Learn via the *XGBRegressor* and *XGBClassifier* classes. \n",
        "\n",
        "Even in this one, we can find the *feature_importances_* property. \n",
        "\n",
        "First, let's install the XGBoost library, with pip:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JNyQBSwlmbhc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "2bd82bcf-d5bc-4a20-c80e-7bbe63a8793f"
      },
      "source": [
        "!pip install xgboost"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: xgboost in /usr/local/lib/python3.6/dist-packages (0.90)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from xgboost) (1.4.1)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from xgboost) (1.18.5)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xjasK8iemgtl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "69614b88-03f9-4ffc-89d1-54f835472ebb"
      },
      "source": [
        "# Check xgboost version\n",
        "import xgboost\n",
        "print(xgboost.__version__)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.90\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qOH7KhfvmpSH"
      },
      "source": [
        "Now, let's take a look at an example of XGBoost for feature importance. \n",
        "\n",
        "**<ins>XGBoost Regression Feature Importance</ins>**\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7AyYTBhOnFgu",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "outputId": "a2aca495-bb0a-4ede-bf69-cb343733f8c3"
      },
      "source": [
        "# xgboost for feature importance on a regression problem\n",
        "from xgboost import XGBRegressor\n",
        "from matplotlib import pyplot as plt\n",
        "# define the model\n",
        "model = XGBRegressor()\n",
        "# fit the model\n",
        "model.fit(X_reg, y_reg)\n",
        "# get importance\n",
        "importance = model.feature_importances_\n",
        "# summarize feature importance\n",
        "for i,v in enumerate(importance):\n",
        "\tprint('Feature: %0d, Score: %.5f' % (i,v))\n",
        "# plot feature importance\n",
        "plt.bar([x for x in range(len(importance))], importance)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[21:06:15] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.\n",
            "Feature: 0, Score: 0.00057\n",
            "Feature: 1, Score: 0.25953\n",
            "Feature: 2, Score: 0.00099\n",
            "Feature: 3, Score: 0.38528\n",
            "Feature: 4, Score: 0.34686\n",
            "Feature: 5, Score: 0.00103\n",
            "Feature: 6, Score: 0.00437\n",
            "Feature: 7, Score: 0.00137\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD6CAYAAACvZ4z8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAATNElEQVR4nO3df6xfd33f8ecLpw5boDRtriZqO7GhpqopU8JunU2s2VTyw1EqG6kgnIkpnZA8pnhLlf2oWatEM0IKVEL9xytY4Il1Td2QrNPVcJulJXRDXcA3P0pmpx43JsXXYsstycgYNMHJe3/cY3ry7XXu8f3h780nz4d0dc/5/Ph+39+b6PU9/pzzPd9UFZKkdr1u3AVIklaXQS9JjTPoJalxBr0kNc6gl6TGGfSS1LhBQZ9kR5ITSWaS7HuFcb+QpJJM9to+3M07keSGlShakjTcRYsNSLIOOABcB8wCR5NMVdXxkXFvBG4Dvtxr2wbsBt4O/DjwB0neVlUvnuv5Lrvsstq8efMSXookvXY9/PDDf15VEwv1LRr0wHZgpqpOAiQ5DOwCjo+M+wjwMeBf9tp2AYer6nng60lmusf77+d6ss2bNzM9PT2gLEnSWUn+7Fx9Q5ZuNgCnevuzXVv/Cd4JbKqqz5/vXEnS6lr2ydgkrwM+AfzzZTzGniTTSabn5uaWW5IkqWdI0J8GNvX2N3ZtZ70R+Gngi0meAv42MNWdkF1sLgBVdbCqJqtqcmJiwSUmSdISDQn6o8DWJFuSrGf+5OrU2c6q+nZVXVZVm6tqM/AQsLOqprtxu5NcnGQLsBX4yoq/CknSOS16MraqziTZC9wPrAMOVdWxJPuB6aqaeoW5x5Lcw/yJ2zPAra90xY0kaeVlrd2meHJysrzqRpLOT5KHq2pyoT4/GStJjTPoJalxBr0kNW7IJ2Olsdq8b/RzeBfOU3fdNLbnllaKR/SS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXGDgj7JjiQnkswk2bdA/4eSPJ7ksSRfSrKta9+c5Htd+2NJPrnSL0CS9MoWvR99knXAAeA6YBY4mmSqqo73ht1dVZ/sxu8EPgHs6PqerKorV7ZsSdJQQ47otwMzVXWyql4ADgO7+gOq6rne7iXA2vrGcUl6DRsS9BuAU7392a7tZZLcmuRJ4OPAP+t1bUnyaJI/SvKzy6pWknTeVuxkbFUdqKq3Ar8M/GrX/E3g8qq6CrgduDvJD4/OTbInyXSS6bm5uZUqSZLEsO+MPQ1s6u1v7NrO5TDwGwBV9TzwfLf9cHfE/zZguj+hqg4CBwEmJydd9tGrht9nq1eDIUf0R4GtSbYkWQ/sBqb6A5Js7e3eBHyta5/oTuaS5C3AVuDkShQuSRpm0SP6qjqTZC9wP7AOOFRVx5LsB6aragrYm+Ra4PvAs8At3fRrgP1Jvg+8BHyoqp5ZjRciSVrYkKUbquoIcGSk7Y7e9m3nmHcfcN9yCpQkLY+fjJWkxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1LhBQZ9kR5ITSWaS7Fug/0NJHk/yWJIvJdnW6/twN+9EkhtWsnhJ0uIWDfok64ADwI3ANuDmfpB37q6qd1TVlcDHgU90c7cBu4G3AzuAf9s9niTpAhlyRL8dmKmqk1X1AnAY2NUfUFXP9XYvAarb3gUcrqrnq+rrwEz3eJKkC+SiAWM2AKd6+7PA1aODktwK3A6sB36uN/ehkbkbllSpJGlJVuxkbFUdqKq3Ar8M/Or5zE2yJ8l0kum5ubmVKkmSxLCgPw1s6u1v7NrO5TDwnvOZW1UHq2qyqiYnJiYGlCRJGmpI0B8FtibZkmQ98ydXp/oDkmzt7d4EfK3bngJ2J7k4yRZgK/CV5ZctSRpq0TX6qjqTZC9wP7AOOFRVx5LsB6aragrYm+Ra4PvAs8At3dxjSe4BjgNngFur6sVVei2SpAUMORlLVR0Bjoy03dHbvu0V5n4U+OhSC5QkLY+fjJWkxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNW7QF49oZWze9/mxPfdTd900tueWNF4e0UtS4wYFfZIdSU4kmUmyb4H+25McT/LVJH+Y5Ipe34tJHut+pkbnSpJW16JLN0nWAQeA64BZ4GiSqao63hv2KDBZVd9N8k+AjwPv7/q+V1VXrnDdkqSBhhzRbwdmqupkVb0AHAZ29QdU1YNV9d1u9yFg48qWKUlaqiFBvwE41duf7drO5YPA7/X2X59kOslDSd6z0IQke7ox03NzcwNKkiQNtaJX3ST5ADAJ/L1e8xVVdTrJW4AvJHm8qp7sz6uqg8BBgMnJyVrJmiTptW7IEf1pYFNvf2PX9jJJrgV+BdhZVc+fba+q093vk8AXgauWUa8k6TwNCfqjwNYkW5KsB3YDL7t6JslVwKeYD/mne+2XJrm4274MeBfQP4krSVpliy7dVNWZJHuB+4F1wKGqOpZkPzBdVVPArwFvAD6XBOAbVbUT+CngU0leYv5N5a6Rq3UkSats0Bp9VR0Bjoy03dHbvvYc8/4YeMdyCpQkLY+fjJWkxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1LhBQZ9kR5ITSWaS7Fug//Ykx5N8NckfJrmi13dLkq91P7esZPGSpMUtGvRJ1gEHgBuBbcDNSbaNDHsUmKyqvwncC3y8m/ujwJ3A1cB24M4kl65c+ZKkxQw5ot8OzFTVyap6ATgM7OoPqKoHq+q73e5DwMZu+wbggap6pqqeBR4AdqxM6ZKkIYYE/QbgVG9/tms7lw8Cv3c+c5PsSTKdZHpubm5ASZKkoVb0ZGySDwCTwK+dz7yqOlhVk1U1OTExsZIlSdJr3pCgPw1s6u1v7NpeJsm1wK8AO6vq+fOZK0laPUOC/iiwNcmWJOuB3cBUf0CSq4BPMR/yT/e67geuT3JpdxL2+q5NknSBXLTYgKo6k2Qv8wG9DjhUVceS7Aemq2qK+aWaNwCfSwLwjaraWVXPJPkI828WAPur6plVeSWSpAUtGvQAVXUEODLSdkdv+9pXmHsIOLTUAiVJy+MnYyWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNW5Q0CfZkeREkpkk+xbovybJI0nOJHnvSN+LSR7rfqZG50qSVtei3xmbZB1wALgOmAWOJpmqquO9Yd8AfhH4Fws8xPeq6soVqFWStARDvhx8OzBTVScBkhwGdgE/CPqqeqrre2kVapQkLcOQpZsNwKne/mzXNtTrk0wneSjJexYakGRPN2Z6bm7uPB5akrSYC3Ey9oqqmgT+AfDrSd46OqCqDlbVZFVNTkxMXICSJOm1Y0jQnwY29fY3dm2DVNXp7vdJ4IvAVedRnyRpmYYE/VFga5ItSdYDu4FBV88kuTTJxd32ZcC76K3tS5JW36JBX1VngL3A/cATwD1VdSzJ/iQ7AZL8TJJZ4H3Ap5Ic66b/FDCd5E+AB4G7Rq7WkSStsiFX3VBVR4AjI2139LaPMr+kMzrvj4F3LLNGSdIy+MlYSWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJatygoE+yI8mJJDNJ9i3Qf02SR5KcSfLekb5bknyt+7llpQqXJA2zaNAnWQccAG4EtgE3J9k2MuwbwC8Cd4/M/VHgTuBqYDtwZ5JLl1+2JGmoIUf024GZqjpZVS8Ah4Fd/QFV9VRVfRV4aWTuDcADVfVMVT0LPADsWIG6JUkDDQn6DcCp3v5s1zbEcuZKklbAmjgZm2RPkukk03Nzc+MuR5KaMiToTwObevsbu7YhBs2tqoNVNVlVkxMTEwMfWpI0xJCgPwpsTbIlyXpgNzA18PHvB65Pcml3Evb6rk2SdIEsGvRVdQbYy3xAPwHcU1XHkuxPshMgyc8kmQXeB3wqybFu7jPAR5h/szgK7O/aJEkXyEVDBlXVEeDISNsdve2jzC/LLDT3EHBoGTVKkpZhTZyMlSStHoNekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGjco6JPsSHIiyUySfQv0X5zkd7r+LyfZ3LVvTvK9JI91P59c2fIlSYtZ9Dtjk6wDDgDXAbPA0SRTVXW8N+yDwLNV9RNJdgMfA97f9T1ZVVeucN2SpIGGHNFvB2aq6mRVvQAcBnaNjNkFfLbbvhd4d5KsXJmSpKUaEvQbgFO9/dmubcExVXUG+DbwY13fliSPJvmjJD+7zHolSedp0aWbZfomcHlVfSvJ3wL+U5K3V9Vz/UFJ9gB7AC6//PJVLkmSXluGHNGfBjb19jd2bQuOSXIR8CbgW1X1fFV9C6CqHgaeBN42+gRVdbCqJqtqcmJi4vxfhSTpnIYE/VFga5ItSdYDu4GpkTFTwC3d9nuBL1RVJZnoTuaS5C3AVuDkypQuSRpi0aWbqjqTZC9wP7AOOFRVx5LsB6aragr4DPCbSWaAZ5h/MwC4Btif5PvAS8CHquqZ1XghkqSFDVqjr6ojwJGRtjt6238BvG+BefcB9y2zRknSMvjJWElqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjRsU9El2JDmRZCbJvgX6L07yO13/l5Ns7vV9uGs/keSGlStdkjTEot8Zm2QdcAC4DpgFjiaZqqrjvWEfBJ6tqp9Ishv4GPD+JNuY/6LwtwM/DvxBkrdV1Ysr/UIkvXps3vf5sT33U3fdNLbnHpchR/TbgZmqOllVLwCHgV0jY3YBn+227wXenSRd++Gqer6qvg7MdI8nSbpAFj2iBzYAp3r7s8DV5xpTVWeSfBv4sa79oZG5G5Zc7QAeKSyNf7f2+N90aVr8uw0J+lWXZA+wp9v9TpITYyrlMuDPlzo5H1vBSv4qa1saa1saa1uacdZ2xbk6hgT9aWBTb39j17bQmNkkFwFvAr41cC5VdRA4OKCWVZVkuqomx13HQqxtaaxtaaxtadZqbUPW6I8CW5NsSbKe+ZOrUyNjpoBbuu33Al+oqurad3dX5WwBtgJfWZnSJUlDLHpE36257wXuB9YBh6rqWJL9wHRVTQGfAX4zyQzwDPNvBnTj7gGOA2eAW73iRpIurEFr9FV1BDgy0nZHb/svgPedY+5HgY8uo8YLaezLR6/A2pbG2pbG2pZmTdaW+RUWSVKrvAWCJDXOoO8sdpuHcUlyKMnTSf7HuGsZlWRTkgeTHE9yLMlt467prCSvT/KVJH/S1fZvxl3TqCTrkjya5D+Pu5a+JE8leTzJY0mmx11PX5IfSXJvkj9N8kSSvzPumgCS/GT39zr781ySXxp3XWe5dMMPbvPwP+nd5gG4eeQ2D2OR5BrgO8C/r6qfHnc9fUneDLy5qh5J8kbgYeA9a+TvFuCSqvpOkh8CvgTcVlUPLTL1gklyOzAJ/HBV/fy46zkryVPAZFUt+Xrw1ZLks8B/q6pPd1cB/vWq+j/jrquvy5PTwNVV9Wfjrgc8oj9ryG0exqKq/ivzVzKtOVX1zap6pNv+v8ATrPInn4eqed/pdn+o+1kzRzVJNgI3AZ8edy2vFkneBFzD/FV+VNULay3kO+8GnlwrIQ8G/VkL3eZhTQTWq0V3x9KrgC+Pt5K/1C2NPAY8DTxQVWumNuDXgX8FvDTuQhZQwH9J8nD3qfW1YgswB/y7bsnr00kuGXdRC9gN/Pa4i+gz6LVsSd4A3Af8UlU9N+56zqqqF6vqSuY/kb09yZpY+kry88DTVfXwuGs5h79bVe8EbgRu7ZYP14KLgHcCv1FVVwH/D1gz59MAuuWkncDnxl1Ln0E/b9CtGvRXdevf9wG/VVX/cdz1LKT75/2DwI5x19J5F7CzWws/DPxckv8w3pL+UlWd7n4/Dfwua+eOs7PAbO9fZvcyH/xryY3AI1X1v8ddSJ9BP2/IbR40ojvh+Rngiar6xLjr6UsykeRHuu2/xvyJ9j8db1XzqurDVbWxqjYz///aF6rqA2MuC4Akl3Qn1umWRa4H1sQVX1X1v4BTSX6ya3o385+6X0tuZo0t28AauXvluJ3rNg9jLguAJL8N/H3gsiSzwJ1V9ZnxVvUD7wL+IfB4txYO8K+7T1KP25uBz3ZXQLwOuKeq1tRljGvU3wB+d/49nIuAu6vq98db0sv8U+C3ugOyk8A/GnM9P9C9MV4H/ONx1zLKyyslqXEu3UhS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIa9/8BL3bbQ4ulwJsAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lXRV3TVxpCAd"
      },
      "source": [
        "**<ins>XGBoost Classification Feature Importance</ins>**\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YKVa-Q-9pGZq",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 403
        },
        "outputId": "b348142b-45fa-46f2-e321-c45c8cd49d79"
      },
      "source": [
        "# xgboost for feature importance on a classification problem\n",
        "from xgboost import XGBClassifier\n",
        "from matplotlib import pyplot\n",
        "# define the model\n",
        "model = XGBClassifier()\n",
        "# fit the model\n",
        "model.fit(X_clf, y_clf)\n",
        "# get importance\n",
        "importance = model.feature_importances_\n",
        "# summarize feature importance\n",
        "for i,v in enumerate(importance):\n",
        "\tprint('Feature: %0d, Score: %.5f' % (i,v))\n",
        "# plot feature importance\n",
        "pyplot.bar([x for x in range(len(importance))], importance)\n",
        "pyplot.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Feature: 0, Score: 0.51614\n",
            "Feature: 1, Score: 0.05840\n",
            "Feature: 2, Score: 0.08998\n",
            "Feature: 3, Score: 0.03381\n",
            "Feature: 4, Score: 0.10906\n",
            "Feature: 5, Score: 0.08295\n",
            "Feature: 6, Score: 0.02555\n",
            "Feature: 7, Score: 0.08412\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAANl0lEQVR4nO3dXYxc912H8ecbu+YlDS2SVyjyS9cCK5IpVRMWF1QUqiZBtlLsSs2FLbVqUCuDVEOqIIEDKKjmpi1S4MZCWElQgKZumjbSQgyhokHQiwSv09DguIat5WJbQJy2EAKiwfTHxR6bybIvs+uxz/rv5yOtMufMXzO/WNGTs+fMGaeqkCRd/a7rewBJ0mgYdElqhEGXpEYYdElqhEGXpEas7uuN165dW+Pj4329vSRdlY4ePfpyVY3N9VxvQR8fH2dqaqqvt5ekq1KSr8/3nKdcJKkRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRvd0peinG9z3Z6/uf+vidvb6/JM3FI3RJaoRBl6RGDBX0JNuSnEgynWTfHM/fneRckue7nw+PflRJ0kIWPYeeZBVwALgDOAMcSTJZVS/OWvqZqtp7GWaUJA1hmCP0rcB0VZ2sqteAQ8DOyzuWJGmphgn6OuD0wPaZbt9s70vylSSPJ9kw1wsl2ZNkKsnUuXPnljGuJGk+o7oo+sfAeFW9DfgC8Mhci6rqYFVNVNXE2Nicf+GGJGmZhgn6WWDwiHt9t++iqvpGVX2723wQ+NHRjCdJGtYwQT8CbE6yKckaYBcwObggyY0DmzuA46MbUZI0jEU/5VJV55PsBZ4CVgEPV9WxJPuBqaqaBH4xyQ7gPPBN4O7LOLMkaQ5D3fpfVYeBw7P23T/w+D7gvtGOJklaCu8UlaRGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGDBX0JNuSnEgynWTfAuvel6SSTIxuREnSMBYNepJVwAFgO7AF2J1kyxzrbgDuAZ4d9ZCSpMUNc4S+FZiuqpNV9RpwCNg5x7rfBD4B/NcI55MkDWmYoK8DTg9sn+n2XZTkFmBDVT250Asl2ZNkKsnUuXPnljysJGl+l3xRNMl1wAPALy22tqoOVtVEVU2MjY1d6ltLkgYME/SzwIaB7fXdvgtuAN4K/GWSU8CPA5NeGJWkK2uYoB8BNifZlGQNsAuYvPBkVf1bVa2tqvGqGgeeAXZU1dRlmViSNKdFg15V54G9wFPAceCxqjqWZH+SHZd7QEnScFYPs6iqDgOHZ+27f56177r0sSRJS+WdopLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUiKGCnmRbkhNJppPsm+P5n0/yQpLnk3wpyZbRjypJWsiiQU+yCjgAbAe2ALvnCPajVfUjVfV24JPAAyOfVJK0oGGO0LcC01V1sqpeAw4BOwcXVNUrA5vXAzW6ESVJw1g9xJp1wOmB7TPAO2YvSvIR4F5gDfDuuV4oyR5gD8DGjRuXOqskaQEjuyhaVQeq6geBXwF+fZ41B6tqoqomxsbGRvXWkiSGC/pZYMPA9vpu33wOAe+9lKEkSUs3TNCPAJuTbEqyBtgFTA4uSLJ5YPNO4B9GN6IkaRiLnkOvqvNJ9gJPAauAh6vqWJL9wFRVTQJ7k9wO/DfwLeCDl3NoSdL/N8xFUarqMHB41r77Bx7fM+K5JElL5J2iktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktSIoYKeZFuSE0mmk+yb4/l7k7yY5CtJ/iLJW0Y/qiRpIYsGPckq4ACwHdgC7E6yZdayLwMTVfU24HHgk6MeVJK0sGGO0LcC01V1sqpeAw4BOwcXVNXTVfWf3eYzwPrRjilJWswwQV8HnB7YPtPtm8+HgD+9lKEkSUu3epQvluT9wATwU/M8vwfYA7Bx48ZRvrUkXfOGOUI/C2wY2F7f7XudJLcDvwbsqKpvz/VCVXWwqiaqamJsbGw580qS5jFM0I8Am5NsSrIG2AVMDi5IcjPwe8zE/KXRjylJWsyiQa+q88Be4CngOPBYVR1Lsj/Jjm7ZbwFvBD6b5Pkkk/O8nCTpMhnqHHpVHQYOz9p3/8Dj20c8lyRpibxTVJIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIasbrvAaSVbnzfk72+/6mP39nr++vqMdQRepJtSU4kmU6yb47nb03yXJLzSe4a/ZiSpMUsGvQkq4ADwHZgC7A7yZZZy/4RuBt4dNQDSpKGM8wpl63AdFWdBEhyCNgJvHhhQVWd6p77zmWYUZI0hGFOuawDTg9sn+n2LVmSPUmmkkydO3duOS8hSZrHFf2US1UdrKqJqpoYGxu7km8tSc0bJuhngQ0D2+u7fZKkFWSYoB8BNifZlGQNsAuYvLxjSZKWatGLolV1Psle4ClgFfBwVR1Lsh+YqqrJJD8GPAF8P/AzST5WVT98WSfXkvl5aqltQ91YVFWHgcOz9t0/8PgIM6diJEk98U5RSdecVn9bNejSVazVMGl5/HIuSWqEQZekRhh0SWqEQZekRhh0SWqEQZekRhh0SWqEQZekRnhj0Yh5o4ekvniELkmNMOiS1AiDLkmNMOiS1AiDLkmNMOiS1AiDLkmNMOiS1AiDLkmN8E5RrQjeYStdOo/QJakRBl2SGmHQJakRnkOXdFl4XeTK8whdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEUMFPcm2JCeSTCfZN8fz35XkM93zzyYZH/WgkqSFLRr0JKuAA8B2YAuwO8mWWcs+BHyrqn4I+G3gE6MeVJK0sGGO0LcC01V1sqpeAw4BO2et2Qk80j1+HLgtSUY3piRpMamqhRckdwHbqurD3fYHgHdU1d6BNX/XrTnTbX+tW/PyrNfaA+zpNm8CTozqX2SJ1gIvL7qqH862PM62PM62PH3O9paqGpvriSt6639VHQQOXsn3nEuSqaqa6HuOuTjb8jjb8jjb8qzU2YY55XIW2DCwvb7bN+eaJKuBNwHfGMWAkqThDBP0I8DmJJuSrAF2AZOz1kwCH+we3wV8sRY7lyNJGqlFT7lU1fkke4GngFXAw1V1LMl+YKqqJoGHgD9MMg18k5nor2S9n/ZZgLMtj7Mtj7Mtz4qcbdGLopKkq4N3ikpSIwy6JDXimgr6Yl9h0KckDyd5qftM/4qSZEOSp5O8mORYknv6numCJN+d5G+S/G0328f6nmlQklVJvpzkT/qeZbYkp5K8kOT5JFN9zzMoyZuTPJ7kq0mOJ/mJvmcCSHJT9+d14eeVJB/te64Lrplz6N1XGPw9cAdwhplP7+yuqhd7HayT5FbgVeAPquqtfc8zKMmNwI1V9VySG4CjwHtXwp9dd0fy9VX1apI3AF8C7qmqZ3oeDYAk9wITwPdV1Xv6nmdQklPAxOwbAFeCJI8Af11VD3afrvveqvrXvuca1DXlLDM3UX6973ng2jpCH+YrDHpTVX/FzCeEVpyq+qeqeq57/O/AcWBdv1PNqBmvdptv6H5WxFFKkvXAncCDfc9yNUnyJuBWZj49R1W9ttJi3rkN+NpKiTlcW0FfB5we2D7DConS1aT7Js2bgWf7neT/dKc1ngdeAr5QVStltt8Bfhn4Tt+DzKOAP09ytPtajpViE3AO+P3udNWDSa7ve6g57AI+3fcQg66loOsSJXkj8Dngo1X1St/zXFBV/1NVb2fmLuatSXo/ZZXkPcBLVXW071kW8JNVdQsz36T6ke6030qwGrgF+N2quhn4D2ClXfNaA+wAPtv3LIOupaAP8xUGmkd3fvpzwKeq6vN9zzOX7tfyp4Ftfc8CvBPY0Z2nPgS8O8kf9TvS61XV2e6fLwFPMHNaciU4A5wZ+E3rcWYCv5JsB56rqn/pe5BB11LQh/kKA82hu/D4EHC8qh7oe55BScaSvLl7/D3MXPT+ar9TQVXdV1Xrq2qcmf/WvlhV7+95rIuSXN9d4KY7nfHTwIr4hFVV/TNwOslN3a7bgN4vwM+ymxV2ugWu8Lct9mm+rzDoeayLknwaeBewNskZ4Deq6qF+p7roncAHgBe6c9UAv1pVh3uc6YIbgUe6TxxcBzxWVSvuI4Ir0A8AT3R/bcFq4NGq+rN+R3qdXwA+1R18nQR+tud5Lur+B3gH8HN9zzLbNfOxRUlq3bV0ykWSmmbQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGvG/zDNyy7hwiP4AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E388oVJbqfEZ"
      },
      "source": [
        "<a id='section2.0.5'></a>\n",
        "### [Permutation Feature Importance](https://christophm.github.io/interpretable-ml-book/feature-importance.html)\n",
        "\n",
        "Permutation feature importance is a technique for calculating relative importance scores that is independent of the model used. \n",
        "It measures the increase in the prediction error of the model after we permuted the feature's values, which breaks the relationship between the feature and the true outcome.\n",
        "\n",
        "The concept is really straightforward: We measure the importance of a feature by calculating the increase in the model's prediction error after permuting the feature. A feature is \"important\" if shuffling its values increases the model error, because in this case the model relied on the feature for the prediction. A feature is \"unimportant\" if shuffling its values leaves the model error unchanged, because in this case the model ignored the feature for the prediction. \n",
        "\n",
        "Permutation feature selection can be used via the [*permutation_importance()*](https://scikit-learn.org/stable/modules/generated/sklearn.inspection.permutation_importance.html) function that take a fit model, a dataset and a scoring function. \n",
        "\n",
        "Let's try this approach with an algorithm that doesn't support feature selection natively, [KNN](https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm) (K-Nearest Neighbors).\n",
        "\n",
        "**<ins>Permutation Feature Importance for Regression</ins>**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "URgfoDbOsXUo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 403
        },
        "outputId": "d01aaad5-cb6e-4dac-975f-faf84cd2a09a"
      },
      "source": [
        "# permutation feature importance with knn for regression\n",
        "from sklearn.neighbors import KNeighborsRegressor\n",
        "from sklearn.inspection import permutation_importance\n",
        "from matplotlib import pyplot as plt \n",
        "# define the model\n",
        "model = KNeighborsRegressor()\n",
        "# fit the model\n",
        "model.fit(X_reg, y_reg)\n",
        "# perform permutation importance\n",
        "results = permutation_importance(model, X_reg, y_reg, scoring='neg_mean_squared_error')\n",
        "# get importance\n",
        "importance = results.importances_mean\n",
        "# summarize feature importance\n",
        "for i,v in enumerate(importance):\n",
        "\tprint('Feature: %0d, Score: %.5f' % (i,v))\n",
        "# plot feature importance\n",
        "plt.bar([x for x in range(len(importance))], importance)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Feature: 0, Score: 39.26371\n",
            "Feature: 1, Score: 1970.76947\n",
            "Feature: 2, Score: 32.07404\n",
            "Feature: 3, Score: 2384.33933\n",
            "Feature: 4, Score: 2505.48063\n",
            "Feature: 5, Score: 53.14065\n",
            "Feature: 6, Score: 64.58060\n",
            "Feature: 7, Score: 44.49132\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAPe0lEQVR4nO3dcayddX3H8fdHim5DNzB0TdfWlZjOpC4ZkJvigjFsRCiwWEwWA8mwIS71D1ggM1mK/+A0LPwxdTNxJBU6S4Y0TCQ22ogdI3H+gXLLGFCQcYcltCm0rk50Jhrcd3+c39Vjue29vb29z8Xf+5WcnOf5Pr/nPN9Tms95+nuec0hVIUnqwxuGbkCStHgMfUnqiKEvSR0x9CWpI4a+JHVk2dANnMi5555ba9euHboNSXpd2bt37/eqavlM25Z06K9du5bJycmh25Ck15UkLxxvm9M7ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOzhn6SNUkeTvJ0kn1Jbmr1jyU5mOTx9rhybJ9bkkwleTbJ5WP1ja02lWTr6XlLkqTjmct9+q8CH6mqx5K8BdibZE/b9umq+tvxwUnWA9cA7wR+B/iXJL/XNn8WeC9wAHg0ya6qenoh3ogkaXazhn5VHQIOteUfJnkGWHWCXTYBO6vqJ8B3k0wBG9q2qap6HiDJzjbW0JekRXJS38hNsha4APgWcDFwY5IPApOM/jXwfUYfCI+M7XaAX3xIvHhM/aIZjrEF2ALwtre97WTakwa1dutXBzv2/tuvGuzYen2Z84XcJG8G7gdurqpXgDuAtwPnM/qXwCcXoqGq2lZVE1U1sXz5jD8dIUmapzmd6Sc5k1Hg31NVXwKoqpfHtn8O+EpbPQisGdt9datxgrokaRHM5e6dAHcBz1TVp8bqK8eGvR94qi3vAq5J8qYk5wHrgG8DjwLrkpyX5I2MLvbuWpi3IUmai7mc6V8MXAc8meTxVvsocG2S84EC9gMfBqiqfUnuY3SB9lXghqr6GUCSG4EHgTOA7VW1bwHfiyRpFnO5e+ebQGbYtPsE+9wG3DZDffeJ9pMknV5+I1eSOmLoS1JHDH1J6oihL0kdWdL/j1zpWH7rVTo1nulLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR2ZNfSTrEnycJKnk+xLclOrvzXJniTPtedzWj1JPpNkKskTSS4ce63NbfxzSTafvrclSZrJXM70XwU+UlXrgXcBNyRZD2wFHqqqdcBDbR3gCmBde2wB7oDRhwRwK3ARsAG4dfqDQpK0OGYN/ao6VFWPteUfAs8Aq4BNwI42bAdwdVveBNxdI48AZydZCVwO7Kmqo1X1fWAPsHFB340k6YROak4/yVrgAuBbwIqqOtQ2vQSsaMurgBfHdjvQaserH3uMLUkmk0weOXLkZNqTJM1izqGf5M3A/cDNVfXK+LaqKqAWoqGq2lZVE1U1sXz58oV4SUlSM6fQT3Imo8C/p6q+1Movt2kb2vPhVj8IrBnbfXWrHa8uSVoky2YbkCTAXcAzVfWpsU27gM3A7e35y2P1G5PsZHTR9gdVdSjJg8DfjF28vQy4ZWHexuvP2q1fHezY+2+/arBjSxrWrKEPXAxcBzyZ5PFW+yijsL8vyYeAF4APtG27gSuBKeDHwPUAVXU0ySeAR9u4j1fV0QV5F5KkOZk19Kvqm0COs/nSGcYXcMNxXms7sP1kGpQkLRy/kStJHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR2ZNfSTbE9yOMlTY7WPJTmY5PH2uHJs2y1JppI8m+TysfrGVptKsnXh34okaTZzOdP/PLBxhvqnq+r89tgNkGQ9cA3wzrbPPyQ5I8kZwGeBK4D1wLVtrCRpES2bbUBVfSPJ2jm+3iZgZ1X9BPhukilgQ9s2VVXPAyTZ2cY+fdIdS5Lm7VTm9G9M8kSb/jmn1VYBL46NOdBqx6u/RpItSSaTTB45cuQU2pMkHWu+oX8H8HbgfOAQ8MmFaqiqtlXVRFVNLF++fKFeVpLEHKZ3ZlJVL08vJ/kc8JW2ehBYMzZ0datxgrokaZHM60w/ycqx1fcD03f27AKuSfKmJOcB64BvA48C65Kcl+SNjC727pp/25Kk+Zj1TD/JvcAlwLlJDgC3ApckOR8oYD/wYYCq2pfkPkYXaF8Fbqiqn7XXuRF4EDgD2F5V+xb83UiSTmgud+9cO0P5rhOMvw24bYb6bmD3SXUnSVpQfiNXkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1JFZQz/J9iSHkzw1Vntrkj1JnmvP57R6knwmyVSSJ5JcOLbP5jb+uSSbT8/bkSSdyFzO9D8PbDymthV4qKrWAQ+1dYArgHXtsQW4A0YfEsCtwEXABuDW6Q8KSdLimTX0q+obwNFjypuAHW15B3D1WP3uGnkEODvJSuByYE9VHa2q7wN7eO0HiSTpNJvvnP6KqjrUll8CVrTlVcCLY+MOtNrx6q+RZEuSySSTR44cmWd7kqSZnPKF3KoqoBagl+nX21ZVE1U1sXz58oV6WUkS8w/9l9u0De35cKsfBNaMjVvdaserS5IW0XxDfxcwfQfOZuDLY/UPtrt43gX8oE0DPQhcluScdgH3slaTJC2iZbMNSHIvcAlwbpIDjO7CuR24L8mHgBeAD7Thu4ErgSngx8D1AFV1NMkngEfbuI9X1bEXhyVJp9msoV9V1x5n06UzjC3ghuO8znZg+0l1J0laUH4jV5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktSRUwr9JPuTPJnk8SSTrfbWJHuSPNeez2n1JPlMkqkkTyS5cCHegCRp7hbiTP+Pqur8qppo61uBh6pqHfBQWwe4AljXHluAOxbg2JKkk3A6pnc2ATva8g7g6rH63TXyCHB2kpWn4fiSpOM41dAv4OtJ9ibZ0morqupQW34JWNGWVwEvju17oNV+SZItSSaTTB45cuQU25MkjVt2ivu/u6oOJvltYE+S74xvrKpKUifzglW1DdgGMDExcVL7SpJO7JTO9KvqYHs+DDwAbABenp62ac+H2/CDwJqx3Ve3miRpkcw79JOcleQt08vAZcBTwC5gcxu2GfhyW94FfLDdxfMu4Adj00CSpEVwKtM7K4AHkky/zheq6mtJHgXuS/Ih4AXgA238buBKYAr4MXD9KRxbkjQP8w79qnoe+IMZ6v8NXDpDvYAb5ns8SdKp8xu5ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSPLhm5AUt/Wbv3qYMfef/tVgx17KIa+1AGDVdMMfUk6jl/FD8tFD/0kG4G/B84A7qyq20/XsX4V/4MtBv/cpF9di3ohN8kZwGeBK4D1wLVJ1i9mD5LUs8W+e2cDMFVVz1fVT4GdwKZF7kGSupWqWryDJX8KbKyqP2/r1wEXVdWNY2O2AFva6juAZxetwV92LvC9gY49G3ubH3ubH3ubnyF7+92qWj7ThiV3IbeqtgHbhu4jyWRVTQzdx0zsbX7sbX7sbX6Wam+LPb1zEFgztr661SRJi2CxQ/9RYF2S85K8EbgG2LXIPUhStxZ1eqeqXk1yI/Ago1s2t1fVvsXs4SQMPsV0AvY2P/Y2P/Y2P0uyt0W9kCtJGpY/uCZJHTH0Jakjhv4MkmxM8mySqSRbh+5nWpLtSQ4neWroXo6VZE2Sh5M8nWRfkpuG7mlakl9L8u0k/9F6++uhezpWkjOS/HuSrwzdy7gk+5M8meTxJJND9zMuydlJvpjkO0meSfKHQ/cEkOQd7c9r+vFKkpuH7muac/rHaD8V8Z/Ae4EDjO44uraqnh60MSDJe4AfAXdX1e8P3c+4JCuBlVX1WJK3AHuBq5fIn1uAs6rqR0nOBL4J3FRVjwzc2s8l+UtgAvjNqvqTofuZlmQ/MFFVS+4LUEl2AP9WVXe2uwF/o6r+Z+i+xrU8OcjoS6gvDN0PeKY/kyX7UxFV9Q3g6NB9zKSqDlXVY235h8AzwKphuxqpkR+11TPbY8mc7SRZDVwF3Dl0L68XSX4LeA9wF0BV/XSpBX5zKfBfSyXwwdCfySrgxbH1AyyR8Hq9SLIWuAD41rCd/EKbPnkcOAzsqaol0xvwd8BfAf83dCMzKODrSfa2n0hZKs4DjgD/2KbF7kxy1tBNzeAa4N6hmxhn6GtBJXkzcD9wc1W9MnQ/06rqZ1V1PqNvgW9IsiSmx5L8CXC4qvYO3ctxvLuqLmT0y7g3tCnGpWAZcCFwR1VdAPwvsGSuvwG0Kaf3Af88dC/jDP3X8qci5qnNl98P3FNVXxq6n5m0KYCHgY1D99JcDLyvzZ3vBP44yT8N29IvVNXB9nwYeIDR9OdScAA4MPYvti8y+hBYSq4AHquql4duZJyh/1r+VMQ8tIuldwHPVNWnhu5nXJLlSc5uy7/O6CL9d4btaqSqbqmq1VW1ltHftX+tqj8buC0AkpzVLsrTpk4uA5bEnWNV9RLwYpJ3tNKlwOA3DRzjWpbY1A4swV/ZHNpS/qmIJPcClwDnJjkA3FpVdw3b1c9dDFwHPNnmzgE+WlW7B+xp2kpgR7uT4g3AfVW1pG6NXKJWAA+MPs9ZBnyhqr42bEu/5C+Ae9rJ2fPA9QP383PtQ/K9wIeH7uVY3rIpSR1xekeSOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI78P5OxI0eBalBSAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xFlEdNEvsXas"
      },
      "source": [
        "**<ins>Permutation Feature Importance for Classification</ins>**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9m00b5nUtjGr",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 403
        },
        "outputId": "302c3117-a4e7-433a-b991-6ecf1c4ffd31"
      },
      "source": [
        "# permutation feature importance with knn for classification\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.inspection import permutation_importance\n",
        "from matplotlib import pyplot as plt\n",
        "# define the model\n",
        "model = KNeighborsClassifier()\n",
        "# fit the model\n",
        "model.fit(X_clf, y_clf)\n",
        "# perform permutation importance\n",
        "results = permutation_importance(model, X_clf, y_clf, scoring='accuracy')\n",
        "# get importance\n",
        "importance = results.importances_mean\n",
        "# summarize feature importance\n",
        "for i,v in enumerate(importance):\n",
        "\tprint('Feature: %0d, Score: %.5f' % (i,v))\n",
        "# plot feature importance\n",
        "plt.bar([x for x in range(len(importance))], importance)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Feature: 0, Score: 0.07960\n",
            "Feature: 1, Score: 0.02720\n",
            "Feature: 2, Score: 0.00660\n",
            "Feature: 3, Score: 0.03100\n",
            "Feature: 4, Score: 0.11100\n",
            "Feature: 5, Score: 0.05320\n",
            "Feature: 6, Score: 0.03720\n",
            "Feature: 7, Score: 0.06260\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAPuklEQVR4nO3df6xfd13H8efLlg0YuuF2Nbhu3iabJEWMYO00IBIaRpfhinFLWvxRzZJiwgyEGCz+MWDyx2YM8w+ncWEjdfzo5pCkcZWJGQlIYPRu/JjdqF5KYa3ouq0Oixml4+0f31P4+vV296z3tue7z56P5Oae8/l8zve8b9O8zrmf8+OmqpAktetHhi5AknRqGfSS1DiDXpIaZ9BLUuMMeklq3MqhC5h03nnn1ezs7NBlSNKzyn333fdoVc0s1Dd1QT87O8vc3NzQZUjSs0qSb5yoz6kbSWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklq3NQ9GSs9m8xuu2uwfe+//vLB9q1nF8/oJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mN6xX0STYk2ZtkPsm2Bfpfk+T+JMeSXDnRtyXJv3VfW5arcElSP4sGfZIVwE3AZcAaYHOSNRPDvgn8LvCRiW1/HHg3cAmwDnh3khcvvWxJUl99zujXAfNVta+qjgI7gI3jA6pqf1V9Bfj+xLZvAD5ZVY9X1WHgk8CGZahbktRTn6A/H3h4bP1A19ZHr22TbE0yl2Tu0KFDPT9aktTHVFyMraqbq2ptVa2dmZkZuhxJakqfoD8IXDC2vqpr62Mp20qSlkGfoN8NXJxkdZIzgE3Azp6ffzdwaZIXdxdhL+3aJEmnyaJBX1XHgGsYBfRDwB1VtSfJdUmuAEjyi0kOAFcBf51kT7ft48CfMDpY7Aau69okSafJyj6DqmoXsGui7dqx5d2MpmUW2vZW4NYl1ChJWoKpuBgrSTp1DHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXG9gj7JhiR7k8wn2bZA/5lJbu/6700y27U/L8n2JA8keSjJu5a3fEnSYhYN+iQrgJuAy4A1wOYkayaGXQ0crqqLgBuBG7r2q4Azq+rlwC8Abzl+EJAknR59zujXAfNVta+qjgI7gI0TYzYC27vlO4H1SQIUcFaSlcALgKPAt5elcklSL32C/nzg4bH1A13bgmOq6hjwBHAuo9D/DvAt4JvAn1XV40usWZL0DKw8xZ+/DngK+CngxcBnkvxTVe0bH5RkK7AV4MILL1zSDme33bWk7Zdi//WXD7ZvSTqRPmf0B4ELxtZXdW0Ljummac4GHgPeDHyiqr5XVY8AnwXWTu6gqm6uqrVVtXZmZuaZ/xSSpBPqE/S7gYuTrE5yBrAJ2DkxZiewpVu+ErinqorRdM3rAJKcBfwS8NXlKFyS1M+iQd/NuV8D3A08BNxRVXuSXJfkim7YLcC5SeaBdwDHb8G8CXhRkj2MDhgfrKqvLPcPIUk6sV5z9FW1C9g10Xbt2PKTjG6lnNzuyELtkqTTxydjJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGrRy6AEmaJrPb7hps3/uvv/yUfK5n9JLUuF5Bn2RDkr1J5pNsW6D/zCS3d/33Jpkd6/u5JJ9LsifJA0mev3zlS5IWs2jQJ1kB3ARcBqwBNidZMzHsauBwVV0E3Ajc0G27EvgQ8PtV9TLgtcD3lq16SdKi+pzRrwPmq2pfVR0FdgAbJ8ZsBLZ3y3cC65MEuBT4SlV9GaCqHquqp5andElSH32C/nzg4bH1A13bgmOq6hjwBHAu8DNAJbk7yf1J3rnQDpJsTTKXZO7QoUPP9GeQJD2NU30xdiXwauA3u++/nmT95KCqurmq1lbV2pmZmVNckiQ9t/QJ+oPABWPrq7q2Bcd08/JnA48xOvv/dFU9WlX/A+wCXrnUoiVJ/fUJ+t3AxUlWJzkD2ATsnBizE9jSLV8J3FNVBdwNvDzJC7sDwK8CDy5P6ZKkPhZ9YKqqjiW5hlForwBurao9Sa4D5qpqJ3ALcFuSeeBxRgcDqupwkvczOlgUsKuqhnsaQZKeg3o9GVtVuxhNu4y3XTu2/CRw1Qm2/RCjWywlSQPwyVhJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhrnX5iSGtXiX0rSyfGMXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjesV9Ek2JNmbZD7JtgX6z0xye9d/b5LZif4LkxxJ8ofLU7Ykqa9Fgz7JCuAm4DJgDbA5yZqJYVcDh6vqIuBG4IaJ/vcD/7D0ciVJz1SfM/p1wHxV7auqo8AOYOPEmI3A9m75TmB9kgAkeRPwdWDP8pQsSXom+gT9+cDDY+sHurYFx1TVMeAJ4NwkLwL+CHjv0+0gydYkc0nmDh061Ld2SVIPp/pi7HuAG6vqyNMNqqqbq2ptVa2dmZk5xSVJ0nPLyh5jDgIXjK2v6toWGnMgyUrgbOAx4BLgyiR/CpwDfD/Jk1X1F0uuXJLUS5+g3w1cnGQ1o0DfBLx5YsxOYAvwOeBK4J6qKuBXjg9I8h7giCEvSafXokFfVceSXAPcDawAbq2qPUmuA+aqaidwC3BbknngcUYHA0nSFOhzRk9V7QJ2TbRdO7b8JHDVIp/xnpOoT1KDZrfdNdi+919/+WD7HopPxkpS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY3r9a4baUi+F0VaGs/oJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuO8vfI08jZBSUPwjF6SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhrXK+iTbEiyN8l8km0L9J+Z5Pau/94ks13765Pcl+SB7vvrlrd8SdJiFg36JCuAm4DLgDXA5iRrJoZdDRyuqouAG4EbuvZHgV+rqpcDW4DblqtwSVI/fc7o1wHzVbWvqo4CO4CNE2M2Atu75TuB9UlSVV+sqn/v2vcAL0hy5nIULknqp0/Qnw88PLZ+oGtbcExVHQOeAM6dGPMbwP1V9d3JHSTZmmQuydyhQ4f61i5J6uG0XIxN8jJG0zlvWai/qm6uqrVVtXZmZuZ0lCRJzxl9gv4gcMHY+qqubcExSVYCZwOPdeurgI8Dv1NVX1tqwZKkZ6ZP0O8GLk6yOskZwCZg58SYnYwutgJcCdxTVZXkHOAuYFtVfXa5ipYk9bdo0Hdz7tcAdwMPAXdU1Z4k1yW5oht2C3BuknngHcDxWzCvAS4Crk3ype7rJ5b9p5AknVCvvzBVVbuAXRNt144tPwlctcB27wPet8QaJUlL4JOxktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuN6/c1YtW92212D7Xv/9ZcPtm/pucAzeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJalyvoE+yIcneJPNJti3Qf2aS27v+e5PMjvW9q2vfm+QNy1e6JKmPRYM+yQrgJuAyYA2wOcmaiWFXA4er6iLgRuCGbts1wCbgZcAG4C+7z5MknSZ9zujXAfNVta+qjgI7gI0TYzYC27vlO4H1SdK176iq71bV14H57vMkSadJn5eanQ88PLZ+ALjkRGOq6liSJ4Bzu/bPT2x7/uQOkmwFtnarR5Ls7VX98jsPePRkN84Ny1jJ/2dtJ8faTo61nZwha/vpE3VMxdsrq+pm4Oah60gyV1Vrh65jIdZ2cqzt5FjbyZnW2vpM3RwELhhbX9W1LTgmyUrgbOCxnttKkk6hPkG/G7g4yeokZzC6uLpzYsxOYEu3fCVwT1VV176puytnNXAx8IXlKV2S1MeiUzfdnPs1wN3ACuDWqtqT5Dpgrqp2ArcAtyWZBx5ndDCgG3cH8CBwDHhrVT11in6W5TD49NHTsLaTY20nx9pOzlTWltGJtySpVT4ZK0mNM+glqXEGfWex1zwMJcmtSR5J8i9D1zIpyQVJPpXkwSR7krxt6JqOS/L8JF9I8uWutvcOXdOkJCuSfDHJ3w9dy7gk+5M8kORLSeaGrmdcknOS3Jnkq0keSvLLQ9cEkOSl3b/X8a9vJ3n70HUd5xw9P3jNw78Cr2f0UNduYHNVPThoYUCS1wBHgL+pqp8dup5xSV4CvKSq7k/yo8B9wJum5N8twFlVdSTJ84B/Bt5WVZ9fZNPTJsk7gLXAj1XVG4eu57gk+4G1VXXSD/6cKkm2A5+pqg90dwG+sKr+a+i6xnV5chC4pKq+MXQ94Bn9cX1e8zCIqvo0ozuZpk5Vfauq7u+W/xt4iAWefB5CjRzpVp/XfU3NWU2SVcDlwAeGruXZIsnZwGsY3eVHVR2dtpDvrAe+Ni0hDwb9cQu95mEqAuvZontj6SuAe4et5Ie6qZEvAY8An6yqqakN+HPgncD3hy5kAQX8Y5L7uteTTIvVwCHgg92U1weSnDV0UQvYBHx06CLGGfRasiQvAj4GvL2qvj10PcdV1VNV9fOMnshel2Qqpr6SvBF4pKruG7qWE3h1Vb2S0Rtr39pNH06DlcArgb+qqlcA3wGm5noaQDeddAXwt0PXMs6gH/FVDSepm//+GPDhqvq7oetZSPfr/acYvSp7GrwKuKKbC98BvC7Jh4Yt6Yeq6mD3/RHg40zPG2cPAAfGfjO7k1HwT5PLgPur6j+HLmScQT/S5zUPmtBd8LwFeKiq3j90PeOSzCQ5p1t+AaML7V8dtqqRqnpXVa2qqllG/9fuqarfGrgsAJKc1V1Yp5sWuRSYiju+quo/gIeTvLRrWs/oqftpspkpm7aBKXl75dBO9JqHgcsCIMlHgdcC5yU5ALy7qm4ZtqofeBXw28AD3Vw4wB9X1a4BazruJcD27g6IHwHuqKqpuo1xSv0k8PHRMZyVwEeq6hPDlvR//AHw4e6EbB/wewPX8wPdgfH1wFuGrmWSt1dKUuOcupGkxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXH/C8wKSSwE5jD+AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wKE0nM86CbhX"
      },
      "source": [
        "**<ins>Feature Selection with Importance</ins>**\n",
        "\n",
        "Feature importance scores can be used to find useful insights and interpret the data, but they can also be used directly to help rank and select features that are most useful. This procedure is usually referred as *Feature Selection*, and we'll look at it in more detail soon.\n",
        "\n",
        "In our case, we can show how is possible to find redundant features by using the previously shown techniques. \n",
        "\n",
        "Firstly, we can split the dataset into train and test sets, train a model on the training set, make predictions on the test set and evaluate the results by employing classification accuracy. We'll use a Logistic Regression model to fit our data. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JT3ayCWgEE15",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "fe3efca1-928f-4dce-da02-01e5e2de9d1c"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.metrics import accuracy_score\n",
        "# split into train and test sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_clf, y_clf, test_size=0.33, random_state=1)\n",
        "# fit the model\n",
        "model = LogisticRegression(solver='liblinear')\n",
        "model.fit(X_train, y_train)\n",
        "# evaluate the model\n",
        "yhat = model.predict(X_test)\n",
        "# evaluate predictions\n",
        "accuracy = accuracy_score(y_test, yhat)\n",
        "print('Accuracy: %.2f' % (accuracy*100))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy: 86.67\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tovo6nqWEOoU"
      },
      "source": [
        "In this case, we can see that our model achieved a classification accuracy of about $86.67 \\%$ using all the features in the dataset. \n",
        "\n",
        "Let's see what happens if we select only relevant features. We could use any of the feature importance scores above, but in this case we'll use the ones provided by random forest. \n",
        "\n",
        "We can use the [*SelectFromModel*](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html) class to define both the model abd the number of features to select. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L7WRDpJAGcYT"
      },
      "source": [
        "from sklearn.feature_selection import SelectFromModel\n",
        "# configure to select a subset of features\n",
        "fs = SelectFromModel(RandomForestClassifier(n_estimators=200), max_features=5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nUGHQMntGyz2"
      },
      "source": [
        "This will calculate the importance scores that can be used to rank all input features. We can then apply the method as a transform to select a subset of 5 most important features from the dataset. This transform will be applied to the training set and the test set. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EaHD0aX-Gxwl"
      },
      "source": [
        "# learn relationship from training data\n",
        "fs.fit(X_train, y_train)\n",
        "# transform train input data\n",
        "X_train_fs = fs.transform(X_train)\n",
        "# transform test input data\n",
        "X_test_fs = fs.transform(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NA4ZY3WTHloZ"
      },
      "source": [
        "We can wrap up every piece and get this code snippet. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XY6luNzeHspU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "433a8c89-313a-4762-ee4e-6eaad2c94d56"
      },
      "source": [
        "# evaluation of a model using 5 features chosen with random forest importance\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.feature_selection import SelectFromModel\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "# feature selection\n",
        "def select_features(X_train, y_train, X_test):\n",
        "\t# configure to select a subset of features\n",
        "\tfs = SelectFromModel(RandomForestClassifier(n_estimators=1000), max_features=5)\n",
        "\t# learn relationship from training data\n",
        "\tfs.fit(X_train, y_train)\n",
        "\t# transform train input data\n",
        "\tX_train_fs = fs.transform(X_train)\n",
        "\t# transform test input data\n",
        "\tX_test_fs = fs.transform(X_test)\n",
        "\treturn X_train_fs, X_test_fs, fs\n",
        "\n",
        "# split into train and test sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_clf, y_clf, test_size=0.33, random_state=1)\n",
        "# feature selection\n",
        "X_train_fs, X_test_fs, fs = select_features(X_train, y_train, X_test)\n",
        "# fit the model\n",
        "model = LogisticRegression(solver='liblinear')\n",
        "model.fit(X_train_fs, y_train)\n",
        "# evaluate the model\n",
        "yhat = model.predict(X_test_fs)\n",
        "# evaluate predictions\n",
        "accuracy = accuracy_score(y_test, yhat)\n",
        "print('Accuracy: %.2f' % (accuracy*100))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy: 86.36\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C0HeQ28sIBbc"
      },
      "source": [
        "In this case, we can see that the model achieves the same performance on the dataset, although with almost half of the features.\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lfhV-Dr7QHl9"
      },
      "source": [
        "<a id='section2.1'></a>\n",
        "## **Feature Selection**\n",
        "\n",
        "> Not all features are created equal\n",
        "\n",
        "> *Zhe Chen*\n",
        "\n",
        "There would always be some features which are less important with respect to a specific problem. Those irrelevant features need to be removed. \n",
        "*Feature selection* addresses these problems by automatically selecting a subset that are most useful to the problem. \n",
        "\n",
        "Most of the times the reduction in the number of input variables shrinks the computational cost of modeling, but sometimes it might happen that it also improves the performance of the model. \n",
        "\n",
        "Among the large amount of feature selection methods we'll focus mainly on statistical-based ones. They involve evaluating the relationship between each input variable and the target variable using statistics. These methods are usually fast and effective, the only issue is that statistical measures depends on the data type of both input and output variables. \n",
        "\n",
        "The classes in the ``sklearn.feature_selection`` module can be used for feature selection/dimensionality reduction on sample sets. \n",
        "\n",
        "<a id='section2.1.0'></a>\n",
        "### [Removing Features with Low Variance](https://chrisalbon.com/machine_learning/feature_selection/variance_thresholding_for_feature_selection/)\n",
        "\n",
        "Whenever you want to go for a simple approach, there's always a *threshold* involved. ``VarianceThreshold``is a simple baseline approach to select features. It removes all features whose variance doesn't reach a certain threshold. \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qiFEa7j6R28S",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        },
        "outputId": "38c9dd1c-3b9f-4c47-8269-8cfdae55df4f"
      },
      "source": [
        "from sklearn import datasets\n",
        "from sklearn.feature_selection import VarianceThreshold\n",
        "#Load Data \n",
        "iris = datasets.load_iris()\n",
        "# Create features and target \n",
        "X = iris.data\n",
        "y = iris.target\n",
        "# Conduct Variance Thresholding \n",
        "thresholder = VarianceThreshold(threshold = .6)\n",
        "X_high_variance = thresholder.fit_transform(X)\n",
        "# View High Variance Features \n",
        "X_high_variance[0:5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[5.1, 1.4],\n",
              "       [4.9, 1.4],\n",
              "       [4.7, 1.3],\n",
              "       [4.6, 1.5],\n",
              "       [5. , 1.4]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_EbcObwlowNX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        },
        "outputId": "cfdf9e1d-53ab-42d8-81c9-95d8acd88baa"
      },
      "source": [
        "X[0:5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[5.1, 3.5, 1.4, 0.2],\n",
              "       [4.9, 3. , 1.4, 0.2],\n",
              "       [4.7, 3.2, 1.3, 0.2],\n",
              "       [4.6, 3.1, 1.5, 0.2],\n",
              "       [5. , 3.6, 1.4, 0.2]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8lQBM66BpDVn"
      },
      "source": [
        "<a id='section2.1.1'></a>\n",
        "### Univariate Feature Selection\n",
        "Univariate feature selection examines each feature individually to determine the strength of the relationship of the feature with the response variable. \n",
        "\n",
        "There are a few different options for univariate selection: \n",
        "\n",
        "* [SelectKBest](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest) removes all but the *k* highest scoring features; \n",
        "* [SelectPercentile](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectPercentile.html#sklearn.feature_selection.SelectPercentile) removes all but a user-specified highest scoring percentage of features; \n",
        "* [SelectFpr](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFpr.html#sklearn.feature_selection.SelectFpr), [SelectFdr](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFdr.html#sklearn.feature_selection.SelectFdr), [SelectFwe](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFwe.html#sklearn.feature_selection.SelectFwe) are some common univariate statistical tests\n",
        "* [GenericUnivariateSelect](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.GenericUnivariateSelect.html#sklearn.feature_selection.GenericUnivariateSelect) allows to select the best univariate selection strategy with hyper-parameter search. \n",
        "\n",
        "We can perform $\\chi^2$ test to the samples to retrieve only the two best features:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8VNEUzQ2nr86",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "0ead5eed-c4fe-4c03-8aa8-1fc8d641301a"
      },
      "source": [
        "from sklearn.datasets import load_iris\n",
        "from sklearn.feature_selection import SelectKBest\n",
        "from sklearn.feature_selection import chi2\n",
        "# Load iris dataset\n",
        "X, y = load_iris(return_X_y=True)\n",
        "print(X.shape)\n",
        "# retrieve the two best features by chi-squared test\n",
        "X_new = SelectKBest(chi2, k=2).fit_transform(X, y)\n",
        "print(X_new.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(150, 4)\n",
            "(150, 2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1cgimZ4rv89R"
      },
      "source": [
        "We have different scoring functions for regression and classification: \n",
        "* Regression: [f_regression](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html#sklearn.feature_selection.f_regression), [mutual_info_regression](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_regression.html#sklearn.feature_selection.mutual_info_regression)\n",
        "\n",
        "* Classification: [chi2](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html#sklearn.feature_selection.chi2), [f_classif](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_classif.html#sklearn.feature_selection.f_classif), [mutual_info_classif](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#sklearn.feature_selection.mutual_info_classif)\n",
        "\n",
        "<a id='section2.1.2'></a>\n",
        "### Recursive Feature Elimination\n",
        "\n",
        "Recursive Feature Elimination (RFE) as its name suggests recursively removes features, builds a model using the remaining attributes and calculates model accuracy. RFE is able to work out the combination of attributes that contribute to the prediction on the target variable. \n",
        "\n",
        "Given an external estimator that assigns weights to features (e.g., the coefficients of a linear model), the goal of recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ep3dTB8vzwil",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        },
        "outputId": "b8307bfd-eb45-4d79-981e-ab5c7a6a5860"
      },
      "source": [
        "from sklearn.svm import LinearSVC\n",
        "from sklearn.feature_selection import RFE\n",
        "from sklearn import datasets\n",
        "# Load Dataset\n",
        "dataset = datasets.load_iris()\n",
        "# Support Vectore Machine classifier\n",
        "svm = LinearSVC()\n",
        "# create the RFE model for the svm classifier \n",
        "# and select attributes\n",
        "rfe = RFE(svm, 2)\n",
        "rfe = rfe.fit(dataset.data, dataset.target)\n",
        "# print summaries for the selection of attributes\n",
        "print(rfe.support_)\n",
        "print(rfe.ranking_)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[False  True False  True]\n",
            "[3 1 2 1]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/svm/_base.py:947: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
            "  \"the number of iterations.\", ConvergenceWarning)\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/svm/_base.py:947: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
            "  \"the number of iterations.\", ConvergenceWarning)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NhdoOMZh6Ail"
      },
      "source": [
        "---\n",
        "<a id='section2.2'></a>\n",
        "## **Feature Extraction (Bonus)**\n",
        "\n",
        "> Feature extraction is very different from Feature selection: the former consists in transforming arbitrary data, such as text or images, into numerical features usable for machine learning. The latter is a machine learning technique applied on these features.\n",
        "\n",
        "<a id='section2.2.1'></a>\n",
        "### Loading Features from Dicts\n",
        "\n",
        "The class [DictVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.DictVectorizer.html#sklearn.feature_extraction.DictVectorizer) transforms lists of feature-value mappings to vectors. \n",
        "\n",
        "In particular, it turns lists of mappings (dict-like objects) of feature names to feature values into *Numpy* arrays or *scipy.sparse* matrices for use with scikit-learn estimators. \n",
        "\n",
        "While not particularly fast to process, Python’s dict has the advantages of being convenient to use, being sparse (absent features need not be stored) and storing feature names in addition to values.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_d7_BhwWDQux",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "0109a37f-96bf-4344-fa35-9daf8075bd26"
      },
      "source": [
        "measurements = [\n",
        "    {'city': 'Milano', 'temperature': 33.},\n",
        "    {'city': 'Torino', 'temperature': 12.},\n",
        "    {'city': 'Roma', 'temperature': 18.},\n",
        "]\n",
        "\n",
        "from sklearn.feature_extraction import DictVectorizer\n",
        "vec = DictVectorizer()\n",
        "print(vec.fit_transform(measurements).toarray())\n",
        "print(vec.get_feature_names())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[ 1.  0.  0. 33.]\n",
            " [ 0.  0.  1. 12.]\n",
            " [ 0.  1.  0. 18.]]\n",
            "['city=Milano', 'city=Roma', 'city=Torino', 'temperature']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yuRcANbaEAPt"
      },
      "source": [
        "DictVectorizer is also a useful representation transformation for training sequence classifiers in Natural Language Processing (NLP).\n",
        "\n",
        "<a id='section2.2.2'></a>\n",
        "### Feature Hashing\n",
        "Named as one of the best hacks in Machine Learning, Feature Hashing is a fast and space-efficient way of vectorizing features, i.e. turning arbitrary features into indices in a vector or matrix. \n",
        "\n",
        "But what is Feature Hashing? In order to answer to this question we should firstly pose another question, what is out-of-core learning? \n",
        "\n",
        "Consider the problem of learning a linear model: an out-of-core algorithm learns the model without loading the whole data set in memory. This makes the algorithm very scalable, a very attractive property when dealing with a huge dataset that doesn't fit in memory. \n",
        "\n",
        "One possible implementation uses a mapping from a feature name (and possibly a value) to its corresponding coefficient. This is nothing but applying one-hot-encoding to a feature. While it looks great at first glance, it relies heavily on the way we implement the mapping from features to coefficients. This incongruence take us to hashing. \n",
        "\n",
        "Hashing usually relies on building a particular data structure, an [hash table](https://www.hackerearth.com/practice/data-structures/hash-tables/basics-of-hash-tables/tutorial/), in our case instances of [FeatureHasher](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html#sklearn.feature_extraction.FeatureHasher) apply a hash function directly to the features to determine their column index in sample matrices. The result is increased speed and reduced memory usage, at the expense of *inspectability*; the hasher does not remember what the input features looked like and has no ``inverse_transform`` method.\n",
        "\n",
        "There are some side effects though, in particular: hash functions can hash different keys to the same integer value, this is known as *collision*. Scikit-learn implements a signed hash function so that the sign of the hash value determines the sign of the value stored in the output matrix for a feature. This way, collisions are likely to cancel out rather than accumulate error, and the expected mean of any output feature’s value is zero (This mechanism is enabled by default with ``alternate_sign=True``)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s6BY479OXtkO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "c7f74a9b-7628-4c34-b458-ceced0112a71"
      },
      "source": [
        "from sklearn.feature_extraction import FeatureHasher\n",
        "\n",
        "# Creating array of dicts\n",
        "data = [\n",
        "    {'dog': -1, 'cat': 2, 'elephant': 4},\n",
        "    {'dog': 2, 'run': 5, 'cat':-7}\n",
        "]\n",
        "\n",
        "h = FeatureHasher(n_features=4)\n",
        "h.transform(data).toarray() "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 0.,  1., -4.,  2.],\n",
              "       [-5., -2.,  0., -7.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_VN4hmTGACkT"
      },
      "source": [
        "\n",
        "<a id='section2.3'></a>\n",
        "## **Feature Construction**\n",
        "There's no strict recipe, it's 99% creativity. We're gonna take a look at some use cases in the next lecture though :)\n",
        "\n",
        "You should take a look at the *Feature Extraction* part of this marvellous notebook from Beluga, one of the best *Competitions Grandmasters* on Kaggle. \n",
        "https://www.kaggle.com/gaborfodor/from-eda-to-the-top-lb-0-367\n",
        "\n",
        "---\n",
        "---\n",
        "\n",
        "<a id='section3'></a>\n",
        "# **Handling missing values**\n",
        "\n",
        "\"*Missingness*\" is almost always informative by itself, but we should tell our algorithm if a value is missing. Even if we build a model to impute our values, we are not adding any real information. We’re just reinforcing the patterns already provided by other features.\n",
        "\n",
        "Basically, there are three categories of missing data:\n",
        "\n",
        "* **MCAR** (Missing Completely At Random) where the pattern of missinginess is statistically independent of the data record. Example: you have a data set on a piece of paper and you spill coffee on the paper destroying part of the data.\n",
        "\n",
        "* **MAR** (Missing At Random) where the probability distribution of the pattern of missingness is functionally dependent upon the observable component in the record. MCAR is a special case of MAR. Example: if a child does not attend an educational assessment because the child is (genuinely) ill, this might be predictable from other data we have about the child’s health, but it would not be related to what we would have measured had the child not been ill. \n",
        "\n",
        "* **MNAR** (Missing Not at Random) which is defined as the case which is NOT MAR, or when the missingness is specifically related to what is missing. Example: a person does not attend a drug test because the person took drugs the night before.\n",
        "\n",
        "Let's see a few strategies to impute missing values, i.e. to infer them from the known part of the data. \n",
        "\n",
        "\n",
        "<a id='section3.1'></a>\n",
        "### **Univariate Feature Imputation**\n",
        "\n",
        "We can rely on scikit-learn's [SimpleImputer](https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html#sklearn.impute.SimpleImputer) class, which provides a few strategies for imputing missing values, such as : imputing by a constant value, by using statistics (mean, median, etc). \n",
        "\n",
        "Let's see how we can replace missing values (``np.nan``) using the mean value of the columns that contain the missing values. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qBIobyNg76Rk",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "f1e7a4ca-9a4f-45a9-a2bd-7a404a9b12cf"
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.impute import SimpleImputer\n",
        "# Missing expected values are Nan, we're using statitistics (mean) to impute \n",
        "imp_mean = SimpleImputer(missing_values=np.nan, strategy='mean')\n",
        "X_train = [[7, 2, 3], [4, np.nan, 6], [10, 5, 9]]\n",
        "imp_mean.fit(X_train)\n",
        "\n",
        "X_test = [[np.nan, 2, 3], [4, np.nan, 6], [10, np.nan, 9]]\n",
        "print(imp_mean.transform(X_test))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[ 7.   2.   3. ]\n",
            " [ 4.   3.5  6. ]\n",
            " [10.   3.5  9. ]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wuZ3Bhpn9d26"
      },
      "source": [
        "SimpleImputer can also be used in conjunction with *pandas* and in particular with data represented as strings or categoricals, by using ``most_frequent``or ``constant``strategy: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OveQ60as99UZ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "0ebfda53-3183-40b5-8465-92d7c4f211c6"
      },
      "source": [
        "import pandas as pd\n",
        "# Create a dataframe (categorical data)\n",
        "df = pd.DataFrame([[\"a\", \"x\"],\n",
        "                   [np.nan, \"y\"],\n",
        "                   [\"a\", np.nan],\n",
        "                   [\"b\", \"y\"]], dtype=\"category\")\n",
        "\n",
        "imp = SimpleImputer(strategy=\"most_frequent\")\n",
        "print(imp.fit_transform(df))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[['a' 'x']\n",
            " ['a' 'y']\n",
            " ['a' 'y']\n",
            " ['b' 'y']]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kbSc5ht5-Myz"
      },
      "source": [
        "\n",
        "<a id='section3.2'></a>\n",
        "### **Multivariate Feature Imputation**\n",
        "\n",
        "Another approach is to use the [IterativeImputer](https://scikit-learn.org/stable/modules/generated/sklearn.impute.IterativeImputer.html#sklearn.impute.IterativeImputer) class, which models each feature with missing values as a function of other features. How does each iteration work? At each step, a feature column is designated as output ``y``and the other feature columns are treated as inputs ``X``. Then, the regressor is used to predict the missing values of ``y``. This is done for each feature in an iterative fashion. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UgHsdX52BIZ6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "a3a6422b-9183-4059-de30-f16c6a05c8cb"
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.experimental import enable_iterative_imputer\n",
        "from sklearn.impute import IterativeImputer\n",
        "# max iter corresponds to imputation rounds\n",
        "imp = IterativeImputer(max_iter=10, random_state=0)\n",
        "X_train = [[1, 2], [3, 6], [4, 8], [np.nan, 3], [7, np.nan]]\n",
        "imp.fit(X_train)\n",
        "\n",
        "X_test = [[np.nan, 2], [6, np.nan], [np.nan, 6]]\n",
        "# the model learns that the second feature is double the first\n",
        "print(np.round(imp.transform(X_test)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[ 1.  2.]\n",
            " [ 6. 12.]\n",
            " [ 3.  6.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uYY_WQ19BuqN"
      },
      "source": [
        "IterativeImputer is very flexible, it allows you to use a variety of estimators, if you want to further delve into this class take a look at [Imputing missing values with variants of IterativeImputer](https://scikit-learn.org/stable/auto_examples/impute/plot_iterative_imputer_variants_comparison.html#sphx-glr-auto-examples-impute-plot-iterative-imputer-variants-comparison-py)\n",
        "\n",
        "\n",
        "<a id='section3.3'></a>\n",
        "### **Nearest Neighbors Imputation**\n",
        "\n",
        "The [KNNImputer](https://scikit-learn.org/stable/modules/generated/sklearn.impute.KNNImputer.html#sklearn.impute.KNNImputer) class offers imputation for filling in missing values using the k-Nearest Neighbors approach. By default it uses an euclidean distance metric that supports missing values, ``nan_euclidean_distances``. \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/MachineLearningJournalClub/HowToTackleAMLCompetition/master/img/Lecture1/1.4.gif\" width=\"600\" height=\"450\">\n",
        "\n",
        "\n",
        "The following code snippet shows how to replace missing values using the mean feature value of the two nearest neighbors of samples with missing values:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mPe1t5kbIPec",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        },
        "outputId": "935c131d-e6c9-48a3-9177-4b36768f9269"
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.impute import KNNImputer\n",
        "X = [[1, 2, np.nan], [3, 4, 3], [np.nan, 6, 5], [8, 8, 7]]\n",
        "imputer = KNNImputer(n_neighbors=2, weights=\"uniform\")\n",
        "print(X)\n",
        "print(imputer.fit_transform(X))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[1, 2, nan], [3, 4, 3], [nan, 6, 5], [8, 8, 7]]\n",
            "[[1.  2.  4. ]\n",
            " [3.  4.  3. ]\n",
            " [5.5 6.  5. ]\n",
            " [8.  8.  7. ]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "enOmLXGkKJX2"
      },
      "source": [
        "\n",
        "<a id='section4'></a>\n",
        "# **References & Additional Material** \n",
        "\n",
        "**Preprocessing Data**\n",
        "* [About Feature Scaling by Sebastian Raschka](https://sebastianraschka.com/Articles/2014_about_feature_scaling.html)\n",
        "* [Scikit-learn's Preprocessing](https://scikit-learn.org/stable/modules/preprocessing.html)\n",
        "\n",
        "**Feature Selection**\n",
        "* [Univariate](https://scikit-learn.org/stable/auto_examples/feature_selection/plot_feature_selection.html#sphx-glr-auto-examples-feature-selection-plot-feature-selection-py)\n",
        "\n",
        "* [Feature Selection using SelectFromModel and LassoCV](https://scikit-learn.org/stable/auto_examples/feature_selection/plot_select_from_model_diabetes.html#sphx-glr-auto-examples-feature-selection-plot-select-from-model-diabetes-py) \n",
        "\n",
        "* [Tree-based feature importance 1](https://scikit-learn.org/stable/auto_examples/feature_selection/plot_select_from_model_diabetes.html#sphx-glr-auto-examples-feature-selection-plot-select-from-model-diabetes-py)\n",
        "\n",
        "* [Tree-based feature importance 2](https://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html#sphx-glr-auto-examples-ensemble-plot-forest-importances-py)\n",
        "\n",
        "\n",
        "**Encoding Categorical Features**\n",
        "* [One Hot Encoding](https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f)\n",
        "\n",
        "* [Feature Hasher vs DictVectorizer](https://scikit-learn.org/stable/auto_examples/text/plot_hashing_vs_dict_vectorizer.html)\n",
        "\n",
        "**Feature Importance**\n",
        "\n",
        "* [Calculate Feature Importance by Machinelearningmastery](https://machinelearningmastery.com/calculate-feature-importance-with-python/)\n",
        "\n",
        "**Feature Selection**\n",
        "\n",
        "* [Feature Selection, Wikipedia](https://en.wikipedia.org/wiki/Feature_selection)\n",
        "\n",
        "* [Feature Selection 1, by Machinelearningmastery](https://machinelearningmastery.com/an-introduction-to-feature-selection/ )\n",
        "\n",
        "* [Feature Selection 2, by Machinelearningmastery](https://machinelearningmastery.com/feature-selection-machine-learning-python/)\n",
        "\n",
        "* [Feature Selection 3, by Machinelearningmastery](https://machinelearningmastery.com/feature-selection-with-real-and-categorical-data/)\n",
        "\n",
        "**Handling Missing Values**\n",
        "\n",
        "* [Marking Imputed Values](https://scikit-learn.org/stable/modules/impute.html#marking-imputed-values)"
      ]
    }
  ]
}