CustomMetrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning with scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the lessons so far, we have mostly just used the default `.score()` method of fitted models.  For most or all classification models, this measures accuracy.  For most or all regression models, this measures $R^2$ (coefficient of determination).  As we have mentioned, the subpackage `sklearn.metrics` contains a large number of other scorers.  Depending on your purpose, one of these might be more appropriate.\n",
    "\n",
    "In this lesson we will look at a few such metrics, but we will also develop a custom metric that is not included in scikit-learn (and presumably never will be, for reasons we will see)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from src.setup import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A slightly unbalanced classification\n",
    "\n",
    "For this lesson, we will look at the [Dermatology Data Set](https://archive.ics.uci.edu/ml/datasets/Dermatology) available from UCI.  This data contains 34 measurements of 366 patients, with each one diagnosed as having one of six skin conditions.  Our purpose in using this data is two-fold. On the one hand, we want to look at a multi-class classification problem, which we have not done extensively in these lessons.  But more interestingly at the end, we want to look at the value of non-top diagnoses, which may have utility for particular domain problems.\n",
    "\n",
    "> <small>Dua, D. and Graff, C. (2019). [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science.</small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Digression on multi-labels\n",
    "\n",
    "Note that what we present here is **not** a multi-label problem.  In some situations it is useful to identify more than one class to which a sample might belong.  In the current domain, that would be patients who have multiple skin conditions at once.  Such is possible, but this dataset is assumed not to contain that situation.  Or in another domain, we might wish to characterize a photographic image by multiple classes.  For example, an image containing both a cat and a dog would get both of these labels, but would get none of the, e.g. other 98 available labels because those things were not in the image.  Multi-label problems can be addressed with scikit-learn.\n",
    "\n",
    "See the official documentation of [multiclass and multilabel algorithms](https://scikit-learn.org/stable/modules/multiclass.html).  Note that multi-output is related to multi-label, but is a somewhat different concept.  In multi-label, any number of labels may be identified (including zero).  This is akin to one-hot encoding, but of the output (maybe \"multi-hot\" would be a good description).  \n",
    "\n",
    "In contrast, mutli-output identifies a fixed number of outputs, which we might think of as orthogonal dimensions of the output.  In a sense this is like the fixed number of input features.  For example, in the photo classification problem, we might always want to predict `(color, subject, lighting)` for every image.  So sometimes it is a \"brown dog in daylight\", other times it is a \"white cat at night.\"\n",
    "\n",
    "Basically all classification models can be transformed into multi-label algorithms by transforming the problem into a collection of one-vs-all classifiers.  For example, one model is cat-vs-not-cat; another model is dog-vs-not-dog.  Similarly for all of the stipulated 100 known classes.  If both the cat-vs-not-cat and dog-vs-not-dog models make a positive prediction, we would assign both those labels.  However, other models are inherently multi-label by their design, so this kind of transformation is irrelevant (and counter-productive, in fact) if you use those."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The dataset\n",
    "\n",
    "We get this data in somewhat raw form.  The `dermatology.data` file is a CSV with no headers.  The `dermatology.names` files contains a bit more than its name might suggest.  Beyond providing the feature names, it gives additional exposition of the dataset, such as value coding, where unknown values occur, and a few other things in prose.  I produced a code-friendly extraction of the relevant information below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Histopathological Attributes: (values 0, 1, 2, 3)\n",
    "# Clinical Attributes: (values 0, 1, 2, 3, unless indicated)\n",
    "features = [\n",
    "    \"erythema\",\n",
    "    \"scaling\",\n",
    "    \"definite borders\",\n",
    "    \"itching\",\n",
    "    \"koebner phenomenon\",\n",
    "    \"polygonal papules\",\n",
    "    \"follicular papules\",\n",
    "    \"oral mucosal involvement\",\n",
    "    \"knee and elbow involvement\",\n",
    "    \"scalp involvement\",\n",
    "    \"family history\",  # 0 or 1\n",
    "    \"melanin incontinence\",\n",
    "    \"eosinophils in the infiltrate\",\n",
    "    \"PNL infiltrate\",\n",
    "    \"fibrosis of the papillary dermis\",\n",
    "    \"exocytosis\",\n",
    "    \"acanthosis\",\n",
    "    \"hyperkeratosis\",\n",
    "    \"parakeratosis\",\n",
    "    \"clubbing of the rete ridges\",\n",
    "    \"elongation of the rete ridges\",\n",
    "    \"thinning of the suprapapillary epidermis\",\n",
    "    \"spongiform pustule\",\n",
    "    \"munro microabcess\",\n",
    "    \"focal hypergranulosis\",\n",
    "    \"disappearance of the granular layer\",\n",
    "    \"vacuolisation and damage of basal layer\",\n",
    "    \"spongiosis\",\n",
    "    \"saw-tooth appearance of retes\",\n",
    "    \"follicular horn plug\",\n",
    "    \"perifollicular parakeratosis\",\n",
    "    \"inflammatory monoluclear inflitrate\",\n",
    "    \"band-like infiltrate\",\n",
    "    \"Age\",  # linear; missing marked '?'\n",
    "    \"TARGET\"  # See mapping\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For reference and later use, the dictionary `targets` contains the class code and name of the skin condition diagnosed.  We also note here the number of observations of each condition.  They are somewhat imbalanced, which might affect the metrics we use.  That is, in this dataset, psorisis is much more common than pilaris.  I am not a dermatologist, and have no idea what the prevalence of these conditions is in the general population; there may have been selection bias in this aggregation.  That is, beyond the obvious selection bias that people with no skin conditions at all are not included."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "targets = {\n",
    "    1:\"psoriasis\",                 # 112 instances\n",
    "    2:\"seboreic dermatitis\",       # 61\n",
    "    3:\"lichen planus\",             # 72\n",
    "    4:\"pityriasis rosea\",          # 49\n",
    "    5:\"cronic dermatitis\",         # 52    \n",
    "    6:\"pityriasis rubra pilaris\",  # 20\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading in the data needs minor massaging to be ready for use.  To have a friendly DataFrame to work with, we attach the names of the features as columns.  But the missing `Age` that is marked with a questino mark needs extra clean-up.  I have decided to impute the median age for missing data. Other approaches are possible, and some models will work with missing data.  As my domain judgement, I chose this approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>erythema</th>\n",
       "      <th>scaling</th>\n",
       "      <th>definite borders</th>\n",
       "      <th>itching</th>\n",
       "      <th>koebner phenomenon</th>\n",
       "      <th>polygonal papules</th>\n",
       "      <th>follicular papules</th>\n",
       "      <th>oral mucosal involvement</th>\n",
       "      <th>knee and elbow involvement</th>\n",
       "      <th>scalp involvement</th>\n",
       "      <th>...</th>\n",
       "      <th>disappearance of the granular layer</th>\n",
       "      <th>vacuolisation and damage of basal layer</th>\n",
       "      <th>spongiosis</th>\n",
       "      <th>saw-tooth appearance of retes</th>\n",
       "      <th>follicular horn plug</th>\n",
       "      <th>perifollicular parakeratosis</th>\n",
       "      <th>inflammatory monoluclear inflitrate</th>\n",
       "      <th>band-like infiltrate</th>\n",
       "      <th>Age</th>\n",
       "      <th>TARGET</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>55.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>26.0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>45.0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>361</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>362</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>363</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>28.0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>364</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>50.0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>365</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>366 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     erythema  scaling  definite borders  itching  koebner phenomenon  \\\n",
       "0           2        2                 0        3                   0   \n",
       "1           3        3                 3        2                   1   \n",
       "2           2        1                 2        3                   1   \n",
       "3           2        2                 2        0                   0   \n",
       "4           2        3                 2        2                   2   \n",
       "..        ...      ...               ...      ...                 ...   \n",
       "361         2        1                 1        0                   1   \n",
       "362         3        2                 1        0                   1   \n",
       "363         3        2                 2        2                   3   \n",
       "364         2        1                 3        1                   2   \n",
       "365         3        2                 2        0                   0   \n",
       "\n",
       "     polygonal papules  follicular papules  oral mucosal involvement  \\\n",
       "0                    0                   0                         0   \n",
       "1                    0                   0                         0   \n",
       "2                    3                   0                         3   \n",
       "3                    0                   0                         0   \n",
       "4                    2                   0                         2   \n",
       "..                 ...                 ...                       ...   \n",
       "361                  0                   0                         0   \n",
       "362                  0                   0                         0   \n",
       "363                  2                   0                         2   \n",
       "364                  3                   0                         2   \n",
       "365                  0                   0                         0   \n",
       "\n",
       "     knee and elbow involvement  scalp involvement  ...  \\\n",
       "0                             1                  0  ...   \n",
       "1                             1                  1  ...   \n",
       "2                             0                  0  ...   \n",
       "3                             3                  2  ...   \n",
       "4                             0                  0  ...   \n",
       "..                          ...                ...  ...   \n",
       "361                           0                  0  ...   \n",
       "362                           0                  0  ...   \n",
       "363                           0                  0  ...   \n",
       "364                           0                  0  ...   \n",
       "365                           3                  3  ...   \n",
       "\n",
       "     disappearance of the granular layer  \\\n",
       "0                                      0   \n",
       "1                                      0   \n",
       "2                                      0   \n",
       "3                                      3   \n",
       "4                                      2   \n",
       "..                                   ...   \n",
       "361                                    0   \n",
       "362                                    1   \n",
       "363                                    0   \n",
       "364                                    0   \n",
       "365                                    2   \n",
       "\n",
       "     vacuolisation and damage of basal layer  spongiosis  \\\n",
       "0                                          0           3   \n",
       "1                                          0           0   \n",
       "2                                          2           3   \n",
       "3                                          0           0   \n",
       "4                                          3           2   \n",
       "..                                       ...         ...   \n",
       "361                                        0           1   \n",
       "362                                        0           1   \n",
       "363                                        3           0   \n",
       "364                                        2           0   \n",
       "365                                        0           0   \n",
       "\n",
       "     saw-tooth appearance of retes  follicular horn plug  \\\n",
       "0                                0                     0   \n",
       "1                                0                     0   \n",
       "2                                2                     0   \n",
       "3                                0                     0   \n",
       "4                                3                     0   \n",
       "..                             ...                   ...   \n",
       "361                              0                     0   \n",
       "362                              0                     0   \n",
       "363                              3                     0   \n",
       "364                              1                     0   \n",
       "365                              0                     0   \n",
       "\n",
       "     perifollicular parakeratosis  inflammatory monoluclear inflitrate  \\\n",
       "0                               0                                    1   \n",
       "1                               0                                    1   \n",
       "2                               0                                    2   \n",
       "3                               0                                    3   \n",
       "4                               0                                    2   \n",
       "..                            ...                                  ...   \n",
       "361                             0                                    2   \n",
       "362                             0                                    2   \n",
       "363                             0                                    2   \n",
       "364                             0                                    2   \n",
       "365                             0                                    3   \n",
       "\n",
       "     band-like infiltrate   Age  TARGET  \n",
       "0                       0  55.0       2  \n",
       "1                       0   8.0       1  \n",
       "2                       3  26.0       3  \n",
       "3                       0  40.0       1  \n",
       "4                       3  45.0       3  \n",
       "..                    ...   ...     ...  \n",
       "361                     0  25.0       4  \n",
       "362                     0  36.0       4  \n",
       "363                     3  28.0       3  \n",
       "364                     3  50.0       3  \n",
       "365                     0  35.0       1  \n",
       "\n",
       "[366 rows x 35 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/dermatology.data', header=None, names=features)\n",
    "df.loc[df.Age == '?', 'Age'] = None\n",
    "df['Age'] = df.Age.astype(float)\n",
    "df.loc[df.Age.isnull(), 'Age'] = df.Age.median()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a model\n",
    "\n",
    "The usual steps can be done here.  We create our X and y arrays for the features and target.  We perform a train/test split on the data.  For this problem, we will use a k-nearest neighbors model.  I have not tried a wide variety of models or hyperparameters, and have no idea what the \"best\" model is.  But KNN is often quite good, and it is a good way to illustrate the concepts here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop('TARGET', axis=1)\n",
    "y = df['TARGET']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9239130434782609"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "knn = KNeighborsClassifier()\n",
    "knn.fit(X_train, y_train)\n",
    "knn.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us also create a collection of predictions against the test data that we will utilize below.  For convenience, we can transform the array result into a Pandas Series so that the index matches between the ground truth of the training data and the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "247              psoriasis\n",
       "127          lichen planus\n",
       "230    seboreic dermatitis\n",
       "162          lichen planus\n",
       "159    seboreic dermatitis\n",
       "              ...         \n",
       "321       pityriasis rosea\n",
       "59     seboreic dermatitis\n",
       "12     seboreic dermatitis\n",
       "312          lichen planus\n",
       "107              psoriasis\n",
       "Length: 92, dtype: object"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = pd.Series(knn.predict(X_test), index=y_test.index)\n",
    "y_pred.map(targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the trained model\n",
    "\n",
    "So far, so good.  The usual fit-then-score steps work as expected.  But in particular, we simply used the default `.score()` method attached to the trained model object.  For this classification, that default is *accuracy*.  We saw earlier, in the introductory material, that depending on our purpose, accuracy might not be the most useful metric.  \n",
    "\n",
    "The decision of a metric is very much driven by our \"business requirement\" and there is not single objective answer.  However, one thing that is absolute is that when we get to comparing different models to each other—whether entirely different styles of models, or different hyperparameters—we need some way of quantifying the **goodness** of a model to choose which one to keep.  In particular, in practical terms we need to reduce the \"goodness\" to a single number we can compare among modeling approaches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Precision, recall, and f1 score\n",
    "\n",
    "How good is our fitted model by some other metrics we dicussed in the very first [\"What Is Machine Learning?\"](WhatIsML.ipynb) lesson?  Perhaps you should review that lesson for the following discussion.  One matter is that the default \"averaging\" technique assumes a binary classification.  For our multi-class model we have to chose something different.  There are three options here:\n",
    "\n",
    "* `'micro'`: Count false positives, false negatives, true positives, true negatives for all observations independently, and simply perform row-wise aggregation of the counts.\n",
    "* `'macro'`: Count the true and false categories grouping by class label, and take the mean of all those scores per label.\n",
    "* `'weighted'`: Similar to macro, but take a weighted average based on the \"support\" (frequency) of each different label.\n",
    "\n",
    "Again, there is no uniformly right answer to which of these is the best.  It depends on your requirements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score, precision_score, recall_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9239130434782609"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_pred, y_test, average='micro')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9240042566129523"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1_score(y_pred, y_test, average='weighted')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.87699709513435"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_pred, y_test, average='macro')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These numbers are not especially far apart, but the different metrics absolutely give us different results.  Which one we choose will give different answers for which model we should choose for the production system.  \n",
    "\n",
    "Let us create a different model and compare it to the first one under different metrics.  A confession here is that I easily identified a number of model types and hyperparameters that are clearly better than the first one under almost any metric.  The `knn` and `knn2` objects are simply \"naive\" attempts that show the pattern I waish to demonstrate.  That is, among these two models, chosing one is sensitive to which metric you prefer.  But for those better models I found, the same general pattern will emerge, just with higher numbers on each metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn2 = KNeighborsClassifier(n_neighbors=2, metric=\"manhattan\")\n",
    "knn2.fit(X_train, y_train)\n",
    "y_pred2 = knn2.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model 1: 0.9239130434782609\n",
      "Model 2: 0.8369565217391305\n"
     ]
    }
   ],
   "source": [
    "print('Model 1:', precision_score(y_pred, y_test, average='micro'))\n",
    "print('Model 2:', precision_score(y_pred2, y_test, average='micro'))   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model 1: 0.9240042566129523\n",
      "Model 2: 0.8411545022822631\n"
     ]
    }
   ],
   "source": [
    "print('Model 1:', f1_score(y_pred, y_test, average='weighted'))\n",
    "print('Model 2:', f1_score(y_pred2, y_test, average='weighted'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model 1: 0.87699709513435\n",
      "Model 2: 0.8525925925925927\n"
     ]
    }
   ],
   "source": [
    "print('Model 1:', recall_score(y_pred, y_test, average='macro'))\n",
    "print('Model 2:', recall_score(y_pred2, y_test, average='macro'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Revisiting accuracy\n",
    "\n",
    "The most notable limitation of using accuracy as a metric is its poor handling of unbalanced classes.  That might be too stark a way to phrase it though.  For *some purposes* class distribution matters. As with all metrics, our choices are driven by task requirements.  However, let us quickly review the problem case we presented in the first lesson:\n",
    "\n",
    "Consider a test/model for an uncommon but serious disease:\n",
    "\n",
    "| Predict/Actual | Positive | Negative |\n",
    "|----------------|----------|----------|\n",
    "| Positive       |    1     |    0     |\n",
    "| Negative       |    2     |   997    | \n",
    "\n",
    "\n",
    "This has an accuracy of 99.8% (997 true negatives, 1 true positive).  But it misses the requirement that detecting the true positives is, by far, the most important goal.  This is a situation where *balanced accuracy* is likely to be more relevant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import balanced_accuracy_score as ba, accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us construct these hypothetical predictions quickly, simply as results.  For ease of creation, we simply put the True values at the start of each array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.998"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "disease_actual = pd.Series([1, 1, 1] + [0]*997)\n",
    "disease_predict = pd.Series([1] + [0]*999)\n",
    "accuracy_score(disease_predict, disease_actual)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to be careful about balanced accuracy because there is a question of \"balanced on what?\"  Notice this difference:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Balanced on prediction: 0.999\n",
      "Balanced on actual: 0.667\n"
     ]
    }
   ],
   "source": [
    "print(f\"Balanced on prediction: \"\n",
    "      f\"{ba(disease_predict, disease_actual):.3f}\")\n",
    "print(f\"Balanced on actual: \"\n",
    "      f\"{ba(disease_actual, disease_predict):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In other words, since predictions are almost always of \"negative\", the balance weighted by prediction pays even more attention to the negative cases.  This is almost surely not what we want.  Rather, we would like to overweight the rare case; or more specifically, the \"positive\" case.\n",
    "\n",
    "We can achieve more fine-tuned control of what we care about by explicitly setting sample weights.  We need not even set the same weight for each prediction of the same class as balanced accuracy does.  It *could be* that we find the result for samples 1, 5, 10, 35, and 72 far more important regardless of which prediction they make, for example.  This is likely an edge case, but it is available.\n",
    "\n",
    "To replicate what balanced accuracy does, we can map weights (per sample/row) to the *reverse* of the actual counts.  Here the order of the actual/prediction arguments does not matter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6666666666666666"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(disease_actual, disease_predict, \n",
    "               sample_weight=disease_actual.map({0:3, 1:997}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the number we get here with weighted accuracy is not precisely the same as any of accuracy, f1 score, precision, or recall.  Perhaps we know, moreover, for this task requirement that positive cases are even more important than their distribution suggests. We can easily use that as our metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.33355481528305425"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(disease_predict, disease_actual, \n",
    "   sample_weight=disease_actual.map({0:1, 1:1_000_000}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Back to multi-class\n",
    "\n",
    "Having more starkly illustrated balance issues with a highly-imbalanced binary classification problem, let us return to the multi-class skin condition model.  We *could* use balanced accuracy as a higher level function, but instead we will jump directly to using explicit sample weights.  This, in principle, allows us to make more nuanced evaluation.\n",
    "\n",
    "However, it is not obvious in the abstract—to repeat the point yet again (that you are perhaps sick of)—what the best balance or weights are.  Is it more important to diagnose the common conditions correctly, or the rare conditions? Is there one condition in particular that it is more important to get right (maybe it is more serious, or requires a more specific treatment)?  You simply have to know the problem domain to make these decisions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a start, we probably want a sense of the distribution of the conditions.  That was given in the metadata descriptions above, but we will repeat it in code.  Moroever, we presumably want the distribution within the overall dataset, not only within the test set.  Using `balanced_accuracy_score()` would not give us the opportunity to look at the wider distribution.  In fact, quite possibly we could know the general distribution of these conditions beyond the 366 training data samples we have, and utilize that information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "psoriasis                   112\n",
       "lichen planus                72\n",
       "seboreic dermatitis          61\n",
       "cronic dermatitis            52\n",
       "pityriasis rosea             49\n",
       "pityriasis rubra pilaris     20\n",
       "Name: TARGET, dtype: int64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts = y.value_counts()\n",
    "counts.index = counts.index.map(targets)\n",
    "counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These counts will allow us to overweight the common classes, but as we have seen, quite often this is exactly the inverse of what we want.  Fortunately, it is easy to construct that inverse.  The scaling of these numbers is not important, simply their ratios."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "psoriasis                   0.008929\n",
       "lichen planus               0.013889\n",
       "seboreic dermatitis         0.016393\n",
       "cronic dermatitis           0.019231\n",
       "pityriasis rosea            0.020408\n",
       "pityriasis rubra pilaris    0.050000\n",
       "Name: TARGET, dtype: float64"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights = 1/counts\n",
    "weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9239130434782609"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_pred, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For convenience of presentation, we create a mapping from the index of the test set to the condition names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "247              psoriasis\n",
       "127          lichen planus\n",
       "230    seboreic dermatitis\n",
       "162          lichen planus\n",
       "159    seboreic dermatitis\n",
       "              ...         \n",
       "321       pityriasis rosea\n",
       "59     seboreic dermatitis\n",
       "12     seboreic dermatitis\n",
       "312          lichen planus\n",
       "107              psoriasis\n",
       "Name: TARGET, Length: 92, dtype: object"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_names = y_test.map(targets)\n",
    "y_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is common that overweighting the frequent target classes produces increased accuracy.  After all, a DummyClassifier can simply predict the most common class with no effort.  Actual models also have a tendency to overpredict common labels; not always, but often enough to be a concern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9382104342757214"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_pred, y_test, sample_weight=y_names.map(counts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The inverse weighting produces a different answer, usually lower than the naive accuracy.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9114370783796502"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_pred, y_test, sample_weight=y_names.map(weights))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, your ultimate goal will not simply be to compare different metrics, but to compare different models once you have determined the appropriate metric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A custom metric\n",
    "\n",
    "The standard metrics in scikit-learn or other machine learning libraries have a lot of uses, and are greatly configurable.  However, for some particular task requirement, you may need a different functional form.  \n",
    "\n",
    "The metric we construct here is closely inspired by an job I worked at.  That was a company that made consumer recommendations.  The basic idea is that sometimes predictions other than the top one are still relevant.  In the context of this dermatological diagnosis problem, we can imagine that it is likewise useful to have a second most likely diagnosis made available to physicians and counting towards the goodness of the model.  For example, perhaps treatments would be considered that would apply to both diagnoses, or an easier transition to a second treatment would be available with the secondary possible diagnosis.\n",
    "\n",
    "Happily, as we have seen before, many types of models can provide not only a single prediction but also probabilities among all the labels.  We get that with the KNN model we used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>psoriasis</th>\n",
       "      <th>seboreic dermatitis</th>\n",
       "      <th>lichen planus</th>\n",
       "      <th>pityriasis rosea</th>\n",
       "      <th>cronic dermatitis</th>\n",
       "      <th>pityriasis rubra pilaris</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    psoriasis  seboreic dermatitis  lichen planus  pityriasis rosea  \\\n",
       "39        0.0                  0.4            0.0               0.6   \n",
       "42        0.0                  0.0            1.0               0.0   \n",
       "79        0.0                  0.0            1.0               0.0   \n",
       "87        0.0                  0.4            0.0               0.6   \n",
       "33        1.0                  0.0            0.0               0.0   \n",
       "35        1.0                  0.0            0.0               0.0   \n",
       "70        0.0                  0.4            0.0               0.0   \n",
       "63        0.0                  0.0            0.0               0.0   \n",
       "20        0.0                  0.2            0.0               0.6   \n",
       "90        0.0                  0.0            1.0               0.0   \n",
       "\n",
       "    cronic dermatitis  pityriasis rubra pilaris  \n",
       "39                0.0                       0.0  \n",
       "42                0.0                       0.0  \n",
       "79                0.0                       0.0  \n",
       "87                0.0                       0.0  \n",
       "33                0.0                       0.0  \n",
       "35                0.0                       0.0  \n",
       "70                0.6                       0.0  \n",
       "63                1.0                       0.0  \n",
       "20                0.2                       0.0  \n",
       "90                0.0                       0.0  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probs = pd.DataFrame(knn.predict_proba(X_test), columns=targets.values())\n",
    "probs.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In some cases, a single diagnosis is assigned a 100% probability.  If correct, that should be worth a lot.  If it is entirely wrong, the model should be penalized.  However, in other samples the evaluation is divided between multiple diagnoses, with various probabilities strictly between 0% and 100%.  We would like to give \"partial credit\" to those rows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function below is implemented for clarity, not for speed.  Specifically, we would figure out how to vectorize functions that operate on NumPy array or Pandas DataFrames or Series if we had a large dataset.  Here we loop through the probabilities and targets to make the code easier to understand.  The algorithm we want here is to give, for each row, a score:\n",
    "\n",
    "* 1.0 if the highest probability is the correct label\n",
    "* 0.8 if two or more labels have equal probability and one is correct\n",
    "* 0.5 if the second highest probability is the correct label\n",
    "* 0.0 otherwise\n",
    "\n",
    "The final score is the mean of all the row scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def first_or_second(estimator, X, y):\n",
    "    assert len(X) == len(y)\n",
    "    \n",
    "    # Get probabilities\n",
    "    probs = estimator.predict_proba(X)\n",
    "    \n",
    "    # Empty array to hold the scores\n",
    "    scores = np.empty(len(y), dtype=float)\n",
    "\n",
    "    # ... map labels to position in the probs array\n",
    "    lab2ndx = {label: ndx \n",
    "               for ndx, label in enumerate(estimator.classes_)}\n",
    "    # y might start as either Pandas Series or NumPy array\n",
    "    target = pd.Series(y).map(lab2ndx).to_numpy()\n",
    "\n",
    "    for i in range(len(target)):\n",
    "        prob = probs[i]\n",
    "        best_val = prob.max()    # highest prob\n",
    "        best_ats = np.where(prob == best_val)[0]\n",
    "        # Only one 2nd place as implemented\n",
    "        the_rest = np.where(prob != best_val)[0]\n",
    "        best_of_rest = np.argmax(the_rest)\n",
    "\n",
    "        if len(best_ats) == 1:  # just one best prediction\n",
    "            # ... and it is correct\n",
    "                if best_ats[0] == target[i]:\n",
    "                    scores[i] = 1.0\n",
    "                # ... or 2nd place is correct\n",
    "                elif best_of_rest == target[i]:\n",
    "                    scores[i] = 0.5\n",
    "                # ... neither 1st nor second is right\n",
    "                else:\n",
    "                    scores[i] = 0.0\n",
    "        else:                   # tie for best prediction\n",
    "            scores[i] = 0.0     # start pessimistic\n",
    "            # if any winners are right, boost score\n",
    "            for ndx in best_ats:\n",
    "                if ndx == target[i]:\n",
    "                    scores[i] = 0.8\n",
    "                    break\n",
    "\n",
    "    return scores.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The concrete result of this metric is slightly higher than others we have looked at, which you would expect since we give credit for \"second recommendation.\"  All that actually matters is that it has the property that higher scores are better.  We can use any numeric range we want for scores, but this metric also falls between 0 and 1 like most others do.  Since we are looking at more than just a single prediction per row, we need to pass in the actual estimator (model) we want to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9286885245901638"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_or_second(knn, X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9721311475409836"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_or_second(knn2, X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The scorer API\n",
    "\n",
    "We have created a useful enough function that can be used manually.  But it is also compliant with the *scorer API* and hence can be used in conjunction with other kinds of validation functions.  For example, we might want to use our custom scorer with cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import make_scorer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember the plain case of cross-validation, which gives us several scores, one for each slice, We could average them or do other operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.87837838, 0.79452055, 0.8630137 , 0.89041096, 0.89041096])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(knn, X, y, cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another option is to generate a scorer from a metric using the `make_scorer()` utility function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.85635755, 0.77090356, 0.84213291, 0.85470189, 0.86390039])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(knn, X, y, cv=5,\n",
    "                scoring=make_scorer(f1_score, average=\"macro\")) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, of interest to us is that we can likewise use the custom `first_or_second()` scorer function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.88108108, 0.82739726, 0.90684932, 0.90821918, 0.89863014])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(knn, X, y, scoring=first_or_second, cv=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.92972973, 0.95616438, 0.95890411, 0.93972603, 0.93424658])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(knn2, X, y, scoring=first_or_second, cv=5)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}