week-7/7-evaluation-metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation Metrics for Classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1- Jaccard Index\n",
    "2- F1-score\n",
    "3- Log loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Jaccard Index (Jaccard Similarity Coefficient)\n",
    "\n",
    "y: actual values\n",
    "y^: predicted values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]\n",
    "y_hat = [1, 1, 0, 0, 0, 1, 1, 1, 1, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6666666666666666\n"
     ]
    }
   ],
   "source": [
    "# jaccard index formula\n",
    "jaccard = 8 / (10 + 10 - 8)\n",
    "print(jaccard)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "High accuracy: 1.0\n",
    "Low accuracy: 0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## F1-score\n",
    "\n",
    "Precision, Recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Precision is a measure of the accuracy\n",
    "Recall is the true positive rate\n",
    "\n",
    "Precision = TP / (TP + FP)\n",
    "Recall = TP / (TP + FN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "TP = 10\n",
    "FN = 2\n",
    "TN = 12\n",
    "FP = 1\n",
    "\n",
    "# class = 1\n",
    "precision_1 = TP / (TP + FP)\n",
    "recall_1 = TP / (TP + FN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "TP = 12\n",
    "FN = 1\n",
    "TN = 10\n",
    "FP = 2\n",
    "\n",
    "# class = 0\n",
    "precision_0 = TP / (TP + FP)\n",
    "recall_0 = TP / (TP + FN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "class=0 0.86 0.92\n",
      "class=1 0.91 0.83\n"
     ]
    }
   ],
   "source": [
    "print(f\"class=0 {precision_0:.2f} {recall_0:.2f}\")\n",
    "print(f\"class=1 {precision_1:.2f} {recall_1:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "F1-score = 2 * (prc * rec) / (prc + rec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "f1_score_0 = 2 * (precision_0 * recall_0) / (precision_0 + recall_0)\n",
    "f1_score_1 = 2 * (precision_1 * recall_1) / (precision_1 + recall_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.89\n",
      "0.87\n"
     ]
    }
   ],
   "source": [
    "print(f\"{f1_score_0:.2f}\")\n",
    "print(f\"{f1_score_1:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The F1 score's best value is 1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Both Jaccard and F1-score can be used for multi-class classifiers as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Log loss\n",
    "If output of classifier is tge probability of a class label instead of the label, we may use log loss.\n",
    "Classifiers with smaller values of log loss are ideal."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}