notebook/5_Automatic_EDA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "009e122d",
   "metadata": {},
   "source": [
    "# Data Loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "366f823e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T11:59:07.392672Z",
     "start_time": "2021-10-27T11:59:03.918384Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff7ed2b2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T11:59:07.527145Z",
     "start_time": "2021-10-27T11:59:07.395875Z"
    }
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import google.colab\n",
    "    data_file = 'https://raw.githubusercontent.com/sesise0307/pydata2021-eda/main/data/preprocessed/adult.feather'\n",
    "except:\n",
    "    data_file = '../data/preprocessed/adult.feather'\n",
    "    \n",
    "df = pd.read_feather(data_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4746a99a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T11:59:07.536550Z",
     "start_time": "2021-10-27T11:59:07.529506Z"
    }
   },
   "outputs": [],
   "source": [
    "df_sample = df.sample(n=500, random_state=47)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cac87194",
   "metadata": {},
   "source": [
    "# [D-Tale](https://github.com/man-group/dtale)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "534189fe",
   "metadata": {},
   "source": [
    "`D-Tale` is the combination of a `Flask` back-end and a `React` front-end to bring you an easy way to view & analyze `Pandas` data structures.\n",
    "\n",
    "It integrates seamlessly with `ipython` notebooks & `python`/`ipython` terminals.\n",
    "\n",
    "Currently this tool supports such `Pandas` objects as `DataFrame`, `Series`, `MultiIndex`, `DatetimeIndex` & `RangeIndex`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4397d4fc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T11:59:26.179222Z",
     "start_time": "2021-10-27T11:59:07.538944Z"
    }
   },
   "outputs": [],
   "source": [
    "import dtale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bcdd8a0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T11:59:26.446964Z",
     "start_time": "2021-10-27T11:59:26.181924Z"
    }
   },
   "outputs": [],
   "source": [
    "dtale.show(df_sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25ef12a0",
   "metadata": {},
   "source": [
    "# [Pandas-profiling](https://github.com/pandas-profiling/pandas-profiling)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19b62b6f",
   "metadata": {},
   "source": [
    "The pandas `df.describe()` function is great but a little basic for serious exploratory data analysis.\n",
    "\n",
    "`pandas_profiling` extends the `pandas` DataFrame with `df.profile_report()` for quick data analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0b329f3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T11:59:33.186982Z",
     "start_time": "2021-10-27T11:59:26.449363Z"
    }
   },
   "outputs": [],
   "source": [
    "from pandas_profiling import ProfileReport"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "520f2cdc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T11:59:33.197245Z",
     "start_time": "2021-10-27T11:59:33.189251Z"
    }
   },
   "outputs": [],
   "source": [
    "profile = ProfileReport(df_sample, title=\"Adult Data Pandas Profiling Report\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e4caa3b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T12:00:00.293668Z",
     "start_time": "2021-10-27T11:59:33.199226Z"
    }
   },
   "outputs": [],
   "source": [
    "profile.to_notebook_iframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91807520",
   "metadata": {},
   "source": [
    "# [Sweetviz](https://github.com/fbdesignpro/sweetviz)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2defaadb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-11T15:17:24.229700Z",
     "start_time": "2021-10-11T15:17:24.225422Z"
    }
   },
   "source": [
    "Sweetviz is an open-source Python library that generates beautiful, high-density visualizations to kickstart EDA (Exploratory Data Analysis) with just two lines of code. Output is a fully self-contained HTML application.\n",
    "\n",
    "The system is built around quickly visualizing target values and comparing datasets. Its goal is to help quick analysis of target characteristics, training vs testing data, and other such data characterization tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d5fb140",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-22T16:04:24.778497Z",
     "start_time": "2021-10-22T16:04:24.591846Z"
    }
   },
   "source": [
    "![Sweetviz](../image/sweetviz.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb69ee32",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T12:00:00.475666Z",
     "start_time": "2021-10-27T12:00:00.296786Z"
    }
   },
   "outputs": [],
   "source": [
    "import sweetviz as sv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "351a4429",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T12:00:04.812811Z",
     "start_time": "2021-10-27T12:00:00.479930Z"
    }
   },
   "outputs": [],
   "source": [
    "sv_report = sv.analyze(df_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56ce52ba",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T12:00:05.093805Z",
     "start_time": "2021-10-27T12:00:04.814581Z"
    }
   },
   "outputs": [],
   "source": [
    "sv_report.show_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6af5db59",
   "metadata": {},
   "source": [
    "#  [Autoviz](https://github.com/AutoViML/AutoViz)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e813c260",
   "metadata": {},
   "source": [
    "AutoViz performs automatic visualization of any dataset with one line.\n",
    "\n",
    "Give any input file (CSV, txt or json) and AutoViz will visualize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f0358d5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T12:00:07.849176Z",
     "start_time": "2021-10-27T12:00:05.095412Z"
    }
   },
   "outputs": [],
   "source": [
    "from autoviz.AutoViz_Class import AutoViz_Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e9a8523",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T12:00:07.854929Z",
     "start_time": "2021-10-27T12:00:07.851410Z"
    }
   },
   "outputs": [],
   "source": [
    "AV = AutoViz_Class()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b758a1f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-27T12:00:17.973657Z",
     "start_time": "2021-10-27T12:00:07.857288Z"
    }
   },
   "outputs": [],
   "source": [
    "AV.AutoViz(\n",
    "    filename='',\n",
    "    dfte=df_sample\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}