04-pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "## Creating Pipelines\n",
    "\n",
    "In notebooks [2](02-feature-engineering.ipynb) and  [3](03-model-logistic-regression.ipynb) we developed and trained a feature engineering technique and a logistic regression model. In this notebook we will combine them into a pipeline. \n",
    "\n",
    "Machine learning pipelines allow you to precisely specify a set of transformations which start with raw data and result in a model. They make it possible to re-train the same model repeatedly, using different parameter values, and to reapply these same transformations to raw data in production, resulting in predictions.  "
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We load in our data: "
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "df = pd.read_parquet(\"fraud-cleaned-sample.parquet\")"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from sklearn import model_selection\n",
    "train, test = model_selection.train_test_split(df, random_state=43)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now we load the pipeline steps we created in earier notebooks. These are `feat_pipeline.pkl` and `lr.pkl`, corresponding to the feature engineering stages and the logisitc regression model, respectively. "
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import cloudpickle as cp\n",
    "feature_pipeline = cp.load(open('feat_pipeline.pkl', 'rb'))\n",
    "model = cp.load(open('lr.pkl', 'rb'))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now we can combine these stages together in a pipeline and fit it to training data:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "pipeline = Pipeline([\n",
    "    ('features', feature_pipeline),\n",
    "    ('model', model)\n",
    "])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "pipeline"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Here you can see all the transformations and parameters used in the pipeline. \n",
    "\n",
    "We can refit the whole pipeline to training data:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "pipeline.fit(train, y = train[\"label\"])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can use this pipeline to make predictions - let's predict for our test set:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "pipeline.predict(test)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's now save this pipeline as one pickled object:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "cp.dump(pipeline, open(\"pipeline.pkl\", \"wb\"))\n"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "With the pipeline saved, we can now start to think about developing a model service that we can interact with, rather than just crunching data through the jupyter notebook. Head over to the 'app' folder inside this file directory to access the next notebooks. "
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}