sdk/python/foundation-models/system/inference/text-classification/text-classification-online-endpoint-oss.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Classification Inference using Online Endpoints\n",
    "\n",
    "This sample shows how to deploy `text-classification` type models to an online endpoint for inference.\n",
    "\n",
    "### Task\n",
    "`text-classification` is generic task type that can be used for scenarios such as sentiment analysis, emotion detection, grammar checking, spam filtering, etc. In this example, we will do sentiment analysis on movie reviews, to determine whether a review is positive or negative. \n",
    "\n",
    "### Inference data\n",
    "We will use the [imdb](https://huggingface.co/datasets/imdb) dataset\n",
    "\n",
    "### Model\n",
    "Look for models tagged with `text-classification` in the system registry. Just looking for `text-classification` is not sufficient, you need to check if the model is specifically finetuned for sentiment analysis by studying the model card and looking at the input/output samples or signatures of the model. In this notebook, we use the `finiteautomata-bertweet-base-sentiment-analysis` model.\n",
    "\n",
    "   \n",
    "\n",
    "### Outline\n",
    "* Set up pre-requisites.\n",
    "* Pick a model to deploy.\n",
    "* Download and prepare data for inference. \n",
    "* Deploy the model for real time inference.\n",
    "* Test the endpoint\n",
    "* Clean up resources."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Set up pre-requisites\n",
    "* Install dependencies\n",
    "* Connect to AzureML Workspace. Learn more at [set up SDK authentication](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-authentication?tabs=sdk). Replace  `<WORKSPACE_NAME>`, `<RESOURCE_GROUP>` and `<SUBSCRIPTION_ID>` below.\n",
    "* Connect to `azureml` system registry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import (\n",
    "    DefaultAzureCredential,\n",
    "    InteractiveBrowserCredential,\n",
    "    ClientSecretCredential,\n",
    ")\n",
    "from azure.ai.ml.entities import AmlCompute\n",
    "import time\n",
    "\n",
    "try:\n",
    "    credential = DefaultAzureCredential()\n",
    "    credential.get_token(\"https://management.azure.com/.default\")\n",
    "except Exception as ex:\n",
    "    credential = InteractiveBrowserCredential()\n",
    "\n",
    "workspace_ml_client = MLClient(\n",
    "    credential,\n",
    "    subscription_id=\"<SUBSCRIPTION_ID>\",\n",
    "    resource_group_name=\"<RESOURCE_GROUP>\",\n",
    "    workspace_name=\"<WORKSPACE_NAME>\",\n",
    ")\n",
    "# The models, fine tuning pipelines and environments are available in the AzureML system registry, \"azureml\"\n",
    "registry_ml_client = MLClient(credential, registry_name=\"azureml\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Pick a model to deploy\n",
    "\n",
    "Browse models in the Model Catalog in the AzureML Studio, filtering by the `fill-mask` task. In this example, we use the `finiteautomata-bertweet-base-sentiment-analysis` model. If you have opened this notebook for a different model, replace the model name and version accordingly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"roberta-large-mnli\"\n",
    "version_list = list(registry_ml_client.models.list(model_name))\n",
    "if len(version_list) == 0:\n",
    "    print(\"Model not found in registry\")\n",
    "else:\n",
    "    model_version = version_list[0].version\n",
    "    foundation_model = registry_ml_client.models.get(model_name, model_version)\n",
    "    print(\n",
    "        \"\\n\\nUsing model name: {0}, version: {1}, id: {2} for fine tuning\".format(\n",
    "            foundation_model.name, foundation_model.version, foundation_model.id\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Download and prepare data for inference.\n",
    "\n",
    "The next few cells show basic data preparation:\n",
    "* Visualize some data rows\n",
    "* Replace numerical categories in data with the actual string labels. This mapping is available in [label.json](./emotion-dataset/label.json). This step is needed because the selected models will return labels such `pos`, `neg`, etc. when running prediction. If the labels in your ground truth data are left as `0`, `1`, `2`, etc., then they would not match with prediction labels returned by the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download a small sample of the dataset into the ./imdb-dataset directory\n",
    "%run ./imdb-dataset/download-dataset.py --download_dir ./imdb-dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "dataset_dir = \"./imdb-dataset\"\n",
    "data_file = \"train.jsonl\"\n",
    "\n",
    "# load the train.jsonl file into a pandas dataframe and show the first 5 rows\n",
    "import pandas as pd\n",
    "\n",
    "pd.set_option(\n",
    "    \"display.max_colwidth\", 0\n",
    ")  # set the max column width to 0 to display the full text\n",
    "df = pd.read_json(os.path.join(dataset_dir, data_file), lines=True)\n",
    "df.head()\n",
    "\n",
    "# load the id2label json element of the label.json file into pandas table with keys as 'label' column of int64 type and values as 'label_string' column as string type\n",
    "import json\n",
    "\n",
    "label_file = \"label.json\"\n",
    "with open(os.path.join(dataset_dir, label_file)) as f:\n",
    "    id2label = json.load(f)\n",
    "    id2label = id2label[\"id2label\"]\n",
    "    label_df = pd.DataFrame.from_dict(\n",
    "        id2label, orient=\"index\", columns=[\"label_string\"]\n",
    "    )\n",
    "    label_df[\"label\"] = label_df.index.astype(\"int64\")\n",
    "    label_df = label_df[[\"label\", \"label_string\"]]\n",
    "\n",
    "# join the train, validation and test dataframes with the id2label dataframe to get the label_string column\n",
    "df = df.merge(label_df, on=\"label\", how=\"left\")\n",
    "# rename the label_string column to ground_truth_label\n",
    "df = df.rename(columns={\"label_string\": \"ground_truth_label\"})\n",
    "\n",
    "import re\n",
    "\n",
    "df[\"text\"] = df[\"text\"].apply(lambda x: re.sub(r\"<.*?>\", \"\", x))\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Deploy the model to an online endpoint\n",
    "Online endpoints give a durable REST API that can be used to integrate with applications that need to use the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time, sys\n",
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineEndpoint,\n",
    "    ManagedOnlineDeployment,\n",
    "    ProbeSettings,\n",
    ")\n",
    "\n",
    "# Create online endpoint - endpoint names need to be unique in a region, hence using timestamp to create unique endpoint name\n",
    "timestamp = int(time.time())\n",
    "online_endpoint_name = \"text-class-\" + str(timestamp)\n",
    "# create an online endpoint\n",
    "endpoint = ManagedOnlineEndpoint(\n",
    "    name=online_endpoint_name,\n",
    "    description=\"Online endpoint for \"\n",
    "    + foundation_model.name\n",
    "    + \", to detect positive or negative sentiment in movie reviews\",\n",
    "    auth_mode=\"key\",\n",
    ")\n",
    "workspace_ml_client.begin_create_or_update(endpoint).wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a deployment\n",
    "demo_deployment = ManagedOnlineDeployment(\n",
    "    name=\"demo\",\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    model=foundation_model.id,\n",
    "    instance_type=\"Standard_DS4_v2\",\n",
    "    instance_count=2,\n",
    "    liveness_probe=ProbeSettings(\n",
    "        failure_threshold=30,\n",
    "        success_threshold=1,\n",
    "        timeout=2,\n",
    "        period=10,\n",
    "        initial_delay=1000,\n",
    "    ),\n",
    "    readiness_probe=ProbeSettings(\n",
    "        failure_threshold=10,\n",
    "        success_threshold=1,\n",
    "        timeout=10,\n",
    "        period=10,\n",
    "        initial_delay=1000,\n",
    "    ),\n",
    ")\n",
    "workspace_ml_client.online_deployments.begin_create_or_update(demo_deployment).wait()\n",
    "endpoint.traffic = {\"demo\": 100}\n",
    "workspace_ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Test the endpoint with sample data\n",
    "\n",
    "We will fetch some sample data from the test dataset and submit to online endpoint for inference. We will then show the display the scored labels alongside the ground truth labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "score_file = \"sample_score.json\"\n",
    "# pick 5 random rows\n",
    "sample_df = df.sample(5)\n",
    "# reset the index of sample_df\n",
    "sample_df = sample_df.reset_index(drop=True)\n",
    "sample_df.drop(columns=[\"label\"], inplace=True)\n",
    "\n",
    "# save the json object to a file named sample_score.json in the\n",
    "test_json = {\n",
    "    \"input_data\": sample_df[\"text\"].tolist(),\n",
    "    \"params\": {\"return_all_scores\": True},\n",
    "}\n",
    "# save the json object to a file named sample_score.json in the ./imdb-dataset folder\n",
    "with open(os.path.join(\".\", dataset_dir, score_file), \"w\") as f:\n",
    "    json.dump(test_json, f)\n",
    "sample_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# score the sample_score.json file using the online endpoint with the azureml endpoint invoke method\n",
    "response = workspace_ml_client.online_endpoints.invoke(\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    deployment_name=\"demo\",\n",
    "    request_file=os.path.join(\".\", dataset_dir, score_file),\n",
    ")\n",
    "print(\"raw response: \\n\", response, \"\\n\")\n",
    "# convert the json response to a pandas dataframe\n",
    "response_df = pd.read_json(response)\n",
    "# rename label column to predicted_label\n",
    "response_df = response_df.rename(columns={0: \"predicted_label\"})\n",
    "response_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# merge the sample_df and response_df dataframes\n",
    "merged_df = sample_df.merge(response_df, left_index=True, right_index=True)\n",
    "merged_df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Delete the online endpoint\n",
    "Don't forget to delete the online endpoint, else you will leave the billing meter running for the compute used by the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workspace_ml_client.online_endpoints.begin_delete(name=online_endpoint_name).wait()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "2f394aca7ca06fed1e6064aef884364492d7cdda3614a461e02e6407fc40ba69"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}