sdk/python/foundation-models/huggingface/inference/text-generation-streaming/text-generation-streaming-online-endpoint.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy text-generation streaming models from HuggingFaceHub to AzureML Online Endpoints\n",
    "\n",
    "This sample shows how to deploy `text-generation` streaming models from the HuggingFaceHub to an online endpoint for inference. Learn more about `text-generation` task: https://huggingface.co/tasks/text-generation\n",
    "\n",
    "A large set of models hosted on [Hugging Face Hub](https://huggingface.co/models) are available in the Hugging Face Hub collection in AzureML Model Catalog. This collection is powered by the Hugging Face Hub community registry. Integration with the AzureML Model Catalog enables seamless deployment of Hugging Face Hub models in AzureML. [Learn more.](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-models-from-huggingface?view=azureml-api-2)",
    "\n",
    "### Outline\n",
    "* Set up pre-requisites.\n",
    "* Pick a model to deploy.\n",
    "* Deploy the model for real time inference.\n",
    "* Try sample inference.\n",
    "* Clean up resources."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up pre-requisites\n",
    "* Install dependencies\n",
    "* Connect to AzureML Workspace. Learn more at [set up SDK authentication](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-authentication?tabs=sdk). Replace  `<WORKSPACE_NAME>`, `<RESOURCE_GROUP>` and `<SUBSCRIPTION_ID>` below.\n",
    "* Connect to `HuggingFaceHub` community registry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1689367816236
    }
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import (\n",
    "    DefaultAzureCredential,\n",
    "    InteractiveBrowserCredential,\n",
    "    ClientSecretCredential,\n",
    ")\n",
    "from azure.ai.ml.entities import AmlCompute\n",
    "import time\n",
    "\n",
    "try:\n",
    "    credential = DefaultAzureCredential()\n",
    "    credential.get_token(\"https://management.azure.com/.default\")\n",
    "except Exception as ex:\n",
    "    credential = InteractiveBrowserCredential()\n",
    "\n",
    "# connect to a workspace\n",
    "workspace_ml_client = None\n",
    "try:\n",
    "    workspace_ml_client = MLClient.from_config(credential)\n",
    "    subscription_id = workspace_ml_client.subscription_id\n",
    "    workspace = workspace_ml_client.workspace_name\n",
    "    resource_group = workspace_ml_client.resource_group_name\n",
    "except Exception as ex:\n",
    "    print(ex)\n",
    "    # Enter details of your workspace\n",
    "    subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "    resource_group = \"<RESOURCE_GROUP>\"\n",
    "    workspace = \"<AML_WORKSPACE_NAME>\"\n",
    "    workspace_ml_client = MLClient(\n",
    "        credential, subscription_id, resource_group, workspace\n",
    "    )\n",
    "# Connect to the HuggingFaceHub registry\n",
    "registry_ml_client = MLClient(credential, registry_name=\"HuggingFace\")\n",
    "print(registry_ml_client)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pick a model to deploy\n",
    "\n",
    "Open the Model Catalog in AzureML Studio and choose the Hugging Face Hub collection. Filter by the `text-generation` task and search any specific models you are interested in. In this example, we use the `tiiuae-falcon-40b` model. If you plan to deploy a different model, replace the model name and version accordingly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1689367836823
    }
   },
   "outputs": [],
   "source": [
    "model_name = \"tiiuae-falcon-40b\"\n",
    "foundation_model = registry_ml_client.models.get(model_name, version=\"5\")\n",
    "print(\n",
    "    \"\\n\\nUsing model name: {0}, version: {1}, id: {2} for inferencing\".format(\n",
    "        foundation_model.name, foundation_model.version, foundation_model.id\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the model to an online endpoint\n",
    "Online endpoints give a durable REST API that can be used to integrate with applications that need to use the model. Create an online endpoint and then create an online deployment. You need to specify the Virtual Machine instance or SKU when creating the deployment. You can find the optimal CPU or GPU SKU for a model by opening the quick deployment dialog from the model page in the AzureML Model Catalog. Specify the SKU in the `instance_type` input in deployment settings below.\n",
    "\n",
    "Typically Online Endpoints require you to provide scoring script and a docker container image (through an AzureML environment), in addition to the model. You don't need to worry about them for HuggingFace Hub models available in AzureML Model Catalog because we have enabled 'no code deployments' for these models by packaging scoring script and container image along with the model.\n",
    "\n",
    "Learn more about Online Endpoints: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-online-endpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time, sys\n",
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineEndpoint,\n",
    "    ManagedOnlineDeployment,\n",
    "    OnlineRequestSettings,\n",
    ")\n",
    "\n",
    "# Create online endpoint - endpoint names need to be unique in a region, hence using timestamp to create unique endpoint name\n",
    "timestamp = int(time.time())\n",
    "online_endpoint_name = \"text-generation-\" + str(timestamp)\n",
    "# create an online endpoint\n",
    "endpoint = ManagedOnlineEndpoint(\n",
    "    name=online_endpoint_name,\n",
    "    description=\"Online endpoint for \"\n",
    "    + foundation_model.name\n",
    "    + \", for text-generation task\",\n",
    "    auth_mode=\"key\",\n",
    ")\n",
    "workspace_ml_client.begin_create_or_update(endpoint).wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a deployment\n",
    "demo_deployment = ManagedOnlineDeployment(\n",
    "    name=\"demo\",\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    model=foundation_model.id,\n",
    "    instance_type=\"Standard_ND96amsr_A100_v4\",\n",
    "    instance_count=1,\n",
    ")\n",
    "workspace_ml_client.online_deployments.begin_create_or_update(demo_deployment).wait()\n",
    "# online endpoints can have multiple deployments with traffic split or shadow traffic. Set traffic to 100% for demo deployment\n",
    "endpoint.traffic = {\"demo\": 100}\n",
    "workspace_ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try sample inference\n",
    "\n",
    "Online endpoints expose a REST API that can be integrated into your applications. Learn how to fetch the scoring REST API and credentials for online endpoints here: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-authenticate-online-endpoint\n",
    "\n",
    "In this example, we will use the Python SDK helper method to invoke the endpoint. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1689367842385
    }
   },
   "outputs": [],
   "source": [
    "# Get the model object from HuggingFaceHub. We can use it to check for sample test data\n",
    "import urllib.request, json\n",
    "\n",
    "raw_data = urllib.request.urlopen(\n",
    "    \"https://huggingface.co/api/models/\" + foundation_model.tags[\"modelId\"]\n",
    ")\n",
    "\n",
    "print(\"https://huggingface.co/api/models/\" + foundation_model.tags[\"modelId\"])\n",
    "data = json.load(raw_data)\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1689367847055
    }
   },
   "outputs": [],
   "source": [
    "# check if there is sample inference data available on HuggingFaceHub for the model, else try with the backup sample data\n",
    "scoring_file = \"./sample_score.json\"\n",
    "prompt = \"\"\n",
    "if \"widgetData\" in data:\n",
    "    for input in data[\"widgetData\"]:\n",
    "        prompt = input[\"text\"]\n",
    "        break\n",
    "    # write the sample_score.json file\n",
    "    score_dict = {\"inputs\": prompt}\n",
    "    with open(scoring_file, \"w\") as outfile:\n",
    "        json.dump(score_dict, outfile)\n",
    "else:\n",
    "    scoring_file = \"./sample_score_backup.json\"\n",
    "\n",
    "# print the sample scoring file\n",
    "print(\"\\n\\nSample scoring file: \")\n",
    "with open(scoring_file) as json_file:\n",
    "    scoring_data = json.load(json_file)\n",
    "    print(scoring_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1689367852666
    }
   },
   "outputs": [],
   "source": [
    "# score the sample_score.json file using the online endpoint with the azureml endpoint invoke method\n",
    "response = workspace_ml_client.online_endpoints.invoke(\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    deployment_name=\"demo\",\n",
    "    request_file=scoring_file,\n",
    ")\n",
    "response_json = json.loads(response)\n",
    "print(json.dumps(response_json, indent=2))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### Stream responses from the online endpoint\n",
    "Text-generation models can take considerable time to handle a request and return generated text. One way to improve the user experience is to stream tokens back to the user as they are generated. Learn more about streaming results from Hugging Face text-generation models here: https://huggingface.co/blog/inference-endpoints-llm.\n",
    "\n",
    "In this example, we will use the Hugging Face python text-generation `InferencingClient` to help stream results from our online endpoint. Learn more about Hugging Face text generation inference here: https://github.com/huggingface/text-generation-inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1689367879226
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# Get the online endpoint scoring uri\n",
    "online_endpoint = workspace_ml_client.online_endpoints.get(online_endpoint_name)\n",
    "online_endpoint_scoring_uri = online_endpoint.scoring_uri\n",
    "print(online_endpoint_scoring_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1689367881833
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# Get the online endpoint api key for authenticating our inferencing request\n",
    "keys = workspace_ml_client.online_endpoints.get_keys(online_endpoint_name)\n",
    "api_key = keys.primary_key\n",
    "print(api_key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1689367886731
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "pip install text-generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1689367890916
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "pip install -U huggingface_hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1689367895354
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "pip install ipywidgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1689368085429
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "from huggingface_hub import InferenceClient\n",
    "\n",
    "# HF Inference Endpoints parameter\n",
    "# Note - do not use the full /generate scoring url here\n",
    "endpoint_url = online_endpoint_scoring_uri.split(\"/generate\")[0]\n",
    "print(\"Online endpoint url: {0}\".format(endpoint_url))\n",
    "headers = {\"Content-Type\": \"application/json\"}\n",
    "\n",
    "# Streaming Client\n",
    "client = InferenceClient(endpoint_url, token=api_key, headers=headers)\n",
    "\n",
    "# generation parameter\n",
    "gen_kwargs = dict(\n",
    "    max_new_tokens=512,\n",
    "    top_k=30,\n",
    "    top_p=0.9,\n",
    "    temperature=0.2,\n",
    "    repetition_penalty=1.02,\n",
    "    stop_sequences=[\"\\nUser:\", \"<|endoftext|>\", \"</s>\"],\n",
    ")\n",
    "\n",
    "# Begin text generation stream\n",
    "print(\"Prompt:\\n{0}\".format(prompt))\n",
    "stream = client.text_generation(prompt, stream=True, details=True, **gen_kwargs)\n",
    "print(\"Stream result:\\n\")\n",
    "# yield each generated token\n",
    "for r in stream:\n",
    "    # skip special tokens\n",
    "    if r.token.special:\n",
    "        continue\n",
    "    # stop if we encounter a stop sequence\n",
    "    if r.token.text in gen_kwargs[\"stop_sequences\"]:\n",
    "        break\n",
    "    # yield the generated token\n",
    "    print(r.token.text, end=\"\")\n",
    "    # yield r.token.text"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the online endpoint\n",
    "Don't forget to delete the online endpoint, else you will leave the billing meter running for the compute used by the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workspace_ml_client.online_endpoints.begin_delete(name=online_endpoint_name).wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python310-sdkv2"
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "microsoft": {
   "host": {
    "AzureML": {
     "notebookHasBeenCompleted": true
    }
   },
   "ms_spell_check": {
    "ms_spell_check_language": "en"
   }
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}