tutorials/21_Customizing_PromptNode.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "BlTn1yekNONP"
   },
   "source": [
    "# Tutorial: Customizing PromptNode for NLP Tasks\n",
    "\n",
    "- **Level**: Intermediate\n",
    "- **Time to complete**: 20 minutes\n",
    "- **Nodes Used**: `PromptNode`, `PromptTemplate`\n",
    "- **Goal**: After completing this tutorial, you will have learned the basics of using PromptNode and PromptTemplates and you'll have added titles to articles from The Guardian and categorized them. \n",
    "\n",
    "> This tutorial is based on Haystack 1.x. If you're using Haystack 2.0 and would like to follow the updated version of this tutorial, check out [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline). \n",
    ">\n",
    "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "ffYFUAjUNONS"
   },
   "source": [
    "## Overview\n",
    "\n",
    "Use large language models (LLMs) through PromptNode and PromptTemplate to summarize and categorize your documents, and find a suitable title for them. In this tutorial, we'll use news from [The Guardian](https://www.theguardian.com/international) as documents, but you can replace them with any text you want.  \n",
    "\n",
    "This tutorial introduces you to the basics of LLMs and PromptNode, showcases the pre-defined \"deepset/summarization\" template, and explains how to use PromptTemplate to generate titles for documents and categorize them with custom prompts."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "ugkQ42iJNONS"
   },
   "source": [
    "## Preparing the Colab Environment\n",
    "\n",
    "- [Enable GPU Runtime in Colab](https://docs.haystack.deepset.ai/docs/enabling-gpu-acceleration#enabling-the-gpu-in-colab)\n",
    "- [Set logging level to INFO](https://docs.haystack.deepset.ai/docs/log-level)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "t0rQfgOVNONS"
   },
   "source": [
    "## Installing Haystack\n",
    "\n",
    "To start, let's install the latest release of Haystack with `pip`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "S-00smnxNONT"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "pip install --upgrade pip\n",
    "pip install farm-haystack[colab]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "VTfmApFpZKn6"
   },
   "source": [
    "### Enabling Telemetry\n",
    "\n",
    "Knowing you're using this tutorial helps us decide where to invest our efforts to build a better product but you can always opt out by commenting the following line. See [Telemetry](https://docs.haystack.deepset.ai/docs/telemetry) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HtImiSaHZKn6"
   },
   "outputs": [],
   "source": [
    "from haystack.telemetry import tutorial_running\n",
    "\n",
    "tutorial_running(21)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "RYtbl6qBNONU"
   },
   "source": [
    "## Trying Out PromptNode\n",
    "\n",
    "The PromptNode is the central abstraction in Haystack's large language model (LLM) support. It uses [`google/flan-t5-base`](https://huggingface.co/google/flan-t5-base) model by default, but you can replace the default model with a flan-t5 model of a different size such as `google/flan-t5-large` or a model by OpenAI such as `gpt-3.5-turbo-instruct`.\n",
    "\n",
    "[Large language models](https://docs.haystack.deepset.ai/docs/language_models#large-language-models-llms) are huge models trained on enormous amounts of data. That’s why these models have general knowledge of the world, so you can ask them anything and they will be able to answer.\n",
    "\n",
    "As a warm-up, let's initialize PromptNode and see what it can do when run stand-alone: "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "2wYbU8WhNX08"
   },
   "source": [
    "1. Initialize a PromptNode instance with [`google/flan-t5-large`](https://huggingface.co/google/flan-t5-large):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wa-I31YZNONU",
    "outputId": "7b101632-4300-43f9-d9d4-78ea78581529"
   },
   "outputs": [],
   "source": [
    "from haystack.nodes import PromptNode\n",
    "\n",
    "prompt_node = PromptNode(model_name_or_path=\"google/flan-t5-large\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "ZE7s-cMRNh5f"
   },
   "source": [
    "> Note: To use PromptNode with an OpenAI model, change the model name and provide an `api_key`: \n",
    "> ```python\n",
    "> prompt_node = PromptNode(model_name_or_path=\"gpt-3.5-turbo-instruct\", api_key=<YOUR_API_KEY>)\n",
    "> ```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "NO5z2H_mO8cg"
   },
   "source": [
    "2. Ask any general question that comes to your mind, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dBxL4odqNg6K",
    "outputId": "c835d67a-f22f-4062-c1ad-7ace72006995"
   },
   "outputs": [],
   "source": [
    "prompt_node(\"What is the capital of Germany?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cg-8tKebDCa0",
    "outputId": "051810e1-3c82-43cc-aef3-2ce9c125a891"
   },
   "outputs": [],
   "source": [
    "prompt_node(\"What is the highest mountain?\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "0MVeOCWSNONV"
   },
   "source": [
    "As `google/flan-t5-large` was trained on school math problems dataset named [GSM8K](https://huggingface.co/datasets/gsm8k) you can also ask some basic math questions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-ZpTIQEUNONW",
    "outputId": "8daa23ba-881e-4ccd-e14d-0b4bb2d107e4"
   },
   "outputs": [],
   "source": [
    "prompt_node(\"If Bob is 20 and Sara is 11, who is older?\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "aybEtTGIQQSX"
   },
   "source": [
    "Now that you've initialized PromptNode and saw how it works, let's see how we can use it for more advanced tasks."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "W6hK9AcsNONW"
   },
   "source": [
    "## Summarizing Documents with PromptNode\n",
    "\n",
    "PromptNode is integrated to [PromptHub](https://prompthub.deepset.ai/) that includes ready-made prompts for the most common NLP tasks such as summarization, question answering, question generation, and more. To use a prompt template from the PromptHub, just provide its name to the PromptNode. \n",
    "\n",
    "For this task, we'll use the `deepset/summarization` template from the PromptHub and news from The Guardian. Let's see how to do it."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "zECQSKkWDYns"
   },
   "source": [
    "1. Define news to use as `documents` for the PromptNode. We'll use these documents throughout the whole tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DjYmEk9_dyHZ"
   },
   "outputs": [],
   "source": [
    "from haystack.schema import Document\n",
    "\n",
    "# https://www.theguardian.com/business/2023/feb/12/inflation-may-have-peaked-but-the-cost-of-living-pain-is-far-from-over\n",
    "news_economics = Document(\n",
    "    \"\"\"At long last, Britain’s annual inflation rate is on the way down. After hitting the highest level since the 1980s, heaping pressure on millions of households as living costs soared, official figures this week could bring some rare good news.\n",
    "City economists expect UK inflation to have cooled for a third month running in January – the exact number is announced on Wednesday – helped by falling petrol prices and a broader decline in the global price of oil and gas in recent months. The hope now is for a sustained decline in the months ahead, continuing a steady drop from the peak of 11.1% seen in October.\n",
    "The message from the Bank of England has been clear. Inflation is on track for a “rapid” decline over the coming months, raising hopes that the worst of Britain’s cost of living crisis is now in the rearview mirror.\n",
    "There are two good reasons for this. Energy costs are moving in the right direction, while the initial rise in wholesale oil and gas prices that followed Russia’s invasion of Ukraine in February last year will soon drop from the calculation of the annual inflation rate.\"\"\"\n",
    ")\n",
    "\n",
    "# https://www.theguardian.com/science/2023/feb/13/starwatch-orions-belt-and-sirius-lead-way-to-hydras-head\n",
    "news_science = Document(\n",
    "    \"\"\"On northern winter nights, it is so easy to be beguiled by the gloriously bright constellations of Orion, the hunter, and Taurus, the bull, that one can overlook the fainter constellations.\n",
    "So this week, find the three stars of Orion’s belt, follow them down to Sirius, the brightest star in the night sky, and then look eastward until you find the faint ring of stars that makes up the head of Hydra, the water snake. The chart shows the view looking south-east from London at 8pm GMT on Monday, but the view will be similar every night this week.\n",
    "Hydra is the largest of the 88 modern constellations covering an area of 1,303 square degrees. To compare, nearby Orion only covers 594 square degrees. Hydra accounts for most of its area by its length, crossing more than 100 degrees of the sky (the full moon spans half a degree).\n",
    "As evening becomes night and into the early hours, the rotation of Earth causes Hydra to slither its way across the southern meridian until dawn washes it from the sky. From the southern hemisphere, the constellation is easily visible in the eastern sky by mid-evening.\"\"\"\n",
    ")\n",
    "\n",
    "# https://www.theguardian.com/music/2023/jan/30/salisbury-cathedral-pipe-organ-new-life-holst-the-planets\n",
    "news_culture = Document(\n",
    "    \"\"\"A unique performance of Gustav Holst’s masterwork The Planets – played on a magnificent pipe organ rather than by an orchestra and punctuated by poems inspired by children’s responses to the music – is to be staged in the suitably vast Salisbury Cathedral.\n",
    "The idea of the community music project is to introduce more people, young and old, to the 140-year-old “Father” Willis organ, one of the treasures of the cathedral.\n",
    "It is also intended to get the children who took part and the adults who will watch and listen thinking afresh about the themes Holst’s suite tackles – war, peace, joy and mysticism – which seem as relevant now as when he wrote the work a century ago.\n",
    "John Challenger, the cathedral’s principal organist, said: “We have a fantastic pipe organ largely as it was when built. It’s a thrilling thing. I view it as my purpose in life to share it with as many people as possible.”\n",
    "The Planets is written for a large orchestra. “Holst calls for huge instrumental forces and an unseen distant choir of sopranos and altos,” said Challenger. But he has transposed the suite for the organ, not copying the effect of the orchestral instruments but finding a new version of the suite.\"\"\"\n",
    ")\n",
    "\n",
    "# https://www.theguardian.com/sport/blog/2023/feb/14/multi-million-dollar-wpl-auction-signals-huge-step-forward-for-womens-sport\n",
    "news_sport = Document(\n",
    "    \"\"\"It was only a few days ago that members of the Australian women’s cricket team were contemplating how best to navigate the impending “distraction” of the inaugural Women’s Premier League auction, scheduled during the first week of the T20 World Cup. “It’s a little bit awkward,” captain Meg Lanning said in South Africa last week. “But it’s just trying to embrace that and understanding it’s actually a really exciting time and you actually don’t have a lot of control over most of it, so you’ve just got to wait and see.”\n",
    "What a pleasant distraction it turned out to be. Lanning herself will be $192,000 richer for three weeks’ work with the Delhi Capitals. Her teammate, Ash Gardner, will earn three times that playing for the Gujarat Giants. The allrounder’s figure of $558,000 is more than Sam Kerr pockets in a season with Chelsea and more than the WNBA’s top earner, Jackie Young.\n",
    "If that sounds like a watershed moment, it’s perhaps because it is. And it is not the only one this past week. The NRLW made its own wage-related headlines on Tuesday, to the effect that the next (agreed in principle) collective bargaining agreement will bring with it a $1.5m salary cap in 2027, at an average salary of $62,500. Women’s rugby, too, is making moves, with news on the weekend that Rugby Australia will begin contracting the Wallaroos.\"\"\"\n",
    ")\n",
    "\n",
    "news = [news_economics, news_science, news_culture, news_sport]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "Ezq9NVmFdzz1"
   },
   "source": [
    "> The token limit for `google/flan-t5-large` is 512. So, all news pieces should be shorter than the limit."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "syB5rl3xIgHr"
   },
   "source": [
    "2. Use the `deepset/summarization` template to generate a summary for each piece of news:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ygxFMwb4NONX",
    "outputId": "cef66f16-f9bc-4286-de68-dac0692a64c1"
   },
   "outputs": [],
   "source": [
    "prompt_node.prompt(prompt_template=\"deepset/summarization\", documents=news)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "lUTckmA1PzLc"
   },
   "source": [
    "Here you go! You have generated summaries of your news articles. But we're missing titles for them. Let's see how PromptNode can help us there."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "f0vQ45tHNONX"
   },
   "source": [
    "## Generating Titles for News Articles with a Custom Template"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "992kb_WKNONX"
   },
   "source": [
    "The biggest benefit of PromptNode is its versatility. You can use it to perform practically any NLP task if you define your own prompt templates for them. By creating your prompt templates, you can extend the model's capabilities and use it for a broader range of NLP tasks in Haystack. \n",
    "\n",
    "You can define custom templates for each NLP task and register them with PromptNode. Let's create a custom template to generate descriptive titles for news:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "en6IbPRsTOTz"
   },
   "source": [
    "1. Initialize a `PromptTemplate` instance by defining the prompt text in `prompt`. To define any parameters for the prompt, add them to the `prompt` wrapped with curly brackets. We need a template to generate titles for our news articles. The only parameter we need is `{news}`, so let's create a PromptTemplate for it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OiOiya2UV4WW"
   },
   "outputs": [],
   "source": [
    "from haystack.nodes import PromptTemplate\n",
    "\n",
    "title_generator = PromptTemplate(\n",
    "    prompt=\"Provide a short, descriptive title for the given piece of news. News: {documents}; Title:\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "jttb_ROnWYwS"
   },
   "source": [
    "2. To use the new template, pass `title_generator` as the `prompt_template` to the `prompt()` method:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zX_nALriWXxx",
    "outputId": "8f8fb802-5116-4e85-b0b3-8ddb1138ae5a"
   },
   "outputs": [],
   "source": [
    "prompt_node.prompt(prompt_template=title_generator, documents=news)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "8t9pdZo9ZKn9"
   },
   "source": [
    "There you go! You should have the titles for your news articles ready. Let's now categorize them."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "eH4QbygONONX"
   },
   "source": [
    "## Categorizing Documents with PromptNode\n",
    "\n",
    "You can customize PromptTemplates as much as you need. Let's try to create a template to categorize the news articles. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "FAELXO0MhbrV"
   },
   "source": [
    "1. Define the `{news}` and `{categories}` parameters. As we will accept an list of strings as `categories`, we need to join the list before injecting categories to the prompt with `\", \".join(categories)` function. See how you can [further customize prompt variables](https://docs.haystack.deepset.ai/docs/prompt_node#prompttemplate-structure) in the documentation.\n",
    "Finally, in the prompt, ask the model not to categorize the news if it doesn't fit in the provided category list: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xHgOjTwkNONY"
   },
   "outputs": [],
   "source": [
    "news_categorizer = PromptTemplate(\n",
    "    prompt=\"Given the categories: {', '.join(categories)}; classify the news: {documents}. Only pick a category from the list, otherwise say: no suitable category\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "nC4gVHCyuXHl"
   },
   "source": [
    "2. Run the `prompt()` method with the `news_categorizer` template:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "taktlLIfNONY",
    "outputId": "1aceac5a-403d-4bca-cfc6-0a28e4d11691"
   },
   "outputs": [],
   "source": [
    "prompt_node.prompt(\n",
    "    prompt_template=news_categorizer, documents=news, categories=[\"sport\", \"economics\", \"culture\"]\n",
    ")  # Answer: ['economics', 'science', 'culture', 'sport']"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "aU1e7ak4ZKn9"
   },
   "source": [
    "Congratulations! You've summarized your documents, generated titles for them, and put them into categories, all using custom prompt templates. "
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "76c0ba09435b0d7ab7f3e4f9fd6b1554cdb12adf349a7242a1470606c432d777"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}