influx-demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "hide_input": false
   },
   "source": [
    "# Getting SQuaSH metrics into InfluxDB\n",
    "\n",
    "InfluxDB is a time series database, Chronograf is the UI for exploratory analysis and dashboarding, and Kapacitor is the component responsible for alerting (it also does series processing in general). This investigation is to evaluate the use of the  InfluxDB + Cronograf + Kapacitor for LSST DM in implementing parts of SQuaSH and for the DM Engineering Facilities Database (DM-EFD) monitoring.\n",
    "\n",
    "\n",
    "## Getting started \n",
    "\n",
    "Check the [InfluxDB start guide](https://docs.influxdata.com/influxdb/v1.6/introduction/getting-started/) to learn the basics of interacting with the database and the [SQL to InfluxDB terminology crosswalk](https://docs.influxdata.com/influxdb/v1.6/concepts/crosswalk/) to understand how InfluxDB is different from a relational database.\n",
    "\n",
    "## Points, measurements, tags and fields\n",
    "\n",
    "**Points** are discrete samples of a metric. Points are written to InfluxDB using the \"Line Protocol\":\n",
    "\n",
    "```\n",
    "<measurement>[,<tag-key>=<tag-value>...] <field-key>=<field-value>[,<field2-key>=<field2-value>...] [unix-nano-timestamp]\n",
    "```\n",
    "\n",
    "**Measurements** act as a container for tags, fields, and the timestamp. The measurement name is the description of the  Measurement names are strings, they describe the data that are stored in the associated fields. A measurement is conceptually similar to an SQL table.\n",
    "\n",
    "**Tags** are used as metadata while **fields** corresponds to your data. An important difference is that tags are indexed while fields are not. So you should consider to turn fields into tags if you are filtering on them.\n",
    "\n",
    "InfluxDB is a schemaless database. You can add new measurements, tags, and fields at any time.\n",
    "\n",
    "Check also InfluxDB [key concepts](https://docs.influxdata.com/influxdb/v1.6/concepts/key_concepts).\n",
    "\n",
    "\n",
    "## Writing data to InfluxDB\n",
    "\n",
    "There are several ways to write data to InfluxDB, here we show three of them using the CLI client, the HTTP API and the Python client.\n",
    "\n",
    "1. Insert a single time series point using the [CLI](https://docs.influxdata.com/influxdb/v1.6/introduction/getting-started/#writing-and-exploring-data)\n",
    "\n",
    "```\n",
    "> USE mydb\n",
    "> INSERT cpu,host=server01,region=us_west value=0.64\n",
    "```\n",
    "\n",
    "2. Inserting a single time series point using the [HTTP API](https://docs.influxdata.com/influxdb/v1.6/guides/writing_data/)\n",
    "\n",
    "```\n",
    "$ curl -XPOST \"http://localhost:8086/write?db=mydb\" -d 'cpu,host=server01,region=uswest value=0.64'\n",
    "```\n",
    "\n",
    "3. Insert a single time series point using the [influxdb Python client](https://github.com/influxdata/influxdb-python)\n",
    "\n",
    "```python\n",
    "data = [\n",
    "    {\n",
    "        \"measurement\": \"cpu\",\n",
    "        \"tags\": {\n",
    "            \"host\": \"server01\",\n",
    "            \"region\": \"us_west\"\n",
    "        },\n",
    "        \"fields\": {\n",
    "            \"value\": 0.64\n",
    "        }\n",
    "    }\n",
    "]\n",
    "from influxdb import InfluxDBClient\n",
    "client = InfluxDBClient('localhost', 8086, 'root', 'root', 'mydb')\n",
    "client.write_points(data)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running InfluxDB + Chronograf + Kapacitor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the `docker-compose` configuration in this repository to run a local instance of InfluxDB + Chronograf + Kapacitor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "docker-compose up -d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the terminal, you can open the InfluxDB CLI with\n",
    "\n",
    "```bash\n",
    "$ docker-compose run influxdb-cli\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or connect to Chronograf at http://localhost:8888. The most interesting functionalities of Chronograf are the \"Data Explorer\", \"Dashboards\", \"Alerting\" and the \"Log Viewer\".\n",
    "\n",
    "SQuaSH metrics follow the concepts developed in [lsst.verify](https://sqr-019.lsst.io/). Here we present two approaches on how SQuaSH metrics can be inserted into InfluxDB. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Approach #1: each metric in SQuaSH is an InfluxDB measurement "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "from influxdb import InfluxDBClient\n",
    "\n",
    "client = InfluxDBClient('localhost', 8086, 'root', 'root', 'squash')\n",
    "client.create_database('squash')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "SQUASH_API_URL = \"https://squash-restful-api-demo.lsst.codes/\"\n",
    "jobs = requests.get(SQUASH_API_URL + \"/jobs\").json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "for job_id in jobs['ids']:\n",
    "\n",
    "    r = requests.get(SQUASH_API_URL + \"/job/{}\".format(job_id)).json()\n",
    "    \n",
    "    # Skip datasets we don't want\n",
    "    if r['ci_dataset'] == 'unknown' or r['ci_dataset'] == 'decam':\n",
    "        continue\n",
    "    \n",
    "    print('Sending point for job {}...'.format(job_id))\n",
    "\n",
    "    points = []\n",
    "    \n",
    "    for meas in r['measurements']:\n",
    "        point = dict()\n",
    "        point['measurement'] = meas['metric']\n",
    "        point['tags'] =  {'filter_name': r['meta']['filter_name'], \n",
    "                          'dataset':  r['ci_dataset']}\n",
    "        point['time'] = r['date_created']\n",
    "        point['fields'] = {'value': meas['value']}\n",
    "        points.append(point)\n",
    "        \n",
    "    client.write_points(points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Approach #2: each verification package in SQuaSH is an InfluxDB measurement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from influxdb import InfluxDBClient\n",
    "\n",
    "client = InfluxDBClient('localhost', 8086, 'root', 'root', 'squash_2')\n",
    "client.create_database('squash_2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "SQUASH_API_URL = \"https://squash-restful-api-demo.lsst.codes/\"\n",
    "jobs = requests.get(SQUASH_API_URL + \"/jobs\").json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "for job_id in jobs['ids']:\n",
    "\n",
    "    r = requests.get(SQUASH_API_URL + \"/job/{}\".format(job_id)).json()\n",
    "    \n",
    "    if r['ci_dataset'] == 'unknown' or r['ci_dataset'] == 'decam':\n",
    "        continue\n",
    "    \n",
    "    print('Sending point for job {}...'.format(job_id))\n",
    "\n",
    "    points = []\n",
    "    \n",
    "    for meas in r['measurements']:\n",
    "        point = dict()\n",
    "        # we could add the verification package in addition to metric to improve \n",
    "        # this\n",
    "        point['measurement'] = meas['metric'].split('.')[0]\n",
    "        point['tags'] =  {'filter_name': r['meta']['filter_name'], \n",
    "                          'dataset':  r['ci_dataset']}                  \n",
    "        point['time'] = r['date_created']\n",
    "        point['fields'] = {meas['metric']: meas['value']}\n",
    "        points.append(point) \n",
    "    \n",
    "    client.write_points(points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It turns out that the second approach fits better along the InfluxDB concepts. You can verify that by exploring the two InfluxDB databases `squash` and `squash_2` just created using [Chronograf](http://localhost:8888/sources/0/chronograf/data-explorer)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alerting\n",
    "\n",
    "Alerting is done with [Kapacitor](https://docs.influxdata.com/kapacitor) which integrates with Chronograf.\n",
    "\n",
    "\n",
    "TODO: demonstrate how to create alerting rules programatically using the metric definition and specifications from the SQuaSH API (similar to what we did when Investigating Honeycomb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion\n",
    "\n",
    "### InfluxDB\n",
    "\n",
    "- Built-in HTTP API (but not RESTful)\n",
    "- CLI\n",
    "- InfluxDB Python client\n",
    "- SQL-like query language\n",
    "- Functions like mean(), median(), min() and max()\n",
    "\n",
    "\n",
    "### Chronograf\n",
    "\n",
    "- Written in Go and React.js, implements most of Grafana fucntionalities but works with InfluxDB only.\n",
    "- Explore mode: query builder is great, we can have multiple queries in a single graph\n",
    "- Query builder: flexible, has the ability to edit and test the actual query before submission\n",
    "- Easy to create dashboards, nice control of the graph properties, has a presentation mode\n",
    "- [Template variables are great to customize dashboards](https://docs.influxdata.com/chronograf/v1.6/guides/dashboard-template-variables/)\n",
    "- [You can export your dashboard definition to JSON](https://www.influxdata.com/blog/chronograf-dashboard-definitions/)\n",
    "- Easy to configure alerting rules\n",
    "- You can export data from the system (download CSV from the UI, from the [HTTP API](http://localhost:8888/docs#tag/measurements) or from thr CLI)\n",
    "- [Combine metrics and logs dashboard](https://docs.influxdata.com/chronograf/v1.6/guides/analyzing-logs/#logs-in-dashboards).\n",
    "- Combine data from two different databases in the same plot/dashboard. That's a really imporant feature in order to correlate metrics from the DM-EFD and SQuaSH. \n",
    "\n",
    "## Kapacitor\n",
    "\n",
    "- Alerting rules\n",
    "- Time series processing \n",
    "- [HTTP API](https://docs.influxdata.com/kapacitor/v1.5/working/api/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "influx-demo",
   "language": "python",
   "name": "influx-demo"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}