Tinybird Organization Metrics Exporter

Monitor Tinybird usage across your Tinybird organization. This project uses Tinybird's Service Data Sources to aggregate and publish organizational metrics as endpoints in Prometheus format for quick integration with common monitoring tools.

Features

• Includes a set of Pipes that consolidate and process organizational metrics.
• Exposes all key metrics through a single Pipe Endpoint in Prometheus format for streamlined monitoring.
• Provides essential insights to help you monitor your usage, detect anomalies, and set up alerts in critical areas such as:
  • Data ingestion: Track the volume and frequency of ingested data.
  • Pipe endpoint usage: Analyze requests and identify unusual traffic patterns.
  • Storage: Monitor storage usage to optimize resources and avoid limits.
  • Jobs: Keep track of the status and performance of scheduled jobs.

Setup

Prerequisites

1. You must have a Tinybird organization.
2. You need to be an admin of the organization to access organization Service Data Sources.

Steps

1. Deploy to a Tinybird Workspace

2. Use the Prometheus Endpoint

• Access the Prometheus metrics endpoint at:

https://api.tinybird.co/v0/pipes/organization_metrics.prometheus

• Replace api.tinybird.co with your Tinybird host if the workspace is in a different region.
• Use admin user@domain Token of an Organization admin to authenticate requests.

To scrape the Tinybird metrics endpoint, you can configure your prometheus.yml file as follows:

scrape_configs:
  - job_name: tinybird_org_metrics
    scrape_interval: 15s  # Adjust the scrape interval as needed
    scheme: 'https'
    static_configs:
      - targets: 
        - 'api.tinybird.co'  # Adjust this for your region if necessary
    metrics_path: '/v0/pipes/organization_metrics.prometheus'
    bearer_token: '<admin-user-token>'  # From an Organization admin

If you use a different setup, such as Datadog + OpenMetrics, you can use the following configuration:

instances:
  - prometheus_url: https://api.tinybird.co/v0/pipes/organization_metrics.prometheus?token=<admin-user-token>
    namespace: tinybird
    metrics:
      - "*"
    tags:
      - tinybird
    max_returned_metrics: 10000 # Adjust this value as needed

Remember to replace api.tinybird.co with your Tinybird host if the Workspace is in a different region.

Start monitoring your Tinybird organization with Prometheus metrics! 🎉

Grafana Quick Start

We've included a sample dashboard config for Grafana to help you get started, see the JSON file.

Import the dashboard with the following steps:

1. Go to Grafana and click on the Dashboards menu
2. Select New => Import
3. Click on Upload dashboard JSON file
4. Select the JSON file and click Upload
5. Select the Prometheus datasource
6. Click Import
7. Done!

• Note: The dashboard is not perfect, it's just a starting point. Modify it as you need!
• Note: The dashboard uses the Prometheus configuration in this repo, with the job name tinybird_org_metrics.

Datadog Quick Start

Add the following configuration to your OpenMetrics Datadog agent conf.yaml file:

instances:
  - prometheus_url: 'https://api.tinybird.co/v0/pipes/organization_metrics.prometheus?token=<admin-user-token>'
    namespace: tinybird_org_metrics
    metrics:
      - "*"
    max_returned_metrics: 700000

• Replace api.tinybird.co with your Tinybird host if the workspace is in a different region.
• Use admin user@domain Token of an Organization admin to authenticate requests.

We've included a sample dashboard config for Datadog that you can use to get started, see the JSON file.

Import the dashboard with the following steps:

1. Go to Datadog and click on the Dashboards menu
2. Select New Dashboard
3. Click on New Dashboard in the modal window
4. Click the settings gear icon (⚙️)
5. Select Import Dashboard
6. Browse to the JSON file
7. Select Yes, replace
8. Done!

• Note: The dashboard is not perfect, it's just a starting point. Modify it as you need!