Skip to content

Latest commit

 

History

History
574 lines (440 loc) · 20.2 KB

README.md

File metadata and controls

574 lines (440 loc) · 20.2 KB
Raw

puppet-st2

Build Status Coverage Status Puppet Forge Endorsement Puppet Forge Version Puppet Forge Downloads puppetmodule.info docs Join our community Slack

Table of Contents

Description

Module to manage StackStorm with Puppet.

Setup

What st2 Affects

The st2 module configures the existing into a complete and dedicated StackStorm node with the following components:

  • StackStorm
  • MongoDB
  • Postgres
  • RabbitMQ
  • Redis
  • Nginx
  • NodeJS

Setup Requirements

This module, similar to normal StackStorm installs, expects to be run on a blank system without any existing configurations. The only hard requirements are on the Operating System and machine specs. See Limitations and the official StackStorm system requirements.

Module Dependencies

This module installs and configures all of the components required for StackStorm. In order to not repeat others work, we've utilized many existing modules from the forge. We manage the module dependenies using a Puppetfile for each OS we support. These Puppetfile can be used both with r10k and librarian-puppet.

Beginning with st2

For a full installation on a single node, a profile already exists to get you setup and going with minimal effort. Simply:

puppet module install stackstorm-st2
puppet apply -e "include st2::profile::fullinstall"

Usage

Reference Documentation

This module uses Puppet Strings as the documentation standard. An live version is available online at puppetmodule.info/m/stackstorm-st2. A markdown version is available directly in this repo in REFERENCE.md.

Configuration

This module aims to provide sane default configurations, but also stay out of your way in the event you need something more custom. To accomplish this, this module uses the Roles/Profiles pattern. Included in this module are several modules that come with sane defaults that you can use directly or use to compose your own site-specific profile for StackStorm installation.

Configuration can be done directly via code composition, or set via Hiera data bindings. A few notable parameters to take note of:

  • st2::version - Version of ST2 to install. This will be set as the ensure value on the st2 packages. The default is present resulting in the most up to date packages being installed initially. If you would like to hard code to an older version you can specify that here (ex: 2.6.0). Note Setting this to latest is NOT recommended. It will cause the StackStorm packages to be automatically updated without the proper upgrade steps being taken (proper steps detailed here: https://docs.stackstorm.com/install/upgrades.html)

  • st2::python_version - Version to Python to use. The default is 'system' and the system python package will be installed, whatever version that is for your OS. To explicitly install Python 3.8 specify '3.8' if on RHEL/CentOS 7. If on Ubuntu 16.04 specify 'python3.8'. Notes

    • RHEL 7 - The Red Hat subscription repo 'rhel-7-server-optional-rpms' will need to be enabled prior to running this module.
    # CentOS/RHEL 7
class { 'st2':
  python_version => '3.8',
}

# Ubuntu 18.04/20.04
class { 'st2':
  python_version            => 'python3.8',
}

contain st2::profile::fullinstall

All other classes are documented with Puppetdoc. Please refer to specific classes for use and configuration.

Profiles

  • st2::profile::client - Profile to install all client libraries for st2
  • st2::profile::fullinstall - Full installation of StackStorm and dependencies
  • st2::profile::mistral - Install of OpenStack Mistral
  • st2::profile::mongodb - st2 configured MongoDB installation
  • st2::profile::nodejs - st2 configured NodeJS installation
  • st2::profile::python - Python installed and configured for st2
  • st2::profile::rabbitmq - st2 configured RabbitMQ installation
  • st2::profile::redis - st2 configured Redis installation
  • st2::proflle::server - st2 server components
  • st2::profile::web - st2 web components
  • st2::profile::chatops - st2 chatops components
  • st2::profile::metrics - st2 set up metrics

Installing and Configuring Packs

StackStorm packs can be installed and configured directly from Puppet. This can be done via the st2::pack and st2::pack::config defined types.

Installation/Configuration via modules:

  # install pack from the exchange
  st2::pack { 'linux': }

  # install pack from a git URL
  st2::pack { 'private':
    repo_url => 'https://private.domain.tld/git/stackstorm-private.git',
  }

  # install pack and apply configuration
  st2::pack { 'slack':
    config   => {
      'post_message_action' => {
        'webhook_url' => 'XXX',
      },
    },
  }

Installation/Configuration via Hiera:

st2::packs:
  linux:
    ensure: present
  private:
    ensure: present
    repo_url: https://private.domain.tld/git/stackstorm-private.git
  slack:
    ensure: present
    config:
      post_message_action:
        webhook_url: XXX

Configuring Authentication

StackStorm uses a pluggable authentication system where auth is delegated to an external service called a "backend". The st2auth service can be configured to use various backends (only one active). For more information on StackStorm authentication see the authentication documentation page.

The following backends are currently available:

  • flat_file - Authenticates against an htpasswd file (default) link
  • keystone - Authenticates against an OpenStack Keystone service link
  • ldap - Authenticates against an LDAP server such as OpenLDAP or Active Directory link
  • mongodb - Authenticates against a collection named users in MongoDB link
  • pam - Authenticates against the PAM Linux service link

By default the flat_file backend is used. To change this you can configure it when instantiating the st2 class in a manifest file:

class { 'st2':
  auth_backend => 'ldap',
}

Or in Hiera:

st2::auth_backend: ldap

Each backend has their own custom configuration settings. The settings can be found by looking at the backend class in the manifests/st2/auth/ directory. These parameters map 1-for-1 to the configuration options defined in each backends GitHub page (links above). Backend configurations are passed in as a hash using the auth_backend_config option. This option can be changed when instantiating the st2 class in a manifest file:

class { 'st2':
  auth_backend        => 'ldap',
  auth_backend_config => {
    host            => 'ldap.domain.tld',
    bind_dn         => 'cn=ldap_stackstorm,ou=service accounts,dc=domain,dc=tld',
    base_dn         => 'dc=domain,dc=tld',
    scope           => 'subtree',
    id_attr         => 'username',
    bind_pw         => 'some_password',
    group_dns       => ['"cn=stackstorm_users,ou=groups,dc=domain,dc=tld"'],
    account_pattern => 'userPrincipalName={username}',
  },
}

Or in Hiera:

st2::auth_backend: "ldap"
st2::auth_backend_config:
  host: "ldaps.domain.tld"
  use_tls: false
  use_ssl: true
  port: 636
  bind_dn: 'cn=ldap_stackstorm,ou=service accounts,dc=domain,dc=tld'
  bind_pw: 'some_password'
  chase_referrals: false
  base_dn: 'dc=domain,dc=tld'
  group_dns:
    - '"cn=stackstorm_users,ou=groups,dc=domain,dc=tld"'
  scope: "subtree"
  id_attr: "username"
  account_pattern: "userPrincipalName={username}"

Configuring ChatOps

Configuration via Hiera:

  # character to trigger the bot that the message is a command
  # example: !help
  st2::chatops_hubot_alias: "'!'"

  # name of the bot in chat, sometimes requires special characters like @
  st2::chatops_hubot_name: '"@RosieRobot"'

  # API key generated by: st2 apikey create
  st2::chatops_api_key: '"xxxxyyyyy123abc"'

  # Public URL used by ChatOps to offer links to execution details via the WebUI.
  st2::chatops_web_url: '"stackstorm.domain.tld"'

  # install and configure hubot adapter (rocketchat, nodejs module installed by nodejs)
  st2::chatops_adapter:
    hubot-adapter:
      package: 'hubot-rocketchat'
      source: 'git+ssh://[email protected]:npm/hubot-rocketchat#master'

  # adapter configuration (hash)
  st2::chatops_adapter_conf:
    HUBOT_ADAPTER: rocketchat
    ROCKETCHAT_URL: "https://chat.company.com:443"
    ROCKETCHAT_ROOM: 'stackstorm'
    LISTEN_ON_ALL_PUBLIC: true
    ROCKETCHAT_USER: st2
    ROCKETCHAT_PASSWORD: secret123
    ROCKETCHAT_AUTH: password
    RESPOND_TO_DM: true

Scaling out services

This module supports scaling out workflowengine, scheduler, rulesengine, and notifier services per the ST2 Documentation.

This would be something that you might consider doing if you have alot of rules running or if you have alot of workflows running in parrallel and/or you have alot of nested workflows and have a server that can be higher on CPU and Memory to allow more processes to run at the same time.

Configuration all services:

class { 'st2':
  python_version     => '3.8',
  workflowengine_num => 4,
  scheduler_num      => 2,
  rulesengine_num    => 1,
  notifier_num       => 1,
}

Or configure individual:

class { 'st2::workflowengine':
  workflowengine_num => 4,
}

Using API Key for pack and K/V management

It is now possible to use StackStorm API Keys to authenticate the st2 commands that are executed during puppet runs, thus replacing the need for username/password and auth token based auth for those commands. To configure, first generate an API key using:

$ st2 apikey create -m '{"used_by": "puppet-st2-cli"}'

Then add the generated API Key to your puppet config:

class { 'st2':
  cli_apikey => 'myapikey'
}

OR in Hiera

st2::cli_apikey: myapikey

Then proceed to using pack and/or k/v functionality normally. When the commands are executed, they will now pass the apikey as auth instead of generated an auth token from the supplied cli_username and cli_password.

Tasks

This module provides several tasks for interacting with StackStorm. These tasks are modeled after the st2 CLI command, names of the tasks and parameters reflect this. Under the hood, the tasks invoke the st2 CLI command so they must be executed on a node where StackStorm is installed.

Task List

  • st2::key_decrypt - Decrypts an encrypted key/value pair
  • st2::key_get - Retrieves the value for a key from the datastore
  • st2::key_load - Loads a list of key/value pairs into the datastore
  • st2::pack_install - Installs a list of packs
  • st2::pack_list - Get a list of installed packs
  • st2::pack_register: Registers a list of packs based from paths on the filesystem
  • st2::pack_remove - Removes a list of packs
  • st2::rule_disable: Disables a rule
  • st2::rule_list: Lists all rules, or just the rules in a given pack
  • st2::run: Runs a StackStorm action

Task Authentication

Tasks that interact with the st2 CLI command require authentication with the StackStorm instance. There are three options for authentication:

  • API Key
  • Auth token
  • Username/password

Using Tasks With API Key

API keys are the recommended way for systems to authenticate with StackStorm. To do this via a task, you would first create an API key in StackStorm:

$ st2 apikey create -m '{"used_by": "bolt"}'

Copy the API key parameter in the output, and then use it when invoking one of the tasks in this module via the api_key parameter:

Usage via command line:

bolt task run st2::key_get key="testkey" api_key='xyz123'

Usage in a plan:

$res = run_task('st2::key_get', $stackstorm_target,
                key        => 'testkey',
                api_key    => $api_key)

Using Tasks With Auth Tokens

Auth tokens can be used by bolt to communicate with StackStorm. First, the user needs to create an auth token, then pass it in via the auth_token parameter

$ st2 auth myuser

Copy the auth token in the output, and then use it when invoking one of the tasks in this module:

Usage via command line:

bolt task run st2::key_get key="testkey" auth_token='xyz123'

Usage in a plan:

$res = run_task('st2::key_get', $stackstorm_target,
                key        => 'testkey',
                auth_token => $auth_token)

Using Tasks With Username and Password

Finally bolt can accept username/passwords to communicate with StackStorm.

Usage via command line:

bolt task run st2::key_get key="testkey" username="myuser" password="xyz123"

Usage in a plan:

$res = run_task('st2::key_get', $stackstorm_target,
                key      => 'testkey',
                username => $username,
                password => $password)

Metrics

This module provides the ability to set up the metrics endpoints to expose different metrics built into StackStorm as documented here: https://docs.stackstorm.com/reference/metrics.html

There is also an addition metric that can be auto created which exposes the basic health for NGINX called a basic status page.

class { 'st2':
  python_version             => '3.8',
  metrics_include            => true,
  nginx_basicstatus_enabled  => true,
}

Once these metrics are setup and enabled it gives the ability to scrape those metrics with an outside source like telegraf or prometheus or any other scraping tool.

Limitations

Supported platforms

  • Ubuntu 18.04
  • Ubuntu 20.04
  • RHEL/CentOS 7

Supported Puppet versions

  • Puppet 6
  • Puppet 7

⚠️ End-of-Support Notice - Mistral

Support for Mistral has been dropped as of StackStorm 3.3.0.

As of version 1.8 this module no longer supports Mistral (and subsequently PostgreSQL) Neither Mistral nor Postgresql will be installed or managed by this module.

⚠️ End-of-Support Notice - Ubuntu 16.04

Support for Ubuntu 16.04 has been dropped as of StackStorm 3.5.0

As of version 2.3 this module no longer supports Ubuntu 16.04

⚠️ End-of-Support Notice - CentOS 6

Support for CentOS 6 has been dropped as of StackStorm 3.3.0.

As of version 1.8 this module no longer supports CentOS 6, so changes will not be tested against this platform.

⚠️ Deprecation Notice - Puppet 5

Puppet 5 reaches End of Life on 2021-12-31. As of version 2.0 use of Puppet 5 with this module is officially deprecated.

  • This module no longer tests against Puppet 5 in its build matrix.
  • The next major release of the module will drop support for Puppet 5 by adjusting the minimum supported Puppet version in metadata.json.

⚠️ Deprecation Notice - Puppet 4

Puppet 4 reached End of Life on 2018-12-31. As of version 1.4 use of Puppet 4 with this module is officially deprecated.

  • As of version 1.5.0 this module no longer tests against Puppet 4 in its build matrix.
  • The next major release of the module will drop support for Puppet 4 by adjusting the minimum supported Puppet version in metadata.json.

⚠️ Deprecation Notice - Puppet 3

This module no longer supports Puppet 3 as of version 1.1

Upgrading StackStorm

By default this module does NOT handle upgrades of StackStorm. It is the responsiblity of the end user to upgrade StackStorm according to the upgrade documenation.

In a future release a Puppet task may be included to perform these update on demand using bolt.

Development

Contributions to this module are more than welcome! If you have a problem with the module or would like to see a new feature, please raise an issue. If you are amazing, find a bug or implement a new feature and want to add it to the module, please submit a Pull Request.

Maintainers

Help

If you're in stuck, our community always ready to help, feel free to:

Your contribution is more than welcome!