design.md

Deployment Manager Design

Overview

Deployment Manager (DM) is a service that runs in a Kubernetes cluster, supported by a command line interface. It provides a declarative YAML-based language for configuring Kubernetes resources, and a mechanism for deploying, updating, and deleting configurations. This document describes the configuration language, the API model, and the service architecture in detail.

Configuration Language

DM uses a YAML-based configuration language with a templating mechanism. A configuration is a YAML file that describes a list of resources. A resource has three properties:

• name: the name to use when managing the resource
• type: the type of the resource being configured
• properties: the configuration properties of the resource

Here's a snippet from a typical configuration file:

resources:
- name: my-rc
  type: ReplicationController
  properties:
    metadata:
      name: my-rc
    spec:
      replicas: 1
    ...
- name: my-service
  type: Service
  properties:
    ...

It describes two resources:

• A replication controller named my-rc, and
• A service named my-service

Types

Resource types are either primitives or templates.

Primitives

Primitives are types implemented by the Kubernetes runtime, such as:

• Pod
• ReplicationController
• Service
• Namespace
• Secret

DM processes primitive resources by passing their properties directly to kubectl to create, update, or delete the corresponding objects in the cluster.

(Note that DM runs kubectl server side, in a container.)

Templates

Templates are abstract types created using Python or Jinja. A template takes a set of properties as input, and must output a valid YAML configuration. Properties are bound to values when a template is instantiated by a configuration.

Templates are expanded before primitive resources are processed. The configuration produced by expanding a template may contain primitive resources and/or additional template invocations. All template invocations are expanded recursively until the resulting configuration is a list of primitive resources.

(Note, however, that DM preserves the template hierarchy and any dependencies between resources in a layout that can be used to reason programmatically about the structure of the resulting collection of resources created in the cluster, as described in greater detail below.)

Here's an example of a template written in Jinja:

resources:
- name: {{ env['name'] }}-service
  type: Service
  properties:
    prop1: {{ properties['prop1'] }}
    ...

As you can see, it's just a YAML file that contains expansion directives. For more information about the kinds of things you can do in a Jinja based template, see the Jina documentation.

Here's an example of a template written in Python:

import yaml

def GenerateConfig(context):
  resources = [{
    'name': context.env['name'] + '-service',
    'type': 'Service',
    'properties': {
      'prop1': context.properties['prop1'],
      ...
    }
  }]

  return yaml.dump({'resources': resources})

Of course, you can do a lot more in Python than in Jinja, but basic things, such as simple parameter substitution, may be easier to implement and easier to read in Jinja than in Python.

Templates provide access to multiple sets of data that can be used to parameterize or further customize configurations:

• env: a map of key/value pairs from the environment, including pairs defined by Deployment Manager, such as deployment, name, and type
• properties: a map of the key/value pairs passed in the properties section of the template invocation
• imports: a map of import file names to file contents for all imports originally specified for the configuration

In Jinja, these variables are available in the global scope. In Python, they are available as properties of the context object passed into the GenerateConfig method.

Template schemas

A template can optionally be accompanied by a schema that describes it in more detail, including:

• info: more information about the template, including long description and title
• imports: any files imported by this template (may be relative paths or URLs)
• required: properties that must have values when the template is expanded
• properties: A JSON Schema description of each property the template accepts

Here's an example of a template schema:

info:
  title: The Example
  description: A template being used as an example to illustrate concepts.

imports:
- path: helper.py

required:
- prop1

properties:
  prop1:
    description: The first property
    type: string
    default: prop-value

When a schema is provided for a template, DM uses it to validate properties passed to the template by its invocation, and to provide default values for properties that were not given values.

Schemas must be supplied to DM along with the templates they describe.

Supplying templates

Templates can be supplied to DM in two different ways:

• They can be passed to DM along with configurations that import them, or
• They can be retrieved by DM from public HTTP endpoints for configurations that reference them.

Template imports

Configurations can import templates using path declarations. For example:

imports:
- path: example.py

resources:
- name: example
  type: example.py
  properties:
    prop1: prop-value

The imports list is not understood by the Deployment Manager service. It's a directive used by client-side tools to specify what additional files should be included when passing the configuration to the API.

If you are calling the Deployment Manager service directly, you must embed the imported templates in the configuration passed to the API.

Template references

Configurations can also reference templates using URLs for public HTTP endpoints. DM will attempt to resolve template references during expansion. For example:

resources:
- name: my-template
  type: https://raw.githubusercontent.com/my-template/my-template.py
  properties:
    prop1: prop-value

When resolving template references, DM assumes that templates are stored in directories, which may also contain schemas, examples and other supporting files. It therefore processes template references as follows:

1. Attempt to fetch the template, and treat it as an import.
2. Attempt to fetch the schema for the template from <base path>/<template name>.schema
3. Attempt to fetch files imported by the schema from <base path>/<import path>

Referring to the previous example,

• the base path is https://raw.githubusercontent.com/my-template,
• the template name is my-template, and
• the schema name is my-template.schema

If we include a configuration that uses the template as an example, then the directory that contains my-template might look like this:

example.yaml
my-template.py
my-template.py.schema
helper.py

Value references

Resources can reference values from other resources. The version of Deployment Manager running in the Google Cloud Platform uses references to understand dependencies between resources and properly order the operations it performs on a configuration.

(Note that this version of DM doesn't yet order operations to satisfy dependencies, but it will soon.)

A reference follows this syntax: $(ref.NAME.PATH), where NAME is the name of the resource being referenced, and PATH is a JSON path to the value in the resource object.

For example:

$(ref.my-service.metadata.name)

In this case, my-service is the name of the resource, and metadata.name is the JSON path to the value being referenced.

API Model

DM exposes a set of RESTful collections over HTTP/JSON.

Deployments

Deployments are the primary resources managed by the Deployment Manager service. The inputs to a deployment are:

• name: the name by which the deployment can be referenced once created
• configuration: the configuration file, plus any imported files (templates, schemas, helper files used by the templates, etc.).

Creating, updating or deleting a deployment creates a new manifest for the deployment. When deleting a deployment, the deployment is first updated to an empty manifest containing no resources, and then removed from the system.

Deployments are available at the HTTP endpoint:

http://manager-service/deployments

Manifests

A manifest is created for a deployment every time it is changed. It contains three key components:

• inputConfig: the original input configuration for the manifest
• expandedConfig: the expanded configuration describing only primitive resources
• layout: the hierarchical structure of the configuration

Manifests are available at the HTTP endpoint:

http://manager-service/deployments/<deployment>/manifests

Expanded configuration

Given a new inputConfig, DM expands all template invocations recursively, until the result is a flat set of primitive resources. This final set is stored as the expandedConfig and is used to instantiate the primitive resources.

Layout

Using templates, callers can build rich, deeply hierarchical architectures in their configurations. Expansion flattens these hierarchies to simplify the process of instantiating the primitive resources. However, the structural information contained in the original configuration has many potential uses, so rather than discard it, DM preserves it in the form of a layout.

The layout looks a lot like the original configuration. It is a YAML file that describes a list of resources. Each resource contains the name, type and properties from the original configuration, plus a list of nested resources discovered during expansion. The resulting structure looks like this:

• name: name of the resource
• type: type of the resource
• properties: properties of the resource, set only for templates
• resources: sub-resources from expansion, set only for templates

Here's an example of a layout:

resources:
- name: rs
  type: replicatedservice.py
  propertes:
    replicas: 2
  resources:
  - name: rs-rc
    type: ReplicationController
  - name: rs-service
    type: Service

In this example, the top level resource is a replicated service named rs, defined by the template named replicatedservice.py. Expansion produced the two nested resources: a replication controller named rs-rc, and a service named rs-service.

Using the layout, callers can discover that rs-rc and rs-service are part of the replicated service named rs. More importantly, if rs was created by the expansion of a larger configuration, such as one that described a complete application, callers could discover that rs-rc and rs-service were part of the application, and perhaps even that they were part of a RabbitMQ cluster in the application's mid-tier.

Types

The types API provides information about types used in the cluster.

It can be used to list all known types used by active deployments:

http://manager-service/types

Or to list all active instances of a specific type in the cluster:

http://manager-service/types/<type>/instances

Passing all as the type name shows all instances of all types in the cluster. The following information is reported for type instances:

• name: name of resource
• type: type of resource
• deployment: name of deployment in which the resource resides
• manifest: name of manifest in which the resource configuration resides
• path: JSON path to the entry for the resource in the manifest layout

Architecture

The Deployment Manager service is manages deployments within a Kubernetes cluster. It has three major components. The following diagram illustrates the components and the relationships between them.

Currently, there are two caveats in the service implementation:

• Synchronous API: the API currently blocks on all processing for a deployment request. In the future, this design will change to an asynchronous operation-based mode.
• In-memory state: the service currently stores all state in memory, so it will lose all knowledge of deployments and related objects on restart. In the future, the service will persist all state in the cluster.

Manager

The manager service acts as both the API server and the workflow engine for processing deployments. It handles a POST to the /deployments collection as follows:

1. Create a new deployment with a manifest containing inputConfig from the user request
2. Call out to the expandybird service to expand the inputConfig
3. Store the resulting expandedConfig and layout
4. Call out to the resourcifier service to instantiate the primitive resources described by the expandedConfig
5. Respond with success or error messages to the original API request

GET, PUT and DELETE operations are processed in a similar manner, except that:

• No expansion is performed for GET or DELETE
• The primitive resources are updated for PUT and deleted for DELETE

The manager is responsible for saving the information associated with deployments, manifests, type instances, and other resources in the Deployment Manager model.

Expandybird

The expandybird service takes in a configuration, performs all necesary template expansions, and returns the resulting flat configuration and layout. It is completely stateless.

Because templates are written in Python or Jinja, the actual expansion process is performed in a sub-process that runs a Python interpreter. A new sub-process is created for every request to expandybird.

Currently, expansion is not sandboxed, but templates should be reproducable, hermetically sealed entities. Future designs may therefore introduce a sandbox to limit external interaction, such as network or disk access, during expansion.

Resourcifier

The resourcifier service takes in a flat expanded configuration describing only primitive resources, and makes the necessary kubectl calls to process them. It is totally stateless, and handles requests synchronously.

The resourcifier runs kubectl in a sub-process within its container. A new sub-process is created for every request to resourcifier.

It returns either success or error messages encountered during resource processing.