Skip to content

Latest commit

 

History

History
178 lines (139 loc) · 7.17 KB

DESIGN.md

File metadata and controls

178 lines (139 loc) · 7.17 KB
Raw

Lambda Builders

Lambda Builders is a separate project that contains scripts to build Lambda functions, given a source location. It was built as part of SAM CLI sam build command.Read aws/aws-sam-cli#743 for design document explaining how Lambda Builders work in context of SAM CLI.

This project has certain attributes that make it unique:

  • Build Actions could be implemented in any programming language. Preferably in the language that they are building.
  • Some build actions simply execute a binary (like Golang) without writing a Go script. We provide generic Python runner to implement such build actions
  • We have one build action for each Programming Language + Framework combination.

Build Actions

A build action is a module that knows how to build for a particular programming language & framework (ex: Python+PIP). Build actions can be implemented in Python or in the native programming language.

If the builder is not written in Python, then it must have a CLI script that can be invoked directly from command line using the programming language binary (ex: python python-pip/cli.py or node nodejs-npm/cli.js). The CLI interface speaks the JSON-RPC protocol over stdin/stdout.

Dependencies

If your build action implementation requires 3rd party libraries, here is how you deal with it:

  • If the action is written in Python, then add the dependency to this project's manifest: requirements/base.txt
  • If the action is not in Python, then vendor in all your dependencies with the action. For example, for JS libraries include the node_modules folder along with your code. In future, we might create a Makefile target that can dynamically pull libraries before release and bundle it with the distribution. We do this to make sure the build actions can work with minimal set of external libraries.

Designs

Each build action has its own design document.

Builders Library

This project is available as a Python Library & a wrapper CLI interface. Given a folder containing function's source code, the selected build action, and path to an artifacts folder, this library will produce built artifacts in the artifacts folder.

Every build follows a standard workflow:

  1. Detect Manifest: Verify the source contains a supported dependency manifest (ex: package.json or pom.xml)
  2. Resolve Dependencies: Install dependencies
  3. Compile: Optionally, compile the code if necessary
  4. Copy Source: Optionally, Copy Lambda function code to the build folder

Build actions can choose to override one or more steps in this workflow. As much as possible, we encourage the actions to make use of the default implementation. It helps reduce the variance in behavior between each action and provides customers with a standard expectation.

Command Line Interface (Internal)

This library provides a wrapper CLI interface for convenience. This interface is not supported at the moment. So we don't provide any guarantees of back compatibility.

It is a very thin wrapper over the library. It is meant to integrate with tools written in other programming languages that can't import Python libraries directly. The CLI provides a JSON-RPC interface over stdin/stdout to invoke the builder and get response.

Request Format

{
  "jsonrpc": "2.0",
  "method": "LambdaBuilder.build",
  "id": 1,
  "params": {
    "__protocol_version": "0.1",  // Expected version of RPC protocol 
    "capability": {
      "language": "<programming language>",
      "dependency_manager": "<programming language framework>",
      "application_framework": "<application framework>"
    },
    "source_dir": "/path/to/source",
    "artifacts_dir": "/path/to/store/artifacts",
    "scratch_dir": "/path/to/tmp",
    "manifest_path": "/path/to/manifest.json",
    "runtime": "Function's runtime ex: nodejs8.10",
    "optimizations": {},  // not supported 
    "options": {}  // not supported
  }
}

Successful Response Format

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {}  // Empty result indicates success
}

Error Response Format

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": "",  // Integer code indicating the problem
    "message": "",  // Contains the Exception name 
    "data": ""   // Contains the exception message
  }  
}

Error Codes:

Error codes returned by the application are similar to HTTP Status Codes.

  • 400 - Similar to HTTP 400. Blame the caller.
  • 500 - Internal server error
  • 505 - RPC Protocol unsupported
  • -32601 - Method unsupported (standard JSON-RPC protocol error code)

Project Meta

Directory Structure

This project's directories are laid as follows:

aws_lambda_builders
├── __init__.py
├── __main__.py <- entrypoint for the CLI
├── action.py   <- This now just has the BaseAction class and any common language-agnostic Actions like CopySourceAction
├── runner.py    <- This is more or less the same as above.
├── workflows  <- Now instead of having all the builders/actions in one place, they are sorted by their language. You now need to use your technique to iterate these submodules and import the builders so that they get included in your registry.
│   ├── __init__.py
│   ├── dotnet_cli
│   │   ├── __init__.py
│   │   ├── actions.py
│   │   └── builders.py
│   ├── javascript_npm
│   │   ├── __init__.py
│   │   ├── npm_packager.js <- custom stuff would live alongside the python in these modules or in more subfolders.
│   │   ├── actions.py
│   │   └── builders.py
│   └── python_pip  <-- files in this directory do not have any convention. You can name them however you want
│       ├── __init__.py
│       ├── packager.py <- low level packager code that is called by the PythonPipResolveAction to do its job.
│       ├── actions.py  <- This now contains python specific actions like PythonPipResolveAction
│       └── builders.py <- The python specific builders would be defined here
├── exceptions.py
└── registry.py

Benefit here is that our high level build/action system is pulled out, and each language-specific piece acts almost like a plugin, its its own self contained directory. Someone could develop their own "package" with the structure

ruby
├── __init__.py
├── packager.rb
├── utils.rb
├── actions.py
└── builders.py

And essentially drop into the builders package (or maybe we can have a notion of a BUILDERS_PATH that is searched for these things and the default entry is this vended builders dir.) to get it to work. This seems the friendliest to me.

Terminologies

  • builder: The entire project is called builder, because it can build Lambda functions
  • workflows: Building for each language+framework combination is defined using a workflow.
  • actions: A workflow is implemented as a chain of actions.