Deploy Rookout on AWS ECS Fargate Cluster using Terraform

This terraform depolying Rookout Controller and Rookout Datastore on AWS ECS Fargate cluster.

The module implements the following architecture (default deployment):

Network architecture (default deployment):

Prerequisites

1. Terraform installed.
2. AWS account inlcuding: AWS CLI installed.
   • The AWS default profile should be set with an access key and secret (reference).
   • Set profile if used non default profile. Run: export AWS_PROFILE="<profile_name>"
3. Create a provider.tf file (reference).
4. Get your organizational Rookout token, and pass it as a variable to this module

   rookout_token = "..."
   

Components

This module deploy the Rookout ETL Controller by default. It also allows deployment of the Rookout Datastore, and a demo application with the Rookout agent.

The components to deploy can be configured with the folloiwng boolean variables:

    deploy_datastore = true/false
    deploy_demo_app = true/false

Certificate and DNS records management types

There are two methods for certificates and DNS record management that will change the network architecture.

Module-managed certificate and CNAME record

For deployments where domain_name is provided, a rookout.YOURDOMAIN subdomain will be created in a route53 public hosted zone, and associated by creating an NS record in your domain's public hosted zone. The subdomain will be used for the controller, datastore (optional) and demo application (optional). A certificate for this subdomain will be created in ACM. ALBs will also be created for those components. The created certificate and DNS records will be associated to those ALBs' domain names.

Note: If you don't use route53 as your DNS registry provider, please contact us.

For this type of deployment, provide the following variable:

    domain_name = "YOUR_DOMAIN"

The internal_controller_alb boolean variable (false by default) can be used to make the communication with the ETL Controller internal.

Self-managed certificate and CNAME record

For self managed certificate deployments CNAME record should be created for the Datastore and/or Controller, so we have two options.

1. Provided ACM certificate for the Datastore (example)

This deployment will use the pre-imported ARN of the certificate in ACM (Body, private key, and chain of certificate are needed). THe certificate will be used by the datastore, therefore a CNAME record of the certificate's domain should be recored at your DNS provider with the Datstore endpoint (output of the module). A Controller will be deployed with an internal load balancer and can be reached from the VPC with the Controller's endpoint (output of the module).

    datastore_acm_certificate_arn = "PRE_IMPORTED_ACM_CERTIFICATE_ARN"

2. Provided ACM certificate for Datastore and Controller (for internet-facing controller) (example)

Same as the previous option, but, the controller is internet facing too. Same procedure of CNAME record registration should be preformed for the Datastore and Controller endpoints that match to their certificate's domain.

    datastore_acm_certificate_arn = "PRE_IMPORTED_ACM_CERTIFICATE_ARN"
    controller_acm_certificate_arn = "PRE_IMPORTED_ACM_CERTIFICATE_ARN"

If a demo application is deployed, it should be configured with the Controller's CNAME record:

    demo_app_controller_host = "YOUR_CONTROLLER_CNAME_RECORD"

Deplyment matrixes

The following matrixes demonsrate application's components network mode by key variables. HTTP means internal traffic with ALB, TLS means external secured traffic with ALB. Demo means demo flask application provided with this module.

Endpoints

controller.PROVIDED_DOMAIN - url of the controller, used for SDK (rooks) when DNS provided.

datastore.PROVIDED_DOMAIN - url to the datastore, used with rookout client (web browser application) when DNS provided.

demo.PROVIDE_DOMAIN - flask demo application for debuging when DNS provided.

Advanced configuration

• Provided Domain + VPC and subnets (example)

  You can configure the module to use an existing VPC (where your application is running) using the following variables:

      vpc_id = "<your's existing vpc id>"
      vpc_public_subnets = ["<first_sub_domain>", "<second_sub_domain>"]
      vpc_private_subnets = ["<first_sub_domain>", "<second_sub_domain>"]
  
  

• Provided Domain + VPC and subnets + ECS cluster (example)

  You can configure the module to use an existing VPC and ECS cluster (where your application is running) using the following variables:

      vpc_id = "<your's vpc id>"
      vpc_public_subnets = ["<first_sub_domain>", "<second_sub_domain>"]
      vpc_private_subnets = ["<first_sub_domain>", "<second_sub_domain>"]
  
      create_cluster = false
      cluster_name = "<your's existing cluster name>"
  

• custom_iam_task_exec_role_arn - This variable can be used to overwrite the existing IAM Role of ECS tasks execution

• deploy_alb - This variable can be set to false to disable the deployment of ALBs. If disabled, DNS subdomain and ACM certificate will not be generated. In this case, the following configuration should be set:

  deploy_alb = false
  controller_target_group_arn = "arn:aws:elasticloadbalancing:AWS_REGION:ACCOUNT_ID:ARN_SUFFIX"
  datastore_target_group_arn = "arn:aws:elasticloadbalancing:AWS_REGION:ACCOUNT_ID:ARN_SUFFIX" #if deploy_datastore=true
  demo_app_target_group_arn = "arn:aws:elasticloadbalancing:AWS_REGION:ACCOUNT_ID:ARN_SUFFIX" #if deploy_demo_app=true
  

  If target groups are not passed, the loadbalancer configuration block in task defenitaion will be disbaled.

• internal - boolean variable wich switches the ALBs to be internal only. if provided domain_name will create private hosted zone us that domain. Usually used with wildcard certificate.

• wildcard certificate can be used with datastore_acm_certificate_arn and controller_acm_certificate_arn variables. when those used, please create CNAME records for controller and datastore out of outputs of this module (controller_endpoint, datastore_endpoint) to match the certificate's domain.

Requirements

Name	Version
terraform	>= 0.14
aws	>= 5.0
awsutils	>= 0.11.0

Providers

Name	Version
aws	5.1.0

Modules

Name	Source	Version
acm	terraform-aws-modules/acm/aws	~> 3.0
vpc	terraform-aws-modules/vpc/aws	4.0.2

Resources

Name	Type
aws_alb.controller	resource
aws_alb.datastore	resource
aws_alb.demo	resource
aws_cloudwatch_log_group.demo	resource
aws_cloudwatch_log_group.rookout	resource
aws_cloudwatch_log_stream.controller_log_stream	resource
aws_cloudwatch_log_stream.datastore_log_stream	resource
aws_cloudwatch_log_stream.demo_log_stream	resource
aws_ecs_cluster.rookout	resource
aws_ecs_service.controller	resource
aws_ecs_service.datastore	resource
aws_ecs_service.demo	resource
aws_ecs_task_definition.controller	resource
aws_ecs_task_definition.datastore	resource
aws_ecs_task_definition.demo	resource
aws_iam_policy.task_exec_role	resource
aws_iam_role.task_exec_role	resource
aws_lb_listener.controller	resource
aws_lb_listener.datastore	resource
aws_lb_listener.demo	resource
aws_lb_target_group.controller	resource
aws_lb_target_group.datastore	resource
aws_lb_target_group.demo	resource
aws_route53_record.controller	resource
aws_route53_record.datastore	resource
aws_route53_record.demo	resource
aws_route53_record.rookout	resource
aws_route53_zone.sub_domain	resource
aws_security_group.alb_controller	resource
aws_security_group.alb_datastore	resource
aws_security_group.alb_demo	resource
aws_security_group.allow_demo	resource
aws_security_group.controller	resource
aws_security_group.datastore	resource
aws_ecs_cluster.provided	data source
aws_region.current	data source
aws_route53_zone.selected	data source

Inputs

Name	Description	Type	Default	Required
additional_controller_env_vars	Additional env variables of contorller, configure as map of key=values	any	{}	no
additional_datastore_env_vars	Additional env variables of contorller, configure as map of key=values	any	{}	no
additional_demo_app_env_vars	Additional env variables of contorller, configure as map of key=values	any	{}	no
cluster_name	ECS cluster name, if we want to deploy to existing one	string	""	no
controller_acm_certificate_arn	ARN of pre-imported SSL certificate to ACM for Rookouts controller public access, if datastore ACM provided controller alb will be internal	string	""	no
controller_alb_sg_igress_cidr_blocks	Ingress CIDRs for controller's ALB security group	list(string)	[ "0.0.0.0/0" ]	no
controller_image	Controller image name	string	"rookout/controller"	no
controller_replicas	Rookout's onprem controller replica count	string	2	no
controller_resource	Rookout's onprem controller resource map	map(any)	{ "cpu": 2048, "memory": 4096 }	no
controller_sg_igress_cidr_blocks	Ingress CIDRs of controller security group	list(string)	[ "0.0.0.0/0" ]	no
controller_target_group_arn	Target group used by controller ECS tasks	string	""	no
controller_version	Controller image version	string	"latest"	no
create_cluster	whether create a cluster or use existing one	bool	true	no
create_vpc	# VPC variables.	bool	true	no
custom_iam_task_exec_role_arn	ECS execution IAM Role overwrite, please pass arn of existing IAM Role	string	""	no
datastore_acm_certificate_arn	ARN of pre-imported SSL certificate to ACM for Rookouts datastore public access	string	""	no
datastore_alb_sg_igress_cidr_blocks	Ingress CIDRs datastore's ALB security group	list(string)	[ "0.0.0.0/0" ]	no
datastore_image	Datastore image name	string	"rookout/data-on-prem"	no
datastore_resource	Rookout's onprem datastore resource map	map(any)	{ "cpu": 2048, "memory": 4096 }	no
datastore_sg_igress_cidr_blocks	Ingress CIDRs of datastore security group	list(string)	[ "0.0.0.0/0" ]	no
datastore_target_group_arn	Target group used by datastore ECS tasks	string	""	no
datastore_version	Datastore image version	string	"latest"	no
demo_app_alb_sg_igress_cidr_blocks	Ingress CIDRs datastore's ALB security group	list(string)	[ "0.0.0.0/0" ]	no
demo_app_controller_host	Host which the demo rook connect to controller using WebSocket	string	""	no
demo_app_sg_igress_cidr_blocks	Ingress CIDRs of datastore security group	list(string)	[ "0.0.0.0/0" ]	no
demo_app_target_group_arn	Target group used by demo applicatino ECS tasks	string	""	no
deploy_alb	Radio button to not deploy ALB for ECS tasks, if false please provide target group for each	bool	true	no
deploy_datastore	(Optional) If true will deploy demo Rookout's datastore locally	bool	true	no
deploy_demo_app	(Optional) If true will deploy demo flask application to start debuging	bool	false	no
domain_name	DNS domain which sub	string	""	no
enforce_token	Whether to enforce the token in controller	bool	true	no
environment	Environment name	string	"rookout"	no
internal	Flag to switch the deployment to be internal	bool	false	no
internal_controller_alb	If domain provided, switching in on will make controller be reachable internaly only	bool	false	no
region	AWS region, using providers region as default	string	""	no
rookout_token	Rookout token	string	""	no
subdomain_vpc_association	flag for association of a hosted zone with a vpc	bool	true	no
vpc_availability_zones	n/a	list(string)	[ "" ]	no
vpc_cidr	n/a	string	"172.30.1.0/25"	no
vpc_id	VPC id should be passed only if create_vpc = false	string	""	no
vpc_private_subnets	n/a	list(string)	[ "172.30.1.0/27", "172.30.1.32/27" ]	no
vpc_public_subnets	n/a	list(string)	[ "172.30.1.64/27", "172.30.1.96/27" ]	no

Outputs

Name	Description
controller_dns	Rookout's on-prem controller dns
controller_endpoint	Rookout's on-prem controller endpoint
datastore_dns	Rookout's on-prem datastore DNS
datastore_endpoint	Rookout's on-prem datastore endpoint
demo_dns	Rookout's flask application DNS
demo_endpoint	Rookout's flask application endpoint
ecs_cluster_id	ECS cluster
vpc_id	VPC id that created