Send CloudWatch Logs to Splunk via Kinesis Firehose

This module configures a Kinesis Firehose, sets up a subscription for a desired CloudWatch Log Group to the Firehose, and sends the log data to Splunk. A Lambda function is required to transform the CloudWatch Log data from "CloudWatch compressed format" to a format compatible with Splunk. This module takes care of configuring this Lambda function.

Usage Instructions

In order to send this data to Splunk you will need to first obtain an HEC Token from your Splunk administrator.

Once you have received the token, you can proceed forward in creating a module resource, such as the one in the Example below. You will use a KMS key of your choice to encrypt the token, as it is sensitive.

Note: the user of this module is responsible for specifying the provider {} block for the AWS Terraform provider. As of v5.0.0 the provider block was removed from this module.

Example

module "kinesis_firehose" {
  source                             = "disney/kinesis-firehose-splunk/aws"
  version                            = "<version>"
  cloudwatch_log_regions             = ["us-east-1", "us-west-2"]
  name_cloudwatch_logs_to_ship       = "/test/test01"
  cloudwatch_log_group_names_to_ship = ["/aws/svc/loggroup1", "log-group-2", "/aws/svc2/loggroup"]
  hec_url                            = "<Splunk_Kinesis_ingest_URL>"
  s3_bucket_name                     = "<mybucketname>"

  ### HEC Token ###
  One of var.hec_token (default) OR var.self_managed_hec_token must be used to pass in the Splunk HEC token.
}

Please see the S3 Life Cycle Rule example if you wish to configure them.

Splunk Cloud Customers

If you are a Splunk Cloud customer, once you have successfully deployed all the resources, you will need to ensure that your Splunk Cloud instance has the Kinesis Data Firehose egress CIDRs allow listed under Server Settings > IP Allow List Management > HEC access for ingestion.

For more details on the relevant CIDRs please reference this article.

Upgrading from v6.0.0 to v7.0.0

If you choose to change the way you pass in your HEC token (see section below) when upgrading from v6.0.0 to v7.0.0, when you run terraform apply, you might run into Terraform reporting that it is going to make changes to resources such as IAM policies when nothing has changed with them. Others have experienced this issue as well, please see this issue.

v7.0.0 Passing in Splunk HEC Token

As of v7.0.0, there are two additional options available to pass in the HEC token:

• You may pass the HEC token in via a variable called var.self_managed_hec_token, which gives you the flexibility to perhaps encrypt the token in your repo with a different tool of your choice. For example, AWS SSM Parameter Store or SOPS.

By DEFAULT, for backwards compatibilty, it will default to using the KMS encrypted HEC token that this module previously required you to configure.

Requirements

Name	Version
terraform	>= 1.0.0
archive	>= 2.3.0, < 3.0.0
aws	>= 5.0.0, < 6.0.0

Providers

Name	Version
archive	2.4.0
aws	5.8.0

Modules

Name	Source	Version
hec_token_kms_secret	./modules/kms_secrets	n/a

Resources

Name	Type
aws_cloudwatch_log_group.firehose_lambda_transform	resource
aws_cloudwatch_log_group.kinesis_logs	resource
aws_cloudwatch_log_stream.kinesis_logs	resource
aws_cloudwatch_log_subscription_filter.cloudwatch_log_filter	resource
aws_cloudwatch_log_subscription_filter.cloudwatch_log_filters	resource
aws_iam_policy.cloudwatch_to_fh_access_policy	resource
aws_iam_policy.kinesis_firehose_iam_policy	resource
aws_iam_policy.lambda_transform_policy	resource
aws_iam_role.cloudwatch_to_firehose_trust	resource
aws_iam_role.kinesis_firehose	resource
aws_iam_role.kinesis_firehose_lambda	resource
aws_iam_role_policy_attachment.cloudwatch_to_fh	resource
aws_iam_role_policy_attachment.kinesis_fh_role_attachment	resource
aws_iam_role_policy_attachment.lambda_policy_role_attachment	resource
aws_kinesis_firehose_delivery_stream.kinesis_firehose	resource
aws_lambda_function.firehose_lambda_transform	resource
aws_s3_bucket.kinesis_firehose_s3_bucket	resource
aws_s3_bucket_acl.kinesis_firehose_s3_bucket	resource
aws_s3_bucket_lifecycle_configuration.this	resource
aws_s3_bucket_object_lock_configuration.kinesis_firehose_s3_lock	resource
aws_s3_bucket_ownership_controls.kinesis_firehose_s3_bucket	resource
aws_s3_bucket_public_access_block.kinesis_firehose_s3_bucket	resource
aws_s3_bucket_server_side_encryption_configuration.kinesis_firehose_s3_bucket	resource
aws_s3_bucket_versioning.kinesis_firehose_s3_bucket_versioning	resource
archive_file.lambda_function	data source
aws_caller_identity.current	data source
aws_iam_policy_document.cloudwatch_to_fh_access_policy	data source
aws_iam_policy_document.cloudwatch_to_firehose_trust_assume_policy	data source
aws_iam_policy_document.kinesis_firehose_policy_document	data source
aws_iam_policy_document.lambda_policy_doc	data source
aws_partition.current	data source
aws_region.current	data source

Inputs

Name	Description	Type	Default	Required
hec_url	Splunk Kinesis URL for submitting CloudWatch logs to splunk	string	n/a	yes
s3_bucket_name	Name of the s3 bucket Kinesis Firehose uses for backups	string	n/a	yes
arn_cloudwatch_logs_to_ship	arn of the CloudWatch Log Group that you want to ship to Splunk.	string	null	no
aws_s3_bucket_versioning	Versioning state of the bucket. Valid values: Enabled, Suspended, or Disabled. Disabled should only be used when creating or importing resources that correspond to unversioned S3 buckets.	string	null	no
cloudwach_log_group_kms_key_id	KMS key ID of the key to use to encrypt the Cloudwatch log group	string	null	no
cloudwatch_log_filter_name	Name of Log Filter for CloudWatch Log subscription to Kinesis Firehose	string	"KinesisSubscriptionFilter"	no
cloudwatch_log_group_names_to_ship	List of CloudWatch Log Group names that you want to ship to Splunk.	list(string)	null	no
cloudwatch_log_regions	List of regions to allow CloudWatch logs to be shipped from. Set in Kinesis Firehose role's trust polucy	list(string)	[]	no
cloudwatch_log_retention	Length in days to keep CloudWatch logs of Kinesis Firehose	number	30	no
cloudwatch_to_fh_access_policy_name	Name of IAM policy attached to the IAM role for CloudWatch to Kinesis Firehose subscription	string	"KinesisCloudWatchToFirehosePolicy"	no
cloudwatch_to_firehose_trust_iam_role_name	IAM Role name for CloudWatch to Kinesis Firehose subscription	string	"CloudWatchToSplunkFirehoseTrust"	no
enable_fh_cloudwatch_logging	Enable kinesis firehose CloudWatch logging. (It only logs errors)	bool	true	no
encryption_context	aws_kms_secrets encryption context	map(string)	{}	no
expected_bucket_owner	The account ID of the expected bucket owner	string	null	no
firehose_name	Name of the Kinesis Firehose	string	"kinesis-firehose-to-splunk"	no
firehose_processing_enabled	Kinesis firehose processing enabled	bool	true	no
firehose_server_side_encryption_enabled	Enable SSE for Kinesis Firehose	bool	false	no
firehose_server_side_encryption_key_arn	ARN of the key to be used for Firehose SSE	string	null	no
firehose_server_side_encryption_key_type	Type of SSE key to be used for encrypting the Firehose. Valid values are AWS_OWNED_CMK and CUSTOMER_MANAGED_CMK	string	null	no
hec_acknowledgment_timeout	The amount of time, in seconds between 180 and 600, that Kinesis Firehose waits to receive an acknowledgment from Splunk after it sends it data.	number	300	no
hec_endpoint_type	Splunk HEC endpoint type; Raw or Event	string	"Raw"	no
hec_token	Splunk security token needed to submit data to Splunk. Required if var.self_managed_hec_token is not specified.	string	null	no
kinesis_firehose_buffer	https://www.terraform.io/docs/providers/aws/r/kinesis_firehose_delivery_stream.html#buffer_size	number	5	no
kinesis_firehose_buffer_interval	Buffer incoming data for the specified period of time, in seconds, before delivering it to the destination	number	300	no
kinesis_firehose_iam_policy_name	Name of the IAM Policy attached to IAM Role for the Kinesis Firehose	string	"KinesisFirehose-Policy"	no
kinesis_firehose_lambda_role_name	Name of IAM Role for Lambda function that transforms CloudWatch data for Kinesis Firehose into Splunk compatible format	string	"KinesisFirehoseToLambaRole"	no
kinesis_firehose_retry_duration	After an initial failure to deliver to Splunk, the total amount of time, in seconds between 0 to 7200, during which Firehose re-attempts delivery (including the first attempt). After this time has elapsed, the failed documents are written to Amazon S3. The default value is 300s. There will be no retry if the value is 0	number	300	no
kinesis_firehose_role_name	Name of IAM Role for the Kinesis Firehose	string	"KinesisFirehoseRole"	no
lambda_function_environment_variables	Environment variables for the lambda function	map(string)	{}	no
lambda_function_memory_size	Amount of memory in MB which Lambda Function can use at runtime. Defaults to 128	number	128	no
lambda_function_name	Name of the Lambda function that transforms CloudWatch data for Kinesis Firehose into Splunk compatible format	string	"kinesis-firehose-transform"	no
lambda_function_timeout	The function execution time at which Lambda should terminate the function.	number	180	no
lambda_iam_policy_name	Name of the IAM policy that is attached to the IAM Role for the lambda transform function	string	"Kinesis-Firehose-to-Splunk-Policy"	no
lambda_kms_key_arn	Amazon Resource Name (ARN) of the AWS Key Management Service (KMS) key that is used to encrypt environment variables.	string	null	no
lambda_processing_buffer_interval_in_seconds	Lambda processing buffer interval in seconds.	number	61	no
lambda_processing_buffer_size_in_mb	Lambda processing buffer size in mb.	number	0.256	no
lambda_reserved_concurrent_executions	Amount of reserved concurrent executions for this lambda function. A value of 0 disables lambda from being triggered and -1 removes any concurrency limitations.	string	null	no
lambda_tracing_config	Configures x-ray tracing for Lambda fuction. See valid values here: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function#mode	string	null	no
lifecycle_rule	List of maps containing configuration of object lifecycle management.	any	[]	no
local_lambda_file	The absolute path to an existing custom Lambda script	string	null	no
local_lambda_file_handler	Allows you to specify Lambda handler if using a local custom file for Lambda function	string	null	no
log_stream_name	Name of the CloudWatch log stream for Kinesis Firehose CloudWatch log group	string	"SplunkDelivery"	no
name_cloudwatch_logs_to_ship	Name of the CloudWatch Log Group that you want to ship to Splunk (single log group; leave empty to not create the subscription filter; see var.cloudwatch_log_group_names_to_ship for creating subscription filters for multiple log groups).	string	null	no
nodejs_runtime	Runtime version of nodejs for Lambda function	string	"nodejs20.x"	no
object_lock_configuration_days	Required if years is not specified. Number of days that you want to specify for the default retention period	number	null	no
object_lock_configuration_mode	Default Object Lock retention mode you want to apply to new objects placed in the specified bucket. Valid values: COMPLIANCE, GOVERNANCE	string	null	no
object_lock_configuration_token	S3 bucket object lock configuration token	string	null	no
object_lock_configuration_years	Required if days is not specified. Number of years that you want to specify for the default retention period	number	null	no
region	DEPRECATED. The region of AWS you want to work in, such as us-west-2 or us-east-1 (deprecated: use var.cloudwatch_log_regions instead)	string	null	no
s3_backup_mode	Defines how documents should be delivered to Amazon S3. Valid values are FailedEventsOnly and AllEvents.	string	"FailedEventsOnly"	no
s3_bucket_block_public_access_enabled	Set to 1 if you would like to add block public access settings for the s3 bucket Kinesis Firehose uses for backups	number	0	no
s3_bucket_key_enabled	Whether or not to use Amazon S3 Bucket Keys for SSE-KMS.	bool	null	no
s3_bucket_object_lock_enabled	Indicates whether this bucket has an Object Lock configuration enabled. Valid values: Enabled.	string	null	no
s3_bucket_server_side_encryption_algorithm	(Required) Server-side encryption algorithm to use. Valid values are AES256 and aws:kms	string	"AES256"	no
s3_bucket_server_side_encryption_kms_master_key_id	AWS KMS master key ID used for the SSE-KMS encryption. This can only be used when you set the value of sse_algorithm as aws:kms. The default aws/s3 AWS KMS master key is used if this element is absent while the sse_algorithm is aws:kms	string	null	no
s3_compression_format	The compression format for what the Kinesis Firehose puts in the s3 bucket	string	"GZIP"	no
s3_prefix	Optional prefix (a slash after the prefix will show up as a folder in the s3 bucket). The YYYY/MM/DD/HH time format prefix is automatically used for delivered S3 files.	string	"kinesis-firehose/"	no
self_managed_hec_token	This variable allows for the user to have additional flexibility in how they pass in the HEC token. Perhaps they want to use a different tool than SSM or KMS encryption in their code base to encrypt it. Required if var.hec_token is not specified.	string	null	no
subscription_filter_pattern	Filter pattern for the CloudWatch Log Group subscription to the Kinesis Firehose. See this for filter pattern info.	string	""	no
tags	Map of tags to put on the resource	map(string)	{}	no

Outputs

Name	Description
cloudwatch_to_firehose_trust_arn	cloudwatch log subscription filter role_arn
destination_firehose_arn	cloudwatch log subscription filter - Firehose destination arn

Acknowledgements

Author

• Mitchell L. Cooper - Maintainer

Reviewers

• Ian Ward
• Justice London