[QUESTION] Issues Ingesting VPC Flow Logs into OpenSearch SIEM with AWS Secure Environment Accelerator

[zahirktk78]

I've recently deployed a SIEM solution on OpenSearch using the AWS Secure Environment Accelerator, specifically following the guidance and resources provided in this repository https://github.com/aws-samples/aws-secure-environment-accelerator/tree/32ee10c50d0489a418888a5bddda7af0e2b9a3c8/reference-artifacts/Add-ons/opensiem .

Problem:

While the setup appears to be correctly ingesting various types of logs, I'm encountering issues specifically with VPC Flow Logs. Despite adjusting the s3_key to point to the appropriate location of the VPC Flow Logs in S3, the logs either fail to be ingested, or I encounter errors. The most telling feedback I've received is a warning message indicating that no entries were successfully loaded:

{
"level": "WARNING",
"message": "No entries were successed to load",
"location": "process_records:346",
"timestamp": "2024-02-21 18:05:00,054+0000",
"service": "os-loader",
"cold_start": false,
"function_name": "OpenSearchSiemStack-SiemProcessorB1FDF325-OFFrKfdLmfiP",
"function_memory_size": "512",
"function_arn": "arn:aws:lambda:Region-1:Account ID:function:OpenSearchSiemStack-SiemProcessor",
"s3_key": "CloudWatchLogs/vpcflowlogs/2024/02/21/18/PBMMAccel-Firehose-Delivery-Stream-Partition-1-2024-02-21-18-03-12-359e52ca"
}
Attempts to Resolve & Questions:
What specific configurations should be applied to the s3_key to accurately reference VPC Flow Logs within the system?
In adjusting the aws.ini file to accommodate various log types for s3_key, are there particular considerations or parameters to be mindful of?
Could you clarify if there's a prescribed format or preprocessing methodology essential for ensuring compatibility of VPC Flow Logs with this SIEM solution?