Skip to content

Commit

Permalink
Rename Chronicle to Google SecOps (#53)
Browse files Browse the repository at this point in the history
* Rename Chronicle to Google Security Operations
* Update contributing guide
  • Loading branch information
rarsan authored Apr 24, 2024
1 parent 1619af0 commit c4853d5
Show file tree
Hide file tree
Showing 57 changed files with 132 additions and 130 deletions.
10 changes: 6 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,12 +39,14 @@ To add a new threat detection or audit question, follow these steps:
### New Query Implementation
To implement a `SQL` query or `YARA-L` rule for an existing or new security question, follow the steps in this section. We'll assume the CSA in question has ID `5.40` and name `sql_tables_most_frequently_accessed`.
1. To provide a BigQuery SQL, implement query in a new SQL file under `sql/` folder, named after corresponding CSA ID and name, which is in our example `5_40_sql_tables_most_frequently_accessed.sql`.
1. To provide a BigQuery SQL query, implement query in a new SQL file under `backends/bigquery/sql/` folder, named after corresponding CSA ID and name, which is in our example `5_40_sql_tables_most_frequently_accessed.sql`.
2. To add a Chronicle YARA-L rule, implement rule in a new YARA-L file under `yaral/` folder, named after corresponding CSA ID and name, which is in our example
1. To provide a Log Analytics SQL query, implement query in a new SQL file under `backends/log_analytics/sql/` folder, named after corresponding CSA ID and name, which is in our example `5_40_sql_tables_most_frequently_accessed.sql`.
2. To add a Google SecOps YARA-L rule, implement rule in a new YARA-L file under `backends/chronicle/yaral/` folder, named after corresponding CSA ID and name, which is in our example
`5_40_sql_tables_most_frequently_accessed.yaral` .
CSA design favors **convention over configuration** when it comes to file naming and docs generation. Therefore, special attention is required for queries/rules file naming where the new filename must match a specific format per above examples, where:
CSA design favors **convention over configuration** when it comes to file naming and docs generation. Therefore, special attention is required for query/rule file naming where the new filename must match a specific format per above examples, where:
- CSA ID and name are concatenated, and
- CSA ID `.` (dot) separator is replaced with `_` (underscore)
Expand Down Expand Up @@ -75,7 +77,7 @@ For example, [CSA `1.01`](./src/1.01/1.01.md), is the first security question fr
The [CSA index](./README.md#security-analytics-use-cases) in README.md as well as individual CSA use case documentation (e.g. [CSA `1.01`](./src/1.01/1.01.md)) are automatically generated based on:
- YAML spec files under `src/` , and,
- Corresponding log samples under `test/fixtures/` folder, and,
- Corresponding query implementations under `sql/` and `yaral/` folders.
- Corresponding query implementations under `backends/*/sql/` and `backends/*/yaral/` folders.
After adding or editing files under any of these directories, here are the steps to regenerate the docs to reflect the changes:
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ As organizations go through the Autonomic Security modernization journey, this r
CSA is a set of foundational security analytics designed to provide organizations with a rich baseline of pre-built queries and rules that they can readily use to start analyzing their Google Cloud logs including Cloud Audit logs, VPC Flow logs, DNS logs, and more using cloud-native or third-party analytics tools. The source code is provided as is, without warranty. See [Copyright & License](#copyright--license) below.

Current release include:
- YARA-L rules for [Chronicle](https://chronicle.security/)
- YARA-L rules for [Google Security Operations](https://chronicle.security/)
- SQL queries for [BigQuery](https://cloud.google.com/bigquery/)
- SQL queries for [Log Analytics](https://cloud.google.com/logging/docs/log-analytics)

Expand All @@ -21,9 +21,9 @@ The security use cases below are grouped in 6 categories depending on underlying
5. :droplet: [Data Usage](#data-usage)
6. :zap: [Network Activity](#network-activity)

To learn more about the variety of Google Cloud logs, how to enable and natively export these logs to destinations like Chronicle or BigQuery for in-depth analytics, refer to Google Cloud [Security and access analytics solution guide](https://cloud.google.com/architecture/exporting-stackdriver-logging-for-security-and-access-analytics).
To learn more about the variety of Google Cloud logs, how to enable and natively export these logs to destinations like BigQuery or Google Security Operations for in-depth analytics, refer to Google Cloud [Security and access analytics solution guide](https://cloud.google.com/architecture/exporting-stackdriver-logging-for-security-and-access-analytics).

**Caution:** CSA is not meant to be a comprehensive set of threat detections, but a collection of community-contributed samples to get you started with detective controls. Use CSA in your threat detection and response capabilities (e.g. [Security Command Center](https://cloud.google.com/security-command-center), Chronicle, BigQuery, Siemplify, or third-party SIEM) in conjunction with threat prevention capabilities (e.g. [Security Command Center](https://cloud.google.com/security-command-center), [Cloud Armor](https://cloud.google.com/armor), [BeyondCorp](https://cloud.google.com/beyondcorp)). To learn more about Google’s approach to modern Security Operations, check out the [Autonomic Security Operations whitepaper](https://services.google.com/fh/files/misc/googlecloud_autonomicsecurityoperations_soc10x.pdf).
**Caution:** CSA is not meant to be a comprehensive set of threat detections, but a collection of community-contributed samples to get you started with detective controls. Use CSA in your threat detection and response capabilities (e.g. [Security Command Center](https://cloud.google.com/security-command-center), [Google Security Operations](https://cloud.google.com/security/products/security-operations), [BigQuery](https://cloud.google.com/bigquery/), or third-party SIEM) in conjunction with threat prevention capabilities (e.g. [Security Command Center](https://cloud.google.com/security-command-center), [Cloud Armor](https://cloud.google.com/armor), [Identity-Aware Proxy](https://cloud.google.com/security/products/iap) and [Chrome Enterprise Premium](https://chromeenterprise.google/products/chrome-enterprise-premium)). To learn more about Google’s approach to modern Security Operations, check out the [Autonomic Security Operations whitepaper](https://services.google.com/fh/files/misc/googlecloud_autonomicsecurityoperations_soc10x.pdf).

## Security Analytics Use Cases
![Security Monitoring](./assets/gcp_security_mon.png)
Expand Down Expand Up @@ -92,9 +92,9 @@ To learn more about the variety of Google Cloud logs, how to enable and natively

The [`dataform` folder](./dataform/) contains the Dataform repo to automate deployment of CSA queries in BigQuery for optimized performance and cost. Use this Dataform repo to operationalize CSA use cases as reports and alerts powered by BigQuery. This Dataform project deploys and orchestrates pre-built ELT pipelines to filter, normalize and model log data leveraging incremental summary tables, lookup tables and views for fast, cost-effective and simpler querying. See underlying [README](./dataform/README.md) for more details.

## CI/CD for CSA on Chronicle
## CI/CD for CSA on Google Security Operations

The [`cicd` folder](./cicd/) contains a set of scripts to help you with storing CSA YARA-L detection rules as code and testing/deploying updates you and your team make in an automated fashion. Whether you use GitHub Actions, Google Cloud Build or Azure DevOps, you can use the corresponding scripts to automatically test and deploy new or modified rules into your Chronicle instance. See underlying [README](./cicd/README.md) for more details.
The [`cicd` folder](./cicd/) contains a set of scripts to help you with storing CSA YARA-L detection rules as code and testing/deploying updates you and your team make in an automated fashion. Whether you use GitHub Actions, Google Cloud Build or Azure DevOps, you can use the corresponding scripts to automatically test and deploy new or modified rules into your Google Security Operations instance. See underlying [README](./cicd/README.md) for more details.

## Support

Expand Down
Binary file modified assets/csa_doc_query.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
32 changes: 16 additions & 16 deletions cicd/README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
# CI/CD for CSA on Chronicle
# CI/CD for CSA on Google Security Operations

## Overview

The [`python` folder](./python/) contains Python helper scripts to gather rules from a local folder and from the remote
Chronicle instance, and to compare them, uploading changed or new files to the Chronicle instance.
Google Security Operations instance, and to compare them, uploading changed or new files to the Google Security Operations instance.

## Pre-installation

Pre-requisites can be installed by copying the files from the folder locally, and running `pip3 install -r requirements.txt`

To run this script you will need a Chronicle API Service Account JSON file. This can be obtained from your Google Chronicle
To run this script you will need a Google SecOps API Service Account JSON file. This can be obtained from your Google Security Operations
account team.

## Usage

First export the path to your Chronicle API key like this:
First export the path to your Google SecOps API key like this:

```bash
export PATH_TO_KEY=~/malachite-abc-7ba40dd4f123.json
Expand Down Expand Up @@ -46,7 +46,7 @@ optional arguments:
differences
-s, --silent supress error messages
-r REGION, --region REGION
Chronicle instance region (leave blank for US)
Google Security Operations instance region (leave blank for US)
```

NOTE: if you provide more than one of `CREDENTIALS_FILE`, `CREDENTIALS_INFO`, `CREDENTIALS_ENV`, then the
Expand Down Expand Up @@ -77,34 +77,34 @@ Example output from making changes (including `-m` flag):

The [`chronicle-rules-cicd.yml`](./github-actions/chronicle-rules-cicd.yml) file located in the
[`github-actions` folder](./github-actions/) in this repository contains an example of using this
Python script to push updates or new detection content to a Chronicle instance. There are two variables
Python script to push updates or new detection content to a Google Security Operations instance. There are two variables
to edit in this script:

Variable Name | Description | Example Value
---|---|---
`region` | The region for your Chronicle instance | us
`region` | The region for your Google Security Operations instance | us
`rules_path` | The relative path from the root of the repository containing the YARA-L rules to work with | rules/yaral

These should be modified for your environment, and the file should be placed in a `.github/workflows` folder in the root of your repository.

In addition to this, the pipeline file requires a secret to be created on your GitHub repository. Details for creating this
can be found [here](https://docs.github.com/en/actions/security-guides/encrypted-secrets). The secret in this case should
be named `SA_CREDENTIAL`. The value is derived from the contents of your Chronicle API key, but line breaks should be
be named `SA_CREDENTIAL`. The value is derived from the contents of your Google SecOps API key, but line breaks should be
removed from the file, and the `"` character should also be replaced with `\"`. The resulting string can then be pasted into the Secrets UI in GitHub.

Now whenever a change is written to the repository the contents of the passed rules folder will be checked and updated/uploaded on the Chronicle instance. To learn more GitHub Actions workflows, see [here](https://docs.github.com/en/actions/using-workflows/about-workflows).
Now whenever a change is written to the repository the contents of the passed rules folder will be checked and updated/uploaded on the Google Security Operations instance. To learn more GitHub Actions workflows, see [here](https://docs.github.com/en/actions/using-workflows/about-workflows).

## Using in a Google Cloud Build pipeline

The [`cloudbuild.yaml`](./cloudbuild/cloudbuild.yaml) file located in the
[`cloudbuild` folder](./cloudbuild/) in this repository contains an example of using this
Python script to push updates or new detection content to a Chronicle instance using Google
Python script to push updates or new detection content to a Google Security Operations instance using Google
Cloud Build.

The pipeline file requires a secret to be created in Secrets Manager, and for this to be
made available to the service account running the build pipeline. Details for creating this
can be found [here](https://cloud.google.com/build/docs/securing-builds/use-secrets). The
value can be copy/pasted from the contents of your Chronicle API key into the Secrets
value can be copy/pasted from the contents of your Google SecOps API key into the Secrets
Manager UI or API.

Your repository should be added in Cloud Build, and a trigger created following [this document](https://cloud.google.com/build/docs/automating-builds/create-manage-triggers),
Expand All @@ -114,7 +114,7 @@ There are four user-defined substitutions to create to support this script:

Substitution Name | Description | Example Value
---|---|---
`_REGION` | The region for your Chronicle instance | us
`_REGION` | The region for your Google Security Operations instance | us
`_RULES_PATH` | The relative path from the root of the repository containing the YARA-L rules to work with | rules/yaral
`_PROJECT_ID` | The project ID containing the secret created earlier | my-project-id
`_SECRET_NAME` | The name of the secret created earlier | bk_api_credential
Expand All @@ -127,24 +127,24 @@ The cloudbuild.yaml file should be placed in the root of your repository.

The [`azure-pipelines.yml`](./azure-devops/azure-pipelines.yml) file located in the
[`azure-devops` folder](./azure-devops/) in this repository contains an example of using this
Python script to push updates or new detection content to a Chronicle instance using Azure DevOps
Python script to push updates or new detection content to a Google Security Operations instance using Azure DevOps
Pipelines.

There are two variables
to edit in this script:

Variable Name | Description | Example Value
---|---|---
`region` | The region for your Chronicle instance | us
`region` | The region for your Google Security Operations instance | us
`rules_path` | The relative path from the root of the repository containing the YARA-L rules to work with | rules/yaral

These should be modified for your environment, and the file should be placed in the root of your repository.

The pool name value should also be updated in the pipelines file, to match the agent pool you want to use to run the code.

In addition to this, the pipeline file requires a secret to be created on your Azure DevOps project. Details for creating this
can be found [here](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/set-secret-variables?view=azure-devops&tabs=yaml%2Cbash). The secret in this case should be named `SA_CREDENTIAL`. The value is the contents of your Chronicle API key, which can just be
can be found [here](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/set-secret-variables?view=azure-devops&tabs=yaml%2Cbash). The secret in this case should be named `SA_CREDENTIAL`. The value is the contents of your Google SecOps API key, which can just be
copy/pasted into the UI, shown in the linked document above, from the JSON file containing the key.

Now whenever a change is written to the repository the contents of the passed rules folder will be checked and updated/uploaded
on the Chronicle instance.
on the Google Security Operations instance.
2 changes: 1 addition & 1 deletion cicd/python/rules.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ def initialize_command_line_args(
parser.add_argument(
"-s", "--silent", action='store_true', help="supress error messages" )
parser.add_argument(
"-r", "--region", required=False, default="us", choices=("asia-southeast1", "europe", "us"), help="Chronicle instance region (leave blank for US)" )
"-r", "--region", required=False, default="us", choices=("asia-southeast1", "europe", "us"), help="Google Security Operations instance region (leave blank for US)" )

parser.set_defaults(make_changes=False)
return parser.parse_args(args)
Expand Down
4 changes: 2 additions & 2 deletions lib/doc_template.md.erb
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@
end%>

## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
<%= generate_query_links(query_paths, 'bigquery') %> | <%= generate_query_links(query_paths, 'chronicle') %> | <%= generate_query_links(query_paths, 'log_analytics') %>
<%= generate_query_links(query_paths, 'bigquery') %> | <%= generate_query_links(query_paths, 'log_analytics') %> | <%= generate_query_links(query_paths, 'chronicle') %>

## Event Generation
<%- if detection['test'] != nil -%>
Expand Down
4 changes: 2 additions & 2 deletions src/1.01/1.01.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ be it from Cloud Console, Admin Console or gcloud CLI.


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/1_01_login_highly_privileged_account.sql) | [YARA-L](../../backends/chronicle/yaral/1_01_login_highly_privileged_account.yaral) | [SQL](../../backends/log_analytics/sql/1_01_login_highly_privileged_account.sql)
[SQL](../../backends/bigquery/sql/1_01_login_highly_privileged_account.sql) | [SQL](../../backends/log_analytics/sql/1_01_login_highly_privileged_account.sql) | [YARA-L](../../backends/chronicle/yaral/1_01_login_highly_privileged_account.yaral)

## Event Generation

Expand Down
4 changes: 2 additions & 2 deletions src/1.02/1.02.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ A suspicious login attempt flagged by Google Workspace, be it from Cloud Console


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/1_02_suspicious_login_attempt.sql) | [YARA-L](../../backends/chronicle/yaral/1_02_suspicious_login_attempt.yaral) | [SQL](../../backends/log_analytics/sql/1_02_suspicious_login_attempt.sql)
[SQL](../../backends/bigquery/sql/1_02_suspicious_login_attempt.sql) | [SQL](../../backends/log_analytics/sql/1_02_suspicious_login_attempt.sql) | [YARA-L](../../backends/chronicle/yaral/1_02_suspicious_login_attempt.yaral)

## Event Generation
No event generation steps provided. [Contribute emulation test](../../CONTRIBUTING.md) to this use case.
Expand Down
4 changes: 2 additions & 2 deletions src/1.03/1.03.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ Multiple login failures (>= 3) detected from any particular user identity in the


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/1_03_excessive_login_failures.sql) | [YARA-L](../../backends/chronicle/yaral/1_03_excessive_login_failures.yaral) | [SQL](../../backends/log_analytics/sql/1_03_excessive_login_failures.sql)
[SQL](../../backends/bigquery/sql/1_03_excessive_login_failures.sql) | [SQL](../../backends/log_analytics/sql/1_03_excessive_login_failures.sql) | [YARA-L](../../backends/chronicle/yaral/1_03_excessive_login_failures.yaral)

## Event Generation

Expand Down
4 changes: 2 additions & 2 deletions src/1.10/1.10.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ see this [VPC Service Controls overview](https://cloud.google.com/vpc-service-co


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/1_10_access_attempts_blocked_by_VPC_SC.sql) | [Contribute rule](../../CONTRIBUTING.md) | [SQL](../../backends/log_analytics/sql/1_10_access_attempts_blocked_by_VPC_SC.sql)
[SQL](../../backends/bigquery/sql/1_10_access_attempts_blocked_by_VPC_SC.sql) | [SQL](../../backends/log_analytics/sql/1_10_access_attempts_blocked_by_VPC_SC.sql) | [Contribute rule](../../CONTRIBUTING.md)

## Event Generation

Expand Down
4 changes: 2 additions & 2 deletions src/1.20/1.20.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ Access attempt blocked by Identity-Aware Proxy (IAP), indicating an initial acce


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/1_20_access_attempts_blocked_by_IAP.sql) | [YARA-L](../../backends/chronicle/yaral/1_20_access_attempts_blocked_by_IAP.yaral) | [SQL](../../backends/log_analytics/sql/1_20_access_attempts_blocked_by_IAP.sql)
[SQL](../../backends/bigquery/sql/1_20_access_attempts_blocked_by_IAP.sql) | [SQL](../../backends/log_analytics/sql/1_20_access_attempts_blocked_by_IAP.sql) | [YARA-L](../../backends/chronicle/yaral/1_20_access_attempts_blocked_by_IAP.yaral)

## Event Generation

Expand Down
Loading

0 comments on commit c4853d5

Please sign in to comment.