Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename Chronicle to Google SecOps #53

Merged
merged 5 commits into from
Apr 24, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
rarsan marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ To implement a `SQL` query or `YARA-L` rule for an existing or new security ques

1. To provide a BigQuery SQL, implement query in a new SQL file under `sql/` folder, named after corresponding CSA ID and name, which is in our example `5_40_sql_tables_most_frequently_accessed.sql`.

2. To add a Chronicle YARA-L rule, implement rule in a new YARA-L file under `yaral/` folder, named after corresponding CSA ID and name, which is in our example
2. To add a Google SecOps YARA-L rule, implement rule in a new YARA-L file under `yaral/` folder, named after corresponding CSA ID and name, which is in our example
`5_40_sql_tables_most_frequently_accessed.yaral` .

CSA design favors **convention over configuration** when it comes to file naming and docs generation. Therefore, special attention is required for queries/rules file naming where the new filename must match a specific format per above examples, where:
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ As organizations go through the Autonomic Security modernization journey, this r
CSA is a set of foundational security analytics designed to provide organizations with a rich baseline of pre-built queries and rules that they can readily use to start analyzing their Google Cloud logs including Cloud Audit logs, VPC Flow logs, DNS logs, and more using cloud-native or third-party analytics tools. The source code is provided as is, without warranty. See [Copyright & License](#copyright--license) below.

Current release include:
- YARA-L rules for [Chronicle](https://chronicle.security/)
- YARA-L rules for [Google Security Operations](https://chronicle.security/)
- SQL queries for [BigQuery](https://cloud.google.com/bigquery/)
- SQL queries for [Log Analytics](https://cloud.google.com/logging/docs/log-analytics)

Expand All @@ -21,9 +21,9 @@ The security use cases below are grouped in 6 categories depending on underlying
5. :droplet: [Data Usage](#data-usage)
6. :zap: [Network Activity](#network-activity)

To learn more about the variety of Google Cloud logs, how to enable and natively export these logs to destinations like Chronicle or BigQuery for in-depth analytics, refer to Google Cloud [Security and access analytics solution guide](https://cloud.google.com/architecture/exporting-stackdriver-logging-for-security-and-access-analytics).
To learn more about the variety of Google Cloud logs, how to enable and natively export these logs to destinations like BigQuery or Google Security Operations for in-depth analytics, refer to Google Cloud [Security and access analytics solution guide](https://cloud.google.com/architecture/exporting-stackdriver-logging-for-security-and-access-analytics).

**Caution:** CSA is not meant to be a comprehensive set of threat detections, but a collection of community-contributed samples to get you started with detective controls. Use CSA in your threat detection and response capabilities (e.g. [Security Command Center](https://cloud.google.com/security-command-center), Chronicle, BigQuery, Siemplify, or third-party SIEM) in conjunction with threat prevention capabilities (e.g. [Security Command Center](https://cloud.google.com/security-command-center), [Cloud Armor](https://cloud.google.com/armor), [BeyondCorp](https://cloud.google.com/beyondcorp)). To learn more about Google’s approach to modern Security Operations, check out the [Autonomic Security Operations whitepaper](https://services.google.com/fh/files/misc/googlecloud_autonomicsecurityoperations_soc10x.pdf).
**Caution:** CSA is not meant to be a comprehensive set of threat detections, but a collection of community-contributed samples to get you started with detective controls. Use CSA in your threat detection and response capabilities (e.g. [Security Command Center](https://cloud.google.com/security-command-center), Google Security Operations, BigQuery, Siemplify, or third-party SIEM) in conjunction with threat prevention capabilities (e.g. [Security Command Center](https://cloud.google.com/security-command-center), [Cloud Armor](https://cloud.google.com/armor), [BeyondCorp](https://cloud.google.com/beyondcorp)). To learn more about Google’s approach to modern Security Operations, check out the [Autonomic Security Operations whitepaper](https://services.google.com/fh/files/misc/googlecloud_autonomicsecurityoperations_soc10x.pdf).
rarsan marked this conversation as resolved.
Show resolved Hide resolved

## Security Analytics Use Cases
![Security Monitoring](./assets/gcp_security_mon.png)
Expand Down Expand Up @@ -92,9 +92,9 @@ To learn more about the variety of Google Cloud logs, how to enable and natively

The [`dataform` folder](./dataform/) contains the Dataform repo to automate deployment of CSA queries in BigQuery for optimized performance and cost. Use this Dataform repo to operationalize CSA use cases as reports and alerts powered by BigQuery. This Dataform project deploys and orchestrates pre-built ELT pipelines to filter, normalize and model log data leveraging incremental summary tables, lookup tables and views for fast, cost-effective and simpler querying. See underlying [README](./dataform/README.md) for more details.

## CI/CD for CSA on Chronicle
## CI/CD for CSA on Google Security Operations

The [`cicd` folder](./cicd/) contains a set of scripts to help you with storing CSA YARA-L detection rules as code and testing/deploying updates you and your team make in an automated fashion. Whether you use GitHub Actions, Google Cloud Build or Azure DevOps, you can use the corresponding scripts to automatically test and deploy new or modified rules into your Chronicle instance. See underlying [README](./cicd/README.md) for more details.
The [`cicd` folder](./cicd/) contains a set of scripts to help you with storing CSA YARA-L detection rules as code and testing/deploying updates you and your team make in an automated fashion. Whether you use GitHub Actions, Google Cloud Build or Azure DevOps, you can use the corresponding scripts to automatically test and deploy new or modified rules into your Google Security Operations instance. See underlying [README](./cicd/README.md) for more details.

## Support

Expand Down
32 changes: 16 additions & 16 deletions cicd/README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
# CI/CD for CSA on Chronicle
# CI/CD for CSA on Google Security Operations

## Overview

The [`python` folder](./python/) contains Python helper scripts to gather rules from a local folder and from the remote
Chronicle instance, and to compare them, uploading changed or new files to the Chronicle instance.
Google Security Operations instance, and to compare them, uploading changed or new files to the Google Security Operations instance.

## Pre-installation

Pre-requisites can be installed by copying the files from the folder locally, and running `pip3 install -r requirements.txt`

To run this script you will need a Chronicle API Service Account JSON file. This can be obtained from your Google Chronicle
To run this script you will need a Google SecOps API Service Account JSON file. This can be obtained from your Google Security Operations
account team.

## Usage

First export the path to your Chronicle API key like this:
First export the path to your Google SecOps API key like this:

```bash
export PATH_TO_KEY=~/malachite-abc-7ba40dd4f123.json
Expand Down Expand Up @@ -46,7 +46,7 @@ optional arguments:
differences
-s, --silent supress error messages
-r REGION, --region REGION
Chronicle instance region (leave blank for US)
Google Security Operations instance region (leave blank for US)
```

NOTE: if you provide more than one of `CREDENTIALS_FILE`, `CREDENTIALS_INFO`, `CREDENTIALS_ENV`, then the
Expand Down Expand Up @@ -77,34 +77,34 @@ Example output from making changes (including `-m` flag):

The [`chronicle-rules-cicd.yml`](./github-actions/chronicle-rules-cicd.yml) file located in the
[`github-actions` folder](./github-actions/) in this repository contains an example of using this
Python script to push updates or new detection content to a Chronicle instance. There are two variables
Python script to push updates or new detection content to a Google Security Operations instance. There are two variables
to edit in this script:

Variable Name | Description | Example Value
---|---|---
`region` | The region for your Chronicle instance | us
`region` | The region for your Google Security Operations instance | us
`rules_path` | The relative path from the root of the repository containing the YARA-L rules to work with | rules/yaral

These should be modified for your environment, and the file should be placed in a `.github/workflows` folder in the root of your repository.

In addition to this, the pipeline file requires a secret to be created on your GitHub repository. Details for creating this
can be found [here](https://docs.github.com/en/actions/security-guides/encrypted-secrets). The secret in this case should
be named `SA_CREDENTIAL`. The value is derived from the contents of your Chronicle API key, but line breaks should be
be named `SA_CREDENTIAL`. The value is derived from the contents of your Google SecOps API key, but line breaks should be
removed from the file, and the `"` character should also be replaced with `\"`. The resulting string can then be pasted into the Secrets UI in GitHub.

Now whenever a change is written to the repository the contents of the passed rules folder will be checked and updated/uploaded on the Chronicle instance. To learn more GitHub Actions workflows, see [here](https://docs.github.com/en/actions/using-workflows/about-workflows).
Now whenever a change is written to the repository the contents of the passed rules folder will be checked and updated/uploaded on the Google Security Operations instance. To learn more GitHub Actions workflows, see [here](https://docs.github.com/en/actions/using-workflows/about-workflows).

## Using in a Google Cloud Build pipeline

The [`cloudbuild.yaml`](./cloudbuild/cloudbuild.yaml) file located in the
[`cloudbuild` folder](./cloudbuild/) in this repository contains an example of using this
Python script to push updates or new detection content to a Chronicle instance using Google
Python script to push updates or new detection content to a Google Security Operations instance using Google
Cloud Build.

The pipeline file requires a secret to be created in Secrets Manager, and for this to be
made available to the service account running the build pipeline. Details for creating this
can be found [here](https://cloud.google.com/build/docs/securing-builds/use-secrets). The
value can be copy/pasted from the contents of your Chronicle API key into the Secrets
value can be copy/pasted from the contents of your Google SecOps API key into the Secrets
Manager UI or API.

Your repository should be added in Cloud Build, and a trigger created following [this document](https://cloud.google.com/build/docs/automating-builds/create-manage-triggers),
Expand All @@ -114,7 +114,7 @@ There are four user-defined substitutions to create to support this script:

Substitution Name | Description | Example Value
---|---|---
`_REGION` | The region for your Chronicle instance | us
`_REGION` | The region for your Google Security Operations instance | us
`_RULES_PATH` | The relative path from the root of the repository containing the YARA-L rules to work with | rules/yaral
`_PROJECT_ID` | The project ID containing the secret created earlier | my-project-id
`_SECRET_NAME` | The name of the secret created earlier | bk_api_credential
Expand All @@ -127,24 +127,24 @@ The cloudbuild.yaml file should be placed in the root of your repository.

The [`azure-pipelines.yml`](./azure-devops/azure-pipelines.yml) file located in the
[`azure-devops` folder](./azure-devops/) in this repository contains an example of using this
Python script to push updates or new detection content to a Chronicle instance using Azure DevOps
Python script to push updates or new detection content to a Google Security Operations instance using Azure DevOps
Pipelines.

There are two variables
to edit in this script:

Variable Name | Description | Example Value
---|---|---
`region` | The region for your Chronicle instance | us
`region` | The region for your Google Security Operations instance | us
`rules_path` | The relative path from the root of the repository containing the YARA-L rules to work with | rules/yaral

These should be modified for your environment, and the file should be placed in the root of your repository.

The pool name value should also be updated in the pipelines file, to match the agent pool you want to use to run the code.

In addition to this, the pipeline file requires a secret to be created on your Azure DevOps project. Details for creating this
can be found [here](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/set-secret-variables?view=azure-devops&tabs=yaml%2Cbash). The secret in this case should be named `SA_CREDENTIAL`. The value is the contents of your Chronicle API key, which can just be
can be found [here](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/set-secret-variables?view=azure-devops&tabs=yaml%2Cbash). The secret in this case should be named `SA_CREDENTIAL`. The value is the contents of your Google SecOps API key, which can just be
copy/pasted into the UI, shown in the linked document above, from the JSON file containing the key.

Now whenever a change is written to the repository the contents of the passed rules folder will be checked and updated/uploaded
on the Chronicle instance.
on the Google Security Operations instance.
2 changes: 1 addition & 1 deletion cicd/python/rules.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ def initialize_command_line_args(
parser.add_argument(
"-s", "--silent", action='store_true', help="supress error messages" )
parser.add_argument(
"-r", "--region", required=False, default="us", choices=("asia-southeast1", "europe", "us"), help="Chronicle instance region (leave blank for US)" )
"-r", "--region", required=False, default="us", choices=("asia-southeast1", "europe", "us"), help="Google Security Operations instance region (leave blank for US)" )

parser.set_defaults(make_changes=False)
return parser.parse_args(args)
Expand Down
4 changes: 2 additions & 2 deletions lib/doc_template.md.erb
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@
end%>

## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
<%= generate_query_links(query_paths, 'bigquery') %> | <%= generate_query_links(query_paths, 'chronicle') %> | <%= generate_query_links(query_paths, 'log_analytics') %>
<%= generate_query_links(query_paths, 'bigquery') %> | <%= generate_query_links(query_paths, 'log_analytics') %> | <%= generate_query_links(query_paths, 'chronicle') %>

## Event Generation
<%- if detection['test'] != nil -%>
Expand Down
4 changes: 2 additions & 2 deletions src/1.01/1.01.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ be it from Cloud Console, Admin Console or gcloud CLI.


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/1_01_login_highly_privileged_account.sql) | [YARA-L](../../backends/chronicle/yaral/1_01_login_highly_privileged_account.yaral) | [SQL](../../backends/log_analytics/sql/1_01_login_highly_privileged_account.sql)
[SQL](../../backends/bigquery/sql/1_01_login_highly_privileged_account.sql) | [SQL](../../backends/log_analytics/sql/1_01_login_highly_privileged_account.sql) | [YARA-L](../../backends/chronicle/yaral/1_01_login_highly_privileged_account.yaral)

## Event Generation

Expand Down
4 changes: 2 additions & 2 deletions src/1.02/1.02.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ A suspicious login attempt flagged by Google Workspace, be it from Cloud Console


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/1_02_suspicious_login_attempt.sql) | [YARA-L](../../backends/chronicle/yaral/1_02_suspicious_login_attempt.yaral) | [SQL](../../backends/log_analytics/sql/1_02_suspicious_login_attempt.sql)
[SQL](../../backends/bigquery/sql/1_02_suspicious_login_attempt.sql) | [SQL](../../backends/log_analytics/sql/1_02_suspicious_login_attempt.sql) | [YARA-L](../../backends/chronicle/yaral/1_02_suspicious_login_attempt.yaral)

## Event Generation
No event generation steps provided. [Contribute emulation test](../../CONTRIBUTING.md) to this use case.
Expand Down
4 changes: 2 additions & 2 deletions src/1.03/1.03.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ Multiple login failures (>= 3) detected from any particular user identity in the


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/1_03_excessive_login_failures.sql) | [YARA-L](../../backends/chronicle/yaral/1_03_excessive_login_failures.yaral) | [SQL](../../backends/log_analytics/sql/1_03_excessive_login_failures.sql)
[SQL](../../backends/bigquery/sql/1_03_excessive_login_failures.sql) | [SQL](../../backends/log_analytics/sql/1_03_excessive_login_failures.sql) | [YARA-L](../../backends/chronicle/yaral/1_03_excessive_login_failures.yaral)

## Event Generation

Expand Down
4 changes: 2 additions & 2 deletions src/1.10/1.10.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ see this [VPC Service Controls overview](https://cloud.google.com/vpc-service-co


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/1_10_access_attempts_blocked_by_VPC_SC.sql) | [Contribute rule](../../CONTRIBUTING.md) | [SQL](../../backends/log_analytics/sql/1_10_access_attempts_blocked_by_VPC_SC.sql)
[SQL](../../backends/bigquery/sql/1_10_access_attempts_blocked_by_VPC_SC.sql) | [SQL](../../backends/log_analytics/sql/1_10_access_attempts_blocked_by_VPC_SC.sql) | [Contribute rule](../../CONTRIBUTING.md)

## Event Generation

Expand Down
4 changes: 2 additions & 2 deletions src/1.20/1.20.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ Access attempt blocked by Identity-Aware Proxy (IAP), indicating an initial acce


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/1_20_access_attempts_blocked_by_IAP.sql) | [YARA-L](../../backends/chronicle/yaral/1_20_access_attempts_blocked_by_IAP.yaral) | [SQL](../../backends/log_analytics/sql/1_20_access_attempts_blocked_by_IAP.sql)
[SQL](../../backends/bigquery/sql/1_20_access_attempts_blocked_by_IAP.sql) | [SQL](../../backends/log_analytics/sql/1_20_access_attempts_blocked_by_IAP.sql) | [YARA-L](../../backends/chronicle/yaral/1_20_access_attempts_blocked_by_IAP.yaral)

## Event Generation

Expand Down
4 changes: 2 additions & 2 deletions src/1.30/1.30.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,9 @@ details are not captured in the audit logs.


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/1_30_console_accesses.sql) | [Contribute rule](../../CONTRIBUTING.md) | [SQL](../../backends/log_analytics/sql/1_30_console_accesses.sql)
[SQL](../../backends/bigquery/sql/1_30_console_accesses.sql) | [SQL](../../backends/log_analytics/sql/1_30_console_accesses.sql) | [Contribute rule](../../CONTRIBUTING.md)

## Event Generation
No event generation steps provided. [Contribute emulation test](../../CONTRIBUTING.md) to this use case.
Expand Down
4 changes: 2 additions & 2 deletions src/2.02/2.02.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ A user member added via Admin Service (Admin Console or Directory API) to a high


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/2_02_user_added_to_privileged_group.sql) | [Contribute rule](../../CONTRIBUTING.md) | [SQL](../../backends/log_analytics/sql/2_02_user_added_to_privileged_group.sql)
[SQL](../../backends/bigquery/sql/2_02_user_added_to_privileged_group.sql) | [SQL](../../backends/log_analytics/sql/2_02_user_added_to_privileged_group.sql) | [Contribute rule](../../CONTRIBUTING.md)

## Event Generation

Expand Down
4 changes: 2 additions & 2 deletions src/2.20/2.20.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ IAM roles granted at the parent project, folder or organization-level.


## Queries or Rules
BigQuery | Chronicle | Log Analytics
BigQuery | Log Analytics | Google SecOps
--- | --- | ---
[SQL](../../backends/bigquery/sql/2_20_permissions_granted_over_SA.sql) | [YARA-L](../../backends/chronicle/yaral/2_20_permissions_granted_over_SA.yaral) | [SQL](../../backends/log_analytics/sql/2_20_permissions_granted_over_SA.sql)
[SQL](../../backends/bigquery/sql/2_20_permissions_granted_over_SA.sql) | [SQL](../../backends/log_analytics/sql/2_20_permissions_granted_over_SA.sql) | [YARA-L](../../backends/chronicle/yaral/2_20_permissions_granted_over_SA.yaral)

## Event Generation

Expand Down
Loading
Loading