Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let product operators control product-specific vector config #794

Open
nightkr opened this issue May 23, 2024 · 0 comments
Open

Let product operators control product-specific vector config #794

nightkr opened this issue May 23, 2024 · 0 comments

Comments

@nightkr
Copy link
Member

nightkr commented May 23, 2024

Currently, op-rs embeds a lot of product-specific knowledge. For example, there are several configuration bits that are only relevant to our specific deployment of OPA that are always embedded in the configuration (

files_opa_bundle_builder:
type: file
include:
- {STACKABLE_LOG_DIR}/bundle-builder/current
files_opa_json:
type: file
include:
- {STACKABLE_LOG_DIR}/opa/current
transforms:
processed_files_opa_bundle_builder:
inputs:
- files_opa_bundle_builder
type: remap
source: |
parsed_event = parse_regex!(strip_whitespace(strip_ansi_escape_codes(string!(.message))), r'(?P<timestamp>[0-9]{{4}}-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])T(2[0-3]|[01][0-9]):[0-5][0-9]:[0-5][0-9].[0-9]{{6}}Z)[ ]+(?P<level>\w+)[ ]+(?P<logger>.+):[ ]+(?P<message>.*)')
.timestamp = parse_timestamp!(parsed_event.timestamp, "%Y-%m-%dT%H:%M:%S.%6fZ")
.level = parsed_event.level
.logger = parsed_event.logger
.message = parsed_event.message
processed_files_opa_json:
inputs:
- files_opa_json
type: remap
source: |
parsed_event = parse_json!(string!(.message))
keys = keys!(parsed_event)
if includes(keys, "timestamp") {{
.timestamp = parse_timestamp!(parsed_event.timestamp, "%Y-%m-%dT%H:%M:%S.%fZ")
}} else {{
.timestamp = parse_timestamp!(parsed_event.time, "%Y-%m-%dT%H:%M:%SZ")
}}
if includes(keys, "decision_id") {{
.logger = "decision"
}} else {{
.logger = "server"
}}
.level = upcase!(parsed_event.level)
.message = string!(parsed_event.msg)
del(parsed_event.time)
del(parsed_event.timestamp)
del(parsed_event.level)
del(parsed_event.msg)
.message = .message + "\n" + encode_key_value(object!(parsed_event), field_delimiter: "\n")
).

It makes sense to offer a few defaults ("this is how we typically configure log4j to behave"), but the final say should be up to the operator and we definitely shouldn't have single-product configurations in the framework crate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant