Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - Enable logging #407

Closed
wants to merge 28 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
77067d0
Implement configuration merge
siegfriedweber Feb 22, 2023
e4dd4f7
Add replicas and selector to CommonConfig
siegfriedweber Feb 22, 2023
cb1cd8c
Remove unused error variants
siegfriedweber Feb 22, 2023
f1fae47
Update changelog
siegfriedweber Feb 22, 2023
ba11a8f
Upgrade operator-rs to version 0.35.0
siegfriedweber Feb 22, 2023
b4dd7fe
Add code comments
siegfriedweber Feb 22, 2023
ccc5f6d
Enable log aggregation
siegfriedweber Feb 23, 2023
7eded34
Add logging test
siegfriedweber Feb 27, 2023
c49233c
Enable log aggregation in authorizer test
siegfriedweber Feb 27, 2023
ede7f9a
Enable log aggregation in hdfs-deep-storage test
siegfriedweber Feb 27, 2023
3a1c662
Enable log aggregation in ingestion-no-s3-ext test
siegfriedweber Feb 27, 2023
5f17132
Enable log aggregation in ingestion-s3-ext test
siegfriedweber Feb 27, 2023
3cd0edb
Enable log aggregation in ldap-authentication test
siegfriedweber Feb 27, 2023
35ccdba
Enable log aggregation in orphand-resources test
siegfriedweber Feb 27, 2023
16257c3
Enable log aggregation in resources test
siegfriedweber Feb 27, 2023
eb052ef
Enable log aggregation in s3-deep-storage test
siegfriedweber Feb 27, 2023
012f54a
Enable log aggregation in smoke test
siegfriedweber Feb 27, 2023
b8183b4
Enable log aggregation in tls test
siegfriedweber Feb 27, 2023
9389036
Fix custom configuration in logging test
siegfriedweber Feb 27, 2023
f95bb6b
Regenerate charts
siegfriedweber Feb 27, 2023
0942915
Upgrade stackableVersions of images
siegfriedweber Feb 27, 2023
2694fe2
Add section about log aggregation to the documentation
siegfriedweber Feb 27, 2023
19dbb67
Update changelog
siegfriedweber Feb 27, 2023
c2bdda5
Merge branch 'main' into enable-logging
siegfriedweber Feb 27, 2023
f92e27e
Optimize code
siegfriedweber Feb 28, 2023
0fcaa74
Use constants for volume names
siegfriedweber Feb 28, 2023
3f11f5c
Set stackableVersion of hadoop to 23.4.0-rc2
siegfriedweber Feb 28, 2023
d205926
Use variables for container names
siegfriedweber Mar 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ All notable changes to this project will be documented in this file.

- Add support for non-TLS LDAP authentication. ([#374])
- Deploy default and support custom affinities ([#406]).
- Log aggregation added ([#407]).

### Changed

Expand All @@ -24,6 +25,7 @@ All notable changes to this project will be documented in this file.
[#387]: https://github.com/stackabletech/druid-operator/pull/387
[#404]: https://github.com/stackabletech/druid-operator/pull/404
[#406]: https://github.com/stackabletech/druid-operator/pull/406
[#407]: https://github.com/stackabletech/druid-operator/pull/407

## [23.1.0] - 2023-01-23

Expand Down
764 changes: 764 additions & 0 deletions deploy/helm/druid-operator/crds/crds.yaml

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/modules/druid/examples/getting_started/druid.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
spec:
image:
productVersion: 24.0.0
stackableVersion: 0.3.0
stackableVersion: 23.4.0-rc2
clusterConfig:
deepStorage:
hdfs:
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/druid/examples/getting_started/hdfs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
spec:
image:
productVersion: 3.3.4
stackableVersion: 0.2.0
stackableVersion: 23.4.0-rc2
zookeeperConfigMapName: simple-hdfs-znode
dfsReplication: 3
nameNodes:
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/druid/examples/getting_started/zookeeper.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
spec:
image:
productVersion: 3.8.0
stackableVersion: 0.8.0
stackableVersion: 23.4.0-rc2
servers:
roleGroups:
default:
Expand Down
37 changes: 36 additions & 1 deletion docs/modules/druid/pages/usage.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ metadata:
spec:
image:
productVersion: 24.0.0
stackableVersion: 0.3.0
stackableVersion: 23.4.0-rc2
clusterConfig:
deepStorage:
hdfs:
Expand Down Expand Up @@ -390,6 +390,41 @@ The operator creates a `ConfigMap` with the name of the cluster which contains c
The managed Druid instances are automatically configured to export Prometheus metrics. See
xref:operators:monitoring.adoc[] for more details.

== Log aggregation

The logs can be forwarded to a Vector log aggregator by providing a discovery
ConfigMap for the aggregator and by enabling the log agent:

[source,yaml]
----
spec:
clusterConfig:
vectorAggregatorConfigMapName: vector-aggregator-discovery
brokers:
config:
logging:
enableVectorAgent: true
coordinators:
config:
logging:
enableVectorAgent: true
historicals:
config:
logging:
enableVectorAgent: true
middleManagers:
config:
logging:
enableVectorAgent: true
routers:
config:
logging:
enableVectorAgent: true
----

Further information on how to configure logging, can be found in
xref:home:concepts:logging.adoc[].

== Configuration & Environment Overrides

The cluster definition also supports overriding configuration properties and environment variables, either per role or per role group, where the more specific override (role group) has precedence over the less specific one (role).
Expand Down
4 changes: 2 additions & 2 deletions examples/psql-s3/psql-s3-druid-cluster.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
spec:
image:
productVersion: 3.8.0
stackableVersion: 0.8.0
stackableVersion: 23.4.0-rc2
servers:
roleGroups:
default:
Expand Down Expand Up @@ -47,7 +47,7 @@ metadata:
spec:
image:
productVersion: 24.0.0
stackableVersion: 0.3.0
stackableVersion: 23.4.0-rc2
clusterConfig:
deepStorage:
s3:
Expand Down
6 changes: 3 additions & 3 deletions examples/psql/psql-hdfs-druid-cluster.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
spec:
image:
productVersion: 3.8.0
stackableVersion: 0.8.0
stackableVersion: 23.4.0-rc2
servers:
roleGroups:
default:
Expand Down Expand Up @@ -35,7 +35,7 @@ metadata:
spec:
image:
productVersion: 3.3.4
stackableVersion: 0.2.0
stackableVersion: 23.4.0-rc2
zookeeperConfigMapName: psql-druid-znode
dfsReplication: 1
nameNodes:
Expand All @@ -58,7 +58,7 @@ metadata:
spec:
image:
productVersion: 24.0.0
stackableVersion: 0.3.0
stackableVersion: 23.4.0-rc2
clusterConfig:
deepStorage:
hdfs:
Expand Down
6 changes: 3 additions & 3 deletions examples/tls/tls-druid-cluster.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
spec:
image:
productVersion: 3.8.0
stackableVersion: 0.8.0
stackableVersion: 23.4.0-rc2
servers:
roleGroups:
default:
Expand All @@ -27,7 +27,7 @@ metadata:
spec:
image:
productVersion: 3.3.4
stackableVersion: 0.2.0
stackableVersion: 23.4.0-rc2
zookeeperConfigMapName: druid-hdfs-znode
dfsReplication: 1
nameNodes:
Expand Down Expand Up @@ -80,7 +80,7 @@ metadata:
spec:
image:
productVersion: 24.0.0
stackableVersion: 0.3.0
stackableVersion: 23.4.0-rc2
clusterConfig:
authentication:
- authenticationClass: druid-mtls-authentication-class
Expand Down
Loading