Ported monitoring stack to k3s #449
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bertiethorpe
previously approved these changes
Oct 29, 2024
|
no image changes since last build so last commit should be ready to merge barring review changes |
sjpb
requested changes
Nov 12, 2024
There was a problem hiding this comment.
Please note in the PR (with better wording!) that
- the "prometheus" group is bascally short for "kube-prometheus-stack" group!
- the monitoring link in CaaS now accesses grafana with anonymous auth (b/c it has to go via OOD), so CaaS users can't change their dashboards
Co-authored-by: Steve Brasier <[email protected]>
…slurm-appliance into feature/k3s-monitoring
|
@wtripp180901 not a high priority but would be nice to know if this PR reduces the size of the data in the image. And/or whether we can reduce the required root disk size at all - which isn't the same thing, b/c e.g. dnf caches which we throw away require additional size during build. I think you'd need qemu-img info to see the former. And monitoring disk usage during build to see the latter. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Monitoring stack (prometheus/node exporter/grafana/alertmanager) binary installs removed from site and fatimage, now installs kube-prometheus-stack Helm chart into k3s cluster during site run. Containers are pre-pulled by podman and exported into k3s during fatimage build.
As a consequence, the
grafana,alertmanagerandnode exportergroups have been removed and associated roles are now all managed by theprometheusrole, which is short for kube_prometheus_stackAlso reduced metrics collected by node exporter down to minimal set described in
docs/monitoring-and-logging.README.md, which was previously unimplementedNote that because of how OOD's proxying interacts with Grafana's server config and kubernetes, OOD being enabled means that Grafana is only accessible through the OOD proxy. In the caas environment, this means that accessing Grafana requires authenticating with OOD's basic auth. Therefore, accessing Grafana through caas no longer logs you in as the admin user, you instead access the dashboards anonymously