Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Adds Prometheus metrics to ODIS combiner daemon (Odis Combiner to k8s) #49

Merged
merged 42 commits into from
Oct 13, 2023

Conversation

alvarof2
Copy link
Member

@alvarof2 alvarof2 commented Sep 29, 2023

Description

  • Adds a new endpoint /metrics to ODIS combiner to expose metrics.
    Docker images:
  • Moves dockerfiles/phone-number-privacy/Dockerfile to dockerfiles/phone-number-privacy/Dockerfile-signer for the signer container.
  • Adds dockerfiles/phone-number-privacy/Dockerfile-combiner for the combiner container.
  • Modifies cloudbuild.yaml accordingly.

Adds the following counters and histograms:

  • Counter combiner_requests_total
  • Counter combiner_responses_total
  • Counter combiner_blockchain_errors_total
  • Counter combiner_bls_compute_errors_total
  • Counter combiner_endpoint_handler_errors_total
  • Counter combiner_not_enough_sig_errors_total
  • Counter combiner_sig_request_errors_total
  • Counter combiner_sig_response_errors_total
  • Counter combiner_sig_inconsistency_errors_total
  • Counter combiner_sig_response_total
  • Counter combiner_unknown_errors_total
  • Counter combiner_warnings_total
  • Histogram combiner_endpoint_latency
  • Histogram combiner_full_node_latency
  • Histogram combiner_signer_latency

I couldn't find a way to create the event loop lag Histogram :(

Current metrics can be seen here: https://clabs.grafana.net/goto/2qF4RaWSg?orgId=1

Other changes

Describe any minor or "drive-by" changes here.

Tested

An explanation of how the changes were tested or an explanation as to why they don't need to be.

Related issues

  • Fixes #[issue number here]

Backwards compatibility

Brief explanation of why these changes are/are not backwards compatible.

Documentation

The set of community facing docs that have been added/modified because of this change

@changeset-bot
Copy link

changeset-bot bot commented Sep 29, 2023

⚠️ No Changeset found

Latest commit: d04bc9a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@jcortejoso
Copy link
Member

I'd be in favor of sunsetting cloud-build and replacing by GitHub Actions.

Happy to address that in a separate PR.

@alvarof2 alvarof2 changed the base branch from alecps/odisCombinerk8s to main October 3, 2023 21:00
@socket-security
Copy link

socket-security bot commented Oct 3, 2023

No top level dependency changes detected. Learn more about Socket for GitHub ↗︎

Copy link
Member

@aaronmgdr aaronmgdr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reminder to add a changeset

@alvarof2 alvarof2 changed the base branch from main to alecps/odisCombinerk8s October 5, 2023 16:58
@alvarof2 alvarof2 changed the base branch from alecps/odisCombinerk8s to main October 5, 2023 16:59
@github-advanced-security
Copy link

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

@alvarof2 alvarof2 changed the base branch from main to alecps/odisCombinerk8s October 6, 2023 15:13
@alvarof2 alvarof2 changed the base branch from alecps/odisCombinerk8s to main October 6, 2023 15:14
@alvarof2 alvarof2 changed the base branch from main to alecps/odisCombinerk8s October 6, 2023 15:57
@alvarof2 alvarof2 marked this pull request as ready for review October 6, 2023 16:34
Copy link
Collaborator

@alecps alecps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks!! Before I merge the odisCombinerK8s branch I'll go through one last time and ensure we aren't double counting errors anywhere

if (!res.headersSent) {
if (err instanceof OdisError) {
sendFailure(err.code, err.status, res, req.url)
} else {
Counters.errors.labels(req.url).inc()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is double counting I believe

config.phoneNumberPrivacy.fullNodeRetryDelayMs
).catch((err) => {
logger.error({ err, account }, 'failed to get on-chain DEK for account')
Counters.errors.labels('getDEK').inc()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a preset for this?

@@ -23,10 +24,12 @@ export function disableDomain(
): ResultHandler<DisableDomainRequest> {
return async (request, response) => {
if (!disableDomainRequestSchema(DomainSchema).is(request.body)) {
Counters.warnings.labels(request.url, WarningMessage.INVALID_INPUT).inc()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this go in errorResult?

@alecps alecps merged commit 9170664 into alecps/odisCombinerk8s Oct 13, 2023
5 checks passed
@alecps alecps deleted the alvarof/prom-odisCombinerk8s branch October 13, 2023 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants