Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metrics: add running metrics for grpc #639

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

bufferflies
Copy link

@bufferflies bufferflies commented Nov 22, 2023

close: #640
the metrics after this pr
7adc4fcd-8c1c-421a-b461-60866449cfec

Signed-off-by: bufferflies <[email protected]>
bufferflies and others added 2 commits November 22, 2023 18:28
Signed-off-by: bufferflies <[email protected]>
Signed-off-by: GitHub <[email protected]>
Copy link
Member

@overvenus overvenus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better not to directly introduce Prometheus. There are two ways to do it:

  1. Use the monitoring infrastructure that comes with grpc (C++ counter/histogram), like this PR Support collecting gRPC core metrics #625.
  2. Add a feature to make Prometheus an optional dependency.

src/env.rs Outdated Show resolved Hide resolved
src/metrics.rs Outdated Show resolved Hide resolved
Signed-off-by: bufferflies <[email protected]>
@bufferflies
Copy link
Author

It's better not to directly introduce Prometheus. There are two ways to do it:

  1. Use the monitoring infrastructure that comes with grpc (C++ counter/histogram), like this PR Support collecting gRPC core metrics #625.
  2. Add a feature to make Prometheus an optional dependency.

good idea, I had add new feature("promethes ") to check the dependency.

@bufferflies
Copy link
Author

/ping @overvenus

Cargo.toml Outdated
@@ -27,6 +27,8 @@ prost = { version = "0.11", optional = true }
bytes = { version = "1.0", optional = true }
log = "0.4"
parking_lot = "0.12"
prometheus = { version = "0.13", default-features = false }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
prometheus = { version = "0.13", default-features = false }
prometheus = { version = "0.13", default-features = false, optional = true }

@@ -42,8 +44,9 @@ members = [
exclude = ["xtask"]

[features]
default = ["protobuf-codec", "boringssl"]
default = ["protobuf-codec", "boringssl","prometheus"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should not enable the "prometheus" by default.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It maybe bring the resouce usage.

Cargo.toml Outdated
_secure = []
prometheus = []
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
prometheus = []
prometheus = ["dep:prometheus"]

src/env.rs Outdated

#[cfg(feature = "prometheus")]
pub struct GRPCRunner {
cq_next_duration_his: Histogram,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better use LocalHistogram and periodically flush(e.g. for every 1k calls) to reduce the overhead.

src/env.rs Outdated
cq_next_duration_his: Histogram,
execute_duration_his: Histogram,
wait_duration_his: Histogram,
event_counter: [GenericCounter<AtomicU64>; 6],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
event_counter: [GenericCounter<AtomicU64>; 6],
event_counter: [Counter; 6],

src/metrics.rs Outdated
"grpc_pool_execute_duration",
"Bucketed histogram of grpc pool execute duration for every time",
&["name"],
exponential_buckets(1e-7, 2.0, 20).unwrap() // 100ns ~ 100ms
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

100ms upper bound is too small here, maybe set the range to 10us~10s is better here.

Signed-off-by: tongjian <[email protected]>
@bufferflies bufferflies force-pushed the metrics branch 2 times, most recently from 2f2cfd2 to cf7cab6 Compare April 28, 2024 09:41
Signed-off-by: tongjian <[email protected]>
@glorv
Copy link
Contributor

glorv commented Apr 29, 2024

@overvenus PTAL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

observability: improve the observability for every cq running task
3 participants