Skip to content

Commit

Permalink
automatically rescale exponential histograms
Browse files Browse the repository at this point in the history
add desired_scale configuration. You can set something like `8` to
try and get fine-grained resolution when your distributions have
close values. When your distributions span a wide range, they will
automatically degrade to the finest resolution that can express them
in exponential histogram format.

The ranges are offset-aware, so if you limit to 160 buckets, you will
still be able to have relatively fine-grained resolution of large numbers
assuming those large numbers are relatively close together.

Scaling down to coarser scales can happen repeatedly or all at once.
The scale workflow is modeled as a single step in-place rescale, followed
by a reingest of the new value. If the new value still doesn't fit,
it reingests again. Thus, the algorithm recursively hadles multi-step
scaling.

Performance is unchanged within the granularity I can measure with
criterion. This should just increase fidelity of histograms for normally
distributed data, like most metrics are expected to approximately be.
  • Loading branch information
kvc0 committed Mar 5, 2024
1 parent 9eaec08 commit ddda202
Show file tree
Hide file tree
Showing 5 changed files with 452 additions and 185 deletions.
1 change: 1 addition & 0 deletions lib/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ ordered-float = { version = "4.1" }

[dev-dependencies]
env_logger = { version = "0.10" }
rand = { version = "0.8" }
test-log = { version = "0.2" }
tokio = { version = "1.33", features = ["rt-multi-thread"]}
tokio-test = { version = "0.4" }
Expand Down
5 changes: 4 additions & 1 deletion lib/benches/benchmarks/aggregation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,10 @@ pub fn aggregation(criterion: &mut Criterion) {
bench_distribution_mode(&mut group, DistributionMode::Histogram);
bench_distribution_mode(
&mut group,
DistributionMode::ExponentialHistogram { max_buckets: 160 },
DistributionMode::ExponentialHistogram {
max_buckets: 160,
desired_scale: 2,
},
);
bench_distribution_mode(&mut group, DistributionMode::TDigest);
}
Expand Down
6 changes: 2 additions & 4 deletions lib/src/downstream/opentelemetry_downstream.rs
Original file line number Diff line number Diff line change
Expand Up @@ -404,17 +404,15 @@ fn as_otel_exponential_histogram(
scale: exponential_histogram.scale() as i32,
zero_count: 0, // I don't do this yet
positive: Some(Buckets {
// TODO: decide what to do about dynamic Scale scaling
offset: 0,
offset: exponential_histogram.bucket_start_offset() as i32,
bucket_counts: exponential_histogram
.take_positives()
.into_iter()
.map(|u| u as u64)
.collect(),
}),
negative: Some(Buckets {
// TODO: decide what to do about dynamic Scale scaling
offset: 0,
offset: exponential_histogram.bucket_start_offset() as i32,
bucket_counts: exponential_histogram
.take_negatives()
.into_iter()
Expand Down
Loading

0 comments on commit ddda202

Please sign in to comment.