Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: introduce new benchmarking script #121

Merged
merged 44 commits into from
Sep 5, 2023
Merged

Conversation

ielashi
Copy link
Contributor

@ielashi ielashi commented Sep 1, 2023

Problem

We've been relying on criterion as our way of running benchmarks and getting reports. However, criterion has been falling short for a number of reasons:

  1. Benchmarks can only have one measurement. In our case, the measurement was "instructions", but that's not good enough. For our use-case, we'd like to have multiple measurements per benchmark (e.g. instructions, stable memory pages, etc.).
  2. criterion always compares the results to the previous run. In almost all cases, I'm interested in measuring the difference in performance relative to the last commit, and there isn't a good of enforcing that workflow in criteron.

Solution

I decided to spend a few hours to replace our reliance on criterion altogether. We now have our very own benchmarking script that runs the benchmarks. A benchmark can have as many measurements as we want, and it neatly persists these benchmarks to our repo along with a report on how the benchmarks have changed.

Here's an example of the report that it prints out to the terminal:
image

Notes:

  • As you can tell from the screen shot above, I added the "stable_memory_size" as a new measurement.
  • I had to bump up the rust version as clap needed a more rust >= 1.70.

…ction in instructions)

```
Benchmarking btreemap_insert_blob_4_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:10.681469 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "3_636_501_975 (78%)",
    "node_save_v2": "621_725_540 (13%)",
}

btreemap_insert_blob_4_1024_v2
                        time:   [4614.6 M Instructions 4614.6 M Instructions 4614.6 M Instructions]
                        change: [-10.016% -10.016% -10.016%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_insert_blob_8_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:12.381113 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "3_888_128_011 (77%)",
    "node_save_v2": "675_563_074 (13%)",
}

btreemap_insert_blob_8_1024_v2
                        time:   [5002.9 M Instructions 5002.9 M Instructions 5002.9 M Instructions]
                        change: [-10.421% -10.421% -10.421%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_insert_blob_16_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:14.230071 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "3_958_532_867 (77%)",
    "node_save_v2": "709_136_673 (13%)",
}

btreemap_insert_blob_16_1024_v2
                        time:   [5115.5 M Instructions 5115.5 M Instructions 5115.5 M Instructions]
                        change: [-10.015% -10.015% -10.015%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_insert_blob_32_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:16.079308 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "4_095_959_285 (77%)",
    "node_save_v2": "745_609_525 (14%)",
}

btreemap_insert_blob_32_1024_v2
                        time:   [5304.5 M Instructions 5304.5 M Instructions 5304.5 M Instructions]
                        change: [-9.8505% -9.8505% -9.8505%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_insert_blob_64_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:17.950616 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "4_528_292_556 (77%)",
    "node_save_v2": "791_326_122 (13%)",
}

btreemap_insert_blob_64_1024_v2
                        time:   [5824.6 M Instructions 5824.6 M Instructions 5824.6 M Instructions]
                        change: [-10.047% -10.047% -10.047%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_insert_blob_128_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:19.897701 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "4_965_928_945 (78%)",
    "node_save_v2": "838_234_275 (13%)",
}

btreemap_insert_blob_128_1024_v2
                        time:   [6358.7 M Instructions 6358.7 M Instructions 6358.7 M Instructions]
                        change: [-9.9985% -9.9985% -9.9985%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_insert_blob_256_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:21.960733 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "5_411_658_559 (77%)",
    "node_save_v2": "901_162_509 (12%)",
}

btreemap_insert_blob_256_1024_v2
                        time:   [6975.5 M Instructions 6975.5 M Instructions 6975.5 M Instructions]
                        change: [-10.178% -10.178% -10.178%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_insert_blob_512_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:24.150752 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "6_876_355_007 (78%)",
    "node_save_v2": "1_058_462_789 (12%)",
}

btreemap_insert_blob_512_1024_v2
                        time:   [8798.0 M Instructions 8798.0 M Instructions 8798.0 M Instructions]
                        change: [-10.020% -10.020% -10.020%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_insert_u64_u64_v2: Warming up for 1.0000 ms
2023-08-23 08:46:26.530747 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "728_775_816 (58%)",
    "node_save_v2": "258_596_339 (20%)",
}

btreemap_insert_u64_u64_v2
                        time:   [1251.5 M Instructions 1251.5 M Instructions 1251.5 M Instructions]
                        change: [-0.0037% -0.0037% -0.0037%] (p = 0.00 < 0.05)
                        Change within noise threshold.

Benchmarking btreemap_insert_u64_blob_8_v2: Warming up for 1.0000 ms
2023-08-23 08:46:27.652437 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "696_823_942 (58%)",
    "node_save_v2": "247_026_223 (20%)",
}

btreemap_insert_u64_blob_8_v2
                        time:   [1187.4 M Instructions 1187.4 M Instructions 1187.4 M Instructions]
                        change: [-0.0038% -0.0038% -0.0038%] (p = 0.00 < 0.05)
                        Change within noise threshold.

Benchmarking btreemap_insert_blob_8_u64_v2: Warming up for 1.0000 ms
2023-08-23 08:46:28.750840 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "758_879_496 (65%)",
    "node_save_v2": "145_886_287 (12%)",
}

btreemap_insert_blob_8_u64_v2
                        time:   [1153.0 M Instructions 1153.0 M Instructions 1153.0 M Instructions]
                        change: [-0.0037% -0.0037% -0.0037%] (p = 0.00 < 0.05)
                        Change within noise threshold.

Benchmarking btreemap_get_blob_4_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:29.831699 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "4_219_207_858 (95%)",
}

btreemap_get_blob_4_1024_v2
                        time:   [4436.9 M Instructions 4436.9 M Instructions 4436.9 M Instructions]
                        change: [-11.542% -11.542% -11.542%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_get_blob_8_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:32.173729 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "3_815_979_475 (94%)",
}

btreemap_get_blob_8_1024_v2
                        time:   [4044.2 M Instructions 4044.2 M Instructions 4044.2 M Instructions]
                        change: [-11.899% -11.899% -11.899%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_get_blob_16_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:34.595119 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "3_851_724_650 (94%)",
}

btreemap_get_blob_16_1024_v2
                        time:   [4081.1 M Instructions 4081.1 M Instructions 4081.1 M Instructions]
                        change: [-12.053% -12.053% -12.053%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_get_blob_32_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:37.017190 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "4_253_072_804 (94%)",
}

btreemap_get_blob_32_1024_v2
                        time:   [4495.5 M Instructions 4495.5 M Instructions 4495.5 M Instructions]
                        change: [-11.570% -11.570% -11.570%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_get_blob_64_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:39.708213 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "4_589_519_380 (94%)",
}

btreemap_get_blob_64_1024_v2
                        time:   [4849.5 M Instructions 4849.5 M Instructions 4849.5 M Instructions]
                        change: [-11.711% -11.711% -11.711%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_get_blob_128_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:42.364900 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "5_006_841_787 (94%)",
}

btreemap_get_blob_128_1024_v2
                        time:   [5272.4 M Instructions 5272.4 M Instructions 5272.4 M Instructions]
                        change: [-11.815% -11.815% -11.815%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_get_u64_u64_v2: Warming up for 1.0000 ms
2023-08-23 08:46:45.151204 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "768_123_966 (83%)",
}

btreemap_get_u64_u64_v2 time:   [922.10 M Instructions 922.10 M Instructions 922.10 M Instructions]
                        change: [-0.0053% -0.0053% -0.0053%] (p = 0.00 < 0.05)
                        Change within noise threshold.

Benchmarking btreemap_get_u64_blob_8_v2: Warming up for 1.0000 ms
2023-08-23 08:46:46.385316 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "729_106_800 (83%)",
}

btreemap_get_u64_blob_8_v2
                        time:   [874.65 M Instructions 874.65 M Instructions 874.65 M Instructions]
                        change: [-0.0056% -0.0056% -0.0056%] (p = 0.00 < 0.05)
                        Change within noise threshold.

Benchmarking btreemap_get_blob_8_u64_v2: Warming up for 1.0000 ms
2023-08-23 08:46:47.594299 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "811_967_401 (82%)",
}

btreemap_get_blob_8_u64_v2
                        time:   [987.62 M Instructions 987.62 M Instructions 987.62 M Instructions]
                        change: [-0.0049% -0.0049% -0.0049%] (p = 0.00 < 0.05)
                        Change within noise threshold.

Benchmarking btreemap_get_blob_256_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:48.774213 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "5_512_886_751 (94%)",
}

btreemap_get_blob_256_1024_v2
                        time:   [5807.4 M Instructions 5807.4 M Instructions 5807.4 M Instructions]
                        change: [-12.998% -12.998% -12.998%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_get_blob_512_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:51.686246 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "6_988_975_712 (95%)",
}

btreemap_get_blob_512_1024_v2
                        time:   [7330.7 M Instructions 7330.7 M Instructions 7330.7 M Instructions]
                        change: [-12.057% -12.057% -12.057%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_remove_blob_4_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:55.065067 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v1": "306_221_077 (29%)",
    "node_save_v1": "416_552_416 (39%)",
}

btreemap_remove_blob_4_1024_v2
                        time:   [1055.5 M Instructions 1055.5 M Instructions 1055.5 M Instructions]
                        change: [+0.0000% +0.0000% +0.0000%] (p = NaN > 0.05)
                        No change in performance detected.

Benchmarking btreemap_remove_blob_8_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:56.830780 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v1": "365_697_986 (27%)",
    "node_save_v1": "555_195_279 (41%)",
}

btreemap_remove_blob_8_1024_v2
                        time:   [1352.3 M Instructions 1352.3 M Instructions 1352.3 M Instructions]
                        change: [+0.0000% +0.0000% +0.0000%] (p = NaN > 0.05)
                        No change in performance detected.

Benchmarking btreemap_remove_blob_16_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:46:58.660166 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "4_691_318_570 (73%)",
    "node_save_v2": "1_237_620_989 (19%)",
}

btreemap_remove_blob_16_1024_v2
                        time:   [6415.9 M Instructions 6415.9 M Instructions 6415.9 M Instructions]
                        change: [-9.2624% -9.2624% -9.2624%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_remove_blob_32_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:47:01.495518 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "4_889_831_082 (72%)",
    "node_save_v2": "1_311_529_745 (19%)",
}

btreemap_remove_blob_32_1024_v2
                        time:   [6715.3 M Instructions 6715.3 M Instructions 6715.3 M Instructions]
                        change: [-9.0911% -9.0911% -9.0911%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_remove_blob_64_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:47:04.425799 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "5_090_099_125 (72%)",
    "node_save_v2": "1_381_087_495 (19%)",
}

btreemap_remove_blob_64_1024_v2
                        time:   [7031.6 M Instructions 7031.6 M Instructions 7031.6 M Instructions]
                        change: [-9.1148% -9.1148% -9.1148%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_remove_blob_128_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:47:07.509173 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "5_468_360_953 (72%)",
    "node_save_v2": "1_467_914_004 (19%)",
}

btreemap_remove_blob_128_1024_v2
                        time:   [7553.3 M Instructions 7553.3 M Instructions 7553.3 M Instructions]
                        change: [-8.8462% -8.8462% -8.8462%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_remove_blob_256_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:47:10.739864 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "6_380_810_655 (73%)",
    "node_save_v2": "1_601_556_578 (18%)",
}

btreemap_remove_blob_256_1024_v2
                        time:   [8714.6 M Instructions 8714.6 M Instructions 8714.6 M Instructions]
                        change: [-9.1626% -9.1626% -9.1626%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_remove_blob_512_1024_v2: Warming up for 1.0000 ms
2023-08-23 08:47:14.235772 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "7_716_480_801 (72%)",
    "node_save_v2": "1_920_254_272 (18%)",
}

btreemap_remove_blob_512_1024_v2
                        time:   [10.603 B Instructions 10.603 B Instructions 10.603 B Instructions]
                        change: [-9.1730% -9.1730% -9.1730%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking btreemap_remove_u64_u64_v2: Warming up for 1.0000 ms
2023-08-23 08:47:18.341386 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "834_031_761 (52%)",
    "node_save_v2": "427_006_161 (27%)",
}

btreemap_remove_u64_u64_v2
                        time:   [1574.5 M Instructions 1574.5 M Instructions 1574.5 M Instructions]
                        change: [-0.0034% -0.0034% -0.0034%] (p = 0.00 < 0.05)
                        Change within noise threshold.

Benchmarking btreemap_remove_u64_blob_8_v2: Warming up for 1.0000 ms
2023-08-23 08:47:19.699772 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "793_821_373 (52%)",
    "node_save_v2": "404_946_198 (27%)",
}

btreemap_remove_u64_blob_8_v2
                        time:   [1499.6 M Instructions 1499.6 M Instructions 1499.6 M Instructions]
                        change: [-0.0036% -0.0036% -0.0036%] (p = 0.00 < 0.05)
                        Change within noise threshold.

Benchmarking btreemap_remove_blob_8_u64_v2: Warming up for 1.0000 ms
2023-08-23 08:47:21.052537 UTC: [Canister rwlgt-iiaaa-aaaaa-aaaaa-cai] {
    "node_load_v2": "866_070_202 (60%)",
    "node_save_v2": "258_699_270 (18%)",
}

btreemap_remove_blob_8_u64_v2
                        time:   [1427.1 M Instructions 1427.1 M Instructions 1427.1 M Instructions]
                        change: [-0.0036% -0.0036% -0.0036%] (p = 0.00 < 0.05)
                        Change within noise threshold.
```
@ielashi ielashi requested a review from a team as a code owner September 1, 2023 11:49
Cargo.toml Outdated Show resolved Hide resolved
benchmarks/benchmark.rs Show resolved Hide resolved
Base automatically changed from ielashi/v2_page_size to main September 5, 2023 11:38
@ielashi ielashi enabled auto-merge (squash) September 5, 2023 11:42
@ielashi ielashi merged commit ab9a9dd into main Sep 5, 2023
3 checks passed
@ielashi ielashi deleted the ielashi/new_benchmarking branch September 5, 2023 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants