-
Notifications
You must be signed in to change notification settings - Fork 95
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
added readme, minor changes, update benchmarks script
- Loading branch information
1 parent
b82fae5
commit e527a78
Showing
7 changed files
with
134 additions
and
28 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,83 @@ | ||
# Degradation Tester | ||
|
||
#### Continuous Benchmarking | ||
|
||
Degradation Tester is a tool for detecting performance degradation in Go projects. It allows comparing current benchmark results against previous ones to identify potential performance issues. | ||
|
||
#### Approving Performance Changes | ||
|
||
If you've made changes that affect performance, whether improvements or degradations, it's essential to regenerate the benchmarks and replace the existing files in the `benchmarks` directory. This ensures that future comparisons accurately reflect the impact of your changes and maintains the integrity of the performance testing process. | ||
|
||
To regenerate and update the benchmarks, use the `update-benchmarks`: | ||
|
||
```bash | ||
make update-benchmarks | ||
``` | ||
|
||
## Configuration | ||
|
||
To use the tool, you need to set up a YAML configuration file. Here is an example configuration: | ||
|
||
```yaml | ||
DefaultOpDelta: 4.0 | ||
DefaultAllocDelta: 0 | ||
Packages: | ||
- Path: "./message/validation" | ||
Tests: | ||
- Name: "VerifyRSASignature" | ||
OpDelta: 3.0 | ||
- Path: "./protocol/v2/types" | ||
Tests: | ||
- Name: "VerifyBLS" | ||
OpDelta: 6.0 | ||
- Name: "VerifyPKCS1v15" | ||
OpDelta: 4.0 | ||
- Name: "VerifyPKCS1v15FastHash" | ||
OpDelta: 6.0 | ||
- Name: "VerifyPSS" | ||
OpDelta: 5.0 | ||
``` | ||
- `DefaultOpDelta` and `DefaultAllocDelta` specify the default performance change thresholds (in percentages) that are allowed for all tests unless specific values are provided. | ||
- `Path` refers to the directory containing the tests. | ||
- `Tests` is a list of tests with names and permissible performance change thresholds. | ||
|
||
## Local Usage | ||
|
||
### Installation | ||
|
||
Install the benchmarks degradation checker tool: | ||
|
||
`cd scripts/degradation-tester/ && go install .` | ||
|
||
### Running Degradation Tests | ||
|
||
To run degradation tests, execute: | ||
|
||
```bash | ||
make degradation-test | ||
``` | ||
|
||
### Updating Benchmarks | ||
|
||
To update benchmarks, execute: | ||
|
||
```bash | ||
make update-benchmarks | ||
``` | ||
|
||
## Usage with GitHub Actions | ||
|
||
`.github/workflows/degradation-test.yml` containing a workflow for automatically running degradation tests as part of the CI/CD process | ||
|
||
### Scripts | ||
|
||
The `./scripts/degradation-tester` directory contains the following scripts: | ||
|
||
- `degradation-check.sh` - running degradation tests. Runs benchmarks, saves the new results, compares benchmarking results per all listed packages in `config.yaml` | ||
|
||
- `update-benchmarks.sh` - A script for updating benchmarks | ||
|
||
### Dependencies | ||
|
||
> Note: Ensure yq is installed in your system for proper YAML config parsing. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,17 +1,17 @@ | ||
DefaultOpDelta: 4.0 | ||
DefaultAllocDelta: 0 | ||
Tests: | ||
- PackagePath: "./message/validation" | ||
TestCases: | ||
Packages: | ||
- Path: "./message/validation" | ||
Tests: | ||
- Name: "VerifyRSASignature" | ||
OpDelta: 3.0 | ||
- PackagePath: "./protocol/v2/types" | ||
TestCases: | ||
OpDelta: 10 | ||
- Path: "./protocol/v2/types" | ||
Tests: | ||
- Name: "VerifyBLS" | ||
OpDelta: 6.0 | ||
- Name: "VerifyPKCS1v15" | ||
OpDelta: 3.0 | ||
OpDelta: 4.0 | ||
- Name: "VerifyPKCS1v15FastHash" | ||
OpDelta: 6.0 | ||
- Name: "VerifyPSS" | ||
OpDelta: 4.0 | ||
OpDelta: 5.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
#!/bin/bash | ||
|
||
prefix="./scripts/degradation-tester" | ||
configFile="$prefix/config.yaml" | ||
benchmarksResults="$prefix/benchmarks" | ||
|
||
packagePaths=($(yq e '.Packages[].Path' $configFile)) | ||
|
||
for pkgPath in "${packagePaths[@]}"; do | ||
packageName=$(basename "$pkgPath") | ||
benchmarksPath="${benchmarksResults}/${packageName}_results_old.txt" | ||
|
||
echo "Updating benchmarks for ${packageName}" | ||
|
||
go test -bench=. -count=10 -benchmem "$pkgPath" | tee "$benchmarksPath" | ||
|
||
echo "✅ Benchmarks updated for ${packageName} package." | ||
done |