-
Notifications
You must be signed in to change notification settings - Fork 99
ci: run benchmarks on every pr and alert if perf is worse #200
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for taking the time to make this PR! This is a great addition. I'm not sure yet why the bench test timed out at 30 minutes. But we should probably timeout the test after 10 minutes.
Makefile
Outdated
@@ -18,6 +18,10 @@ test: | |||
go tool cover -html coverage.out -o coverage.html; \ | |||
exit $$ret) | |||
|
|||
.PHONY: bench | |||
bench: | |||
go test ./... -bench . -benchtime 5s -timeout 0 -run=XXX -cpu 1 -benchmem |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some of the tests bench concurrent access. I don't think we need -run=XXX -cpu 1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 I'll remove -cpu 1
but keep -run=XXX
since that is supposed to ensure that all benchmarks in all folders run
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I've used -run '^$'
for this purpose to match an empty string which is (probably) impossible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know what i'm doing wrong, but i get an error
[21/11/23 4:36:35] ~/GitHub/gubernator (run-benchmarks) $ make bench
go test ./... -bench . -benchtime 5s -timeout 0 -run '^ -benchmem
/bin/sh: -c: line 0: unexpected EOF while looking for matching `''
/bin/sh: -c: line 1: syntax error: unexpected end of file
make: *** [bench] Error 2
[21/11/23 4:36:44] ~/GitHub/gubernator (run-benchmarks) $
@thrawn01 I think either
Each benchmark is configured to run for 5 seconds and the benchmark step in the github action are set to either 15 or 30 minutes |
@miparnisari Could we test the |
@Baliedge i would need to merge this first into |
6bb2816
to
ad1c5f4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, Does this require a GitHub API key in secrets?
Never mind, I just answered my own question 😜 |
Close #199