ci: run benchmarks on every pr and alert if perf is worse

[miparnisari]

Close #199

[thrawn01]

Thank you for taking the time to make this PR! This is a great addition. I'm not sure yet why the bench test timed out at 30 minutes. But we should probably timeout the test after 10 minutes.

[thrawn01]

Some of the tests bench concurrent access. I don't think we need -run=XXX -cpu 1

[miparnisari]

👍 I'll remove -cpu 1 but keep -run=XXX since that is supposed to ensure that all benchmarks in all folders run

[Baliedge]

nit: I've used -run '^$' for this purpose to match an empty string which is (probably) impossible.

[miparnisari]

I don't know what i'm doing wrong, but i get an error

[21/11/23 4:36:35] ~/GitHub/gubernator (run-benchmarks) $ make bench
go test ./... -bench . -benchtime 5s -timeout 0 -run '^ -benchmem
/bin/sh: -c: line 0: unexpected EOF while looking for matching `''
/bin/sh: -c: line 1: syntax error: unexpected end of file
make: *** [bench] Error 2
[21/11/23 4:36:44] ~/GitHub/gubernator (run-benchmarks) $ 

[miparnisari]

I'm not sure yet why the bench test timed out at 30 minutes.

@thrawn01 I think either BenchmarkLRUCache/Concurrent_writes or https://github.com/miparnisari/gubernator/blob/61e6cd7bd0e2d876431b8d085c5d0226ea16a894/lrucache_test.go#L551 timed out. Probably because it executed all the reads serially? 🤷‍♀️

But we should probably timeout the test after 10 minutes.

Each benchmark is configured to run for 5 seconds and the benchmark step in the github action are set to either 15 or 30 minutes

[Baliedge]

@miparnisari Could we test the go-bench check to show it fails when exceeding the threshold? Maybe make a temporary change to add sleeps in the benchmarks?

[miparnisari]

@Baliedge i would need to merge this first into master so that I have something to compare with :) But i can do that in a different PR.

[thrawn01]

Looks good to me, Does this require a GitHub API key in secrets?

[thrawn01]

Never mind, I just answered my own question 😜