✨Add benchmarks

[cowboyd]

Motivation

In order to maintain performance guarantees in Effection we need to have a standard by which to measure. It will not only help us improve performance, but also help to make sure we don't regress any gains that we've made.

Approach

This adapts the async-benchmark suite developed by @rixtox for inclusion directly into the effection repository. While the content of the benchmark examples is mostly the same, it does not measure memory usage (yet). Also, instead of using a spawned process in a separate file it uses a Worker. This makes it easy to cancel if you want to ctrl-c for any reason.

We've wrapped Deno and a CLI around it:

% deno task bench --help
Task bench deno run -A tasks/bench.ts "--help"
Usage: bench [options]

Run Effection benchmarks

Options:
  -h, --help              Show help
  -V, --version           Show version
      --include <string>  include only scenarios matching REGEXP
      --exclude <string>  exclude all scenanios matching REGEXP
  -n, --repeat <number>   number of times to repeat (default: 10)
  -d, --depth <number>    number of levels of recursion to run (default: 100)

To run with a recursive depth of 500, do the following:

% deno task bench --depth 500
Task bench deno run -A tasks/bench.ts "--depth" "500"
┌──────────────────────────────────────────┐
│ Basic Recursion                          │
├────────────────────┬─────────────────────┤
│ Library            │ Avg (ms)            │
└────────────────────┴─────────────────────┘
  effection            1.956070900000001

  rxjs                 0.7305832999999993

  co                   0.4338708999999998

  async+await          0.16547500000000018
┌──────────────────────────────────────────┐
│ Recursive Events                         │
├────────────────────┬─────────────────────┤
│ Library            │ Avg (ms)            │
└────────────────────┴─────────────────────┘
  effection            489.69844580000006

  rxjs                 135.5726289

  add-event-listener   13.211841899999996

You can adjust the number of times that it repeats and the depth of the recursion. Each scenario is named by a file, and you can filter with the --include and --exclude regexp flags. For example, to include only the effection and rxjs, you can say:

% deno task bench --include '(effection|rxjs)'
Task bench deno run -A tasks/bench.ts "--include" "(effection|rxjs)"
┌─────────────────────────────────┐
│ Basic Recursion                 │
├───────────┬─────────────────────┤
│ Library   │ Avg (ms)            │
└───────────┴─────────────────────┘
  effection   0.7081123999999995

  rxjs        0.37000859999999847
┌─────────────────────────────────┐
│ Recursive Events                │
├───────────┬─────────────────────┤
│ Library   │ Avg (ms)            │
└───────────┴─────────────────────┘
  effection   158.8061377

  rxjs        120.2107459

run everything but the rxjs recursion scenario

% deno task bench --exclude rxjs.recursion
Task bench deno run -A tasks/bench.ts "--exclude" "rxjs.recursion"
┌──────────────────────────────────────────┐
│ Basic Recursion                          │
├────────────────────┬─────────────────────┤
│ Library            │ Avg (ms)            │
└────────────────────┴─────────────────────┘
  effection            0.7624042000000006

  co                   0.18677059999999948

  async+await          0.09880399999999981
┌──────────────────────────────────────────┐
│ Recursive Events                         │
├────────────────────┬─────────────────────┤
│ Library            │ Avg (ms)            │
└────────────────────┴─────────────────────┘
  effection            158.6162375

  rxjs                 120.17651679999999

  add-event-listener   3.8213539999999995

[taras]

🔥

[neurosnap]

This is awesome!

[cowboyd]

Preliminary results are encouraging. When I implemented iterator hoisting, Effection out performed both rxjs and co, for very deeply nested recursion:

% deno task bench --include recursion --depth 750
┌───────────────────────────────────────┐
│ Basic Recursion                       │
├─────────────────┬─────────────────────┤
│ Library         │ Avg (ms)            │
└─────────────────┴─────────────────────┘
  effection         2.7688957999999997

  effection+hoist   0.520741799999999

  rxjs              0.8812332000000012

  co                0.6309749999999991

  async+await       0.17363750000000042

Interestingly, rxjs is not capable of going deeper than 750 because it bombs out with a stack overflow.