Benchmarking

[dogonthehorizon]

No description provided.

[IamfromSpace]

An obvious performance overhead comparison is https://github.com/awslabs/aws-lambda-rust-runtime as they have the same HTTP runtime interfacing constraints. Rust is a lower level language and should perform better, but we'd like to minimize this gap.

In over 10k events configured at 128MB, split between a "Hello, World" Rust lambda and a "Hello, World" Haskell lambda there are a couple of findings:

The data set is very noisy.

Haskell warm starts have outliers early in their container's life.

Visually, it seems obvious that both Rust and Haskell lambdas can end up running in "fast" or "slow" containers. Even with n > 10k and outliers removed, we only see a trend p < 0.21 that these two results are statistically different.

If we assume the trend to be correct, comparing fast/fast and slow/slow containers between Rust and Haskell, we see the following results:

Haskell (fast container): 11.2ms +/- 8.1
Rust (fast container): 7.8 +/- 6.8

Haskell (slow container): 27.6ms +/- 18.5
Rust (slow container): 22.5 +/- 17.0

The difference seems to be about 20% in both cases, with similar standard deviation (though with such large standard deviation, statistically we don't have much confidence in their difference).

It still seems that more exploration is warranted, especially at higher CPU allowance. It seems unlikely to be able to determine the likelihood of hitting a fast/slow container, based on the difficulty to collect the data. Seems even harder to make any reasonable claim that one language is more likely to fall into one versus another.