Making Nevergrad's CI parallel

[teytaud]

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Motivation and Context / Related issue

How Has This Been Tested (if it applies)

Checklist

• The documentation is up-to-date with the changes I made.
• I have read the CONTRIBUTING document and completed the CLA (see CLA).
• All tests passed, and additional code has been covered with new tests.

[teytaud]

The idea here is that it is both faster and more economical / ecological, because bugs are found after less tests when the order is not sequential. This is because most errors typically breaks all tests in one of the block of tests, so a randomized (or parallel) order will be more likely to find early. It would be great to have a clever order though (I'm trying to do that, ideally I'd like to do one test per test file first).

[teytaud]

Hum I see that we can go up to 300. Should we use 10 ?

[jrapin]

does it actually go faster?
22min is the standard time I think, so this does not change anything right now

[teytaud]

I guess it's much more complicated than that... we need to split, otherwise CircleCI just runs all tests on all machines, which is not what we want to do :-)

[teytaud]

(it's cool for testing flakiness though...)
i'll see what I can do for this.

[jrapin]

(it's cool for testing flakiness though...) i'll see what I can do for this.

In term of environment it's a disaster though, i'd rather avoid it

[jrapin]

the best option for shortening the CI is to shorten (not remove) the slowest tests. I'm pretty sure we can go below 10min this way in a day or so