Added ipv8 walk_interval scaling

[qstokkink]

Fixes #5384

This PR addresses scaling back IPv8's bandwidth and CPU use while the main thread is congested. Concretely, this PR makes the following changes:

• Adds a configuration option for enabling IPv8 auto scaling (enabled by default).
• Adds a configuration option for the IPv8 walk_interval.
• Adds a configuration option for the upper limit the walk_interval may grow to, if scaling is enabled.
• Adds the IPv8Monitor class (and tests), to measure the main thread choking and scale the walk_interval accordingly.
• Adds the IPv8Monitor to the Session IPv8 initialization.
• Updates the IPv8 initialization in Session to use the new ConfigBuilder.
• Updates the IPv8 health debug screen to show the current target walk_interval.
• Updates the IPv8 pointer (fixes a bug where inspecting the debug IPv8 health interferes with this code).

[devos50]

Unfortunately, these tests are still flaky :/

[devos50]

retest this please

[devos50]

I suspect that last one is caused by having multiple test instances running at the same time; I didn't test that thoroughly yet.

[devos50]

retest this please

[qstokkink]

Alright, I'm done with this PR. @devos50 do you want to use this PR to do flaky test hunting?

[devos50]

I'll do that in another PR.

[qstokkink]

@devos50 https://jenkins-ci.tribler.org/job/GH_Tribler_PR_Tests/job/PR_win64_pytest/21/consoleFull

Is that due to [gw5] [ 99%] FAILED src/tribler-core/tribler_core/modules/tunnel/test_full_session/test_tunnel_community.py::test_anon_download ?

[devos50]

Crap, there seems to be a critical bug in flaky: box/flaky#166 😢

[devos50]

These tests are really driving me crazy. At this point, I'm totally fine with disabling them... @qstokkink your thoughts?

[qstokkink]

@devos50 🤷 "If you can't fix it, skip it." Though, of course, it would be better if these tests were fixed.

[devos50]

Well, let me first revert the flaky stuff

[devos50]

Ok, it seems that the pytest-rerunfailures works. I'll use that instead 👍

[qstokkink]

@devos50 cool, tell me when to rebase 👍

[devos50]

Noooooooo pytest-rerunfailures does not play nicely with pytest-timeout (see here) 😢

[qstokkink]

So.. back to skipping?

[qstokkink]

To be fair, if anything, it seems like test_anon_download is now consistently failing 😄

[devos50]

I won't have time to look into it further today. It also seems that getting logging output from pytest-xdist is a hell. I didn't foresee these issues.

For now, I finished #5584 so that should be ready for merge. Please rebase then and try to get the tests passing I guess?

I look at it later.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities (and 0 Security Hotspots to review)
0 Code Smells

No Coverage information
0.5% Duplication

[qstokkink]

The gods of testing have smiled upon me: none of the unrelated tests have failed during this testing run. 🙏

This PR is once again ready for review.

[devos50]

Looks good. Not sure how it would behave in the wild but there's only one way to find out 👍

[qstokkink]

Looks good. Not sure how it would behave in the wild but there's only one way to find out 👍

I have spent quite some hours running this and tweaking it.. but, should worst come to worst, we can disable this by default in the config.