Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skipping flaky tests #1848

Merged
merged 5 commits into from
Feb 18, 2022
Merged

Skipping flaky tests #1848

merged 5 commits into from
Feb 18, 2022

Conversation

hbandura
Copy link
Contributor

@hbandura hbandura commented Feb 17, 2022

This PR comments/skips tests, builds and benchmarks that are known to be flaky.

Unit tests:

Benchmarks:

Circleci builds:

While creating this PR, the istanbul-e2e-coverage failed once, then worked when it was re-run. I'm not skipping this one right now since I'd like to test it a bit more. Most likely it is because the parent commit has flaky tests, which are run for coverage report.

codecov/project is failing because we are removing a few tests. It is expected.

@hbandura hbandura requested a review from a team as a code owner February 17, 2022 22:42
@hbandura hbandura requested review from gastonponti and 37ng and removed request for a team February 17, 2022 22:42
@piersy
Copy link
Contributor

piersy commented Feb 17, 2022

Coverage from tests in ./e2e_test/... for ./consensus/istanbul/... at commit 6033f85

coverage: 51.7% of statements across all listed packages
coverage:  65.6% of statements in consensus/istanbul
coverage:  41.8% of statements in consensus/istanbul/announce
coverage:  55.4% of statements in consensus/istanbul/backend
coverage:   0.0% of statements in consensus/istanbul/backend/backendtest
coverage:  24.3% of statements in consensus/istanbul/backend/internal/replica
coverage:  71.7% of statements in consensus/istanbul/core
coverage:  45.0% of statements in consensus/istanbul/db
coverage:   0.0% of statements in consensus/istanbul/proxy
coverage:  75.3% of statements in consensus/istanbul/uptime
coverage: 100.0% of statements in consensus/istanbul/uptime/store
coverage:  51.8% of statements in consensus/istanbul/validator
coverage:  79.2% of statements in consensus/istanbul/validator/random
CommentID: 5dd1388110

@codecov
Copy link

codecov bot commented Feb 17, 2022

Codecov Report

Merging #1848 (6033f85) into master (22e80e7) will decrease coverage by 0.09%.
The diff coverage is 64.49%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1848      +/-   ##
==========================================
- Coverage   54.28%   54.18%   -0.10%     
==========================================
  Files         671      673       +2     
  Lines       88964    88989      +25     
==========================================
- Hits        48296    48222      -74     
- Misses      37012    37129     +117     
+ Partials     3656     3638      -18     
Impacted Files Coverage Δ
consensus/istanbul/announce/manager.go 0.00% <0.00%> (ø)
consensus/istanbul/announce/util.go 0.00% <0.00%> (ø)
consensus/istanbul/types.go 13.10% <0.00%> (-0.21%) ⬇️
miner/block.go 54.30% <0.00%> (-2.13%) ⬇️
params/version.go 25.71% <ø> (ø)
consensus/istanbul/backend/backend.go 53.13% <50.00%> (-0.54%) ⬇️
consensus/istanbul/backend/engine.go 63.89% <77.77%> (+0.02%) ⬆️
core/tx_list.go 80.83% <80.00%> (+4.30%) ⬆️
core/sys_context.go 78.04% <83.33%> (+11.38%) ⬆️
core/tx_multicurrency_priceheap.go 96.77% <96.77%> (ø)
... and 28 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 300e836...6033f85. Read the comment docs.

@piersy
Copy link
Contributor

piersy commented Feb 18, 2022

@hbandura Looks good, can you make the associated tickets before we merge this?

Copy link
Contributor

@gastonponti gastonponti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mergify mergify bot merged commit c78a51f into master Feb 18, 2022
@mergify mergify bot deleted the hbandura/flakyness branch February 18, 2022 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants