Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: perturbation/metamorphic/backfill failed #133374

Open
cockroach-teamcity opened this issue Oct 24, 2024 · 1 comment · May be fixed by #133393
Open

roachtest: perturbation/metamorphic/backfill failed #133374

cockroach-teamcity opened this issue Oct 24, 2024 · 1 comment · May be fixed by #133393
Assignees
@andrewbaptist
Labels
A-testing Testing tools and infrastructure B-runtime-assertions-enabled branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 24, 2024 •
edited by cockroach-jira-scripts
Loading

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.perturbation/metamorphic/backfill failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

(cluster.go:2449).Run: full command output in run_172538.337172712_n31-32_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/perturbation/metamorphic/backfill/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=2
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-43561
@cockroach-teamcity cockroach-teamcity added B-runtime-assertions-enabled branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Oct 24, 2024
@miraradeva
Copy link
Contributor

The test failed due to a serialization error:

Error: ERROR: result is ambiguous: error=ba: Put [/Table/106/1/7889463034662031696/0], EndTxn(parallel commit) [/Table/106/1/7889463034662031696/0], [txn: 9266c39a], [can-forward-ts] RPC error: grpc: error reading from server: read tcp 10.142.1.76:53622->10.142.1.91:26257: use of closed network connection [code 14/Unavailable] [propagate] (last error: TransactionRetryError: retry txn (RETRY_SERIALIZABLE): "sql txn" meta={id=9266c39a key=/Table/106/1/7889463034662031696/0 iso=Serializable pri=0.02816496 epo=0 ts=1729790905.319193092,2 min=1729790902.078664092,0 seq=2} lock=true stat=PENDING rts=1729790902.078664092,0 wto=false gul=1729790902.578664092,0) (SQLSTATE 40003)

This failed right after running:

2024/10/24 17:25:05 admission_control_latency.go:341: waiting for replicas to be in place
2024/10/24 17:25:38 cluster.go:2469: running cmd `./cockroach workload run kv...` on nodes [:31-32]

That workload command is:

"./cockroach workload run kv --db backfill --duration=%s --max-block-bytes=%d --min-block-bytes=%d --concurrency=100 {pgurl%s}"

@andrewbaptist shouldn't this include --tolerate-errors?

miraradeva added a commit to miraradeva/cockroach that referenced this issue Oct 24, 2024
@miraradeva
roachtest: deflake perturbation/metamorphic/backfill
be485b2 
The test runs a `cockroach workload run kv --db backfill ...` workload
to set the test up. It can fail if it encounters a transaction
retryable error.

This commit adds the `--tolerate-errors` flag to the workload command.

Fixes: cockroachdb#133374

Release note: None
@miraradeva miraradeva linked a pull request Oct 24, 2024 that will close this issue
Open
@miraradeva miraradeva added A-testing Testing tools and infrastructure P-3 Issues/test failures with no fix SLA and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Oct 24, 2024
@kvoli kvoli mentioned this issue Oct 25, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andrewbaptist andrewbaptist

Labels
A-testing Testing tools and infrastructure B-runtime-assertions-enabled branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

roachtest: deflake perturbation/metamorphic/backfill miraradeva/cockroach
3 participants
@cockroach-teamcity @andrewbaptist @miraradeva