[Bug]: chaos test tpcc verify failed

[tom-csf]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

main

Commit ID

d354fe6

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

Chaos test, every 5min kill one log, tpcc test is continues ,after 1.5h tpcc verify failed


dn pod restart ,too

https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22GYP%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-chaos-d354fe6-202410110956%5C%22%7D%20%7C%3D%20%60ERROR%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-12h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

https://github.com/matrixorigin/mo-nightly-regression/actions/runs/11284918759/job/31386877122

Additional information

No response

[LeftHandCold]

未投入

[LeftHandCold]

未投入

[aressu1985]

暂时复现不了，也没有定位手段，考虑DEALY到下个版本，继续观察

[LeftHandCold]

可能是操作引起的failed，后面复现过一次因为tpcc 任务没有关闭，但是tpcc数据重新load了，并且create table和load不是一个txn

[LeftHandCold]

tpcc 任务运行期间，出现了从新create table和load data的情况

[LeftHandCold]

本地使用launch-with-proxy启动5个进程，运行tpcc，kill -9 tn再重启tn，可以复现这个问题
debug后结论如下：一个事务中有update两个表的操作，提交到tn这里，只有一个表的update的操作，所以一致性出现问题。
后续在cn中添加log，发现run的时候执行了两个update操作，但是写入workspace只有一个表的数据。

[tom-csf]

在chaos环境构建kill DN+ tpcc 测试场景复现

https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22O2c%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-chaos-2212882-202410311802%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221730368972000%22,%22to%22:%221730371972000%22%7D%7D%7D&schemaVersion=1&orgId=1

[XuPeng-SH]

As discussed with @ouyuanning offline, @ouyuanning will investigate this issue later.

[volgariver6]

fixed, please test again.

[tom-csf]

验证 ok