[Bug]:[1105 main tke regression] tpcc test report ' panic runtime error: invalid memory address or nil pointer dereference' #19789

Ariznawlll · 2024-11-05T02:21:11Z

Is there an existing issue for the same bug?

I have checked the existing issues.

Branch Name

main

Commit ID

5e4d063

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/11666410553/job/32491369265

先报错了个孤儿事务，后面报错panic，不确定是否有联系，需定位

2024-11-04 18:15:25 INFO jTPCC:805 - Term-00, Term-00, Running Average tpmTOTAL: 45608.73 Current tpmTOTAL: 6823332 Memory Usage: 59MB / 167MB
2024-11-04 18:15:25 INFO jTPCC:805 - Term-00, Term-00, Running Average tpmTOTAL: 45608.67 Current tpmTOTAL: 6823308 Memory Usage: 59MB / 167MB
2024-11-04 18:15:26 INFO jTPCC:805 - Term-00, Term-00, Running Average tpmTOTAL: 45573.60 Current tpmTOTAL: 6823320 Memory Usage: 60MB / 167MB
2024-11-04 18:15:27 FATAL jTPCCTerminal:328 - [UNEXPECTED][TT_NEW_ORDER][EXECUTION] ErrorCode : 20705, ErrorMessage : cannot commit a orphan transaction
2024-11-04 18:15:27 INFO jTPCC:805 - Term-00, Term-00, Running Average tpmTOTAL: 45536.11 Current tpmTOTAL: 6823320 Memory Usage: 61MB / 167MB
2024-11-04 18:15:27 FATAL jTPCCTerminal:214 - [UNEXPECTED][TT_PAYMENT][EXECUTION] ErrorCode : 1105, ErrorMessage : internal error: panic runtime error: invalid memory address or nil pointer dereference:
runtime.panicmem
/usr/local/go/src/runtime/panic.go:262
runtime.sigpanic
/usr/local/go/src/runtime/signal_unix.go:900
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*deletedBlocks).clean
/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/types.go:369
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*Transaction).compactionBlksLocked.func1
/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/txn.go:1025
runtime.gopanic
/usr/local/go/src/runtime/panic.go:785
runtime.panicmem
/usr/local/go/src/runtime/panic.go:262
runtime.sigpanic
/usr/local/go/src/runtime/signal_unix.go:900
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*deletedBlocks).iter
/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/types.go:375
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*Transaction).compactionBlksLocked
/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/txn.go:1027
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*Transaction).mergeTxnWorkspaceLocked
/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/txn.go:1016
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*Transaction).IncrStatementID
/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/types.go:519
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt.func1
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2583
github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).ExecuteFuncWithRecover
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_buffer.go:840
github.com/matrixorigin/matrixone/pkg/frontend.init.func12
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_buffer.go:824
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2582
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2555
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithTxn
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2447
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithResponse
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2411
github.com/matrixorigin/matrixone/pkg/frontend.doComQuery
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:3041
github.com/matrixorigin/matrixone/pkg/frontend.ExecRequest
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:3177
github.com/matrixorigin/matrixone/pkg/frontend.(*Routine).handleRequest
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/routine.go:287
github.com/matrixorigin/matrixone/pkg/frontend.(*RoutineManager).Handler
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/routine_manager.go:385
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleRequest
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/server.go:515
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleMessage
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/server.go:483
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleLoop
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/server.go:212
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleConn
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/server.go:208
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1700
2024-11-04 18:15:27 FATAL jTPCCTerminal:214 - [UNEXPECTED][TT_PAYMENT][EXECUTION] ErrorCode : 1105, ErrorMessage : internal error: panic runtime error: invalid memory address or nil pointer dereference:
runtime.panicmem
/usr/local/go/src/runtime/panic.go:262
runtime.sigpanic
/usr/local/go/src/runtime/signal_unix.go:900
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*deletedBlocks).clean
/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/types.go:369
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*Transaction).compactionBlksLocked.func1
/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/txn.go:1025
runtime.gopanic
/usr/local/go/src/runtime/panic.go:785
runtime.panicmem
/usr/local/go/src/runtime/panic.go:262
runtime.sigpanic
/usr/local/go/src/runtime/signal_unix.go:900
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*deletedBlocks).iter
/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/types.go:375
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*Transaction).compactionBlksLocked
/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/txn.go:1027
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*Transaction).mergeTxnWorkspaceLocked
/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/txn.go:1016
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*Transaction).IncrStatementID
/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/types.go:519
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt.func1
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2583
github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).ExecuteFuncWithRecover
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_buffer.go:840
github.com/matrixorigin/matrixone/pkg/frontend.init.func12
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_buffer.go:824
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2582
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2555
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithTxn
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2447
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithResponse
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2411
github.com/matrixorigin/matrixone/pkg/frontend.doComQuery
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:3041
github.com/matrixorigin/matrixone/pkg/frontend.ExecRequest
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:3177
github.com/matrixorigin/matrixone/pkg/frontend.(*Routine).handleRequest
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/routine.go:287
github.com/matrixorigin/matrixone/pkg/frontend.(*RoutineManager).Handler
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/routine_manager.go:385
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleRequest
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/server.go:515
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleMessage
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/server.go:483
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleLoop
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/server.go:212
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleConn
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/server.go:208
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1700

tpcc100-100-----tpcc500-1000测试期间panic日志（UTC时间）
log：https://grafana.ci.matrixorigin.cn/explore?panes=%7B%221u3%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-main-nightly-5e4d0638a-20241104%5C%22%7D%20%7C%3D%20%60panic%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221730744117000%22,%22to%22:%221730749177000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

trigger daily test workflow on tke

Additional information

No response

The text was updated successfully, but these errors were encountered:

jensenojs · 2024-11-05T02:34:14Z

表象是deletedBlocks在调用clean时已经为nil引发的panic, 请飞哥帮忙看一下 @triump2020 . 孤儿事务不知道跟这个panic相关与否, 需要再看

triump2020 · 2024-11-05T10:24:33Z

初步原因： txn1 commit 出现error , 然后rollback , 将txn.deletedBlocks = nil ；启动txn2， txn2不清楚为啥复用了txn1 的workspace, 然后执行 IncrStatementID, 出现了panic

triump2020 · 2024-11-05T13:49:03Z

@zhangxu19830126

zhangxu19830126 · 2024-11-06T01:09:30Z

fixed

Ariznawlll · 2024-11-06T02:06:11Z

今天测试观察一下

Ariznawlll · 2024-11-07T02:25:42Z

main分支冻结了，修复pr没进去，今天还有这个问题

Ariznawlll added kind/bug Something isn't working needs-triage severity/s-1 labels Nov 5, 2024

Ariznawlll added this to the 2.0.1 milestone Nov 5, 2024

Ariznawlll assigned matrix-meow Nov 5, 2024

jensenojs assigned triump2020 and unassigned matrix-meow Nov 5, 2024

matrix-meow removed the needs-triage label Nov 5, 2024

daviszhen mentioned this issue Nov 5, 2024

Revert "reuse txnop (#19776)" #19814

Closed

7 tasks

triump2020 assigned zhangxu19830126 and unassigned triump2020 Nov 5, 2024

This was referenced Nov 6, 2024

fix txn status invalid #19816

Merged

fix txn status invalid to 2.0 #19817

Merged

zhangxu19830126 assigned Ariznawlll and unassigned zhangxu19830126 Nov 6, 2024

matrix-meow added the phase/testing label Nov 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]:[1105 main tke regression] tpcc test report ' panic runtime error: invalid memory address or nil pointer dereference' #19789

[Bug]:[1105 main tke regression] tpcc test report ' panic runtime error: invalid memory address or nil pointer dereference' #19789

Ariznawlll commented Nov 5, 2024

jensenojs commented Nov 5, 2024

triump2020 commented Nov 5, 2024

triump2020 commented Nov 5, 2024

zhangxu19830126 commented Nov 6, 2024

Ariznawlll commented Nov 6, 2024

Ariznawlll commented Nov 7, 2024

[Bug]:[1105 main tke regression] tpcc test report ' panic runtime error: invalid memory address or nil pointer dereference' #19789

[Bug]:[1105 main tke regression] tpcc test report ' panic runtime error: invalid memory address or nil pointer dereference' #19789

Comments

Ariznawlll commented Nov 5, 2024

Is there an existing issue for the same bug?

Branch Name

Commit ID

Other Environment Information

Actual Behavior

Expected Behavior

Steps to Reproduce

Additional information

jensenojs commented Nov 5, 2024

triump2020 commented Nov 5, 2024

triump2020 commented Nov 5, 2024

zhangxu19830126 commented Nov 6, 2024

Ariznawlll commented Nov 6, 2024

Ariznawlll commented Nov 7, 2024