[Bug]: MO does not return result in 60s after run bvt test for about 12 times

[aressu1985]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Environment

- Version or commit-id (e.g. v0.1.0 or 8b23a93):b908fa0cf540b81cac22b82570542a7dace4ab34
- Hardware parameters:
- OS type:Centos 8
- Others: 16core, 61G

Actual Behavior

as the bvt test lasting, the cost of each test turn increased , as following:

and at last , the mo can not return result in 60s ：
[ERROR]
[SCRIPT FILE]: /mnt/datadisk0/actions-runner/_work/mo-nightly-regression/mo-nightly-regression/head/test/distributed/cases/table/system_table_cases.sql
[ROW NUMBER]: 99
[SQL STATEMENT]: SELECT COUNT(0) FROM (SELECT * FROM sys_cpu_combined_percent LIMIT 10) AS temp;
[EXPECT RESULT]:
count(0)
10
[ACTUAL RESULT]:
MO does not return result in 60000 ms.

[ERROR]
[SCRIPT FILE]: /mnt/datadisk0/actions-runner/_work/mo-nightly-regression/mo-nightly-regression/head/test/distributed/cases/table/system_table_cases.sql
[ROW NUMBER]: 100
[SQL STATEMENT]: SELECT COUNT('') FROM (SELECT * FROM sys_cpu_combined_percent LIMIT 10) AS temp;
[EXPECT RESULT]:
count()
10
[ACTUAL RESULT]:
SQL parser error: table "sys_cpu_combined_percent" does not exist

[ERROR]
[SCRIPT FILE]: /mnt/datadisk0/actions-runner/_work/mo-nightly-regression/mo-nightly-regression/head/test/distributed/cases/table/system_table_cases.sql
[ROW NUMBER]: 101
[SQL STATEMENT]: SELECT COUNT(NULL) FROM (SELECT * FROM sys_cpu_combined_percent LIMIT 10) AS temp;
[EXPECT RESULT]:
count(null)
0
[ACTUAL RESULT]:
SQL parser error: table "sys_cpu_combined_percent" does not exist

Expected Behavior

No response

Steps to Reproduce

run bvt test circularly

Additional information

No response

[nnsgmsone]

I'll take a look.

[nnsgmsone]

After testing, I found that there was a very old transaction that never ended, causing cn's gc to never work.

[nnsgmsone]

Look at this issue after the transaction bug has been fixed.

[nnsgmsone]

No progress at the moment

[nnsgmsone]

No progress at the moment

[nnsgmsone]

After pr 7669 fixed some of cn's memory issues, there are still a lot of memory leaks in the system, but I think the test can try to run a daily run first

[aressu1985]

tracking for a more day

[aressu1985]

do not fixed in commitid:ce1aae5e38bfab63d644440fb8113c455f31e975

[nnsgmsone]

no process

[nnsgmsone]

wait for memory leak fix and memtable refactor

[nnsgmsone]

This can be converted to a feature

[aressu1985]

#8327

[nnsgmsone]

等处理完https://github.com/matrixorigin/matrixone/issues/7891在处理这个。

[nnsgmsone]

等处理完https://github.com/matrixorigin/matrixone/issues/7891在处理这个。

[nnsgmsone]

等处理完https://github.com/matrixorigin/matrixone/issues/7891在处理这个。

[nnsgmsone]

等处理完https://github.com/matrixorigin/matrixone/issues/7891在处理这个。

[nnsgmsone]

After running bvt x times, i get the following pprof:

after running bvt x+1 times, i get the following pprof:

[nnsgmsone]

The gc of memtable on cn looks like there are some problems

[nnsgmsone]

这个问题本质是commit越来越慢的问题，等plan重构后处理这个问题

[jensenojs]

跟 #7601 应该是类似的，我一起跟一下，目前相关进度的同步在 #7601 中

[jensenojs]

相关进展可以看步到 #11018 和 #7601

[sukki37]

traced by #7601, close this one