[Tech Request]: Umbrella issue of memory optimization

[sukki37]

Is there an existing issue for enhancement?

• I have checked the existing issues.

What would you like to be added?

We experienced many out-of-memory errors, leading to system instability. Memory management needs optimization for preventing OOM scenarios and for better control over memory usage. This issue is a summary of related bug and task issues:

• [Bug]: index of sysbench oltp & tpcc cases are very unstable on distributed #10509
• [Bug]: cache anomalies #10945
• [Bug]: Run tpch10 sql cn oom(aliyun) #11225
• [Bug]:run TPCH100 on eks RC with 1CN: Can't connect to MySQL server on 'ip:6001' #11221
• [Bug]: 3c4.5g Resource Execution tpcc causes CN OOM. #10659
• [Bug]: mo killed by oom during stability test on standlone mode #10212
• [Bug]: lost connect when run SSB 100G Q4.3 no filter query  #7505
• [Enhancement]: Optimizing garbage collection #9008
• [Tech Request]: Implement Efficient Session-Level Memory Control  #11509
• https://github.com/matrixorigin/MO-Cloud/issues/984
• https://github.com/matrixorigin/MO-Cloud/issues/952
• https://github.com/matrixorigin/MO-Cloud/issues/952
• [Bug]: cache anomalies #10945
• [Bug]: cn oom when test create account and alter account, retry many times #12204
• [Bug]: mo cloud environment cn instance occurs oom #12193
• https://github.com/matrixorigin/MO-Cloud/issues/1584
• [Subtask]: reduce shuffle memory allocation #11529

Dependencies:

• [Subtask]: logtail protocol changes  #10370
• [Tech Request]: Enhance hakeeper to Save Node Resource Information #11511
• [Tech Request]: mo_tables, mo_columns and mo_databases need to be persisted #11650
• [Tech Request]: A buffer for allocating fractional objects #11651
• [Tech Request]: Need a module for getting resource information for the cluster #11652
• [Enhancement]: Hash join/Hash agg spill #3433
• [Tech Request]: Quantitative cache implementation #11658
• [Tech Request]: optimize hash build by not allocate sels #11865
• [Tech Request]: optimize hash build by not merging input batches #11578
• [Bug]: mo killed by oom during stability test on standlone mode #10212

Why is this needed?

No response

Additional information

No response

[nnsgmsone]

处理daily run的问题中。。还未处理

[nnsgmsone]

讨论data race以及pipeline重构的一些事情，尚未处理

[nnsgmsone]

编写pipeline重构文档中，还未处理。

[nnsgmsone]

编写pipeline重构文档中，还未处理。

[nnsgmsone]

思考pipeline重构的设计中，还未处理

[nnsgmsone]

继续整理pipeline的文档中，还未处理

[nnsgmsone]

明天提交第一个pr，用于管理session的内存

[nnsgmsone]

session buffer的pr已经合入

[nnsgmsone]

先处理mpool的bug

[nnsgmsone]

先处理多session insert报错和daily run的问题

[nnsgmsone]

写软著中。。cache已经实现了一个，不过还没合入mo

[nnsgmsone]

在处理s-1。。。还未处理

[nnsgmsone]

在处理s-1。。。还未处理

[nnsgmsone]

开始编写代码，重新实现新的cache

[nnsgmsone]

内存问题等待#12532

[nnsgmsone]

内存问题等待#12532

[nnsgmsone]

内存问题等待#12532

[nnsgmsone]

no process

[nnsgmsone]

pr等待测试

[nnsgmsone]

pr等待测试和合入

[nnsgmsone]

等待pr合ru

[nnsgmsone]

等待pr和入

[nnsgmsone]

处理事务泄露中

[nnsgmsone]

定位问题中

[nnsgmsone]

no process

[nnsgmsone]

等待pr合入

[nnsgmsone]

cache pr已经合入，相关issue测试中

[nnsgmsone]

cache pr已经合入，相关issue测试中

[nnsgmsone]

这一系列以后远宁跟踪把

[ouyuanning]

待沟通跟进方式

[ouyuanning]

待沟通跟进方式

[ouyuanning]

在处理prepare重构，还没时间看

[ouyuanning]

先处理DML重构

[ouyuanning]

未投入