[Bug]: cache anomalies

[sukki37]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Environment

- Version or commit-id (e.g. v0.1.0 or 8b23a93): 333d855
- Hardware parameters:
- OS type:
- Others:

Actual Behavior

@nnsgmsone encountered some unexpected cache behaviors. Specifically, there are instances where the cache suddenly exceeds the configured cache limit. This behavior is unexpected and could potentially lead to resource-related issues in production environments.

Several memory-related issues we've recently faced have been traced back to this cache anomaly. We have temporarily circumvented these specific issues by modifying configurations, but it's a makeshift solution. It's imperative to identify the root cause and fix it. Hence, this issue is being opened explicitly for tracking this behavior.

Related Issues:
#10659
https://github.com/matrixorigin/MO-Cloud/issues/984

Expected Behavior

All cache works as expected.

Steps to Reproduce

No response

Additional information

No response

[nnsgmsone]

我会处理这个问题。

[jensenojs]

使用单机mo，将cn, dn, log的metacache / fileservice.cache的memory-capacity都设置为1MB之后，开始轮bvt。

下图是在跑完第八次bvt后做的pprof

[jensenojs]

使用单机mo，将cn, dn, log的metacache / fileservice.cache的memory-capacity都设置为1MB之后，开始轮bvt。

下图是在跑完第八次bvt后做的pprof

补充一个diff_base下的heap_pprof的对比，可以更直观的看到跑了若干次bvt后，heap增长有一部分来源于HandleRowsDelete，的调用。它会突破内存限制的原因是：cn需要dn的信号来清理掉这些被delete的数据，这个问题现在被 #10370 跟踪。

其他内存上涨有必要会继续追踪。

[fengttt]

because another S-1 #10659 depends on this one.

[nnsgmsone]

https://github.com/matrixorigin/docs/blob/main/mem/mo%20memory.md cache相关分析的文档

[nnsgmsone]

等明天讨论后确定方案。

[nnsgmsone]

正在处理

[nnsgmsone]

正在编码中。。

[nnsgmsone]

正在编码中。。

[nnsgmsone]

代码基本完成，正在测试

[nnsgmsone]

debug中。

[nnsgmsone]

继续测试中

[nnsgmsone]

先出design doc，然后评审后继续开发.

[nnsgmsone]

继续整理设计文档。

[aressu1985]

tracking by #11553 ，该ISSUE降级到s1