[Bug]: TPMC of 1000w is 20% lower than in mo 1.2.0

[aressu1985]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

main

Commit ID

30195c0

Other Environment Information

- Hardware parameters:
3*CN: 16C 64G
1*DN: 16C 64G
3*LOG: 4C 16G
1*PROXY: 3C 7G
- OS type:
- Others:

Actual Behavior

main metrics:
https://grafana.ci.matrixorigin.cn/d/ee086o0rtlq0wc/txn-metrics?orgId=1&var-interval=1m&var-namespace=mo-main-nightly-30195c002-20241018&var-pod=All&from=1729281003000&to=1729283446000

mo-1.2 metrics:
https://grafana.ci.matrixorigin.cn/d/ee086o0rtlq0wc/txn-metrics?orgId=1&var-interval=1m&var-namespace=mo-branch-nightly-c3c1ce0f5-20241018&var-pod=All&from=1729281003000&to=1729284046000

Expected Behavior

No response

Steps to Reproduce

run tpcc 1000w test

Additional information

No response

[badboynt1]

tpcc 1000仓，数据量比较大，memcache不是完全命中，需要频繁进行替换。

观察main分支跑起来，tps稳定后，cpu利用率不到一半。应该是加锁等待的开销比较大。抓了fgprof以后看到，确实主要是memcache加锁开销大。需要优化。
再观察cpu profile情况，主要开销都是在io上，这种情况只能通过sharding来提高cache命中率，进而提高tps。
(pprof) top
Showing nodes accounting for 370.37s, 55.70% of 664.90s total
Dropped 4010 nodes (cum <= 3.32s)
Showing top 10 nodes out of 311
flat flat% sum% cum cum%
119.69s 18.00% 18.00% 119.69s 18.00% runtime.memmove
71.97s 10.82% 28.83% 71.97s 10.82% internal/runtime/syscall.Syscall6
67.10s 10.09% 38.92% 138.40s 20.82% github.com/pierrec/lz4/v4/internal/lz4block.decodeBlock
26.38s 3.97% 42.88% 28.35s 4.26% runtime.findObject
23.80s 3.58% 46.46% 23.80s 3.58% runtime.memclrNoHeapPointers
16.98s 2.55% 49.02% 16.98s 2.55% sync/atomic.(*Int64).Add
15.32s 2.30% 51.32% 66.54s 10.01% runtime.scanobject
10.19s 1.53% 52.85% 18.04s 2.71% sync.(*poolChain).popTail
9.52s 1.43% 54.29% 9.70s 1.46% runtime.(*gcBits).bitp (inline)
9.42s 1.42% 55.70% 9.42s 1.42% sync/atomic.(*Uint32).CompareAndSwap

prof_1.zip

[reusee]

上面的fgprof profile有问题，需要更新到最新的版本。
从 cpu profile 看，内存缓存的锁开销并不大。

[reusee]

无进展

[reusee]

继续优化

[reusee]

已优化