Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: mo killed by oom during stability test on standlone mode #10212

Closed
1 task done
Tracked by #11553
aressu1985 opened this issue Jun 25, 2023 · 71 comments
Closed
1 task done
Tracked by #11553

[Bug]: mo killed by oom during stability test on standlone mode #10212

aressu1985 opened this issue Jun 25, 2023 · 71 comments
Assignees
@aressu1985
Labels
area/memory area/stability kind/bug Something isn't working severity/s1 High impact: Logical errors or data errors that must occur team/c2 to-next-release
Milestone
1.1.0

Comments

@aressu1985
Copy link
Contributor

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Environment

- Version or commit-id (e.g. v0.1.0 or 8b23a93):69d60980cb6b81b86a33c8e7f8180b1fadd5bc12
- Hardware parameters:16C 64G
- OS type:
- Others:

Actual Behavior

during stability test on standlone, the mo was killed b oom :
[Sat Jun 24 22:02:51 2023] [ 26370] 0 26370 265094 16433 450560 0 0 YDService
[Sat Jun 24 22:02:51 2023] [ 58535] 0 58535 588310 4173 339968 0 0 sh
[Sat Jun 24 22:02:51 2023] [3720175] 1000 3720175 3181 466 77824 0 0 run.sh
[Sat Jun 24 22:02:51 2023] [2527897] 1000 2527897 3181 447 65536 0 0 run-helper.sh
[Sat Jun 24 22:02:51 2023] [2527902] 1000 2527902 960475 17412 602112 0 0 Runner.Listener
[Sat Jun 24 22:02:51 2023] [2631990] 1000 2631990 18543527 15107219 128045056 0 500 mo-service
[Sat Jun 24 22:02:51 2023] [2634882] 1000 2634882 4280 470 73728 0 500 bash
[Sat Jun 24 22:02:51 2023] [2635046] 1000 2635046 5561 1778 77824 0 500 bash
[Sat Jun 24 22:02:51 2023] [2635189] 1000 2635189 4247 467 69632 0 500 bash
[Sat Jun 24 22:02:51 2023] [2635199] 1000 2635199 5898738 76189 1683456 0 500 java
[Sat Jun 24 22:02:51 2023] [2635421] 1000 2635421 4280 468 69632 0 500 bash
[Sat Jun 24 22:02:51 2023] [2635427] 1000 2635427 10636863 406768 4804608 0 500 java
[Sat Jun 24 22:02:51 2023] [2660982] 1000 2660982 1183993 110043 1384448 0 500 java
[Sat Jun 24 22:02:51 2023] [2680104] 0 2680104 15460 2209 159744 0 0 barad_agent
[Sat Jun 24 22:02:51 2023] [2680105] 0 2680105 313644 2919 311296 0 0 barad_agent
[Sat Jun 24 22:02:51 2023] [2680205] 1000 2680205 5561 1572 73728 0 500 bash
[Sat Jun 24 22:02:51 2023] [2680206] 1000 2680206 22528 317 143360 0 500 mysql
[Sat Jun 24 22:02:51 2023] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-14965.scope,task=mo-service,pid=2631990,uid=1000
[Sat Jun 24 22:02:51 2023] Out of memory: Killed process 2631990 (mo-service) total-vm:74174108kB, anon-rss:60428876kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:125044kB oom_score_adj:500

mo log:
mo-service.log.tar.gz

the last 3 times proflies:
2023-06-24_21_38_20.tar.gz
2023-06-24_21_44_24.tar.gz
2023-06-24_21_50_26.tar.gz

Expected Behavior

No response

Steps to Reproduce

run a mo tester on stanlone and run the following test
1. bvt loop
   git clone mo-tester
   ./run.sh -n -g -p  xxxxx/matrixone/test/distributed/cases -e ddl -t 100

2. tpch loop 
   git clone mo-tpch, laod tpch 10G data
   ./run.sh -q all -s 10 -t 50000

3. sysbench mixed
   git clone mo-load, and init 10-talbes-100000-per-table 
   ./start.sh -m SYSBENCH -n 10 -s 100000
   ./start.sh -c cases/sysbench/mixed_10_100000/ -d 5000 -g

4. tpcc 10-10 long run
   git clone mo-tpcc, and load tpcc 10 warehouse data, and modify runMins to 5000 in props.mo
   ./runBenchmark.sh props.mo

Additional information

No response
@aressu1985 aressu1985 added this to the V0.8.0 milestone Jun 25, 2023
@LeftHandCold
Copy link
Contributor

image image

@xzxiong
Copy link
Contributor

xzxiong commented Jun 25, 2023

watch

1 similar comment
@qingxinhome
Copy link
Contributor

watch

@daviszhen daviszhen assigned daviszhen and unassigned jianwan0214 Jun 26, 2023
@daviszhen
Copy link
Contributor

2023-06-24_21_38_20.tar.gz
里面的的mem profile。

截屏2023-06-26 10 11 17

@daviszhen
Copy link
Contributor

2023-06-24_21_44_24.tar.gz
里面的mem profile。

截屏2023-06-26 10 16 37

@daviszhen
Copy link
Contributor

2023-06-24_21_50_26.tar.gz
里面的mem profile。

截屏2023-06-26 10 21 17

@daviszhen
Copy link
Contributor

daviszhen commented Jun 26, 2023 •
edited
Loading

image image

这个大概率不是真实原因。goetty最近没改过。内存消耗应该跟以前一样大。

@fengttt
Copy link
Contributor

fengttt commented Jun 26, 2023

@daviszhen your goetty switch merged. Now we are pulling from matrixorigin goetty repo. Please check if they are synced.

@fengttt
Copy link
Contributor

fengttt commented Jun 26, 2023

#9674

@daviszhen
Copy link
Contributor

daviszhen commented Jun 26, 2023 •
edited
Loading

@daviszhen your goetty switch merged. Now we are pulling from matrixorigin goetty repo. Please check if they are synced.

They are same. I have checked it.

  1. this issue's (69d6098) goetty version is github.com/fagongzi/goetty/v2 v2.0.3-0.20230520035916-bc1fed6f5e26

  2. the main : a67f9d9
    image

@daviszhen
Copy link
Contributor

daviszhen commented Jun 26, 2023 •
edited
Loading

测试不能正常完成。1小时20分钟后，oom kill。

跑了半小时后，出了下面的问题：

1，tpch找不到表。
Screenshot from 2023-06-26 16-09-35

2，sysbench(mo-load)找不到表。

Screenshot from 2023-06-26 16-06-16

3，tpcc事务报错。

Screenshot from 2023-06-26 16-07-29

4，登录看。sysbench, tpch的库都不存在了
Screenshot from 2023-06-26 16-14-12

@daviszhen
Copy link
Contributor

mo-tester的mo.yaml
中
builtindb: mo_catalog,tpcc_10,tpch_10g,sbtest,mo_task,information_schema,mysql,system_metrics,system,query_result

@cnutshell
Copy link
Contributor

According to the OOM killer log, mo-service took 60428876kB memory, which was roughly 60GB memory.
According to the memory profile of 2023-06-24_21_50_26.tar.gz, the inuse_space of mo-service is 22.78GB.

If mo-service didn't allocated memory repeatedly, it shouldn't be killed(22.78GB << 60GB). The reason why mo-service was killed by OOM should be: certain components of mo-service allocs memory so quickly that the scavenger of golang couldn't release free-memory back to OS in time.

Advice: just pay attention to those code that allocated too much memory:
image

@daviszhen
Copy link
Contributor

daviszhen commented Jun 27, 2023 •
edited
Loading

昨天试了仅有tpcc,tpch,sysbench 混合跑的情况。

大概跑2小时后，内存oom。比带上bvt出oom的情况要慢。

Screenshot from 2023-06-27 10-23-18

最后几分中的内存情况。

Screenshot from 2023-06-27 10-32-36

@daviszhen
Copy link
Contributor

期间watermark 的情况。

2023/06/26 19:38:26.576611 +0800 INFO logtail/mgr.go:160 init waterline to 1687779506576560208-0
2023/06/26 19:38:26.576674 +0800 INFO dn-service.logtail-push-service service/server.go:375 init waterline {"uuid": "dd4dccb4-4d3c-41f8-b482-5251dc7a41bf", "server-id": "6a0f9299-6294-4c42-be8f-8843a21c9174", "to": "PhysicalTime:1687779506576560208 "}
2023/06/26 19:39:28.837201 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 1073995872
2023/06/26 19:39:28.850698 +0800 INFO mpool/mpool.go:81 MPool pipeline-1b737384-1416-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1073763840
2023/06/26 19:39:35.092109 +0800 INFO mpool/mpool.go:81 MPool pipeline-1eb9493c-1416-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1073752760
2023/06/26 19:39:41.635527 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 2147520901
2023/06/26 19:39:45.355942 +0800 INFO mpool/mpool.go:81 MPool pipeline-2508d8d3-1416-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1073799992
2023/06/26 19:40:06.946052 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 3221225512
2023/06/26 19:40:14.797836 +0800 INFO mpool/mpool.go:81 MPool pipeline-289aa2c4-1416-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1073846024
2023/06/26 19:40:17.146592 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 4295107656
2023/06/26 19:41:42.267377 +0800 INFO mpool/mpool.go:81 MPool pipeline-6a62a06c-1416-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1295272360
2023/06/26 19:42:00.353682 +0800 INFO mpool/mpool.go:81 MPool pipeline-760b0778-1416-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1074287800
2023/06/26 19:42:03.983918 +0800 INFO mpool/mpool.go:81 MPool pipeline-77fa4d99-1416-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1263125760
2023/06/26 19:42:07.163203 +0800 INFO mpool/mpool.go:81 MPool pipeline-77fa4d99-1416-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 2206582016
2023/06/26 19:42:23.634502 +0800 INFO mpool/mpool.go:81 MPool pipeline-82fb492c-1416-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1291712424
2023/06/26 19:42:52.508115 +0800 INFO mpool/mpool.go:81 MPool pipeline-94ba0441-1416-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1076434104
2023/06/26 19:43:27.981915 +0800 INFO mpool/mpool.go:81 MPool pipeline-a7f3d62f-1416-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1082656000
2023/06/26 19:43:40.874400 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 1075824672
2023/06/26 19:43:41.160988 +0800 INFO mpool/mpool.go:81 MPool pipeline-a7f3d62f-1416-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 2204910848
2023/06/26 19:43:56.866236 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 1073741856
2023/06/26 19:44:17.982667 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 5371429356
2023/06/26 19:44:48.475176 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 2156995744
2023/06/26 19:46:46.417191 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 3227241880
2023/06/26 19:47:08.559735 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 6447184088
2023/06/26 19:47:11.651694 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 2147566992
2023/06/26 19:48:02.132056 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 7518267680
2023/06/26 19:48:07.072031 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 4296428928
2023/06/26 19:49:01.434509 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 8595399833
2023/06/26 19:49:05.856064 +0800 INFO mpool/mpool.go:81 MPool pipeline-70183db6-1417-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1121776384
2023/06/26 19:49:06.264939 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 9668877078
2023/06/26 19:49:12.115521 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 3221394848
2023/06/26 19:50:19.511329 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 5371455448
2023/06/26 19:51:02.018465 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 10737420890
2023/06/26 19:51:06.923200 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 4295027384
2023/06/26 19:51:54.729821 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 11811190448
2023/06/26 19:52:22.150770 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 6442523408
2023/06/26 19:52:37.176088 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 12886583062
2023/06/26 19:52:51.874957 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 5368802136
2023/06/26 19:53:07.175586 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 13958689412
2023/06/26 19:53:46.959002 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 7518419104
2023/06/26 19:54:23.097671 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 15033106120
2023/06/26 19:54:26.654850 +0800 INFO mpool/mpool.go:81 MPool pipeline-2f27da9f-1418-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1085797120
2023/06/26 19:54:27.447032 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 16108075280
2023/06/26 19:55:01.701438 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 6442461328
2023/06/26 19:55:27.895723 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 8592318648
2023/06/26 19:55:58.832286 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 17181424378
2023/06/26 19:56:32.031103 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 18254357401
2023/06/26 19:56:34.620532 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 9664470944
2023/06/26 19:56:36.863270 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 7516203656
2023/06/26 19:57:01.784998 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 19336272680
2023/06/26 19:57:20.717683 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 10741074888
2023/06/26 19:57:38.829063 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 20401536576
2023/06/26 19:58:31.987669 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 21475736512
2023/06/26 19:58:42.299592 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 8589943568
2023/06/26 19:58:50.387510 +0800 INFO mpool/mpool.go:81 MPool pipeline-cd63f86b-1418-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1086386944
2023/06/26 19:58:50.500887 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 22570144544
2023/06/26 19:58:53.878373 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 11820817248
2023/06/26 19:59:50.615828 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 12888630792
2023/06/26 20:00:02.246466 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 23623682712
2023/06/26 20:00:31.384713 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 13964515248
2023/06/26 20:00:31.497299 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 24703743144
2023/06/26 20:00:52.145522 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 9663690624
2023/06/26 20:01:03.844787 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 25773894248
2023/06/26 20:01:22.585483 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 15036939936
2023/06/26 20:01:37.035832 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 27247965672
2023/06/26 20:02:35.469571 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 16110839632
2023/06/26 20:02:45.248514 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 27918348396
2023/06/26 20:02:54.140658 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 28996586174
2023/06/26 20:02:54.551932 +0800 INFO mpool/mpool.go:81 MPool pipeline-5f1c6272-1419-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1114114816
2023/06/26 20:03:20.077451 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 17181569928
2023/06/26 20:03:57.272672 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 10737447024
2023/06/26 20:04:01.810447 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 30064938677
2023/06/26 20:04:21.121478 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 18254143216
2023/06/26 20:04:52.369795 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 31157174736
2023/06/26 20:05:13.280508 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 19332839584
2023/06/26 20:05:30.188584 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 32322450656
2023/06/26 20:05:56.104789 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 20401967928
2023/06/26 20:06:05.541895 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 33286675460
2023/06/26 20:06:12.784490 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 34396009820
2023/06/26 20:06:12.952690 +0800 INFO mpool/mpool.go:81 MPool pipeline-d57ed4bc-1419-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1084584704
2023/06/26 20:06:46.037007 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 21478896472
2023/06/26 20:07:19.355030 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 35434083109
2023/06/26 20:07:21.759849 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 11811173152
2023/06/26 20:07:28.013861 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 22552185736
2023/06/26 20:07:53.951362 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 36514829760
2023/06/26 20:08:16.656783 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 37581082525
2023/06/26 20:08:28.454052 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 23626640536
2023/06/26 20:09:12.134095 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 24698503824
2023/06/26 20:09:22.430510 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 38660678896
2023/06/26 20:09:33.703943 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 39747661952
2023/06/26 20:09:33.894870 +0800 INFO mpool/mpool.go:81 MPool pipeline-4d55d394-141a-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1103131392
2023/06/26 20:10:06.450171 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 25776233504
2023/06/26 20:11:06.589064 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 40809705836
2023/06/26 20:11:12.281436 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 26846968376
2023/06/26 20:11:29.616769 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 41886128296
2023/06/26 20:11:31.730372 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 12884949048
2023/06/26 20:11:51.698090 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 27919249472
2023/06/26 20:12:15.256360 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 42968228704
2023/06/26 20:12:37.468716 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 44067237360
2023/06/26 20:12:37.516413 +0800 INFO mpool/mpool.go:81 MPool pipeline-bb3ae217-141a-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1075139328
2023/06/26 20:12:49.169376 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 28992850088
2023/06/26 20:13:29.314614 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 30069510096
2023/06/26 20:13:42.818146 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 45097275744
2023/06/26 20:14:10.646788 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 31145272536
2023/06/26 20:14:28.929796 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 46177787992
2023/06/26 20:15:23.401532 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 47288425080
2023/06/26 20:15:23.695629 +0800 INFO mpool/mpool.go:81 MPool pipeline-1e2ceb42-141b-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1101263616
2023/06/26 20:15:32.131829 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 32213135344
2023/06/26 20:16:18.799098 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 33288323144
2023/06/26 20:16:22.801335 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 48325548600
2023/06/26 20:17:08.652495 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 49685661104
2023/06/26 20:17:31.817586 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 34363010840
2023/06/26 20:18:07.549172 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 50490448000
2023/06/26 20:18:07.859352 +0800 INFO mpool/mpool.go:81 MPool pipeline-8043deb1-141b-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1101268992
2023/06/26 20:18:26.250435 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 35436589128
2023/06/26 20:19:37.059658 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 51542354464
2023/06/26 20:19:43.073237 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 36509294768
2023/06/26 20:20:11.496808 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 52618546992
2023/06/26 20:20:42.153495 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 37586514016
2023/06/26 20:21:15.110427 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 53688669175
2023/06/26 20:21:16.465278 +0800 INFO mpool/mpool.go:81 MPool pipeline-ef7dd040-141b-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1125659392
2023/06/26 20:21:18.536648 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 54761593432
2023/06/26 20:21:59.531257 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 38656000832
2023/06/26 20:22:52.776314 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 39730620664
2023/06/26 20:23:06.009165 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 56244611608
2023/06/26 20:23:51.796853 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 40804978952
2023/06/26 20:24:03.818573 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 56958284484
2023/06/26 20:24:03.980485 +0800 INFO mpool/mpool.go:81 MPool pipeline-54b78db6-141c-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1080619776
2023/06/26 20:24:50.494260 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 41878387200
2023/06/26 20:25:22.684597 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 57986593376
2023/06/26 20:25:40.003634 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 42956064152
2023/06/26 20:25:59.326564 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 59546250848
2023/06/26 20:26:50.311633 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 44024643448
2023/06/26 20:26:52.767368 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 60163132192
2023/06/26 20:26:53.020356 +0800 INFO mpool/mpool.go:81 MPool pipeline-b912f353-141c-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1107833600
2023/06/26 20:27:21.772418 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 13958668136
2023/06/26 20:27:36.139566 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 45098117576
2023/06/26 20:27:59.545078 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 61209354128
2023/06/26 20:28:41.561995 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 46172449384
2023/06/26 20:28:42.158790 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 62596131344
2023/06/26 20:29:34.540514 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 47248081976
2023/06/26 20:29:38.762300 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 63383416448
2023/06/26 20:29:38.966444 +0800 INFO mpool/mpool.go:81 MPool pipeline-1c35de5e-141d-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1085147904
2023/06/26 20:30:16.085642 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 48319002712
2023/06/26 20:30:33.547331 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 64483386288
2023/06/26 20:31:14.977538 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 49393967936
2023/06/26 20:31:20.529967 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 65498655463
2023/06/26 20:31:57.476875 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 50470342704
2023/06/26 20:32:03.884199 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 66572760576
2023/06/26 20:32:24.317212 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 67654853456
2023/06/26 20:32:24.646464 +0800 INFO mpool/mpool.go:81 MPool pipeline-7ee1789e-141d-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1077310208
2023/06/26 20:32:36.389535 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 51546152800
2023/06/26 20:33:27.455150 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 52617681704
2023/06/26 20:33:34.910393 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 68786140052
2023/06/26 20:34:02.316329 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 69801795928
2023/06/26 20:34:03.818850 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 53694003296
2023/06/26 20:34:40.581708 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 54761742096
2023/06/26 20:34:44.737406 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 70872917792
2023/06/26 20:35:00.899566 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 71942950720
2023/06/26 20:35:01.111446 +0800 INFO mpool/mpool.go:81 MPool pipeline-dcbe49fc-141d-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1098887936
2023/06/26 20:35:34.759365 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 55837503376
2023/06/26 20:36:25.397632 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 56909715472
2023/06/26 20:36:32.970111 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 73015725436
2023/06/26 20:36:46.619425 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 74309303376
2023/06/26 20:37:04.545276 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 57985601864
2023/06/26 20:37:41.252118 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 75173537312
2023/06/26 20:37:42.144960 +0800 INFO mpool/mpool.go:81 MPool pipeline-3bd45002-141e-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1090333440
2023/06/26 20:37:55.863548 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 59063184568
2023/06/26 20:38:37.563077 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 60132022376
2023/06/26 20:38:38.276369 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 76268910124
2023/06/26 20:39:10.266313 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 61203537400
2023/06/26 20:39:12.735583 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 77311323040
2023/06/26 20:39:30.120303 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 78593063224
2023/06/26 20:39:44.007936 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 62279699624
2023/06/26 20:40:23.214034 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 79482775792
2023/06/26 20:40:23.450495 +0800 INFO mpool/mpool.go:81 MPool pipeline-9cbea751-141e-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1120203520
2023/06/26 20:40:36.232137 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 63353335744
2023/06/26 20:41:14.059966 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 64432183624
2023/06/26 20:41:15.960615 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 80530986886
2023/06/26 20:41:47.510785 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 81675308700
2023/06/26 20:41:48.322042 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 65500449616
2023/06/26 20:42:13.769561 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 82683941212
2023/06/26 20:42:35.560540 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 66577631232
2023/06/26 20:43:04.869695 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 83795225104
2023/06/26 20:43:05.166030 +0800 INFO mpool/mpool.go:81 MPool pipeline-fc9f4186-141e-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1086206720
2023/06/26 20:43:27.788458 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 67649503360
2023/06/26 20:44:03.090647 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 84831354004
2023/06/26 20:44:08.357743 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 68722887296
2023/06/26 20:44:43.449646 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 85902014508
2023/06/26 20:44:59.687997 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 69795539984
2023/06/26 20:45:32.559152 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 86973718812
2023/06/26 20:45:41.010016 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 70871317088
2023/06/26 20:45:45.650469 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 88050768708
2023/06/26 20:45:45.672257 +0800 INFO mpool/mpool.go:81 MPool pipeline-5cc36c57-141f-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1115859712
2023/06/26 20:46:27.799548 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 71943043752
2023/06/26 20:46:51.784275 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 89124072532
2023/06/26 20:46:59.875678 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 73018882744
2023/06/26 20:47:33.875612 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 90194848522
2023/06/26 20:47:56.997007 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 74091470840
2023/06/26 20:48:25.156402 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 91281035052
2023/06/26 20:48:29.564199 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 92353153673
2023/06/26 20:48:29.931283 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 75162000000
2023/06/26 20:48:29.943228 +0800 INFO mpool/mpool.go:81 MPool pipeline-be531620-141f-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1105163008
2023/06/26 20:49:06.476055 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 76236096856
2023/06/26 20:49:43.248441 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 93420248764
2023/06/26 20:49:52.889248 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 77313612480
2023/06/26 20:50:12.922985 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 94964064873
2023/06/26 20:50:35.481580 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 78389842488
2023/06/26 20:51:06.293509 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 95577094380
2023/06/26 20:51:08.969684 +0800 INFO mpool/mpool.go:81 MPool pipeline-1d38f64f-1420-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1116058368
2023/06/26 20:51:09.045404 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 96675117260
2023/06/26 20:51:20.454791 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 79458062304
2023/06/26 20:52:01.205036 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 80530733320
2023/06/26 20:52:41.489619 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 97712349004
2023/06/26 20:52:54.009545 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 81607046808
2023/06/26 20:53:17.322452 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 98786286028
2023/06/26 20:53:35.765681 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 82679830152
2023/06/26 20:53:46.441177 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 99918188092
2023/06/26 20:53:46.686986 +0800 INFO mpool/mpool.go:81 MPool pipeline-7b250a04-1420-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1117506304
2023/06/26 20:54:15.288978 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 83755541272
2023/06/26 20:54:39.791382 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 100935372364
2023/06/26 20:54:51.117999 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 84825822224
2023/06/26 20:55:12.956077 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 102008125120
2023/06/26 20:55:36.069675 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 103099226563
2023/06/26 20:55:39.335524 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 85899692504
2023/06/26 20:56:25.751610 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 104183512284
2023/06/26 20:56:25.798491 +0800 INFO mpool/mpool.go:81 MPool pipeline-da47663c-1420-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1119734528
2023/06/26 20:56:35.496525 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 86974522536
2023/06/26 20:57:12.692192 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 88053372856
2023/06/26 20:57:39.806851 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 105228753668
2023/06/26 20:57:53.675660 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 89125286552
2023/06/26 20:58:10.094432 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 106816103164
2023/06/26 20:58:46.319607 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 90195125144
2023/06/26 20:59:04.514619 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 107382068172
2023/06/26 20:59:06.729171 +0800 INFO mpool/mpool.go:81 MPool pipeline-3a20123c-1421-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1118204672
2023/06/26 20:59:07.361724 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 108448674011
2023/06/26 20:59:30.656660 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 91270571208
2023/06/26 21:00:11.543965 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 92347518280
2023/06/26 21:00:47.487559 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 109522384684
2023/06/26 21:01:09.967595 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 93416100752
2023/06/26 21:01:30.390266 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 110648149684
2023/06/26 21:01:49.889811 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 94491121792
2023/06/26 21:01:52.644591 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 111674185580
2023/06/26 21:01:52.849226 +0800 INFO mpool/mpool.go:81 MPool pipeline-9ce92e1a-1421-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1085141760
2023/06/26 21:02:27.398999 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 95564660400
2023/06/26 21:02:47.358672 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 112802586700
2023/06/26 21:03:14.118899 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 96636864304
2023/06/26 21:03:32.510025 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 113817694880
2023/06/26 21:03:57.718613 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 97711582000
2023/06/26 21:04:07.774055 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 114898000112
2023/06/26 21:04:35.971875 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 98789664816
2023/06/26 21:04:36.308081 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 116019532908
2023/06/26 21:04:36.620436 +0800 INFO mpool/mpool.go:81 MPool pipeline-fe7a754e-1421-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1118548736
2023/06/26 21:05:29.192551 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 99863760504
2023/06/26 21:05:44.800426 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 117042717228
2023/06/26 21:06:08.853563 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 100935639432
2023/06/26 21:06:16.220425 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 118120926300
2023/06/26 21:06:43.394279 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 102014940976
2023/06/26 21:06:56.336318 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 119185924332
2023/06/26 21:07:15.049875 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 120277743980
2023/06/26 21:07:15.240357 +0800 INFO mpool/mpool.go:81 MPool pipeline-5cb71b5d-1422-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1118554880
2023/06/26 21:07:22.968513 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 103084179504
2023/06/26 21:08:05.602745 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 104154343888
2023/06/26 21:08:26.928685 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 121332920132
2023/06/26 21:08:47.891634 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 105227743128
2023/06/26 21:08:54.379955 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 122470850744
2023/06/26 21:09:26.139375 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 106300543416
2023/06/26 21:09:35.093057 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 123482378076
2023/06/26 21:09:53.304395 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 124597747756
2023/06/26 21:09:53.600580 +0800 INFO mpool/mpool.go:81 MPool pipeline-bb703813-1422-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1085229824
2023/06/26 21:10:12.475923 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 107375492032
2023/06/26 21:10:59.459568 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 108449445552
2023/06/26 21:11:06.646451 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 125706935768
2023/06/26 21:11:36.379740 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 127064031684
2023/06/26 21:11:43.085734 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 109525482264
2023/06/26 21:12:27.266020 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 110597473224
2023/06/26 21:12:33.353251 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 127779939860
2023/06/26 21:12:36.355706 +0800 INFO mpool/mpool.go:81 MPool pipeline-1c5cac19-1423-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1077236480
2023/06/26 21:12:36.419191 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 128864788252
2023/06/26 21:13:09.440011 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 111675683136
2023/06/26 21:13:56.115552 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 112743576480
2023/06/26 21:13:56.770357 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark
high water mark : 15032827800
2023/06/26 21:14:01.786514 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 129923722882
2023/06/26 21:14:30.116543 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 131365162444
2023/06/26 21:14:34.337963 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 113820753176
2023/06/26 21:15:30.436119 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 132130940188
2023/06/26 21:15:30.635887 +0800 INFO mpool/mpool.go:81 MPool pipeline-8513a54d-1423-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1078315776
2023/06/26 21:15:43.956076 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 114892414784
2023/06/26 21:16:26.666993 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 115966516944
2023/06/26 21:16:33.399004 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 133145557876
2023/06/26 21:17:01.897961 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 117041079664
2023/06/26 21:17:10.023848 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 134227994668
2023/06/26 21:17:59.870522 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 118113683800
2023/06/26 21:18:07.019899 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 135295378172
2023/06/26 21:18:08.959181 +0800 INFO mpool/mpool.go:81 MPool pipeline-e2ed26d0-1423-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1074795264
2023/06/26 21:18:08.964217 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 136369212516
2023/06/26 21:19:00.555493 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 119185780320
2023/06/26 21:19:40.834262 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 120261063800
2023/06/26 21:19:45.276899 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 137441219228
2023/06/26 21:20:28.490634 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 121333400648
2023/06/26 21:20:30.819496 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 138577752316
2023/06/26 21:20:51.506824 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 139612742052
2023/06/26 21:20:51.610626 +0800 INFO mpool/mpool.go:81 MPool pipeline-43e5b54e-1424-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1139987200
2023/06/26 21:21:07.174849 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 122410275720
2023/06/26 21:21:41.508054 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 123484118544
2023/06/26 21:21:45.247397 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 140662554176
2023/06/26 21:22:19.785120 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 141734684588
2023/06/26 21:22:19.785197 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 124556912776
2023/06/26 21:23:22.467440 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 142808085916
2023/06/26 21:23:24.296788 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 125632526064
2023/06/26 21:23:36.533559 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 143888450500
2023/06/26 21:23:36.562793 +0800 INFO mpool/mpool.go:81 MPool pipeline-a5b4121c-1424-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1089794816
2023/06/26 21:24:10.146574 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 126706509480
2023/06/26 21:24:49.121562 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 127776032392
2023/06/26 21:24:53.046703 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 144955551373
2023/06/26 21:25:22.068727 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 146323614660
2023/06/26 21:25:42.731408 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 128851186864
2023/06/26 21:26:13.573587 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 147168623106
2023/06/26 21:26:13.838646 +0800 INFO mpool/mpool.go:81 MPool pipeline-038e821a-1425-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1099844352
2023/06/26 21:26:16.559956 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 148182512100
2023/06/26 21:26:23.581717 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 129927695240
2023/06/26 21:27:08.641023 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 130998173544
2023/06/26 21:27:57.435766 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 149250637564
2023/06/26 21:28:06.130989 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 132071808288
2023/06/26 21:28:49.877492 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 133147543632
2023/06/26 21:28:53.070421 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 150325434916
2023/06/26 21:28:56.785919 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 151422290495
2023/06/26 21:28:56.961012 +0800 INFO mpool/mpool.go:81 MPool pipeline-64870244-1425-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1120875264
2023/06/26 21:29:32.251421 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 134221201760
2023/06/26 21:30:17.363843 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 135297049736
2023/06/26 21:30:18.759142 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 152471365929
2023/06/26 21:30:42.857487 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 153699512284
2023/06/26 21:31:07.411994 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 136367423440
2023/06/26 21:31:38.016335 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 154625766388
2023/06/26 21:31:40.213519 +0800 INFO mpool/mpool.go:81 MPool pipeline-c63338f7-1425-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1119636224
2023/06/26 21:31:40.458198 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 155734312148
2023/06/26 21:31:42.330779 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 137449708512
2023/06/26 21:32:30.508948 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 138515487056
2023/06/26 21:33:14.165506 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 156769065044
2023/06/26 21:33:18.881007 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 139587378800
2023/06/26 21:33:58.165212 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 157848470276
2023/06/26 21:34:03.780823 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 140663760712
2023/06/26 21:34:22.118481 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 158961923388
2023/06/26 21:34:22.501538 +0800 INFO mpool/mpool.go:81 MPool pipeline-26e7460f-1426-11ee-bf5c-7070fc0034e7 new high watermark
high water mark : 1122104064
2023/06/26 21:34:42.929065 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 141735051080
2023/06/26 21:35:20.860254 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 159991181068
2023/06/26 21:35:27.703471 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 142812096400
2023/06/26 21:36:05.879998 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
high water mark : 161061301782
2023/06/26 21:36:12.952202 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark
high water mark : 143889170192

@daviszhen
Copy link
Contributor

中午用 dacfe86.
tpcc+tpch+sysbench。

跑了2个小时，没出现oom。内存控制30G左右。

Screenshot from 2023-06-27 13-55-15

@daviszhen
Copy link
Contributor

daviszhen commented Jun 27, 2023 •
edited
Loading

试验了 23日最后一个版本。 f0e71a0
跑了1个小时，内存有泄漏的现象出现。

Screenshot from 2023-06-27 15-24-53

@daviszhen
Copy link
Contributor

daviszhen commented Jun 27, 2023 •
edited
Loading

试验了22日最后一个版本。 bf23f2a
oom了。超过2个半小时。
Screenshot from 2023-06-27 18-55-22

@daviszhen
Copy link
Contributor

昨晚试验了版本 c5a8d23
没跑出来oom。（从 2023/06/27 20:31:12 到 2023/06/28 09:50:15）。
Screenshot from 2023-06-28 09-38-42

@xzxiong
Copy link
Contributor

xzxiong commented Aug 23, 2023 •
edited
Loading

update:
1） 昨晚我做了测试，未复现
测试case ：insert 1小时后，做delete from ... where ... 操作，重复多次
-- 未复现 api.(*PrecommitWriteCmd).Marshal 的堆栈消耗。只有 BTree 相关的内存消耗（这个内存也很夸张，会导致oom）

2）这里的 profile.tar.gz 的有个细节：goroutine 没有 buildInPurgeLog 的堆栈
里面有连续 5次profile的文件，heap的信息都有 buildInPurgeLog的使用，goroutine 里面没有。
profile文件详见：#10212 (comment)

  • heap, inuse 中 api.(*PrecommitWriteCmd).Marshal 占用的大量内存
    image
  • 如下图，goroutine.log 文件中没有 buildInPurgeLog 相关的对账
    image

===>
这里应该是 sqlExecutor 本身有内存泄露的问题 @nnsgmsone 麻烦一起确认下~

@aressu1985
Copy link
Contributor Author

update on 8.23
panic by out of memory

mo log:
mo-service-log.tar.gz

mo profile:
profile.tar.gz

@nnsgmsone
Copy link
Contributor

nnsgmsone commented Aug 28, 2023 •
edited by LeftHandCold
Loading

从最新的pprof来看，purelog的内存占用确实没了，不过ckpt的内存占用在不断的增加，最后oom:
截图 2023-08-28 09-57-31

截图 2023-08-28 09-57-04
@XuPeng-SH

@LeftHandCold
Copy link
Contributor

从最新的pprof来看，purelog的内存占用确实没了，不过ckpt的内存占用在不断的增加，最后oom: 截图 2023-08-28 09-57-31

截图 2023-08-28 09-57-04 @XuPeng-SH

#11423 应该是修复了这个bug，可以继续观察

@aressu1985
Copy link
Contributor Author

OOM问题依旧存在

@sukki37 sukki37 mentioned this issue Aug 30, 2023
Open
28 tasks
@nnsgmsone
Copy link
Contributor

等待重现后的pprof

@nnsgmsone
Copy link
Contributor

@aressu1985 复现了吗。苏兄。。复现的话，贴一下pprof吧

1 similar comment
@nnsgmsone
Copy link
Contributor

@aressu1985 复现了吗。苏兄。。复现的话，贴一下pprof吧

@aressu1985 aressu1985 added severity/s1 High impact: Logical errors or data errors that must occur and removed severity/s-1 labels Sep 8, 2023
@aressu1985
Copy link
Contributor Author

tracking by #11553 ，该ISSUE降级到s1

@sukki37 sukki37 modified the milestones: 1.0.0, 1.1.0 Oct 7, 2023
@aressu1985
Copy link
Contributor Author

update on 10.16
mo carsh by :
88671 fatal error: runtime: out of memory
88672
88673 runtime stack:
88674 runtime.throw({0x36a3b85?, 0x0?})
88675 /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7ff389ff7000 sp=0x7ff389ff6fd0 pc=0x4cc13d
88676 runtime.sysMapOS(0xcad4400000, 0x37800000?)

Uploading mo-panic-service.tar.gz…

@aressu1985
Copy link
Contributor Author

关闭该issue，使用新的issue跟踪#12142

@matrix-meow matrix-meow reopened this Dec 15, 2023
@LiSong0214 LiSong0214 assigned aressu1985 and unassigned nnsgmsone Dec 15, 2023
@LiSong0214 LiSong0214 modified the milestones: 1.1.0, 1.2.0 Dec 15, 2023
@sukki37 sukki37 modified the milestones: 1.2.0, 1.1.0 Dec 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aressu1985 aressu1985

Labels
area/memory area/stability kind/bug Something isn't working severity/s1 High impact: Logical errors or data errors that must occur team/c2 to-next-release
Projects
None yet
Milestone
1.1.0
Development

No branches or pull requests