Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](cloud-mow) Fix the issue of compaction holding mow table lock for a long time #40431

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

hust-hhb
Copy link
Contributor

@hust-hhb hust-hhb commented Sep 5, 2024

Now compaction use the same lock id (i.e. -1) when getting mow table lock from ms, if there are a large number of continuous compaction tasks to be executed, compaction will hold the lock for a long time, which will make loading task hang on waiting mow table lock when commiting txn, my test show it may lead to commit txn fail.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Sep 5, 2024

run buildall

Copy link
Contributor

github-actions bot commented Sep 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Sep 5, 2024

run buildall

Copy link
Contributor

github-actions bot commented Sep 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38443 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9fdfda03d0d5fe0d853cd116d08eb591bd2d66a6, data reload: false

------ Round 1 ----------------------------------
q1	17620	4476	4391	4391
q2	2013	191	184	184
q3	11782	981	1115	981
q4	10509	713	755	713
q5	7728	2866	2835	2835
q6	229	142	140	140
q7	958	606	601	601
q8	9531	2089	2081	2081
q9	7002	6571	6581	6571
q10	7013	2252	2263	2252
q11	459	246	243	243
q12	403	224	225	224
q13	17833	3131	3053	3053
q14	269	249	232	232
q15	536	493	474	474
q16	603	531	504	504
q17	987	702	729	702
q18	7410	6858	6873	6858
q19	1390	1040	1116	1040
q20	700	326	339	326
q21	3878	3362	3032	3032
q22	1097	1006	1027	1006
Total cold run time: 109950 ms
Total hot run time: 38443 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4368	4328	4292	4292
q2	381	270	277	270
q3	2923	2662	2678	2662
q4	1996	1745	1711	1711
q5	5703	5680	5863	5680
q6	235	137	141	137
q7	2275	1891	1860	1860
q8	3309	3419	3530	3419
q9	8882	8882	8853	8853
q10	3690	3454	3336	3336
q11	603	519	532	519
q12	838	684	713	684
q13	16270	3319	3305	3305
q14	323	288	282	282
q15	535	490	512	490
q16	628	586	596	586
q17	1862	1577	1515	1515
q18	8247	7925	7895	7895
q19	1756	1735	1529	1529
q20	2166	1933	1963	1933
q21	5756	5563	5292	5292
q22	1144	1083	1059	1059
Total cold run time: 73890 ms
Total hot run time: 57309 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193593 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9fdfda03d0d5fe0d853cd116d08eb591bd2d66a6, data reload: false

query1	1267	901	924	901
query2	6465	1968	1898	1898
query3	10633	3987	3882	3882
query4	59534	23853	23271	23271
query5	5440	507	516	507
query6	410	160	164	160
query7	5820	298	297	297
query8	330	229	225	225
query9	8825	2522	2512	2512
query10	485	281	272	272
query11	18119	14932	15191	14932
query12	154	113	105	105
query13	1602	428	420	420
query14	11133	7890	7440	7440
query15	238	166	179	166
query16	7024	472	490	472
query17	1142	591	647	591
query18	1228	309	303	303
query19	301	162	166	162
query20	125	113	113	113
query21	206	111	108	108
query22	4613	4483	4534	4483
query23	35453	34049	33633	33633
query24	5931	2874	2790	2790
query25	482	393	389	389
query26	672	156	150	150
query27	1760	277	278	277
query28	3949	2063	2045	2045
query29	627	414	406	406
query30	236	156	155	155
query31	945	809	787	787
query32	80	54	57	54
query33	467	286	272	272
query34	882	484	461	461
query35	863	738	720	720
query36	1068	947	943	943
query37	152	94	89	89
query38	4071	3872	3878	3872
query39	1432	1395	1380	1380
query40	193	113	116	113
query41	48	46	43	43
query42	113	94	95	94
query43	507	456	473	456
query44	1110	759	744	744
query45	199	165	162	162
query46	1086	786	731	731
query47	1940	1802	1853	1802
query48	382	296	291	291
query49	803	437	451	437
query50	842	411	409	409
query51	7117	6901	6887	6887
query52	97	87	85	85
query53	249	180	181	180
query54	564	456	468	456
query55	78	73	73	73
query56	276	248	265	248
query57	1176	1095	1071	1071
query58	219	228	233	228
query59	3099	2849	2868	2849
query60	291	264	270	264
query61	104	99	102	99
query62	737	643	669	643
query63	216	192	185	185
query64	2897	684	663	663
query65	3209	3141	3157	3141
query66	626	342	335	335
query67	15511	15379	15390	15379
query68	2947	590	589	589
query69	404	283	285	283
query70	1164	1129	1144	1129
query71	338	284	269	269
query72	5812	4228	3976	3976
query73	758	331	322	322
query74	9289	8807	8953	8807
query75	3373	2680	2710	2680
query76	1361	961	1013	961
query77	562	324	332	324
query78	10788	9360	9192	9192
query79	1047	533	591	533
query80	688	506	521	506
query81	468	246	231	231
query82	250	147	149	147
query83	169	146	148	146
query84	258	78	76	76
query85	669	278	282	278
query86	300	294	306	294
query87	4336	4426	4281	4281
query88	3003	2379	2392	2379
query89	396	296	287	287
query90	2001	188	187	187
query91	124	99	99	99
query92	62	52	50	50
query93	1060	532	544	532
query94	690	280	293	280
query95	353	252	251	251
query96	586	275	266	266
query97	3234	3077	3038	3038
query98	228	208	211	208
query99	1544	1283	1281	1281
Total cold run time: 306115 ms
Total hot run time: 193593 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9fdfda03d0d5fe0d853cd116d08eb591bd2d66a6, data reload: false

query1	0.04	0.05	0.04
query2	0.07	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.07	0.08
query5	0.52	0.51	0.51
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.55	0.50	0.49
query10	0.55	0.57	0.55
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	2.06	2.08	2.06
query15	0.85	0.83	0.83
query16	0.37	0.36	0.38
query17	1.06	1.04	1.05
query18	0.21	0.20	0.21
query19	1.88	1.88	1.75
query20	0.02	0.01	0.01
query21	15.39	0.67	0.67
query22	4.66	8.26	1.17
query23	18.27	1.25	1.23
query24	2.14	0.22	0.22
query25	0.15	0.09	0.08
query26	0.26	0.18	0.17
query27	0.07	0.08	0.08
query28	13.28	1.03	0.99
query29	12.64	3.35	3.34
query30	0.25	0.05	0.06
query31	2.88	0.39	0.40
query32	3.25	0.48	0.48
query33	2.97	2.96	3.02
query34	17.04	4.32	4.39
query35	4.44	4.48	4.39
query36	0.66	0.49	0.51
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.17	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 111.29 s
Total hot run time: 31.23 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.84% (9395/25503)
Line Coverage: 28.28% (77465/273951)
Region Coverage: 27.67% (39977/144464)
Branch Coverage: 24.31% (20341/83668)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9fdfda03d0d5fe0d853cd116d08eb591bd2d66a6_9fdfda03d0d5fe0d853cd116d08eb591bd2d66a6/report/index.html

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Sep 5, 2024

run buildall

Copy link
Contributor

github-actions bot commented Sep 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.84% (9395/25503)
Line Coverage: 28.27% (77438/273951)
Region Coverage: 27.67% (39975/144464)
Branch Coverage: 24.31% (20343/83668)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3f3d16eac8a58c8bff61c1bb53994f19f2e2662b_3f3d16eac8a58c8bff61c1bb53994f19f2e2662b/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 37891 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f3d16eac8a58c8bff61c1bb53994f19f2e2662b, data reload: false

------ Round 1 ----------------------------------
q1	17782	4412	4309	4309
q2	2029	191	179	179
q3	11789	946	1141	946
q4	10512	727	726	726
q5	7777	2858	2823	2823
q6	227	138	140	138
q7	957	604	609	604
q8	9522	2067	2066	2066
q9	7297	6526	6539	6526
q10	7001	2168	2227	2168
q11	458	247	242	242
q12	394	217	218	217
q13	17754	3087	3073	3073
q14	282	238	239	238
q15	533	499	491	491
q16	613	525	510	510
q17	981	635	747	635
q18	7310	6773	6871	6773
q19	1383	1011	1065	1011
q20	680	339	351	339
q21	3866	2920	2866	2866
q22	1097	1034	1011	1011
Total cold run time: 110244 ms
Total hot run time: 37891 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4372	4300	4254	4254
q2	385	281	265	265
q3	2932	2643	2662	2643
q4	1964	1711	1679	1679
q5	5623	5690	5730	5690
q6	232	140	134	134
q7	2269	1848	1840	1840
q8	3349	3439	3485	3439
q9	8811	8923	8874	8874
q10	3571	3704	3389	3389
q11	613	530	519	519
q12	838	650	668	650
q13	14091	3240	3286	3240
q14	313	315	288	288
q15	518	497	489	489
q16	631	581	576	576
q17	1846	1553	1514	1514
q18	8172	7903	8024	7903
q19	1730	1680	1445	1445
q20	2180	1911	2004	1911
q21	5840	5451	5575	5451
q22	1125	1037	1046	1037
Total cold run time: 71405 ms
Total hot run time: 57230 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192484 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f3d16eac8a58c8bff61c1bb53994f19f2e2662b, data reload: false

query1	1250	867	864	864
query2	6203	1869	1861	1861
query3	10610	4091	4004	4004
query4	59771	25395	23338	23338
query5	5241	511	493	493
query6	402	183	161	161
query7	5763	307	294	294
query8	309	222	211	211
query9	8944	2494	2489	2489
query10	507	275	262	262
query11	17948	15019	15308	15019
query12	151	102	102	102
query13	1589	420	400	400
query14	10962	7374	7497	7374
query15	223	175	193	175
query16	7107	436	467	436
query17	1115	574	579	574
query18	1972	299	299	299
query19	283	152	152	152
query20	121	111	116	111
query21	205	104	105	104
query22	4559	4359	4399	4359
query23	34292	33673	33009	33009
query24	5965	2937	2850	2850
query25	540	404	398	398
query26	653	152	151	151
query27	1733	273	276	273
query28	3886	2075	2054	2054
query29	646	403	406	403
query30	232	156	155	155
query31	926	765	760	760
query32	75	52	57	52
query33	441	287	290	287
query34	881	476	464	464
query35	839	729	730	729
query36	1063	933	928	928
query37	144	90	87	87
query38	4045	3872	3910	3872
query39	1461	1383	1383	1383
query40	200	119	115	115
query41	50	49	47	47
query42	116	98	99	98
query43	506	479	469	469
query44	1074	753	741	741
query45	201	172	167	167
query46	1098	746	739	739
query47	1912	1810	1808	1808
query48	391	306	305	305
query49	779	464	467	464
query50	849	413	425	413
query51	7060	6857	6960	6857
query52	101	89	89	89
query53	253	189	187	187
query54	577	461	472	461
query55	79	78	79	78
query56	287	270	297	270
query57	1200	1060	1069	1060
query58	233	238	244	238
query59	3014	2779	2687	2687
query60	304	280	280	280
query61	121	118	117	117
query62	751	669	650	650
query63	218	189	189	189
query64	2456	753	749	749
query65	3223	3133	3224	3133
query66	682	348	370	348
query67	15288	15111	15030	15030
query68	3332	606	575	575
query69	402	279	288	279
query70	1137	1148	1089	1089
query71	355	290	281	281
query72	6452	4223	4146	4146
query73	753	324	331	324
query74	9296	8778	8824	8778
query75	3381	2645	2662	2645
query76	1571	974	1011	974
query77	592	316	330	316
query78	9801	9004	9061	9004
query79	2755	541	550	541
query80	1150	508	496	496
query81	569	242	236	236
query82	605	144	152	144
query83	178	148	150	148
query84	261	78	75	75
query85	912	312	282	282
query86	423	289	302	289
query87	4545	4308	4296	4296
query88	3917	2375	2363	2363
query89	383	288	289	288
query90	1847	191	187	187
query91	120	96	97	96
query92	65	48	48	48
query93	2510	545	535	535
query94	842	303	311	303
query95	353	255	259	255
query96	607	269	276	269
query97	3163	3029	3116	3029
query98	213	201	197	197
query99	1571	1270	1275	1270
Total cold run time: 310141 ms
Total hot run time: 192484 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.38 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f3d16eac8a58c8bff61c1bb53994f19f2e2662b, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.65	0.08	0.08
query5	0.50	0.50	0.51
query6	1.18	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.55	0.58	0.55
query11	0.15	0.12	0.11
query12	0.16	0.13	0.12
query13	0.60	0.59	0.59
query14	2.07	2.14	2.05
query15	0.85	0.82	0.84
query16	0.36	0.38	0.37
query17	1.06	1.04	1.07
query18	0.20	0.20	0.20
query19	1.97	1.83	1.81
query20	0.02	0.01	0.01
query21	15.38	0.67	0.67
query22	4.24	7.25	2.13
query23	18.32	1.36	1.27
query24	2.15	0.22	0.22
query25	0.14	0.08	0.08
query26	0.28	0.17	0.18
query27	0.08	0.08	0.08
query28	13.25	1.01	1.00
query29	12.57	3.34	3.33
query30	0.24	0.06	0.05
query31	2.87	0.40	0.40
query32	3.27	0.47	0.48
query33	2.94	3.03	2.98
query34	17.18	4.39	4.40
query35	4.48	4.44	4.46
query36	0.65	0.48	0.46
query37	0.19	0.17	0.15
query38	0.16	0.14	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 111.08 s
Total hot run time: 32.38 s

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Sep 5, 2024

run buildall

Copy link
Contributor

github-actions bot commented Sep 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38844 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8492f6425acadaceae6ca5be9c88dcc914b9fb8b, data reload: false

------ Round 1 ----------------------------------
q1	18185	4610	4433	4433
q2	2621	190	186	186
q3	11009	1158	1141	1141
q4	10497	774	726	726
q5	7777	2983	2876	2876
q6	233	143	145	143
q7	982	641	651	641
q8	9557	2106	2086	2086
q9	7122	6568	6559	6559
q10	6991	2210	2234	2210
q11	451	242	240	240
q12	394	230	228	228
q13	17755	3062	3066	3062
q14	284	246	247	246
q15	538	509	482	482
q16	518	453	440	440
q17	971	749	802	749
q18	7427	7012	6934	6934
q19	1398	1080	1133	1080
q20	676	342	341	341
q21	4066	3204	3053	3053
q22	1088	988	1007	988
Total cold run time: 110540 ms
Total hot run time: 38844 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4311	4291	4296	4291
q2	372	281	262	262
q3	2911	2679	2660	2660
q4	1925	1667	1664	1664
q5	5403	5411	5387	5387
q6	229	133	133	133
q7	2175	1721	1768	1721
q8	3230	3406	3363	3363
q9	8492	8499	8461	8461
q10	3450	3187	3217	3187
q11	605	516	512	512
q12	789	611	618	611
q13	8106	3058	3043	3043
q14	305	270	274	270
q15	531	476	492	476
q16	519	497	474	474
q17	1816	1496	1485	1485
q18	7690	7396	7358	7358
q19	1699	1530	1644	1530
q20	2091	1826	1814	1814
q21	5427	5338	5156	5156
q22	1108	1042	1026	1026
Total cold run time: 63184 ms
Total hot run time: 54884 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187247 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8492f6425acadaceae6ca5be9c88dcc914b9fb8b, data reload: false

query1	919	377	375	375
query2	6484	1914	1857	1857
query3	6652	207	222	207
query4	27838	23091	23136	23091
query5	4160	508	505	505
query6	252	165	168	165
query7	4575	315	304	304
query8	274	217	231	217
query9	8599	2474	2477	2474
query10	449	290	278	278
query11	17885	15016	15108	15016
query12	150	101	106	101
query13	1629	382	369	369
query14	9852	7514	7061	7061
query15	263	166	185	166
query16	8120	460	472	460
query17	1593	574	565	565
query18	2150	316	280	280
query19	183	141	145	141
query20	117	112	106	106
query21	211	107	98	98
query22	4363	4085	4274	4085
query23	33772	33253	33207	33207
query24	11222	2872	2886	2872
query25	624	371	382	371
query26	1301	155	158	155
query27	2901	274	274	274
query28	7727	2054	2058	2054
query29	875	408	402	402
query30	310	159	148	148
query31	975	754	793	754
query32	99	55	54	54
query33	758	288	287	287
query34	989	480	476	476
query35	842	733	717	717
query36	1071	922	896	896
query37	154	84	77	77
query38	3929	3954	3837	3837
query39	1452	1397	1383	1383
query40	273	112	115	112
query41	48	47	47	47
query42	137	96	99	96
query43	501	465	460	460
query44	1285	763	747	747
query45	199	163	169	163
query46	1093	718	735	718
query47	1870	1754	1781	1754
query48	375	305	299	299
query49	1088	430	432	430
query50	820	408	422	408
query51	7061	6875	6889	6875
query52	96	84	86	84
query53	253	182	183	182
query54	905	453	452	452
query55	75	77	76	76
query56	286	250	255	250
query57	1211	1064	1059	1059
query58	237	229	232	229
query59	2928	2768	2746	2746
query60	330	262	264	262
query61	100	97	99	97
query62	829	658	646	646
query63	222	182	185	182
query64	5245	683	653	653
query65	3217	3140	3153	3140
query66	1436	354	387	354
query67	15479	15348	15410	15348
query68	3571	588	581	581
query69	385	274	291	274
query70	1165	1079	1127	1079
query71	341	281	276	276
query72	6267	4161	3976	3976
query73	752	333	329	329
query74	9131	8867	8837	8837
query75	3367	2679	2635	2635
query76	1944	1006	962	962
query77	456	304	320	304
query78	9962	9027	8976	8976
query79	1041	530	522	522
query80	695	498	499	498
query81	449	234	231	231
query82	236	141	142	141
query83	170	154	147	147
query84	223	79	74	74
query85	730	287	269	269
query86	302	297	294	294
query87	4371	4253	4487	4253
query88	2945	2354	2349	2349
query89	391	291	283	283
query90	1965	195	197	195
query91	132	103	103	103
query92	63	51	49	49
query93	1031	538	533	533
query94	687	305	296	296
query95	344	265	253	253
query96	585	274	265	265
query97	3156	3065	3108	3065
query98	233	196	205	196
query99	1475	1248	1269	1248
Total cold run time: 281883 ms
Total hot run time: 187247 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.5 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8492f6425acadaceae6ca5be9c88dcc914b9fb8b, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.05
query3	0.23	0.04	0.04
query4	1.68	0.08	0.08
query5	0.53	0.50	0.51
query6	1.14	0.74	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.54	0.50	0.49
query10	0.55	0.57	0.52
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	1.39	1.42	1.44
query15	0.85	0.80	0.83
query16	0.35	0.38	0.38
query17	1.03	1.08	0.97
query18	0.21	0.20	0.22
query19	1.91	1.82	1.87
query20	0.02	0.01	0.01
query21	15.43	0.66	0.66
query22	4.65	6.50	2.03
query23	18.25	1.38	1.33
query24	2.07	0.24	0.22
query25	0.15	0.08	0.08
query26	0.26	0.18	0.17
query27	0.07	0.07	0.08
query28	13.23	1.01	1.01
query29	12.63	3.34	3.28
query30	0.24	0.05	0.06
query31	2.88	0.40	0.39
query32	3.25	0.48	0.48
query33	2.98	3.01	2.95
query34	17.03	4.40	4.43
query35	4.44	4.41	4.44
query36	0.66	0.48	0.48
query37	0.19	0.15	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.04	0.04
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.5 s
Total hot run time: 31.5 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.78% (9381/25507)
Line Coverage: 28.21% (77350/274190)
Region Coverage: 27.62% (39939/144585)
Branch Coverage: 24.27% (20328/83754)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8492f6425acadaceae6ca5be9c88dcc914b9fb8b_8492f6425acadaceae6ca5be9c88dcc914b9fb8b/report/index.html

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.90% (9456/25629)
Line Coverage: 28.26% (77808/275306)
Region Coverage: 27.66% (40156/145195)
Branch Coverage: 24.27% (20405/84084)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bcc07001b56b1730fa62848bc9c748933768348a_bcc07001b56b1730fa62848bc9c748933768348a/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 39815 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bcc07001b56b1730fa62848bc9c748933768348a, data reload: false

------ Round 1 ----------------------------------
q1	17640	4289	4295	4289
q2	2039	186	181	181
q3	10480	1289	1380	1289
q4	10351	980	1020	980
q5	7729	3137	3139	3137
q6	227	145	140	140
q7	1057	636	634	634
q8	9454	1998	2014	1998
q9	6823	6313	6316	6313
q10	7009	2488	2460	2460
q11	431	252	251	251
q12	408	240	238	238
q13	17763	3033	3105	3033
q14	286	245	251	245
q15	542	505	481	481
q16	512	427	418	418
q17	969	940	941	940
q18	7394	6695	6941	6695
q19	1381	1231	1239	1231
q20	596	343	333	333
q21	3916	3549	3551	3549
q22	1115	1041	980	980
Total cold run time: 108122 ms
Total hot run time: 39815 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4218	4257	4183	4183
q2	340	240	239	239
q3	2926	2897	2886	2886
q4	1989	1975	2028	1975
q5	5762	5734	5782	5734
q6	232	139	141	139
q7	2155	1781	1779	1779
q8	3365	3374	3404	3374
q9	8844	8840	8842	8840
q10	3536	3595	3652	3595
q11	596	499	486	486
q12	818	614	633	614
q13	10810	3279	3100	3100
q14	337	283	266	266
q15	553	493	505	493
q16	501	487	461	461
q17	1831	1753	1751	1751
q18	8580	7867	8145	7867
q19	1766	1750	1754	1750
q20	2164	1913	1905	1905
q21	5865	5680	5659	5659
q22	1156	1024	1026	1024
Total cold run time: 68344 ms
Total hot run time: 58120 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199326 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bcc07001b56b1730fa62848bc9c748933768348a, data reload: false

query1	1251	886	905	886
query2	6238	1749	1760	1749
query3	10807	3906	3999	3906
query4	55858	25315	24020	24020
query5	5120	562	541	541
query6	352	173	171	171
query7	5675	320	305	305
query8	292	234	223	223
query9	6100	2452	2463	2452
query10	416	292	275	275
query11	15861	15488	15479	15479
query12	157	112	102	102
query13	1445	392	375	375
query14	10241	6880	7374	6880
query15	204	178	184	178
query16	7009	527	481	481
query17	1172	590	602	590
query18	1882	313	299	299
query19	202	155	156	155
query20	136	111	114	111
query21	212	109	106	106
query22	4816	4416	4464	4416
query23	34549	34149	33505	33505
query24	6209	3096	3108	3096
query25	507	388	380	380
query26	608	156	155	155
query27	1580	282	283	282
query28	2789	2017	1998	1998
query29	663	427	421	421
query30	223	150	151	150
query31	943	738	797	738
query32	78	53	57	53
query33	442	308	301	301
query34	899	480	491	480
query35	850	743	737	737
query36	1063	913	939	913
query37	144	93	86	86
query38	4122	3902	3912	3902
query39	1472	1417	1408	1408
query40	203	113	113	113
query41	46	45	46	45
query42	123	98	99	98
query43	463	446	418	418
query44	1193	768	749	749
query45	197	174	172	172
query46	1089	862	832	832
query47	1897	1790	1796	1790
query48	357	292	286	286
query49	727	456	448	448
query50	959	453	446	446
query51	6986	6925	6944	6925
query52	103	90	87	87
query53	254	189	187	187
query54	567	466	473	466
query55	79	79	78	78
query56	293	270	284	270
query57	1208	1061	1086	1061
query58	233	243	248	243
query59	2725	2459	2381	2381
query60	322	285	304	285
query61	105	101	98	98
query62	800	687	673	673
query63	222	193	195	193
query64	1633	679	676	676
query65	3252	3180	3215	3180
query66	678	341	338	338
query67	15892	15447	15505	15447
query68	1873	869	848	848
query69	427	340	340	340
query70	1170	1137	1144	1137
query71	354	353	359	353
query72	4703	3583	3541	3541
query73	589	581	587	581
query74	9106	9241	9062	9062
query75	3113	3017	3055	3017
query76	959	887	873	873
query77	498	431	434	431
query78	9446	9316	9381	9316
query79	936	889	886	886
query80	869	884	857	857
query81	444	269	274	269
query82	273	271	276	271
query83	202	195	312	195
query84	209	151	109	109
query85	649	421	412	412
query86	324	330	316	316
query87	4490	4353	4416	4353
query88	4088	4024	4008	4008
query89	380	369	374	369
query90	1400	317	311	311
query91	125	128	129	128
query92	82	80	79	79
query93	1033	1034	1033	1033
query94	551	364	404	364
query95	453	443	436	436
query96	471	470	473	470
query97	3127	3145	3124	3124
query98	244	231	234	231
query99	1382	1323	1316	1316
Total cold run time: 289163 ms
Total hot run time: 199326 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.14 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bcc07001b56b1730fa62848bc9c748933768348a, data reload: false

query1	0.04	0.05	0.04
query2	0.07	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.07	0.06
query5	0.49	0.50	0.50
query6	1.14	0.72	0.72
query7	0.02	0.02	0.02
query8	0.06	0.05	0.04
query9	0.56	0.51	0.50
query10	0.58	0.58	0.55
query11	0.16	0.12	0.12
query12	0.15	0.13	0.13
query13	0.62	0.60	0.60
query14	1.44	1.46	1.46
query15	0.89	0.88	0.87
query16	0.35	0.37	0.37
query17	1.04	1.04	1.00
query18	0.16	0.17	0.17
query19	1.90	1.88	1.86
query20	0.01	0.01	0.01
query21	15.40	0.68	0.68
query22	3.86	7.35	1.35
query23	18.02	1.30	1.31
query24	2.26	0.23	0.22
query25	0.19	0.08	0.07
query26	0.28	0.18	0.18
query27	0.07	0.08	0.08
query28	13.16	1.01	0.99
query29	12.57	3.36	3.36
query30	0.24	0.06	0.06
query31	2.89	0.42	0.42
query32	3.24	0.49	0.50
query33	3.01	3.07	3.06
query34	15.46	4.32	4.33
query35	4.37	4.37	4.35
query36	0.69	0.49	0.49
query37	0.20	0.17	0.16
query38	0.16	0.15	0.16
query39	0.04	0.05	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.05	0.04
Total cold run time: 108.06 s
Total hot run time: 31.14 s

@hust-hhb hust-hhb changed the title [fix](cloud-mow) compaciton should not use the same lock id when geting mow table lock from ms [fix](cloud-mow) Fix the issue of compaction holding mow table lock for a long time Sep 12, 2024
@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.89% (9462/25647)
Line Coverage: 28.25% (77785/275352)
Region Coverage: 27.66% (40175/145241)
Branch Coverage: 24.28% (20418/84110)
Coverage Report: http://coverage.selectdb-in.cc/coverage/39693808c042a34b6fd2621a5230b0c97dd110ec_39693808c042a34b6fd2621a5230b0c97dd110ec/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 42821 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39693808c042a34b6fd2621a5230b0c97dd110ec, data reload: false

------ Round 1 ----------------------------------
q1	17597	7297	7228	7228
q2	2058	187	182	182
q3	10468	1240	1376	1240
q4	10403	954	1009	954
q5	7712	3189	3105	3105
q6	246	158	158	158
q7	1033	650	641	641
q8	9459	2016	2010	2010
q9	6792	6320	6283	6283
q10	6997	2574	2553	2553
q11	440	260	253	253
q12	416	237	231	231
q13	17775	3001	3016	3001
q14	289	247	261	247
q15	574	541	534	534
q16	523	423	425	423
q17	1015	960	941	941
q18	7327	6745	6959	6745
q19	1382	1250	1243	1243
q20	614	343	332	332
q21	3899	3629	3548	3548
q22	1032	969	983	969
Total cold run time: 108051 ms
Total hot run time: 42821 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7231	7853	7209	7209
q2	355	246	243	243
q3	3065	3119	3023	3023
q4	2094	2113	2031	2031
q5	5740	5604	5688	5604
q6	241	151	151	151
q7	2156	1771	1806	1771
q8	3380	3418	3396	3396
q9	8791	8899	8830	8830
q10	3451	3591	3587	3587
q11	563	480	471	471
q12	785	613	665	613
q13	10304	3220	3199	3199
q14	310	285	302	285
q15	594	519	556	519
q16	545	489	483	483
q17	1804	1752	1788	1752
q18	8533	7895	8055	7895
q19	1778	1746	1748	1746
q20	2141	1894	1878	1878
q21	6028	5573	5771	5573
q22	1135	1025	1028	1025
Total cold run time: 71024 ms
Total hot run time: 61284 ms

@hust-hhb
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.41% (9661/25827)
Line Coverage: 28.69% (80179/279490)
Region Coverage: 28.11% (41452/147456)
Branch Coverage: 24.71% (21109/85420)
Coverage Report: http://coverage.selectdb-in.cc/coverage/70aa3ccfd0587cdf4ed04a18ee4400f3c8085324_70aa3ccfd0587cdf4ed04a18ee4400f3c8085324/report/index.html

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants