Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: CN lost connection #9647

Closed
1 task done
fengttt opened this issue May 23, 2023 · 14 comments
Closed
1 task done

[Bug]: CN lost connection #9647

fengttt opened this issue May 23, 2023 · 14 comments
Assignees
Labels
area/storage impact/1.1 kind/bug Something isn't working long-term-job phase/testing severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Milestone

Comments

@fengttt
Copy link
Contributor

fengttt commented May 23, 2023

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Environment

- Version or commit-id (e.g. v0.1.0 or 8b23a93):
- Hardware parameters:
- OS type:
- Others:

Actual Behavior

Running same test in #9646, but with a better machine (32G to 64G memory), you will see cn lost connection.

ERROR 20503 (HY000) at line 40: stream closed

Expected Behavior

No response

Steps to Reproduce

No response

Additional information

No response

@volgariver6
Copy link
Contributor

in local test:

create database if not exists db1;

use db1

drop table if exists t;

create table t (i int, j int);

insert into t values (1, 1), (2, 2), (3, 3), (4, 4), (5, null), (null, 5);

insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
select count(*) from t;

insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
select count(*) from t;

insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
select count(*) from t;

insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
select count(*) from t;

insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
select count(*) from t;

delete from t where i = 1;
delete from t where i = 2;
select count(*) from t;

insert into t select * from t;
select count(*) from t;

above script will result in:
2023/05/24 12:21:17.672100 +0800 ERROR logservicedriver/appender.go:70 append failed: internal error: message body 116720140 is too large, max is 104857600

@volgariver6
Copy link
Contributor

dup with #9447

@volgariver6
Copy link
Contributor

该问题的分析见#9447 (comment)

后续由 @triump2020 再做一些优化。

@volgariver6
Copy link
Contributor

ERROR 20503 (HY000) at line 40: stream closed

出现该错误或者其他的连接关闭的错误的原因是,rpc框架中的gc任务会检查每个连接的活跃状态,当超过一定时间(默认1分钟)没有数据时,就会关闭这个连接。

delete语句的commit时间长,导致其连接被gc给close了,所以执行失败,需要优化。

@triump2020
Copy link
Contributor

depends on #9996

@XuPeng-SH
Copy link
Contributor

@jiangxinmeng1 delete related issue

@XuPeng-SH
Copy link
Contributor

#10418 cannot fix this issue. This depends on relatively significant refactoring.

@sukki37 sukki37 added severity/s0 Extreme impact: Cause the application to break down and seriously affect the use priority/p0.5 labels Jul 31, 2023
@sukki37 sukki37 modified the milestones: 1.0.0, 1.1.0 Aug 18, 2023
@jiangxinmeng1
Copy link
Contributor

还没有进展

@XuPeng-SH XuPeng-SH assigned triump2020 and unassigned jiangxinmeng1 Oct 9, 2023
@triump2020
Copy link
Contributor

depends on V1.1

@triump2020
Copy link
Contributor

depends on V1.1

@XuPeng-SH XuPeng-SH added the duplicate This issue or pull request already exists label Oct 23, 2023
@XuPeng-SH
Copy link
Contributor

it depends on #11805 #11804

@sukki37 sukki37 removed the duplicate This issue or pull request already exists label Oct 23, 2023
@sukki37 sukki37 modified the milestones: 1.1.0, 1.2.0 Nov 20, 2023
@aressu1985 aressu1985 modified the milestones: 1.2.0, 1.3.0-Backlog Apr 24, 2024
@XuPeng-SH
Copy link
Contributor

XuPeng-SH commented Jun 26, 2024

depends on #11471

@jiangxinmeng1
Copy link
Contributor

depends on #11805 #11804

@sukki37 sukki37 modified the milestones: 1.3.0-Backlog, 1.3.0 Jul 1, 2024
@XuPeng-SH
Copy link
Contributor

fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/storage impact/1.1 kind/bug Something isn't working long-term-job phase/testing severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Projects
None yet
Development

No branches or pull requests

9 participants