Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

performance regression after update to the newest master commit #650

Closed
glorv opened this issue May 22, 2024 · 2 comments
Closed

performance regression after update to the newest master commit #650

glorv opened this issue May 22, 2024 · 2 comments

Comments

@glorv
Copy link
Contributor

glorv commented May 22, 2024

Describe the bug
Recent I was trying to update the grpcio version from v0.10.4 to the newest master commit for tikv. And when I did some benchmark using sysbench I found there are performance regression within different workloads(oltp_read_only, oltp_read_write and oltp_write_only), the qps dropped around 5% ~ 10%.
The v0.10.4 build is based on commit: tikv/tikv@966e134
The master build is based on: https://github.com/glorv/tikv/commits/grpc-master/

From the metrics I saw master grpc threads consumed more cpu time than v0.10.4:
20240522-172954

From the cpu profiling, I saw the cq_next call consumed more cpu time while other part are almost the same:
(v0.10.4)
image

(master)
image

With v0.10.4, grpc cost about 14.78% of all cpu time:
CallTag::resolve(4.12%), CompletionQueue::next(1.45%), UnfinishedWork::finish(9.22%)

With master commit, gRPC cost about 21.35% of all cpu time:
CallTag::resolve(5.87%), CompletionQueue::next(4.77%), UnfinishedWork::finish(10.71%)

It seems that some changes in CompletionQueue may be the root cause of this regression.
To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

System information

  • CPU architecture:
  • Distribution and kernel version:
  • SELinux on?:
  • Any other system details we should know?:

Additional context
Add any other context about the problem here.

@glorv
Copy link
Contributor Author

glorv commented May 22, 2024

/cc @Connor1996 @overvenus @LykxSassinator @SpadeA-Tang
Please help take a look if you have any ideas

@glorv
Copy link
Contributor Author

glorv commented May 22, 2024

Seems this is duplicated with #643

@glorv glorv closed this as completed May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant