Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

posix_data_source_impl: adaptive prefetch size #4

Open
wants to merge 1 commit into
base: ceph-octopus
Choose a base branch
from
Open

posix_data_source_impl: adaptive prefetch size #4

wants to merge 1 commit into from

Conversation

cyx1231st
Copy link
Member

@cyx1231st cyx1231st commented Apr 29, 2019 •
edited
Loading

4K & depth=1 case will be a little slower because ceph::net::posix_data_source_impl will always allocate 64K sized buffer for prefetch (https://github.com/ceph/seastar/blob/master/net/posix-stack.cc#L286), but there are at most 4K data on the wire if the depth is 1. And it isn't common case. For other cases, the performance will be notably better, because of larger prefetched buffer.

Implemented adaptive prefetch buffer based on last read size:

  • No magic number needed to be configured any more.
  • Won't waste memory space when there are not that much to prefetch.
  • Can have bigger prefetch buffer instantly whenever needed.

2-way 4K read/write

saturated connection (throughput):
$perf_crimson_msgr --round=4194304 --cbs=4096 --sbs=4096 --depth=512 -c 3

run 1st 2nd 3rd
fixed 8K (reference point) 18.4555s 18.5185s 18.4318s
fixed 64K 17.5711s 17.6602s 17.694s
adaptive 17.11s 17.21s 17.28s

depth=1 (latency):
$perf_crimson_msgr --round=1048576 --cbs=4096 --sbs=4096 --depth=1 -c 3

run 1st 2nd 3rd
fixed 8K (reference point) 20.48s 20.47s 20.64s
fixed 64K 21.3099s 21.2366s 21.2861s
adaptive 20.52s 20.28s 20.41s

2-way 1M read/write

saturated connection (throughput):
$perf_crimson_msgr --round=65536 --cbs=1048576 --sbs=1048576 --depth=512 -c 3

run 1st 2nd 3rd
fixed 8K (reference point) 24.7577s 24.8432s 24.8043s
fixed 64K 18.5113s 18.4522s 18.4962s
adaptive 17.92s 17.66s 17.83s

depth=1 (latency):
$perf_crimson_msgr --round=65536 --cbs=1048576 --sbs=1048576 --depth=1 -c 3

run 1st 2nd 3rd
fixed 8K (reference point) 31.0175s 30.9355s 30.8921s
fixed 64K 19.9777s 19.9308s 19.9492s
adaptive 19.73s 19.63s 19.68s

@cyx1231st cyx1231st mentioned this pull request Apr 29, 2019
Closed
@cyx1231st cyx1231st mentioned this pull request Apr 29, 2019
Closed
@cyx1231st cyx1231st changed the title posix_data_source_impl: increase buffer size posix_data_source_impl: adaptive prefetch size Apr 30, 2019
@cyx1231st
Copy link
Member Author

thanks! LGTM

@tchaikov
Copy link

tchaikov commented May 9, 2019

see https://docs.google.com/spreadsheets/d/19YfTwHol3f77A_5nMGxAPLDrm4c-nIvfKoJkQcShCYM/edit#gid=0. i think it's not a pure win.

could you help look at it?

@cyx1231st cyx1231st mentioned this pull request Jun 11, 2019
Open
cyx1231st pushed a commit to cyx1231st/seastar that referenced this pull request Jan 2, 2020
@avikivity
Revert "task: stop wrapping tasks with unique_ptr"
8fd8568 
This reverts commit 33406cf. It
introduces memory leaks:

Direct leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7fb773b389d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    ceph#1 0x108f0d4 in seastar::reactor::poller::~poller() ../src/core/reactor.cc:2879
    ceph#2 0x11c1e59 in std::experimental::fundamentals_v1::_Optional_base<seastar::reactor::poller, true>::~_Optional_base() /usr/include/c++/9/experimental/optional:288
    ceph#3 0x118f2d7 in std::experimental::fundamentals_v1::optional<seastar::reactor::poller>::~optional() /usr/include/c++/9/experimental/optional:491
    ceph#4 0x108c5a5 in seastar::reactor::run() ../src/core/reactor.cc:2587
    ceph#5 0xf1a822 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) ../src/core/app-template.cc:199
    ceph#6 0xf1885d in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) ../src/core/app-template.cc:115
    ceph#7 0xeb2735 in operator() ../src/testing/test_runner.cc:72
    ceph#8 0xebb342 in _M_invoke /usr/include/c++/9/bits/std_function.h:300
    ceph#9 0xf3d8b0 in std::function<void ()>::operator()() const /usr/include/c++/9/bits/std_function.h:690
    ceph#10 0x1034c72 in seastar::posix_thread::start_routine(void*) ../src/core/posix.cc:52
    ceph#11 0x7fb7738804e1 in start_thread /usr/src/debug/glibc-2.30-13-g919af705ee/nptl/pthread_create.c:479

Reported-by: Rafael Avila de Espindola <[email protected]>
@cyx1231st
Copy link
Member Author

Seastar community already merged the same idea in posix socket implementation, see 9e62799

I think this means #5 may probably be an easier path to be accepted by seastar, and for us to read aligned buffer faster in POSIX stack.

tchaikov pushed a commit that referenced this pull request May 13, 2021
@bhalevy @avikivity
test: closeable: allocate variables accessed by continuations using d…
4d69526 
…o_with

Fixes failures in debug mode:
```
$ build/debug/tests/unit/closeable_test -l all -t deferred_close_test
WARNING: debug mode. Not for benchmarking or production
random-seed=3064133628
Running 1 test case...
Entering test module "../../tests/unit/closeable_test.cc"
../../tests/unit/closeable_test.cc(0): Entering test case "deferred_close_test"
../../src/testing/seastar_test.cc(43): info: check true has passed
==9449==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
terminate called after throwing an instance of 'seastar::broken_promise'
  what():  broken promise
==9449==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fbf1f49f000; bottom 0x7fbf40971000; size: 0xffffffffdeb2e000 (-558702592)
False positive error reports may follow
For details see google/sanitizers#189
=================================================================
==9449==AddressSanitizer CHECK failed: ../../../../libsanitizer/asan/asan_thread.cpp:356 "((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0)
    #0 0x7fbf45f39d0b  (/lib64/libasan.so.6+0xb3d0b)
    #1 0x7fbf45f57d4e  (/lib64/libasan.so.6+0xd1d4e)
    #2 0x7fbf45f3e724  (/lib64/libasan.so.6+0xb8724)
    #3 0x7fbf45eb3e5b  (/lib64/libasan.so.6+0x2de5b)
    #4 0x7fbf45eb51e8  (/lib64/libasan.so.6+0x2f1e8)
    #5 0x7fbf45eb7694  (/lib64/libasan.so.6+0x31694)
    #6 0x7fbf45f39398  (/lib64/libasan.so.6+0xb3398)
    #7 0x7fbf45f3a00b in __asan_report_load8 (/lib64/libasan.so.6+0xb400b)
    #8 0xfe6d52 in bool __gnu_cxx::operator!=<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > >(__gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&, __gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&) /usr/include/c++/10/bits/stl_iterator.h:1116
    #9 0xfe615c in dl_iterate_phdr ../../src/core/exception_hacks.cc:121
    #10 0x7fbf44bd1810 in _Unwind_Find_FDE (/lib64/libgcc_s.so.1+0x13810)
    #11 0x7fbf44bcd897  (/lib64/libgcc_s.so.1+0xf897)
    #12 0x7fbf44bcea5f  (/lib64/libgcc_s.so.1+0x10a5f)
    #13 0x7fbf44bcefd8 in _Unwind_RaiseException (/lib64/libgcc_s.so.1+0x10fd8)
    #14 0xfe6281 in _Unwind_RaiseException ../../src/core/exception_hacks.cc:148
    scylladb#15 0x7fbf457364bb in __cxa_throw (/lib64/libstdc++.so.6+0xaa4bb)
    scylladb#16 0x7fbf45e10a21  (/lib64/libboost_unit_test_framework.so.1.73.0+0x1aa21)
    scylladb#17 0x7fbf45e20fe0 in boost::execution_monitor::execute(boost::function<int ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2afe0)
    scylladb#18 0x7fbf45e21094 in boost::execution_monitor::vexecute(boost::function<void ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2b094)
    scylladb#19 0x7fbf45e43921 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d921)
    scylladb#20 0x7fbf45e5eae1  (/lib64/libboost_unit_test_framework.so.1.73.0+0x68ae1)
    scylladb#21 0x7fbf45e5ed31  (/lib64/libboost_unit_test_framework.so.1.73.0+0x68d31)
    scylladb#22 0x7fbf45e2e547 in boost::unit_test::framework::run(unsigned long, bool) (/lib64/libboost_unit_test_framework.so.1.73.0+0x38547)
    scylladb#23 0x7fbf45e43618 in boost::unit_test::unit_test_main(bool (*)(), int, char**) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d618)
    scylladb#24 0x44798d in seastar::testing::entry_point(int, char**) ../../src/testing/entry_point.cc:77
    scylladb#25 0x4134b5 in main ../../include/seastar/testing/seastar_test.hh:65
    scylladb#26 0x7fbf44a1b1e1 in __libc_start_main (/lib64/libc.so.6+0x281e1)
    scylladb#27 0x4133dd in _start (/home/bhalevy/dev/seastar/build/debug/tests/unit/closeable_test+0x4133dd)
```

Signed-off-by: Benny Halevy <[email protected]>
Message-Id: <[email protected]>
tchaikov pushed a commit that referenced this pull request Nov 21, 2022
@syuu1228 @xemul
fsqual: stop causing memory leak error on LeakSanitizer
fe8fcd6 
When we enable the sanitizer, we get following error while running
iotune:

==86505==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 4096 byte(s) in 1 object(s) allocated from:
    #0 0x5701b8 in aligned_alloc (/home/syuu/seastar.2/build/sanitize/apps/iotune/iotune+0x5701b8) (BuildId: 411f9852d64ed8982d5b33d02489b5932d92b8b7)
    #1 0x6d0813 in seastar::filesystem_has_good_aio_support(seastar::basic_sstring<char, unsigned int, 15u, true>, bool) /home/syuu/seastar.2/src/core/fsqual.cc:74:16
    #2 0x5bcd0d in main::$_0::operator()() const::'lambda'()::operator()() const /home/syuu/seastar.2/apps/iotune/iotune.cc:742:21
    #3 0x5bb1f1 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/syuu/seastar.2/include/seastar/core/future.hh:2118:28
    #4 0x5bb039 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/syuu/seastar.2/include/seastar/core/thread.hh:258:13
    #5 0x5bb039 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/syuu/seastar.2/include/seastar/util/noncopyable_function.hh:124:20
    #6 0x8e0a77 in seastar::thread_context::main() /home/syuu/seastar.2/src/core/thread.cc:299:9
    #7 0x7f30ff8547bf  (/lib64/libc.so.6+0x547bf) (BuildId: 85c438f4ff93e21675ff174371c9c583dca00b2c)

SUMMARY: AddressSanitizer: 4096 byte(s) leaked in 1 allocation(s).

This is because we don't free buffer which allocated at filesystem_has_good_aio_support(), we should free it to avoid such error.

And this is needed to test Scylla machine image with debug mode binary,
since it tries to run iotune with the sanitizer and fails.

Closes scylladb#1284
Matan-B pushed a commit that referenced this pull request Jul 7, 2024
@tchaikov @xemul
apps/httpd: free allocated http_server_control
2b47c97 
in main(), we creates an instance of `http_server_control` using
new, but we never destroy it. this is identified by ASan

```
==2190125==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x55e21cf487bd in operator new(unsigned long) /home/kefu/dev/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:86:3
    #1 0x55e21cf6cf31 in main::$_0::operator()() const::'lambda'()::operator()() const /home/kefu/dev/seastar/apps/httpd/main.cc:121:27
    #2 0x55e21cf6b4cc in int std::__invoke_impl<int, main::$_0::operator()() const::'lambda'()>(std::__invoke_other, main::$_0::operator()() const::'lambda'()&&) /usr/lib/gcc/x86_64-redhat-linux/14/../../../../incl
ude/c++/14/bits/invoke.h:61:14
    #3 0x55e21cf6b46c in std::__invoke_result<main::$_0::operator()() const::'lambda'()>::type std::__invoke<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&) /usr/lib/gcc/x86_
64-redhat-linux/14/../../../../include/c++/14/bits/invoke.h:96:14
    #4 0x55e21cf6b410 in decltype(auto) std::__apply_impl<main::$_0::operator()() const::'lambda'(), std::tuple<>>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&, std::integer_sequence<unsigned long, .
..>) /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/tuple:2921:14
    #5 0x55e21cf6b3b2 in decltype(auto) std::apply<main::$_0::operator()() const::'lambda'(), std::tuple<>>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /usr/lib/gcc/x86_64-redhat-linux/14/../../../
../include/c++/14/tuple:2936:14
    #6 0x55e21cf6b283 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/kefu/dev/seastar/include/sea
star/core/future.hh:2005:28
    #7 0x55e21cf6b043 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0:
:operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/kefu/dev/seastar/include/seastar/core/thread.hh:260:13
    #8 0x55e21cf6ae74 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()()
 const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/kefu/dev/seastar/include/seastar/util/noncopyable
_function.hh:129:20
    #9 0x7f5d757a0fb3 in seastar::noncopyable_function<void ()>::operator()() const /home/kefu/dev/seastar/include/seastar/util/noncopyable_function.hh:215:16
    #10 0x7f5d75ef5611 in seastar::thread_context::main() /home/kefu/dev/seastar/src/core/thread.cc:311:9
    #11 0x7f5d75ef50eb in seastar::thread_context::s_main(int, int) /home/kefu/dev/seastar/src/core/thread.cc:287:43
    #12 0x7f5d72f8a18f  (/lib64/libc.so.6+0x5a18f) (BuildId: b098f1c75a76548bb230d8f551eae07a2aeccf06)
```

so, in this change, let's hold it using a smart pointer, so we
can destroy it when it leaves the lexical scope.

Signed-off-by: Kefu Chai <[email protected]>

Closes scylladb#2224
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rzarzynski rzarzynski Awaiting requested review from rzarzynski

@tchaikov tchaikov Awaiting requested review from tchaikov

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@cyx1231st @tchaikov