Manticore crashes on distributed index

[keepalot]

Bug Description:

Crash happens when executing query with many results and does not crash on queries with few results

------- FATAL: CRASH DUMP -------
[Thu Dec  5 13:55:21.523 2024] [2464571]

--- crashed SphinxAPI request dump ---
AAABJgAAARgAAAAVAAAAAQAAAEAAAAAAAAAD6AAAAAYAAAAAAAAABAAAAAxAd2VpZ2h0IGRlc2MAAAAGaXBob25l
AAAAAAAAAAtvZmZlcl9sb2NhbAAAAAEAAAAAAAAAAP//////////AAAAAAAAAAQAAAAAAAAD6AAAAA1A
Z3JvdXBieSBkZXNj/////wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJpZAAA
AAAAAAAAAAAAAAAAAAD/////AAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAA
AAJpZAAAAAJpZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
--- request dump end ---
--- local index:offer_main_08
Manticore 6.3.8 d17bd2b6b@24112202 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 16.0.6
Configured with flags: Configured with these definitions: -DDISTR_BUILD=rhel9 -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmariadb.so.3 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (rhel9) (cross-compiled)
Stack bottom = 0x7f8490099a60, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x20000)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x20000, stack=0x7f84900a0000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x227)[0x565487795ec7]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x364)[0x56548760afe4]
/lib64/libc.so.6(+0x3e730)[0x7f85a683e730]
/usr/bin/searchd(_ZNK13CSphIndex_VLN13MatchExtendedILb1ELb0ELb0ELb0ELb0ELb0ELb0EEEvR16CSphQueryContextRK9CSphQueryRK11VecTraits_TIP15ISphMatchSorterEP10ISphRankeriii+0xdf)[0x56548775a12f]
/usr/bin/searchd(_ZNK13CSphIndex_VLN16ParsedMultiQueryERK9CSphQueryR15CSphQueryResultRK11VecTraits_TIP15ISphMatchSorterERK9XQQuery_t17CSphRefcountedPtrI8CSphDictERK18CSphMultiQueryArgsP18CSphQueryNodeCachel+0x1c7e)[0x5654876c465e]
/usr/bin/searchd(+0xfc4cb1)[0x56548773bcb1]
/usr/bin/searchd(_ZNK13CSphIndex_VLN19RunParsedMultiQueryEiR17CSphRefcountedPtrI8CSphDictEbRK9CSphQueryR15CSphQueryResultR11VecTraits_TIP15ISphMatchSorterERK9XQQuery_tRK18CSphMultiQueryArgsl+0x103)[0x5654876bf7d3]
/usr/bin/searchd(_ZNK13CSphIndex_VLN10MultiQueryER15CSphQueryResultRK9CSphQueryRK11VecTraits_TIP15ISphMatchSorterERK18CSphMultiQueryArgs+0x1973)[0x5654876c1183]
/usr/bin/searchd(_ZNK13CSphIndex_VLN12MultiQueryExEiPK9CSphQueryP15CSphQueryResultPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x53)[0x5654876c1f53]
/usr/bin/searchd(+0xef9e70)[0x565487670e70]
/usr/bin/searchd(+0x1243cb2)[0x5654879bacb2]
/usr/bin/searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_8__invokeESB_+0x1c)[0x5654887b2d4c]
/usr/bin/searchd(make_fcontext+0x2f)[0x5654887f4c3f]
Trying boost backtrace:
 0# sphBacktrace(int, bool) in /usr/bin/searchd
 1# CrashLogger::HandleCrash(int) in /usr/bin/searchd
 2# 0x00007F85A683E730 in /lib64/libc.so.6
 3# void CSphIndex_VLN::MatchExtended<true, false, false, false, false, false, false>(CSphQueryContext&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, ISphRanker*, int, int, int) const in /usr/bin/searchd
 4# CSphIndex_VLN::ParsedMultiQuery(CSphQuery const&, CSphQueryResult&, VecTraits_T<ISphMatchSorter*> const&, XQQuery_t const&, CSphRefcountedPtr<CSphDict>, CSphMultiQueryArgs const&, CSphQueryNodeCache*, long) const in /usr/bin/searchd
 5# 0x000056548773BCB1 in /usr/bin/searchd
 6# CSphIndex_VLN::RunParsedMultiQuery(int, CSphRefcountedPtr<CSphDict>&, bool, CSphQuery const&, CSphQueryResult&, VecTraits_T<ISphMatchSorter*>&, XQQuery_t const&, CSphMultiQueryArgs const&, long) const in /usr/bin/searchd
 7# CSphIndex_VLN::MultiQuery(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&) const in /usr/bin/searchd
 8# CSphIndex_VLN::MultiQueryEx(int, CSphQuery const*, CSphQueryResult*, ISphMatchSorter**, CSphMultiQueryArgs const&) const in /usr/bin/searchd
 9# 0x0000565487670E70 in /usr/bin/searchd
10# 0x00005654879BACB2 in /usr/bin/searchd
11# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd
12# make_fcontext in /usr/bin/searchd

-------------- backtrace ends here ---------------

Manticore Search Version:

Manticore 6.3.8 d17bd2b6b@24112202 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)

Operating System Version:

CentOS Stream release 9

Have you tried the latest development version?

No

[sanikolaev]

Hello @keepalot

Crash happens when executing query with many results

Can you please share your table files and the query which leads to the crash? You can upload to our write-only S3: https://manual.manticoresearch.com/dev/Reporting_bugs#Uploading-your-data

[keepalot]

Index files have been uploaded to S3

Config file

table offer_main {
    type                = plain
    path                = /data_disk/manticore/offer_main_00
}
searchd {
    listen              = 9306:mysql
    log                 = /var/log/manticore/searchd.log
    pid_file            = /var/run/manticore/searchd.pid
    pseudo_sharding     = 0
}

Simple queries do work like:

SELECT * FROM offer_main WHERE usd_price > 10;

But crashes on MATCH() queries like:

SELECT * FROM offer_main WHERE MATCH('elena');

Setting pseudo_sharding = 1 all queries do work without crashing

[sanikolaev]

Thank you for sharing the files @keepalot

We'll look into the issue.