crash on s390x

[akostadinov]

Bug Description:

Hello. Manticore search crashes when run on s390x architecture. It is enough to insert a single index entry and then select *. The following is how you can run it. The image I've built is public (to run with docker and no selinux, use docker instead of podman and remove option --security-opt=label=disable).

# podman run --init --ulimit core=-1 --mount type=bind,source=/tmp/,target=/tmp/ --rm --security-opt=label=disable -p 9306:9306 quay.io/3scale/searchd:s390x638jammy --coredump
[Sun Dec 15 20:47:55.150 2024] [2] using config file '/etc/manticoresearch/manticore.conf' (5197 chars)...
[Sun Dec 15 20:47:55.152 2024] [2] Using local time zone '/etc/localtime'
[Sun Dec 15 20:47:55.152 2024] [2] starting daemon version '6.3.8 004fd4d0d@24112209' ...
starting daemon version '6.3.8 004fd4d0d@24112209' ...
[Sun Dec 15 20:47:55.152 2024] [2] listening on all interfaces for mysql, port=9306
listening on all interfaces for mysql, port=9306
Manticore 6.3.8 004fd4d0d@24112209
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)

WARNING: key 'workers' is deprecated in /etc/manticoresearch/manticore.conf line 12; use 'default value' instead.
precaching table 'account_core'
precaching table 'backend_api_core'
precaching table 'cms_page_core'
precaching table 'email_configuration_core'
precaching table 'metric_core'
precaching table 'plan_core'
[Sun Dec 15 20:47:55.164 2024] [3] WARNING: table 'proxy_rule_core': min_infix_len must be greater than 1, changed to 2
WARNING: table 'proxy_rule_core': min_infix_len must be greater than 1, changed to 2
precaching table 'proxy_rule_core'
precaching table 'service_core'
precaching table 'topic_core'
[Sun Dec 15 20:47:55.165 2024] [4] prereading 9 tables
prereading 9 tables
[Sun Dec 15 20:47:55.165 2024] [4] preread 9 tables in 0.000 sec
preread 9 tables in 0.000 sec
[Sun Dec 15 20:47:55.165 2024] [2] WARNING: [BUDDY] no SPHINX or HTTP listeners found, disabled
WARNING: [BUDDY] no SPHINX or HTTP listeners found, disabled
[Sun Dec 15 20:47:55.166 2024] [2] accepting connections
accepting connections

At this point I connect with mysql cli like this:

# mysql -P 9306 -h 127.0.0.1
<...>
mysql> INSERT INTO backend_api_core (id, name, system_name) VALUES (1 ,'garga',
'roshava');
Query OK, 1 row affected (0.00 sec)

mysql> SELECT * FROM backend_api_core;
ERROR 2013 (HY000): Lost connection to MySQL server during query

The result in the container is:

/* Sun Dec 15 20:48:05.148 2024 conn 1 (10.88.0.1:54750) real 0.000 wall 0.000 found 1 */ SELECT * FROM backend_api_core;
------- FATAL: CRASH DUMP -------
[Sun Dec 15 20:48:04.678 2024] [    2]

--- crashed SphinxQL request dump ---
SELECT * FROM backend_api_core
--- request dump end ---
--- local index:����▒api_core
Manticore 6.3.8 004fd4d0d@24112209
Handling signal 11
Crash!!! Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 16.0.6
Configured with flags: Configured with these definitions: -DDISTR_BUILD=jammy -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux s390x (jammy) (cross-compiled)
Stack bottom = 0x3ff8c023f80, thread stack size = 0x20000
Trying manual backtrace:
Frame pointer is null, manual backtrace failed (did you build with -fomit-frame-pointer?)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x230)[0x2aa400d3a10]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x558)[0x2aa3fe8b218]
linux-vdso64.so.1(__kernel_sigreturn+0x0)[0x3ffeb4d3498]
/lib/s390x-linux-gnu/libc.so.6(__libc_free+0x10)[0x3ffa3427330]
/usr/bin/searchd(_ZNK16CSphSchemaHelper12FreeDataPtrsER9CSphMatch+0x6e)[0x2aa41a21ffe]
/usr/bin/searchd(_ZN14OneResultset_tD1Ev+0x5e)[0x2aa3fea3c7e]
/usr/bin/searchd(_ZN12AggrResult_tD2Ev+0xfa)[0x2aa3fda3eaa]
/usr/bin/searchd(_ZN15SearchHandler_cD1Ev+0x528)[0x2aa3feadf78]
/usr/bin/searchd(_ZN15ClientSession_c7ExecuteESt4pairIPKciER11RowBuffer_i+0x1a82)[0x2aa3ff0e4d2]
/usr/bin/searchd(_ZN7session7ExecuteESt4pairIPKciER11RowBuffer_i+0x36)[0x2aa3ff179d6]
/usr/bin/searchd(_Z8SqlServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EE+0x210a)[0x2aa3fd8584a]
/usr/bin/searchd(+0x8ffc0a)[0x2aa3fd7fc0a]
/usr/bin/searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_8__invokeESB_+0x38)[0x2aa41ba06f8]
/usr/bin/searchd(make_fcontext+0x26)[0x2aa41c16aae]
Trying boost backtrace:
 0# sphBacktrace(int, bool) in /usr/bin/searchd
 1# CrashLogger::HandleCrash(int) in /usr/bin/searchd
 2# __kernel_sigreturn in linux-vdso64.so.1
 3# __libc_free in /lib/s390x-linux-gnu/libc.so.6
 4# CSphSchemaHelper::FreeDataPtrs(CSphMatch&) const in /usr/bin/searchd
 5# OneResultset_t::~OneResultset_t() in /usr/bin/searchd
 6# AggrResult_t::~AggrResult_t() in /usr/bin/searchd
 7# SearchHandler_c::~SearchHandler_c() in /usr/bin/searchd
 8# ClientSession_c::Execute(std::pair<char const*, int>, RowBuffer_i&) in /usr/bin/searchd
 9# session::Execute(std::pair<char const*, int>, RowBuffer_i&) in /usr/bin/searchd
10# SqlServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >) in /usr/bin/searchd
11# 0x000002AA3FD7FC0A in /usr/bin/searchd
12# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd
13# make_fcontext in /usr/bin/searchd

-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the manual
(https://manual.manticoresearch.com/Reporting_bugs)
Dump with GDB via watchdog
--- active threads ---
thd 0 (work_0), proto mysql, state query, command select
--- Totally 2 threads, and 1 client-working threads ---
------- CRASH DUMP END -------

Find the core dump attached core.work_0.2.gz

# export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com"
# gdb /usr/bin/searchd /tmp/core.work_0.2 
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
<...>
Core was generated by `/usr/bin/searchd --pidfile --config /etc/manticoresearch/manticore.conf --nodet'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000003ffa3427330 in free () from /lib/s390x-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x3ffa3fe7840 (LWP 3))]


(gdb) info threads
  Id   Target Id                    Frame 
* 1    Thread 0x3ffa3fe7840 (LWP 3) 0x000003ffa3427330 in free () from /lib/s390x-linux-gnu/libc.so.6
  2    Thread 0x3ffa3f3d840 (LWP 6) 0x000003ffa349043a in epoll_wait () from /lib/s390x-linux-gnu/libc.so.6
  3    Thread 0x3ffa3fedc00 (LWP 2) 0x000003ffa348892e in select () from /lib/s390x-linux-gnu/libc.so.6
  4    Thread 0x3ffa3f7f840 (LWP 4) 0x000003ffa3412ae6 in ?? () from /lib/s390x-linux-gnu/libc.so.6
  5    Thread 0x3ffa3f5e840 (LWP 5) 0x000003ffa3412ae6 in ?? () from /lib/s390x-linux-gnu/libc.so.6


(gdb) thread apply all bt

Thread 5 (Thread 0x3ffa3f5e840 (LWP 5)):
#0  0x000003ffa3412ae6 in ?? () from /lib/s390x-linux-gnu/libc.so.6
#1  0x000003ffa3415a64 in pthread_cond_timedwait () from /lib/s390x-linux-gnu/libc.so.6
#2  0x000002aa3ff65452 in AutoEvent_T<true>::WaitEvent (this=0x2aa42152498 <g_TinyTimer()::tTimer+64>, iMsec=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/std/autoevent.cpp:207
#3  0x000002aa40440b84 in TinyTimer_c::Loop (this=0x2aa42152458 <g_TinyTimer()::tTimer>, this@entry=<error reading variable: value has been optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/mini_timer.cpp:153
#4  0x000002aa403ecabe in std::function<void ()>::operator()() const (this=<optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590
#5  RuntimeThreadContext_t::Run (this=0x2aa42685260, pStack=0x3ffa3f5de6e) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:1623
#6  ThreadProcWrapper_fn (pArg=0x2aa42685260) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:1648
#7  0x000003ffa3416296 in ?? () from /lib/s390x-linux-gnu/libc.so.6
#8  0x000003ffa348ff8e in ?? () from /lib/s390x-linux-gnu/libc.so.6

Thread 4 (Thread 0x3ffa3f7f840 (LWP 4)):
#0  0x000003ffa3412ae6 in ?? () from /lib/s390x-linux-gnu/libc.so.6
#1  0x000003ffa34156fa in pthread_cond_wait () from /lib/s390x-linux-gnu/libc.so.6
#2  0x000002aa403f1ca0 in sph::Event_c::Wait<CSphScopedLock<CSphMutex> > (this=0x2aa42671c48, dLock=...) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/event.h:241
#3  Threads::Service_t::do_run_one (this=<optimized out>, this_thread=..., bBusy=..., dLock=...) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:342
#4  Threads::Service_t::run (this=0x2aa42671c10, bBusy=std::atomic<bool> = { false }) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:323
#5  0x000002aa403f1a2e in Threads::ThreadPool_c::loop (this=0x2aa42671c00, iChild=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:756
#6  Threads::ThreadPool_c::ThreadPool_c(unsigned long, char const*)::{lambda()#1}::operator()() const (this=<optimized out>, this@entry=<error reading variable: value has been optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:778
#7  std::__invoke_impl<void, Threads::ThreadPool_c::ThreadPool_c(unsigned long, char const*)::{lambda()#1}&>(std::__invoke_other, Threads::ThreadPool_c::ThreadPool_c(unsigned long, char const*)::{lambda()#1}&) (__f=..., __f@entry=<error reading variable: value has been optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
#8  std::__invoke_r<void, Threads::ThreadPool_c::ThreadPool_c(unsigned long, char const*)::{lambda()#1}&>(Threads::ThreadPool_c::ThreadPool_c(unsigned long, char const*)::{lambda()#1}&) (__fn=..., __fn@entry=<error reading variable: value has been optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111
#9  std::_Function_handler<void (), Threads::ThreadPool_c::ThreadPool_c(unsigned long, char const*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=..., __functor@entry=<error reading variable: value has been optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290
#10 0x000002aa403efa9e in std::function<void ()>::operator()() const (this=<optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590
#11 Threads::CreateQ(unsigned long*, std::function<void ()>, bool, char const*, int)::$_0::operator()() const (this=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:1825
#12 std::__invoke_impl<void, Threads::CreateQ(unsigned long*, std::function<void ()>, bool, char const*, int)::$_0&>(std::__invoke_other, Threads::CreateQ(unsigned long*, std::function<void ()>, bool, char const*, int)::$_0&) (__f=...) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
#13 std::__invoke_r<void, Threads::CreateQ(unsigned long*, std::function<void ()>, bool, char const*, int)::$_0&>(Threads::CreateQ(unsigned long*, std::function<void ()>, bool, char const*, int)::$_0&) (__fn=...) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111
#14 std::_Function_handler<void (), Threads::CreateQ(unsigned long*, std::function<void ()>, bool, char const*, int)::$_0>::_M_invoke(std::_Any_data const&) (__functor=..., __functor@entry=<error reading variable: value has been optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290
#15 0x000002aa403ecabe in std::function<void ()>::operator()() const (this=<optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590
#16 RuntimeThreadContext_t::Run (this=0x2aa42672d70, pStack=0x3ffa3f7ee6e) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:1623
#17 ThreadProcWrapper_fn (pArg=0x2aa42672d70) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:1648
#18 0x000003ffa3416296 in ?? () from /lib/s390x-linux-gnu/libc.so.6
#19 0x000003ffa348ff8e in ?? () from /lib/s390x-linux-gnu/libc.so.6

Thread 3 (Thread 0x3ffa3fedc00 (LWP 2)):
#0  0x000003ffa348892e in select () from /lib/s390x-linux-gnu/libc.so.6
#1  0x000002aa3ff6f384 in sphSleepMsec (iMsec=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/sphinx.cpp:1390
#2  0x000002aa3ff207c0 in TickHead () at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/searchd.cpp:19667
#3  0x000002aa3ff2ccde in ServiceMain (argc=<optimized out>, argv=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/searchd.cpp:21480
#4  0x000002aa3ff2e964 in mainimpl (argv=<optimized out>, argc=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/searchd.cpp:21529
#5  main (argc=<optimized out>, argv=0x3ffeb47a138) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/searchd.cpp:21535

Thread 2 (Thread 0x3ffa3f3d840 (LWP 6)):
#0  0x000003ffa349043a in epoll_wait () from /lib/s390x-linux-gnu/libc.so.6
#1  0x000002aa3fdaadae in PollTraits_t::poll_events (pEvents=<optimized out>, timeoutUS=-1, iPoll=<optimized out>, iEventNum=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/netpoll.cpp:207
#2  NetPooller_c::Impl_c::Wait (this=0x2aa4266e4e0, iUS=-1) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/netpoll.cpp:350
#3  0x000002aa3fd7bb38 in CSphNetLoop::Impl_c::Poll (this=0x2aa42670990, tmLastWaitUS=287053222143) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/networking_daemon.cpp:249
#4  CSphNetLoop::Impl_c::LoopNetPoll (this=0x2aa42670990) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/networking_daemon.cpp:261
#5  0x000002aa3ff47b54 in ServiceMain(int, char**)::$_3::operator()() const (this=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/searchd.cpp:21450
#6  Threads::details::CompletionHandler_c<ServiceMain(int, char**)::$_3>::DoComplete(void*, Threads::details::SchedulerOperation_t*) (pOwner=0x2aa426748a0, pOwner@entry=<error reading variable: value has been optimized out>, pBase=<optimized out>, pBase@entry=<error reading variable: value has been optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils_impl.h:246
#7  0x000002aa403f1dac in Threads::details::SchedulerOperation_t::Complete (this=0x2aa426ed000, pOwner=0x2aa426748a0) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils_impl.h:202
#8  Threads::Service_t::do_run_one (this=<optimized out>, this_thread=..., bBusy=..., dLock=...) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:357
#9  Threads::Service_t::run (this=0x2aa426748a0, bBusy=std::atomic<bool> = { true }) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:323
#10 0x000002aa403f1a2e in Threads::ThreadPool_c::loop (this=0x2aa42674890, iChild=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:756
#11 Threads::ThreadPool_c::ThreadPool_c(unsigned long, char const*)::{lambda()#1}::operator()() const (this=<optimized out>, this@entry=<error reading variable: value has been optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:778
#12 std::__invoke_impl<void, Threads::ThreadPool_c::ThreadPool_c(unsigned long, char const*)::{lambda()#1}&>(std::__invoke_other, Threads::ThreadPool_c::ThreadPool_c(unsigned long, char const*)::{lambda()#1}&) (__f=..., __f@entry=<error reading variable: value has been optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
#13 std::__invoke_r<void, Threads::ThreadPool_c::ThreadPool_c(unsigned long, char const*)::{lambda()#1}&>(Threads::ThreadPool_c::ThreadPool_c(unsigned long, char const*)::{lambda()#1}&) (__fn=..., __fn@entry=<error reading variable: value has been optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111
#14 std::_Function_handler<void (), Threads::ThreadPool_c::ThreadPool_c(unsigned long, char const*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=..., __functor@entry=<error reading variable: value has been optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290
#15 0x000002aa403efa9e in std::function<void ()>::operator()() const (this=<optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590
#16 Threads::CreateQ(unsigned long*, std::function<void ()>, bool, char const*, int)::$_0::operator()() const (this=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:1825
#17 std::__invoke_impl<void, Threads::CreateQ(unsigned long*, std::function<void ()>, bool, char const*, int)::$_0&>(std::__invoke_other, Threads::CreateQ(unsigned long*, std::function<void ()>, bool, char const*, int)::$_0&) (__f=...) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
#18 std::__invoke_r<void, Threads::CreateQ(unsigned long*, std::function<void ()>, bool, char const*, int)::$_0&>(Threads::CreateQ(unsigned long*, std::function<void ()>, bool, char const*, int)::$_0&) (__fn=...) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111
#19 std::_Function_handler<void (), Threads::CreateQ(unsigned long*, std::function<void ()>, bool, char const*, int)::$_0>::_M_invoke(std::_Any_data const&) (__functor=..., __functor@entry=<error reading variable: value has been optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290
#20 0x000002aa403ecabe in std::function<void ()>::operator()() const (this=<optimized out>) at /sysroot/root/usr/lib/gcc/s390x-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590
#21 RuntimeThreadContext_t::Run (this=0x2aa426ed690, pStack=0x3ffa3f3ce6e) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:1623
#22 ThreadProcWrapper_fn (pArg=0x2aa426ed690) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/threadutils.cpp:1648
#23 0x000003ffa3416296 in ?? () from /lib/s390x-linux-gnu/libc.so.6
#24 0x000003ffa348ff8e in ?? () from /lib/s390x-linux-gnu/libc.so.6

Thread 1 (Thread 0x3ffa3fe7840 (LWP 3)):
PC not saved



(gdb) thread 3
[Switching to thread 3 (Thread 0x3ffa3fedc00 (LWP 2))]
#0  0x000003ffa348892e in select () from /lib/s390x-linux-gnu/libc.so.6



(gdb) bt full
#0  0x000003ffa348892e in select () from /lib/s390x-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000002aa3ff6f384 in sphSleepMsec (iMsec=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/sphinx.cpp:1390
        tvTimeout = {tv_sec = 0, tv_usec = 500000}
#2  0x000002aa3ff207c0 in TickHead () at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/searchd.cpp:19667
        tmSleep = 500
#3  0x000002aa3ff2ccde in ServiceMain (argc=<optimized out>, argv=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/searchd.cpp:21480
        sError = {m_sValue = 0x0, static EMPTY = "", static SAFETY_GAP = 4}
        sKNNError = {m_sValue = 0x0, static EMPTY = <same as static member of an already seen type>, static SAFETY_GAP = 4}
        sTZWarning = {m_sValue = 0x2aa4214d188 <CSphString::EMPTY> "", static EMPTY = <same as static member of an already seen type>, static SAFETY_GAP = 4}
        conf = {m_dHash = {0x0 <repeats 256 times>}, m_pFirstByOrder = 0x0, m_pLastByOrder = 0x0, m_iLength = 0}
        dConfig = {<VecTraits_T<char>> = {m_pData = 0x0, m_iCount = 0}, <sph::DefaultStorage_T<char>> = {static is_sized = <optimized out>, static is_constructed = true, static is_owned = <optimized out>}, <sph::DefaultRelimit> = {static SANE_SIZE = 1073741823, static MAGIC_INITIAL_LIMIT = 8}, m_iLimit = 0}
        dListenerDescs = {<VecTraits_T<ListenerDesc_t>> = {m_pData = 0x0, m_iCount = 0}, <sph::DefaultStorage_T<ListenerDesc_t>> = {static is_sized = <optimized out>, static is_constructed = true, static is_owned = <optimized out>}, <sph::DefaultRelimit> = {static SANE_SIZE = 1073741823, static MAGIC_INITIAL_LIMIT = 8}, m_iLimit = 0}
        sSslCert = {m_sValue = 0x2aa4214d188 <CSphString::EMPTY> "", static EMPTY = <same as static member of an already seen type>, static SAFETY_GAP = 4}
        sSslKey = {m_sValue = 0x2aa4214d188 <CSphString::EMPTY> "", static EMPTY = <same as static member of an already seen type>, static SAFETY_GAP = 4}
        sSslCa = {m_sValue = 0x2aa4214d188 <CSphString::EMPTY> "", static EMPTY = <same as static member of an already seen type>, static SAFETY_GAP = 4}
        dExactIndexes = {<VecTraits_T<CSphString>> = {m_pData = 0x0, m_iCount = 0}, <sph::DefaultStorage_T<CSphString>> = {static is_sized = <optimized out>, static is_constructed = true, static is_owned = <optimized out>}, <sph::DefaultRelimit> = {static SANE_SIZE = 1073741823, static MAGIC_INITIAL_LIMIT = 8}, m_iLimit = 0}
        bColumnarError = <optimized out>
        bSecondaryError = <optimized out>
        bKNNError = <optimized out>
        bTestMode = <optimized out>
        bOptListen = <optimized out>
        bOptPort = <optimized out>
        i = <optimized out>
        uReplayFlags = <optimized out>
        szCmdConfigFile = <optimized out>
        sOptListen = <optimized out>
        iOptPort = <optimized out>
        dOptIndexes = <optimized out>
        hSearchdpre = <optimized out>
        bVisualLoad = <optimized out>
        bWatched = <optimized out>
        iDevNull = 3
        hSearchd = @0x2aa42676858: {<CSphOrderedHash<CSphVariant, CSphString, CSphStrHashFunc, 256>> = {m_dHash = {0x0 <repeats 138 times>, 0x2aa426773f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2aa426772a0, 0x0 <repeats 42 times>, 0x2aa42677350, 0x0, 0x0, 0x0, 0x0, 0x2aa42677160, 0x0 <repeats 19 times>, 0x2aa42677200, 0x0 <repeats 35 times>, 0x2aa426770b0, 0x0, 0x0}, m_pFirstByOrder = 0x2aa426770b0, m_pLastByOrder = 0x2aa426773f0, m_iLength = 6}, m_iTag = 6}
        iRtFlushPeriodUs = <optimized out>
        thMain = <optimized out>
        bOptStop = <optimized out>
        bOptStopWait = <optimized out>
        bOptStatus = <optimized out>
        bOptPIDFile = <optimized out>
        bOptDebugQlog = <optimized out>
        bForcedPreread = <optimized out>
        bNewCluster = <optimized out>
        bNewClusterForce = <optimized out>
        bForcePseudoSharding = <optimized out>
        wFakeLock = <optimized out>
        hConf = <optimized out>
#4  0x000002aa3ff2e964 in mainimpl (argv=<optimized out>, argc=<optimized out>) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/searchd.cpp:21529
        cTopOfMainStack = 16 '\020'
#5  main (argc=<optimized out>, argv=0x3ffeb47a138) at /manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/src/searchd.cpp:21535
No locals.
(gdb) info locals
No symbol table info available.

To build manticore on s390 with the existing semi-cross compile mechanism, one needs a few modificaitons: akostadinov@8494125

The deb packages can be downloaded here: https://filetransfer.io/data-package/IzVHQlul#link

This is the manticode configuration file: manticore.conf.gz

To build the image that I've built, place the debs in a directory name debs and the following Dockerfile :
Dockerfile.gz

You can build everything on x86/arm64 machine instead of real hardware:

• install qemu-static package of your distro
• use quay.io/3scale/searchd:external_toolchain_s390x (if you don't want to build your own)
• modifying dist/build_dockers/cross/sysroots/debian.sh and finalize.sh to enforce s390x architecture
• run all docker/podman commands with --arch s390x

But debugging under the qemu emulation didn't yield any meaningful results for me.

Please let me know in case I can provide any additional information.

Manticore Search Version:

6.3.8

Operating System Version:

RHEL9, Ubuntu jammy, Fedora 40

Have you tried the latest development version?

None

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed
• Changelog updated

[sanikolaev]

Hello @akostadinov

Thank you for your detailed report and efforts to provide all the necessary information.

Unfortunately, Manticore Search is not compatible with the s390x architecture. We have never officially supported this platform, and there were no plans to do so in the past or in the foreseeable future. Additionally, we have not conducted any tests or development to make Manticore compatible with s390x, and we currently do not have access to the required hardware for testing or debugging purposes.

If support for the s390x architecture is mission-critical for your company, we invite you to explore our professional services. Through these services, we can discuss potential options to address your specific requirements.

Alternatively, we would be glad to review a pull request that adds support for s390x and includes the necessary tests in the CI pipeline.

We appreciate your understanding, and if you have any further questions or need clarification, please let us know.

[klirichek]

@akostadinov
we build on x86-64 and on arm64. For both ready tools exist, so, it perfectly compiled with our 'vanilla' build system for s390x. The only small is providing right UNALIGNED_RAM_ACCESS_EXITCODE, however that is already done for another 'alien' arches, so just add a line with this arch is enough.
Essential feature of s390x is that it is big-endian.
Lucky, we have some support of it since sphinx fork.
FYU, on big-endian build there are some features:

• text protocols (http, mysql) works as usual.
• binary protocol (sphinxapi) works generally ok, except 'packed factors'. They are sent in native order, and so, if you have distributed (master + agents), and they're different endianess, and you use packedfactors, they will be broken.
• index files are not compatible between machines with different endianess. I.e. you can't just move table build on intel (LE) arch to your s390 (BE).
• language packs (for morphology) are compatible with both endianess (we build them on LE, and then added code which patches them on-the-fly during loading, if you works on BE)
• ICU support is unknown. Everything depends on whether icudata file is endianess-agnostic or not.
• basic core functions are endianess-aware. However, aside libs like 'columnar', 'secondary', 'knn' and whatever are in unknown state.

Your backtrace includes enough to locate the problem. Actually, that is just a few lines to be changed, and it is explicitly marked already (you can locate by exact match of 'since such cast gives garbage' in the sources).
Just a few notes for you:

• crash was in thread 1. Which is 'LWP 3'. That confused you to dump variables of thread 3
• when looking to coredump with gdb, it is usually more preferable to use symbols file instead of original binary. Symbols are placed separately as 'debug symbols package' (they are present on the folder with packages you've provided). Another approach - is to extract debuginfo and place aside of binary (gdb will find and load it also).

Commit 96b5a5c includes few fixes addressing big-endian arch. You can try it.

[akostadinov]

@klirichek , wow, thanks a lot for this update!

I just managed to try it out and my trivial INSERT/SELECT did work. I'll have to get it examined more thoroughly and will update you.

FYI this is the image that I've built quay.io/3scale/searchd:s390x_be

Update: my intention is to follow-up on this once folks have a chance to test it a little bit more. This means January or even more likely February. Happy holidays to all.