Manticore 6.3.8 crash

[ztest1209]

Bug Description:

[Fri Dec 20 17:32:27.324 2024] [88877] FATAL: invalid meta file /xxx/searchd/binlog/binlog.meta

--- crashed invalid query ---

--- request dump end ---
--- local index:
Manticore 6.3.8 004fd4d0db@241122 (columnar 2.3.0 619550d@240522) (secondary 2.3.0 619550d@240522) (knn 2.3.0 619550d@240522)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with GNU 13.3.0
Configured with flags: Configured with these definitions: -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.20 -DLOCALDATADIR=/var -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64
Stack bottom = 0x7fa640040f00, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x7fa640040f00)
Stack looks OK, attempting backtrace.
0x35
Something wrong in frame pointers, manual backtrace failed (fp=0)
Trying system backtrace:
[Fri Dec 20 17:32:27.327 2024] [88864] watchdog: main process 88877 killed dirtily with signal 11, will be restarted
[Fri Dec 20 17:32:27.327 2024] [88864] watchdog: main process 88883 forked ok

Manticore Search Version:

Manticore 6.3.8

Operating System Version:

linux 5.15.175

Have you tried the latest development version?

None

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed
• Changelog updated

[ztest1209]

The problem occurred after restarting, but I cannot reproduce it

[ztest1209]

Manticore 6.3.8 004fd4d@241122 (columnar 2.3.0 619550d@240522) (secondary 2.3.0 619550d@240522) (knn 2.3.0 619550d@240522)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with GNU 13.3.0
Configured with flags: Configured with these definitions: -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.20 -DLOCALDATADIR=/var -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64
Stack bottom = 0x7f30bc040f00, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x7f30bc040f00)
Stack looks OK, attempting backtrace.
0x35
Something wrong in frame pointers, manual backtrace failed (fp=0)
Trying system backtrace:
begin of system symbols:
searchd(_Z12sphBacktraceib+0xc4) [0x55d51dd78894]
searchd(_ZN11CrashLogger11HandleCrashEi+0x2c3) [0x55d51daca783]
/lib/libc.so.6(+0x3b460) [0x7f30d2ee8460]
/lib/libc.so.6(+0xa88ee) [0x7f30d2f558ee]
searchd(+0x9d9bcd) [0x55d51db75bcd]
searchd(+0x9d41a6) [0x55d51db701a6]
searchd(ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_4_FUNESB+0x1c) [0x55d51ef95cbc]
searchd(make_fcontext+0x2f) [0x55d51efeb58f]
Trying boost backtrace:
0# sphBacktrace(int, bool) in searchd
1# CrashLogger::HandleCrash(int) in searchd
2# 0x00007F30D2EE8460 in /lib/libc.so.6
3# 0x00007F30D2F558EE in /lib/libc.so.6
4# 0x000055D51DB75BCD in searchd
5# 0x000055D51DB701A6 in searchd
6# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::_FUN(boost::context::detail::transfer_t) in searchd
7# make_fcontext in searchd

-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the manual
(https://manual.manticoresearch.com/Reporting_bugs)
Dump with GDB via watchdog
--- active threads ---
[Fri Dec 20 17:37:09.941 2024] [92379] watchdog: main process 92692 killed dirtily with signal 11, will be restarted

[tomatolog]

could you provide full searchd.log attached?

[ztest1209]

searchd.log

[ztest1209]

您能提供附件的完整 searchd.log 吗？

searchd.log

[tomatolog]

seems your binlog file is corrupted somehow

FATAL: invalid meta file /xxx/searchd/binlog/binlog.meta

you need to remove all binlog files prior to start the daemon and repopulate data that could be affected.

Or provide these files along with indexes for investigation

[ztest1209]

你的 binlog 文件似乎已损坏

FATAL: invalid meta file /xxx/searchd/binlog/binlog.meta

您需要在启动守护进程之前删除所有 binlog 文件并重新填充可能受影响的数据。

或者提供这些文件以及索引以供调查

I don't know why the binlog file got corrupted; I only performed a machine restart. I also believe that even if the binlog file is corrupted, the program should not crash.

[tomatolog]

daemon can not start with the data corrupted. dev op should take action for that case either delete binlog files and repopulate data or restore data from the backup