Skip to content

fdbserver 7.x on sqlite OOM on RHEL9

Jump to bottom
Michael Stack edited this page Aug 13, 2024 · 2 revisions

Moving from OEL7 to RHEL9, after running for a number of hours (10+), resident memory (RSS in ps, and RES in top) would inexplicably start to rise for fdbservers on sqlite storage. Eventually, after an hour or so, we’d hit the fdbserver configured maximum memory limit and it would throw an OOM.

Undoing global TransparentHugePages (THP) stopped the OOMs. Instead of the RHEL9 default of 'always', we set it to 'madvise' so applications that could make use of THP still had access but THP was not default.

Disabling THP for databases is a common recommendation for various reasons; the phenomenon we observed most closely matched Transparent Huge Pages and Alternative Memory Allocators: A Cautionary Tale(fdb uses jemalloc since version 7.x). THP support in FDB is recognized as a TODO.

See How to use, monitor, and disable transparent hugepages in Red Hat Enterprise Linux 6 ,7 and 8? for how to dynamically or permanently disable THP.

References

  1. Disable Transparent HugePages when you run OracleDB, Transparent Huge Pages: Why We Disable It for Databases | PingCAP, Settling the Myth of Transparent HugePages for Databases
  2. This page was helpful: Huge Pages are a Good Idea (evanjones.ca).
Clone this wiki locally