Seg Faults with PyPy #2781

samuelcolvin · 2022-11-24T16:17:26Z

samuelcolvin
Nov 24, 2022

I'm seeing intermittent but (I think) increasing common segmentation faults with pypy on CI.

Builds examples:

The error appears at the end of running tests, and output generally looks something like:

Results (36.97s):
      2467 passed
        39 skipped
/opt/hostedtoolcache/PyPy/3.9.12/x64/lib/pypy3.9/site-packages/coverage/pytracer.py:290: CoverageWarning: Trace function changed, data is likely wrong: None != <bound method PyTracer._trace of <PyTracer at 0x14a58a0: 1013 lines in 2 files>> (trace-changed)
  self.warn(
make: *** [Makefile:79: test] Segmentation fault (core dumped)

Any idea what's going on here? I'm not suggesting this is an issue with pyo3, but not sure where else to ask.

I saw seg faults early on with pypy when pushing deep recursive validation models, but once I set a lower threshold for recursion in pypy the errors stopped (well paused, as they're happening again).

However these errors are happening when pytest finished, not on the recursion tests. So I'm a bit stumped.

Answered by adamreichold

Jun 11, 2023

@samuelcolvin The following diff fixes the stack overflows / segmentation faults for (but it leads to assertion errors as the assertions depend on the depth limit):

diff --git a/src/serializers/extra.rs b/src/serializers/extra.rs
index dac8f86..06393f7 100644
--- a/src/serializers/extra.rs
+++ b/src/serializers/extra.rs
@@ -13,6 +13,7 @@ use super::errors::{PydanticSerializationUnexpectedValue, UNEXPECTED_TYPE_SER_MA
 use super::ob_type::ObTypeLookup;
 use super::shared::CombinedSerializer;
 use crate::definitions::Definitions;
+use crate::validators::BACKUP_GUARD_LIMIT;
 
 /// this is ugly, would be much better if extra could be stored in `SerializationState`
 /// then `SerializationStat…

View full answer

messense · 2022-11-24T16:31:28Z

messense
Nov 24, 2022
Maintainer

Try use the tmate action to obtain and inspect the core dump file?

- uses: mxschmitt/action-tmate@v3
  if: failure()

8 replies

messense Nov 25, 2022
Maintainer

You'd need to ssh into the runner and run ulimit -c unlimited, then try to reproduce the crash to generate a full core dump file if you can't find a existing one. Use gdb -c core-dump-file-path to debug it.

samuelcolvin Dec 3, 2022
Author

Good news! I have at last caught a failure before the session expires. The core dump (zipped) can be downloaded from here, this link will expire after 14 days, but I have a copy locally incase it's needed.

This is a zip of /var/crash/_opt_hostedtoolcache_PyPy_3.7.13_x64_bin_pypy3.1001.crash which seems to have been generated by the original test run.

I also noted while playing around in the tmate session that I always get the seg fault when running coverage run -m pytest, but not when running pytest.

samuelcolvin Dec 3, 2022
Author

humm gdb is saying it doesn't recognize that file as a core dump, so I'm not sure what it is.

messense Dec 4, 2022
Maintainer

https://askubuntu.com/a/947532

messense Dec 4, 2022
Maintainer

You will have to extract the core dump and use gdb to get backtrace on GitHub runner, otherwise gdb can't resolve symbols:

$ root@ubuntu:~/pypy-crash# gdb -c CoreDump
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/bin/pypy3 during file-backed mapping note processing

warning: Can't open file /home/runner/work/pydantic-core/pydantic-core/pydantic_core/_pydantic_core.pypy37-pp73-x86_64-linux-gnu.so during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib/liblzma.so.5 during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib/libsqlite3.so.0 during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib_pypy/_pypy_openssl.pypy37-pp73-x86_64-linux-gnu.so during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib/libffi.so.6 during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib_pypy/_pwdgrp_cffi.pypy37-pp73-x86_64-linux-gnu.so during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib_pypy/_lzma_cffi.pypy37-pp73-x86_64-linux-gnu.so during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib_pypy/_sqlite3_cffi.pypy37-pp73-x86_64-linux-gnu.so during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib_pypy/_sha3/_sha3_cffi.pypy37-pp73-x86_64-linux-gnu.so during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib_pypy/_blake2/_blake2s_cffi.pypy37-pp73-x86_64-linux-gnu.so during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib_pypy/_blake2/_blake2b_cffi.pypy37-pp73-x86_64-linux-gnu.so during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib/libtinfow.so.6 during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib/libncursesw.so.6 during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib/libexpat.so.1 during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/lib_pypy/_pypy_util_cffi_inner.pypy37-pp73-x86_64-linux-gnu.so during file-backed mapping note processing

warning: Can't open file /opt/hostedtoolcache/PyPy/3.7.13/x64/bin/libpypy3-c.so during file-backed mapping note processing
[New LWP 1897]
Core was generated by `/opt/hostedtoolcache/PyPy/3.7.13/x64/bin/python /opt/hostedtoolcache/PyPy/3.7.1'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fd9a15bec65 in ?? ()
(gdb) bt
#0  0x00007fd9a15bec65 in ?? ()
#1  0x000000630490d230 in ?? ()
#2  0x0000000000001000 in ?? ()
#3  0x00007fd9a1364040 in ?? ()
#4  0x000000000000000b in ?? ()
#5  0x000000000000010f in ?? ()
#6  0x0000000000001030 in ?? ()
#7  0x0000000000000040 in ?? ()
#8  0x0000007000000101 in ?? ()
#9  0x0000000000000008 in ?? ()
#10 0x0000000000000001 in ?? ()
#11 0x0000000000000000 in ?? ()
(gdb)

mattip · 2023-06-08T15:12:21Z

mattip
Jun 8, 2023

Sorry I missed this. We just got a report to the PyPy issue tracker where pydantic core tests crash when built in dev mode. Maybe related, maybe something else? I do see that there are test failures in dev mode when I skip the segfault-ing tests

SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

This comes from modsupport.c where we have exactly the same code as CPython, so I wonder what is going on. Is there some code in PyO3 or pydantic that uses the # formatter and has a condition for PyPy?

8 replies

adamreichold Jun 11, 2023
Maintainer

I fear that the PY_SSIZE_T_CLEAN errors are only an effect of memory corruption elsewhere as I can't find any usage of the PyArg_Parse* oder Py_BuildValue in our code base?

adamreichold Jun 11, 2023
Maintainer

Running the first crashing test test_cycle_change under Valgrind, I get the following hopefully more useful backtrace:

==22649== Invalid read of size 8
==22649==    at 0x6BD3E0C: pypy_debug_catch_fatal_exception (in /home/adam/Downloads/pypy3.9-v7.3.11-linux64/bin/libpypy3.9-c.so)
==22649==    by 0x58DC8F3: pypy_g_ccall_pypy_debug_catch_fatal_exception____1 (in /home/adam/Downloads/pypy3.9-v7.3.11-linux64/bin/libpypy3.9-c.so)
==22649==    by 0x60B00DB: pypy_g_unexpected_exception (in /home/adam/Downloads/pypy3.9-v7.3.11-linux64/bin/libpypy3.9-c.so)
==22649==    by 0x60780D1: pypy_g_wrapper_second_level__star_3_3 (in /home/adam/Downloads/pypy3.9-v7.3.11-linux64/bin/libpypy3.9-c.so)
==22649==    by 0xF0B2B88: pyo3::types::any::PyAny::call (any.rs:437)
==22649==    by 0xF0B30A8: pyo3::types::any::PyAny::call1 (any.rs:508)
==22649==    by 0xF0036F9: _pydantic_core::serializers::infer::infer_to_python_known (infer.rs:212)
==22649==    by 0xF0007F3: _pydantic_core::serializers::infer::infer_to_python (infer.rs:32)
==22649==    by 0xF003768: _pydantic_core::serializers::infer::infer_to_python_known (infer.rs:213)
==22649==    by 0xF0007F3: _pydantic_core::serializers::infer::infer_to_python (infer.rs:32)
==22649==    by 0xF003768: _pydantic_core::serializers::infer::infer_to_python_known (infer.rs:213)
==22649==    by 0xF0007F3: _pydantic_core::serializers::infer::infer_to_python (infer.rs:32)
==22649==  Address 0x18 is not stack'd, malloc'd or (recently) free'd
==22649==

i.e. this is from ffi::PyObject_Call involving the fallback and from the address 0x18 I guess a null pointer is involved?

adamreichold Jun 11, 2023
Maintainer

I also do not get any PY_SSIZE_T_CLEAN-related errors if I skip test_cycle_change and test_fallback_cycle_change.

adamreichold Jun 11, 2023
Maintainer

@samuelcolvin The following diff fixes the stack overflows / segmentation faults for (but it leads to assertion errors as the assertions depend on the depth limit):

diff --git a/src/serializers/extra.rs b/src/serializers/extra.rs
index dac8f86..06393f7 100644
--- a/src/serializers/extra.rs
+++ b/src/serializers/extra.rs
@@ -13,6 +13,7 @@ use super::errors::{PydanticSerializationUnexpectedValue, UNEXPECTED_TYPE_SER_MA
 use super::ob_type::ObTypeLookup;
 use super::shared::CombinedSerializer;
 use crate::definitions::Definitions;
+use crate::validators::BACKUP_GUARD_LIMIT;
 
 /// this is ugly, would be much better if extra could be stored in `SerializationState`
 /// then `SerializationState` got a `serialize_infer` method, but I couldn't get it to work
@@ -365,8 +366,6 @@ pub struct SerRecursionGuard {
 }
 
 impl SerRecursionGuard {
-    const MAX_DEPTH: u16 = 200;
-
     pub fn add(&self, value: &PyAny, def_ref_id: usize) -> PyResult<usize> {
         // https://doc.rust-lang.org/std/collections/struct.HashSet.html#method.insert
         // "If the set did not have this value present, `true` is returned."
@@ -374,7 +373,7 @@ impl SerRecursionGuard {
         let mut info = self.info.borrow_mut();
         if !info.ids.insert((id, def_ref_id)) {
             Err(PyValueError::new_err("Circular reference detected (id repeated)"))
-        } else if info.depth > Self::MAX_DEPTH {
+        } else if info.depth > BACKUP_GUARD_LIMIT {
             Err(PyValueError::new_err("Circular reference detected (depth exceeded)"))
         } else {
             info.depth += 1;
diff --git a/src/validators/definitions.rs b/src/validators/definitions.rs
index 5b5208c..daa5a75 100644
--- a/src/validators/definitions.rs
+++ b/src/validators/definitions.rs
@@ -172,8 +172,8 @@ impl Validator for DefinitionRefValidator {
 // see #143 this is a backup in case the identity check recursion guard fails
 // if a single validator "depth" (how many times it's called inside itself) exceeds the limit,
 // we raise a recursion error.
-const BACKUP_GUARD_LIMIT: u16 = if cfg!(PyPy) || cfg!(target_family = "wasm") {
-    123
+pub(crate) const BACKUP_GUARD_LIMIT: u16 = if cfg!(PyPy) || cfg!(target_family = "wasm") {
+    50
 } else {
     255
 };
diff --git a/src/validators/mod.rs b/src/validators/mod.rs
index 377f7af..d6a5135 100644
--- a/src/validators/mod.rs
+++ b/src/validators/mod.rs
@@ -56,6 +56,8 @@ mod with_default;
 
 pub use with_default::DefaultType;
 
+pub(crate) use self::definitions::BACKUP_GUARD_LIMIT;
+
 use self::definitions::DefinitionRefValidator;
 
 #[pyclass(module = "pydantic_core._pydantic_core", name = "Some")]

I suspect that dev builds need significantly more stack space per call frame, i.e. already 60 instead of 50 leads to crashes again.

@mattip I also had the impression that this was independent of ulimit -s. Does PyPy manage stack size itself somehow?

Answer selected by davidhewitt

mattip Jun 11, 2023

There are hueristics to track used stack and possibly emit a StackOverflow error to the interpreter when stack is full. I am not aware of compile-time stack tweaks, except on windows.

adamreichold Jun 11, 2023
Maintainer

I guess this is pytest spawning new threads to run the tests and managing therefore having to manage their stack size. I did not find a know to tune that though...

davidhewitt · 2023-07-05T14:06:19Z

davidhewitt
Jul 5, 2023
Maintainer

I think I have a resolution for this in pydantic/pydantic-core#744

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Seg Faults with PyPy #2781

{{title}}

Replies: 3 comments 16 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Seg Faults with PyPy #2781

samuelcolvin Nov 24, 2022

Replies: 3 comments · 16 replies

messense Nov 24, 2022 Maintainer

messense Nov 25, 2022 Maintainer

samuelcolvin Dec 3, 2022 Author

samuelcolvin Dec 3, 2022 Author

messense Dec 4, 2022 Maintainer

messense Dec 4, 2022 Maintainer

mattip Jun 8, 2023

adamreichold Jun 11, 2023 Maintainer

adamreichold Jun 11, 2023 Maintainer

adamreichold Jun 11, 2023 Maintainer

adamreichold Jun 11, 2023 Maintainer

mattip Jun 11, 2023

adamreichold Jun 11, 2023 Maintainer

davidhewitt Jul 5, 2023 Maintainer

samuelcolvin
Nov 24, 2022

Replies: 3 comments 16 replies

messense
Nov 24, 2022
Maintainer

messense Nov 25, 2022
Maintainer

samuelcolvin Dec 3, 2022
Author

samuelcolvin Dec 3, 2022
Author

messense Dec 4, 2022
Maintainer

messense Dec 4, 2022
Maintainer

mattip
Jun 8, 2023

adamreichold Jun 11, 2023
Maintainer

adamreichold Jun 11, 2023
Maintainer

adamreichold Jun 11, 2023
Maintainer

adamreichold Jun 11, 2023
Maintainer

adamreichold Jun 11, 2023
Maintainer

davidhewitt
Jul 5, 2023
Maintainer