Even less safe way to access PyLong digits for blazing speed

[iliya-malecki]

I was exploring packing pythons digits into normal u32 digits the other day (see this function), then i noticed pyo3 already has support for BigInts with all possible safety features and got super happy that i can compare my code to something good. However, i immeadiately noticed the buffer that not only gets zeroed out right before being written to, but also gets discarded right afterwards. I understand that it is absolutely unrealistic to have an int of such mind-boggling size that copying it twice is a performance concern, however, im still curious about this approach from an unreasonable idealistic perspective. Are there any guarantees that we get by using the ffi rather than just looking at the bytes directly (which would save us from the unnecessary allocation and reduce the wall time by 70% for a halfway-reasonable int 10**300)?

[adamreichold]

I think this is mainly about being independent from as many implementation details of CPython as reasonably possible, e.g. working with different versions of CPython, with its limited API, but also with alternative implementations like PyPy.

(We could probably skip zero initialization, but that makes the code significantly more complex as we have to make sure to not pass uninitialized data where it isn't expected.)

[iliya-malecki]

Then I don't really understand what limited API is. The function is underscored and a quick google without getting into weeds (which could explain everything) shows that apparently the function could change at any moment. Is it still safer and further from implementation details than looking at bytes?
And, right away, a quick idea: maybe reinterpret the Vec<u8> as Vec<u32>, with some padding? That sounds mighty fast, if only the ffi function didn't return the silly two's complement for negative numbers...

[davidhewitt]

You are right that we could optimize this further; the tradeoff is against maintenance. The current implementation works well across the variety of Python versions we support. I won't claim it is perfect.

In PyO3 our general pattern is not to use pointer offsetting but instead define the expected struct layout and occasionally perform reads through FFI functions. See https://github.com/python/cpython/blob/main/Include/cpython/longintrepr.h for the layout of the PyLongObject structure.

You are welcome to contribute to PyO3 a fast-path for specific CPython versions which adds the appropriate FFI definitions. You should be aware that the general trend from CPython's API is that structure layouts are becoming implementation-private and we should not build functionality which relies on them. In practice we can use them when not doing abi3 builds (i.e. we are building for the ABI of a specific Python version).

[iliya-malecki]

Sounds like good fun, I'll try! As I'm usually using python and not rust, my first pr is bound to be an abomination but I'll get there eventually :)