I'd love some help with pydantic-core

[samuelcolvin]

First of all, thank you again for pyo3 - it's amazing.

I've building the new validation core for pydantic in rust using pyo3, see pydantic-core.

I'd love some help making it as safe and performant as possible while the API can still be easily changed.

I've described my questions and concerns in pydantic/pydantic-core#35.

Any help greatly appreciated!

Bit of background on pydantic:

• it does data validation and coercion in python using type hints
• it's one of the primary dependencies of the very popular FastAPI
• it has 9.8k stars on github, is used by 60k repos and is downloaded 21M times a month
• it's used by all the big tech companies, and many of the small ones

Which is all just to say that making pydantic better has a real effect, making it faster should save the planet a non-trivial amount of carbon...

[davidhewitt]

Hey! Sorry for the slow response from me; time is stretched a bit thin at the moment. I'm very excited to hear about your choice to use PyO3 for pydantic-core - I'm a user of both pydantic and FastAPI in my professional time 😄

I see that lots of people have already contributed to the discussion on the pydantic-core repo, so I think that the time I can offer to review for performance refactoring wouldn't be worth it for now. That said, if you want to have my direct input on any of the discussions please do to ping me and I'll do my best to give useful insight!

Similarly, if you run into shortcomings of the PyO3 API which pydantic-core would find useful, please do fire suggestions back here.

Which is all just to say that making pydantic better has a real effect, making it faster should save the planet a non-trivial amount of carbon...

🤩 this is actually exactly one of the reasons I tell folks when they ask me why I contribute to PyO3! (As well as it being great fun!)

[davidhewitt]

Thanks - I took a look at your second bullet and proposed pydantic/pydantic-core#55, which you may find interesting. It may not be exactly right because I think it will also call __init__.

I think the second argument needs to be a ptr to an empty tuple, not a null pointer - happy to create an issue if if helps

Hehe yes there's a FIXME just above. PyO3 needs a slightly smarter API to pass real arguments in also (e.g. for dict subclasses where the dictionary itself could receive initial values in its __new__).

there's no pyo3 type for dict_key or dict_values

See #2358 :)

[samuelcolvin]

@davidhewitt any chance you could have a quick look at #2729 and confirm whether what I'm trying to do is impossible?

I suspect it is, but if I was somehow able to create a class in rust, subclass it in python, then access the underlying struct of the subclass in rust it would be a massive win for pydantic, hence I'd like to be sure.

[davidhewitt]

Sorry for the long delay, had covid and have a sleepless baby at the moment, been zeroing out my capacity to think on more complex questions!

So I think the comments in that discussion are correct - perhaps one clarifying point is that the "struct of the subclass" is a bit of a murky concept. Because it's a Python-defined subclass, the memory layout will be the same as the Rust-defined base class, maybe with one extra pointer at the end to store a dict of Python instance attributes.

If you want to call methods of the subclass, they will all be stored as attributes on the subclass' type object, so you can just look them up with getattr and call them as normal Python functions.

Not sure if any of that is helpful...?

[samuelcolvin]

Thanks so much.

I'm sorry to hear it, poor you, you have all my sympathies as I've experienced both. Hope you feel better soon.

[davidhewitt]

Thanks, we're not sick any more but definitely still sleepless, so I'm catching up with topics slowly 😂