fix: use PyCapsule Interface instead of Dataframe Interchange Protocol #3782
Conversation
MarcoGorelli
force-pushed
the
pycapsule
branch
from
aa6132d
to
7c03e13
Compare
MarcoGorelli
force-pushed
the
pycapsule
branch
from
7c03e13
to
cb86e7a
Compare
|
Implementation wise I think this looks great. Nice work @MarcoGorelli |
MarcoGorelli
force-pushed
the
pycapsule
branch
3 times, most recently
from
31146c8
to
9599662
Compare
MarcoGorelli
force-pushed
the
pycapsule
branch
2 times, most recently
from
cf4ce2c
to
f516630
Compare
MarcoGorelli
force-pushed
the
pycapsule
branch
from
f516630
to
0bd8507
Compare
| try: | ||
| import pyarrow | ||
| except ImportError as err: | ||
| msg = "PyArrow is required for non-pandas Dataframe support." |
There was a problem hiding this comment.
Is this generally a dependency of non-pandas dataframe libraries now? Or could this change introduce a regression for e.g. polars users who are currently leveraging the dataframe interchange protocol?
There was a problem hiding this comment.
Thanks for your review!
Polars doesn't depend on PyArrow, but polars.DataFrame.to_pandas always requires PyArrow. So, in practice, anyone working with both dataframe libraries may well already have PyArrow already installed
To avoid requiring PyArrow for the cases when it's not necessary, one way could be to do something like:
- try using the interchange protocol
- if it raises, then fall back to the PyCapsule Interface (which currently requires PyArrow)
This has the upside of not requiring PyArrow in some cases, but the downside of hiding issues where the interchange protocol silently produces invalid results
It may be possible to do this PyCapsule Interface conversion in the future without PyArrow but with something lighter instead, like arro3 by @kylebarron (who I'm ccing in case he has comments too)
What would be your preference?
There was a problem hiding this comment.
Some polars users may not have pyarrow installed. If seaborn needs to get pandas data, the only production-ready way to do Arrow -> pandas that I know of is using pyarrow.
As Marco mentions I'm working on arro3, which is a minimal library for Arrow in Python, but Pandas interop is not a primary concern, and it's not production-ready today.
There was a problem hiding this comment.
FWIW pandas 3.x is going to strongly incentivize users to install PyArrow, although it stops short of outright requiring it. In theory, the only people that shouldn't have PyArrow installed are those that operate in space/resource constrained environments, probably in headless environments like AWS Lambda where seaborn won't be used
Of course up to you how much you want to support non-PyArrow configurations, but the dataframe interchange protocol is relatively buggy and gets very little support, so you may find it easier altogether to force users towards PyArrow
| def test_data_interchange(self, mock_long_df, long_df): | ||
| pytest.importorskip( |
closes #3756
closes #3533
I'm hoping that this can supersede #3534
This means that you get support for quite a lot more, e.g.:
cuDF (their interchange protocol implementation is currently broken anyway [BUG] pd.api.interchange.from_dataframe fails with simple cuDF dataframe rapidsai/cudf#17282)
Polars: it fixes the issue reported in Try
to_pandasrather than erroring if interchanging to pandas doesn't work? #3533, because the PyCapsule interface actually supports nested data types:In addition, this has no effect on existing pandas users, as there's already an early return for pandas https://github.com/MarcoGorelli/seaborn/blob/0bd85071284d45f38cbf419b8cf1efb2179eda24/seaborn/_core/data.py#L284-L285
I'm sorry for having introduced the Interchange Protocol in the first place. It's turned out to be fairly problematic, see pandas-dev/pandas#56732 (comment) as the associated discussion for more context
cc @WillAyd for comments