[Bug]: Tests failing when trying to use the right logic for categoricals in Pandas

[caneff]

What happened?

In #28623 I tried to update the categorical checks to the non-deprecated way dictated by pandas, but when I do I get failing tests. It makes frames_test.py::DeferredFrameTest::test_categorical_groupby fail for AssertionError: NonParallelOperation not raised.

Getting rid of the nonparallel=True option in those tests fails with

E       AssertionError: Expression does not preserve partitioning!
E                       Expression: ComputedExpression[post_combine_sum_DataFrame_140482246047888]
E                       Requires: Index
E                       Preserves: Arbitrary
E                       Input partitioning: Index
E                       Expected output partitioning: Index

Issue Priority

Priority: 2

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[caneff]

R: @tvalentyn

[robertwb]

The problem here is that the replacement for the deprecated method should be isinstance(to_group.get_level_values(i).dtype, pd.CategoricalDtype) rather than isinstance(to_group.get_level_values(i), pd.CategoricalDtype).

Without the correct detection that the columns are categorical this test doesn't attempt to do this as a non-parallel operation (hence the failure to raise the NonParallelOperation error) and gets the wrong result (due to duplicate default values when partitioned, as per BEAM-11190, which the categorical-column detection was put in place to resolve).

Checking the type of the column's dtype should solve the issue.

[caneff]

Yep that fixes it. Thanks!

[tvalentyn]

Yep that fixes it. Thanks!

should we keep this open until we merge a fix?