Empty slicing behaviour before and after ak.to_numpy is different

[pfackeldey]

Version of Awkward Array

'1.4.0'

Description and code to reproduce

Dear awkward developers,

we noticed an unexpected behaviour in fancy indexing empty dimensions before and after ak.to_numpy call.
I think the following snippet makes it a bit more clear:

# before
ak.to_numpy(ak.Array([[1], [2]])[[]])
>> array([], shape=(0, 0), dtype=int64)

# after
ak.to_numpy(ak.Array([[1], [2]]))[[]]
>> array([], shape=(0, 1), dtype=int64) # <-- this is what we expected

The shape discrepancy is unexpected for us.
What is your take, is this behaviour intended?

Best, Peter

[jpivarski]

It's because ak.Array([[1], [2]]) is an array of variable-length lists and np.array([[1], [2]]) is an array of fixed-length lists:

>>> ak.Array([[1], [2]])                   # note the "var"
<Array [[1], [2]] type='2 * var * int64'>
>>> 
>>> ak.Array(np.array([[1], [2]]))         # now it's regular
<Array [[1], [2]] type='2 * 1 * int64'>

The empty slice maintains var on the var-type array and the regular length of the regular array:

>>> ak.Array([[1], [2]])[[]]
<Array [] type='0 * var * int64'>
>>> 
>>> ak.Array(np.array([[1], [2]]))[[]]
<Array [] type='0 * 1 * int64'>

And this is preserved when converting to NumPy. Var-type arrays can only be converted to NumPy if they happen to be regular in the instance you're trying to convert. (Regular arrays can always be converted to NumPy.) The lengths of the inner dimensions of a var-type array are determined from what those list lengths happen to be in the particular instance. Here, they happen to be zero because they're empty.

>>> ak.to_numpy(ak.Array([[1], [2]])[[]])
array([], shape=(0, 0), dtype=int64)
>>> 
>>> ak.to_numpy(ak.Array(np.array([[1], [2]]))[[]])
array([], shape=(0, 1), dtype=int64)

It's unfortunate that we have to have this subtlety—that you, as user, have to be aware of the distinction between var-type and regular-type nested lists, but this is because only the regular-type can perfectly correspond to NumPy's behavior; var-type is new territory.

Let me know if I should reopen this, but I think it answers your question. Actually, no, I'm going to convert it to a Q&A discussion so that it remains visible. Others might encounter this, and I don't want to bury it by making it a closed issue.

[pfackeldey]

Thanks for the detailed reply! Alright I think I got the idea. Is var-type then always converted to an empty dimension, when converting to NumPy? I guess there is no correct "default", since this is new territory...

[jpivarski]

No, var-type is converted to however many list elements a particular array has. For instance,

>>> ak.to_numpy(ak.Array([[], [], []]))
array([], shape=(0, 0), dtype=float64)
>>> ak.to_numpy(ak.Array([[1], [1], [1]]))
array([[1],
       [1],
       [1]])
>>> ak.to_numpy(ak.Array([[1, 2], [1, 2], [1, 2]]))
array([[1, 2],
       [1, 2],
       [1, 2]])
>>> ak.to_numpy(ak.Array([[1, 2, 3], [1, 2, 3], [1, 2, 3]]))
array([[1, 2, 3],
       [1, 2, 3],
       [1, 2, 3]])

But it's entirely possible for a var-type to have an inconsistent number of list elements (which is the point of it, after all). In that case, it can't be converted into NumPy:

>>> ak.to_numpy(ak.Array([[1, 2, 3], [1, 2, 3, 4, 5, 6, 7, 8], [1, 2, 3]]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jpivarski/miniconda3/lib/python3.8/site-packages/awkward/operations/convert.py", line 204, in to_numpy
    return to_numpy(array.layout, allow_missing=allow_missing)
  File "/home/jpivarski/miniconda3/lib/python3.8/site-packages/awkward/operations/convert.py", line 332, in to_numpy
    return to_numpy(array.toRegularArray(), allow_missing=allow_missing)
ValueError: in ListOffsetArray64, cannot convert to RegularArray because subarray lengths are not regular

---

Actually, that first example (ak.to_numpy(ak.Array([[], [], []]))) should be converted into a NumPy array of shape (3, 0), rather than (0, 0). That might actually be a bug. It's possible to express an array of shape (3, 0) with Awkward Array:

>>> ak.Array(ak.layout.RegularArray(
...     ak.layout.NumpyArray(np.array([])),
...     size=0,
...     zeros_length=3,
... ))
<Array [[], [], []] type='3 * 0 * float64'>

That's what the var-type list ak.Array([[], [], []]) should have been converted into. It could be that the conversion code is missing the zeros_length argument. (The zeros_length argument was added later to patch up missing expressiveness: the ability to say that an array of regular, zero-length lists is not empty. In NumPy, the shape can be given any tuple of nonnegative integers, but Awkward Array has to construct these out of nested tree nodes, which complicates some corner-cases.)

[jpivarski]

I've written up the bug as #1048.