How to combine a list of awkward arrays ?

[MoAly98]

Hi,

If I have a list of awkward arrays, for example:

[{'x': 20, y: [1, 2, 3, 4, 5]}, {'x':30, 'y':[6, 7, 8, 9, 10]}] 

And I would like to combine the elements of the list, such that I have

{'x': [20,30], 'y': [[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]] }

Is there a function which does that? I tried to use ak.concatenatebut I get back something like:

['x', 'y', 'x', 'y']

Thanks,
Mo

[jpivarski]

Now I wish the string representation of Awkward Arrays and Records didn't look so much like Python lists and dicts, because Python data type is important for this question!

ak.concatenate is the right function to use, though you can only use it without qualification if you really do have two Awkward Arrays:

>>> one = ak.Array([{"x": 20, "y": [1, 2, 3, 4, 5]}])
>>> two = ak.Array([{"x": 30, "y": [6, 7, 8, 9, 10]}])
>>> onetwo = ak.concatenate([one, two])
>>> onetwo.tolist()
[{'x': 20, 'y': [1, 2, 3, 4, 5]}, {'x': 30, 'y': [6, 7, 8, 9, 10]}]

You wanted the x data and the y data in separate arrays, so

>>> onetwo.x
<Array [20, 30] type='2 * int64'>
>>> onetwo.y
<Array [[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]] type='2 * var * int64'>

or maybe

>>> ak.Record({"x": onetwo.x, "y": onetwo.y}).tolist()
{'x': [20, 30], 'y': [[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]]}

To end up with lists of field names, I think you might be iterating over Python dicts or Awkward Records, rather than concatenating Arrays. Maybe this?

>>> one = ak.Record({"x": 20, "y": [1, 2, 3, 4, 5]})
>>> two = ak.Record({"x": 30, "y": [6, 7, 8, 9, 10]})
>>> ak.concatenate([one, two])
<Array ['x', 'y', 'x', 'y'] type='4 * string'>

Like all functions that take ak.Array input, if ak.concatenate gets an argument that is not an ak.Array, it will try to coerce it into being an ak.Array. A NumPy array, for instance, would be converted with ak.from_numpy. Anything with no recognizable type other than being a Python iterable will be iterated over, using ak.from_iter.

ak.Record, like a Python dict, iterates over its field names:

>>> list({"x": 20, "y": [1, 2, 3, 4, 5]})
['x', 'y']
>>> list(ak.Record({"x": 20, "y": [1, 2, 3, 4, 5]}))
['x', 'y']

[MoAly98]

Ha! That's exactly what was happening. I'm a bit confused as to why I ended up with a Record object though. I had started with data from data = uproot.concatenate(..) then I was testing my code on just one event by using data[0] where data should be similar to the example I gave above -- should this not return an Array object? Is there a general rule as to when a slicing operation on an awkward array return a Record object ?

Thank you so much, that's been extremely helpful!

[jpivarski]

If data is an ak.Array, data[0] might be an ak.Array, an ak.Record, or a numerical/boolean/datetime/string/bytes scalar:

>>> array = ak.Array([[{"x": 1, "y": 1.1}], [], [{"x": 2, "y": 2.2}]])
>>> array
<Array [[{x: 1, y: 1.1}], ... [{x: 2, y: 2.2}]] type='3 * var * {"x": int64, "y"...'>
>>> array[2]
<Array [{x: 2, y: 2.2}] type='1 * {"x": int64, "y": float64}'>
>>> array[2, 0]
<Record {x: 2, y: 2.2} type='{"x": int64, "y": float64}'>
>>> array[2, 0, "y"]
2.2

It depends on the array.type:

>>> array.type
3 * var * {"x": int64, "y": float64}

Successive unpacking will get simpler objects until they can't be unpacked anymore.

I also answered your other question on scikit-hep/uproot5/discussions/550 (for cross-referencing, since they're related).

[MoAly98]

I see, that makes sense. Is there a scenario where we use uproot.concatenate or uproot.iterate where the return type ends up being for example:

N * var * {"branch1": int64, "branch2": float64}

rahter than

N  * {"branch1": int64, "branch2": float64}

where N is the number of events in the file?

I also wonder if there is any documentation on how to drop fields from an awkward arrays and how to change field names, if either operations are possible?

Thank you very much.

[jpivarski]

I'm hesitating before making such a categorical statement, but I think there is no way that uproot.concatenate, uproot.iterate, uproot.lazy, uproot.TTree.arrays, or uproot.TTree.iterate can make a data type like

N * var * XYZ

All of those 5 functions (all array-fetching functions except for uproot.TBranch.array) get data from multiple TBranches, so there would always be a top-level record structure for the separate branches, even if there's only one of them, at least to specify their names (or its name). The question is whether any of those are ever made with ak.zip without restricting depth_limit, such that a structure that's originally

N * {field1: var * X, field2: var * Y, ...}

might be returned as

N * var * {field1: X, field2: Y, ...}

and I think the answer is "no." Whether or not it can do that, even in principle, depends on whether or not the "var" before "X" has the same number of subentries for each entry as the "var" before "Y", and that's something that we can't know about two unconnected branches. Returning a different type depending on the values of data in a file would be bad—the type must be fully determined by what's in the TTree metadata, before reading any TBaskets.

Actually, now that I think about it, there is type-level information (in the TTree metadata) that would say whether one branch's "var" is the same as another branch's "var": if the two TBranches' TLeaves have the same TLeaf::GetLeafCount. In principle, the array type could depend on that and not violate the rule against types depending on values. However, I don't think I implemented anything like that—such a rule would be more subtle than Uproot ought to be.

I found a good test case:

>>> import uproot, skhep_testdata
>>> tree = uproot.open(skhep_testdata.data_path("uproot-small-evnt-tree-fullsplit.root"))["tree"]

The "SliceF32" and "SliceF64" branches have the same TLeaf::GetLeafCount. In C++, the function returns the same pointer; in Python, it returns references to the same object, which can be identified by the "at 0x..." in its repr:

>>> tree["evt/SliceF32"].member("fLeaves")[0].member("fLeafCount")
<TLeafElement (version 1) at 0x7f8803f3a580>
>>> tree["evt/SliceF64"].member("fLeaves")[0].member("fLeafCount")
<TLeafElement (version 1) at 0x7f8803f3a580>

Even though these have the same leaf count and could in principle be 100 * var * {"evt/SliceF32": float32, "evt/SliceF64": float64}, they aren't:

>>> tree.arrays(["evt/SliceF32", "evt/SliceF64"]).type
100 * {"evt/SliceF32": var * float32, "evt/SliceF64": var * float64}

So no: those 5 functions will always return arrays with a record structure at top-level, even if they could have been merged into an array of lists of records.

---

Dropping fields is done by slicing. One of the allowed slice types is a list of strings, which would pull out more than one field. You can use this to pull out just the fields you want.

>>> array = ak.Array([{"x": 1, "y": 1.1, "z": "one"}, {"x": 2, "y": 2.2, "z": "two"}])
>>> array["x"]
<Array [1, 2] type='2 * int64'>
>>> array[["x", "y"]]
<Array [{x: 1, y: 1.1}, {x: 2, y: 2.2}] type='2 * {"x": int64, "y": float64}'>
>>> array[["z", "x"]]
<Array [{z: 'one', x: 1}, {z: 'two', x: 2}] type='2 * {"z": string, "x": int64}'>

It's documented under ak.Array.getitem as "nested projection."

The list given to a projection is an inclusive list, so you have to write down which ones you want to keep, rather than the ones you want to drop. You can get a full list of fields from ak.fields.

You can also drop fields, changing the ak.Array or ak.Record object in place. Nope, apparently not: you can assign in-place with __setitem__ (possibly replacing and old field), but the corresponding __delitem__ (which would be quite similar) apparently hasn't been implemented. That would be a good feature request, though it would likely only go into v2 at this point.

The assignment with __setitem__ is implemented through ak.with_field if you want a non-in-place version. Internally, Awkward Arrays are implemented with immutable but structurally shared pieces (only the high-level ak.Array and ak.Record can actually be changed in-place), so making a new array with a different field uses the same memory as replacing a field in an old array. It's just syntactic sugar for your convenience.

[MoAly98]

Brilliant, thanks a lot for the comprehensive response and advice.

I will raise a feature request for __delitem__ as i think it would be useful. I delete the fields now via a list comprehension where I drop I don't process the fields I don't want.

Thanks a lot.
Mo