Remove unwanted rows in array?

[CaueSousa]

Hello, dear community!

I'm trying to solve the following issue:

I have data about a particle cloud in a NumPy array with shape (1250,91,19) where the first index is related to the number of events(1250) and the other two with the rows (particles) and columns (particles features) of my data.

Each event has a variable number of particles, the event with the highest number has 91 particles. Because of this zero padding was used to fill the rows up to 91.

I converted this NumPy array to an awkward array and now I'm trying to get rid of the rows filled with zero.

I tried to assign to each "zero" a None element and after this use the a[~ak.is_none(a)] to remove these rows:

    input_events = ak.from_numpy(data_array_final)

    for event in range(1250):
        for row in range(91):
    
        #Some values of the columns can be zero in a valid row(like mass, vx, vy,vz), but
        #if the second collum(collum of PDG ID) is zero, the entire row is zero:
        if(input_events[event,row,1] == 0):
            for collum in range(19):
                input_events[event,row,collum] = None

but this attempt would be an In-place assignment, which isn't supported as a design choice.

Is there any way to remove an entire row similar to np.delete() from Numpy? Or to select only those rows filled with zeros and remove them from the overall array?

An example of the structure of the events can be found in the following file:
events_example.txt

My apologies if this is a simple question, I've searched for some solution, but I found only the "None element" one.

[agoose77]

I have to make some assumptions here; let me know if they are wrong!

• I understand that your data can contain zeros, but the second index of the inner dimension (array[:, :, 1]) is zero if that "row" (axis=1) is missing.
• I assume that your intended Awkward type is 1250 * var * 19, i.e. your final dimension is regular of length 19.

What you probably want to use here is Awkward's ragged indexing. We'll construct an array that indexes up to the second dimension, and ignore the empty rows.

First, let's convert the array to an Awkward Array

array = ak.from_numpy(input_events)

Now we can identify which rows are not padding, according to their second element

row_is_not_padding = array[:, :, 1] != 0

Subsequently, we can keep only these rows. We need to convert the index into a ragged array, so that Awkward's ragged indexing takes place. I.e., to go from

>>> row_is_not_padding.type.show()
1250 * 91 * bool

to

>>> ak.from_regular(row_is_not_padding).type.show()
1250 * var * bool

Finally, we can index into the array with this ragged index

array_ragged = array[
    ak.from_regular(row_is_not_padding)
]

The result has a ragged shape:

>>> array_ragged.type.show()
1250 * var * 19 * float64

[jpivarski]

As a question about interpreting the problem, if you have an array like this:

[
    [1.1, 2.2, 3.3, 4.4, 5.5, 0.0, 0.0, 0.0, 0.0, 0.0],
    [1.1, 2.2, 0.0, 0.0, 3.3, 4.4, 0.0, 0.0, 0.0, 0.0],
    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.1, 0.0],
    [1.1, 0.0, 2.2, 3.3, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
]

Would you want it to be unpadded like the following (remove all zeros)?

[
    [1.1, 2.2, 3.3, 4.4, 5.5],
    [1.1, 2.2, 3.3, 4.4],
    [1.1],
    [1.1, 2.2, 3.3],
]

Or would you want it to be unpadded like the following (remove all contiguous zeros, starting from the end)?

[
    [1.1, 2.2, 3.3, 4.4, 5.5],
    [1.1, 2.2, 0.0, 0.0, 3.3, 4.4],
    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.1],
    [1.1, 0.0, 2.2, 3.3],
]

In the first case, 0.0 is never a valid value. In the second case, you can't distinguish between a valid 0.0 at the end of the valid region and where the valid region ends. Like, if a row is supposed to look like

    [1.1, 2.2, 3.3, 4.4, 5.5, 0.0],

you wouldn't be able to tell.

Since @agoose77's row_is_not_padding is false for any 0.0 anywhere, it implements the first case. The second case is trickier, but possible. (It would involve ak.run_lengths.)

[CaueSousa]

Hi, Angus!

Your two assumptions are correct!

• A valid row can contain zeros, but not on the second collum since this collum is the particle id according to PDG. If a row has zero on the second collum, it necessarily came from the zero padding and has to be removed!
• Precisely, the number of rows (second index) should be var but the number of collums is fixed (third index)

Thank you very much! I'll read about the functions you just used since I'm starting to use Awkward. Thank you!

[CaueSousa]

Hi, Jim!

In the example you gave, all the rows should be maintained, since none of them are completely filled with zeros. So given an array like:

[
    [0, 22, 3.3, 4.4, 5.5, 0.0, 0.0, 0.0, 0.0, 0.0],
    [1,-211, 0.0, 0.0, 3.3, 4.4, 0.0, 0.0, 0.0, 0.0],
    [2,130, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.1, 0.0],
    [3,-11, 2.2, 3.3, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
]

the idea is to keep the first 4 rows and remove the last 3 (since they only have zeros):

[
    [0, 22, 3.3, 4.4, 5.5, 0.0, 0.0, 0.0, 0.0, 0.0],
    [1,-211, 0.0, 0.0, 3.3, 4.4, 0.0, 0.0, 0.0, 0.0],
    [2,130, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.1, 0.0],
    [3,-11, 2.2, 3.3, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
]

Thank you guys for your support! Now I can go back to my master's degree project!

[jpivarski]

In that case, does row_is_not_padding need an ak.all with axis=-1 in its definition?

[CaueSousa]

I don't know, Jim, but I think Angus's solution is quite right.

For example, my first event had 32 valid rows and 59 filled with zeros. After applying Angus solution, this is how the first event looks like:

[
    [0, 22, 0, -1, -1, -1, -1, 0, 0, ..., -2.8, 0.26, 0.353, 0.094, -3, 0, 0, 0, 0],
    [1, 22, 0, -1, -1, -1, -1, 0, 0, ..., 0.563, 0.259, 0.163, -2.54, 0, 0, 0, 0],
    [2, 22, 0, -1, -1, -1, -1, 0, 0, ..., 1.62, 0.477, 4.76, 2.46, 13, 0, 0, 0, 0],
    [3, 22, 0, -1, -1, -1, -1, 0, 0, ..., -2.25, -0.809, -1.01, 4.17, 0, 0, 0, 0],
    [4, 22, 0, -1, -1, -1, -1, 0, 0, ..., -2.34, -0.255, -0.263, 2.33, 0, 0, 0, 0],
    [5, 130, 0, -1, -1, -1, -1, 0, 0, ..., 0.311, 0.902, 0.29, -7.12, 0.14, 0, 0, 0],
    [6, 130, 0, -1, -1, -1, -1, 0, ..., -0.47, 0.773, -0.393, 2.09, 0.14, 0, 0, 0],
    [7, -11, 0, -1, -1, -1, -1, 0, 0, ..., 0.531, 3.44, 2.02, 9.66, 0, 0, 0, 0],
    [8, 11, 0, -1, -1, -1, -1, 0, 0, ..., -0.852, 0.343, -0.393, -4.08, 0, 0, 0, 0],
    [9, 211, 0, -1, -1, -1, -1, 0, 0, ..., -1.34, 0.09, -0.377, 3.51, 0.14, 0, 0, 0],
    [10, 211, 0, -1, -1, -1, -1, 0, ..., -2.16, -2.85, -4.27, 12.2, 0.14, 0, 0, 0],
    [11, -211, 0, -1, -1, -1, -1, 0, ..., -2.21, -3.17, -4.26, 12.1, 0.14, 0, 0, 0],
    [12, -211, 0, -1, -1, -1, -1, 0, 0, ..., 0.346, 4.35, 1.57, 10.5, 0.14, 0, 0, 0],
    [13, 211, 0, -1, -1, -1, -1, 0, ..., -2.45, -2.73, -2.27, 9.11, 0.14, 0, 0, 0],
    [14, 211, 0, -1, -1, -1, -1, 0, ..., -2.25, -1.75, -2.17, 9.33, 0.14, 0, 0, 0],
    [15, -13, 0, -1, -1, -1, -1, ..., 2.37, -0.007, 5.53, 0.106, 0, -0.018, -4.78],
    [16, -211, 0, -1, -1, -1, -1, 0, ..., 1.77, -0.449, 2.17, 15.7, 0.14, 0, 0, 0],
    [17, 211, 0, -1, -1, -1, -1, 0, 0, ..., -2.21, -1.21, -1.64, 4.3, 0.14, 0, 0, 0],
    [18, -211, 0, -1, -1, -1, -1, 0, 0, ..., -1.29, 0.55, -1.88, 19, 0.14, 0, 0, 0],
    [19, -211, 0, -1, -1, -1, -1, 0, 0, ..., 0.87, 1.16, 1.38, 6.83, 0.14, 0, 0, 0],
    [20, 211, 0, -1, -1, -1, -1, 0, ..., -2.88, -1.72, -0.471, -15.6, 0.14, 0, 0, 0],
    [21, 211, 0, -1, -1, -1, -1, 0, ..., -2.27, -1.12, -1.33, 4.19, 0.14, 0, 0, 0],
    [22, 211, 0, -1, -1, -1, -1, 0, ..., -1.76, -0.321, -1.65, 9.01, 0.14, 0, 0, 0],
    [23, 211, 0, -1, -1, -1, -1, 0, ..., -2.98, -1.42, -0.225, -7.68, 0.14, 0, 0, 0],
    [24, 211, 0, -1, -1, -1, -1, 0, ..., -1.82, -0.333, -1.32, 6.99, 0.14, 0, 0, 0],
    [25, 211, 0, -1, -1, -1, -1, 0, ..., 0.809, 0.941, 0.985, 3.07, 0.14, 0, 0, 0],
    [26, -211, 0, -1, -1, -1, -1, 0, ..., -1.35, 0.286, -1.28, -5.75, 0.14, 0, 0, 0],
    [27, -211, 0, -1, -1, -1, -1, 0, ..., -2.18, -0.763, -1.09, 4.88, 0.14, 0, 0, 0],
    [28, -211, 0, -1, -1, -1, -1, 0, ..., -2.17, -0.72, -1.06, 6.36, 0.14, 0, 0, 0],
    [29, 211, 0, -1, -1, -1, -1, 0, 0, ..., 1.23, 0.423, 1.2, 5.19, 0.14, 0, 0, 0],
    [30, -211, 0, -1, -1, -1, -1, 0, ..., -1.09, 0.578, -1.11, 8.47, 0.14, 0, 0, 0],
    [31, 211, 0, -1, -1, -1, -1, 0, 0, ..., 1.09, 0.564, 1.08, 2.77, 0.14, 0, 0, 0]
]

only the rows filled entirely with zeros were removed.