Sliceable MetaData Class

[DanRyanIrish]

Description

This PR introduces a sliceable Meta class based on dict, which allows metadata to be associated with data axes and the whole class to be sliced using the normal Python slicing API. Metadata not associated with any axes is not touched by the slicing mechanism.

This PR introduces two types of axis-aware metadata: axis-aligned and grid-aligned. Axis-aligned metadata gives one value for each of its associated axes, and is scalar or a string if only associated with one axis. This type of metadata is only changed if the axis/axes within which it is associated is dropped by slicing. In this case, the values corresponding to the dropped axes are also dropped.

Grid-aligned metadata provides a value for each data array element in its associated axis/axes. This metadata cannot be scalar. Since this metadata mirrors at least part of the data array, it is sliced as if it were data, and its shape is kept consistent with its associated data axes. If all its axes are sliced away, the scalar value at the row/column location at which it was sliced is kept, and its axis-awareness is dropped, i.e. it becomes a piece of normal non-axis-aware scalar metadata.

API Summary

>>> import astropy.units as u
>>> from ndcube.meta import Meta

# Define a Meta object associated with data of shape (3, 4, 5)
>>> header = {"salutation": "hello",
              "name": "world",
              "exposure_time": ([2] * 3) * u.s,  # 1-dimension grid-aligned (pixel-wise) metadata
              "instrument_axes": ["slit step", "slit", "dispersion"],  # axis-aligned (scalar/string-per-axis) metadata
              "pixel_efficiency": np.random.ones(4, 5)  # multi-dimension grid-aligned (pixel-wise) metadata}
>>> comments = {"salutation": "This is polite."}
>>> axes = {"exposure_time": 0,
              "instrument_axes": (0, 1, 2),
              "pixel_efficiency": (1, 2)}
>>> meta = Meta(header, axes=axes, comments=comments, data_shape=(3, 4, 5))

# Use slice item that you would apply to the cube on the Meta object.
>>> sliced_meta = meta[0:2, 2]  # Reduce length of 0th axis and drop 1st axis.
>>> sliced_meta.shape  # Shape has been updated.
(2, 5)
>>> sliced_meta["exposure_time"]  # Exposure time now length of new 0th axis
<Quantity [2., 2.] s >
>>> sliced_meta["instrument_axes"]  # instrument axes has dropped slit entry
["slit step", "dispersion"]
>>> sliced_meta.axes["instrument_axes"]
(0, 1)
>>> sliced_meta["pixel_efficiency"]  # pixel efficiency is now 1-D of length 5
array([1., 1., 1., 1., 1.])
>>> sliced_meta.axes["pixel_efficiency"]
(1,)

# Rebin the Meta object
>>> rebinned_meta = meta.rebin((1, 2, 1))
>>> rebinned_meta["exposure_time"]  # grid-aligned metadata associated with non-rebinned axes remains unchanged...
<Quantity [2., 2., 2.] s >
>>> rebinned_meta.axes["exposure_time"]  # ...and still associated with its axis/axes
(0,)
>>> rebinned_meta["pixel_efficiency"]  # grid-aligned metadata associated with rebinned axes remains...
np.random.ones(4, 5)
>>> rebinned_meta.axes["pixel_efficiency"]  # ...but grid-awareness is removed
None
>>> rebinned_meta["instrument_axes"]  # axis-aligned (scalar/string-per-axis) metadata unchanged...
["slit step", "slit", "dispersion"]
>>> rebinned_meta.axes["instrument_axes"]  # ...and still associated with axes as they haven't been dropped
(0, 1, 2)

To Do

• Implement __delitem__ and remove remove method.
• Implement setter for NDMeta.data_shape which verifies that it's compatible with pre-existing axis- and grid-aligned metadata.
• Improve narrative docs

[pep8speaks]

Hello @DanRyanIrish! Thanks for updating this PR.

• In the file ndcube/meta.py:

Line 71:70: W504 line break after binary operator

• In the file ndcube/tests/helpers.py:

Line 136:38: E721 do not compare types, use 'isinstance()'
Line 118:67: W504 line break after binary operator

Comment last updated at 2021-11-18 14:32:19 UTC

[DanRyanIrish]

@nabobalis Could you give this PR a review? It should be very similar if not the same as what's in your sunraster PR.

[DanRyanIrish]

@Cadair this PR is ready for review. To pre-empt possible concerns regarding DKIST tools, the introduction of this new class does not require that metadata be sliceable. Instead it allows for it if desired.

In order for the Meta object to be sliceable, Meta.shape must be set via the data_shape kwarg during intialization. This must be set to the shape of the data array with which the metadata object is associated. If Meta.shape is None, the object is not sliceable.

Not all entries in the Meta object have to be sliceable. Only those who have a corresponding entry in the Meta.axes dict will be. This gives the axes with which the metadata is associated and is therefore used to work out how to slice it. If Meta.shape is set, but there are no entries in Meta.axes, then slicing has no effect on the Meta object.

Therefore, as far as I understand, this new class should not threaten your assumption in the DKIST tools that .meta never changes.

[DanRyanIrish]

@Cadair, this is ready to merge as far as I can see. Do you still want more time to look at it given your DKIST concerns? As I above, I don't think this causes an issues as the new functionalities are optional and don't have to change previous behaviour.

[DanRyanIrish]

ping @Cadair. A friendly reminder for a review on this PR :)

[Cadair]

Can't see any major issues.

[DanRyanIrish]

@nabobalis Can you interpret this doc error for me?

[samaloney]

I think you need to add the warning to the nitpick-exceptions file so something like this but no 100% sure of the syntax.

py:class Update D from mapping/iterable E and F