Sliceable MetaData Class

changelog/455.feature.rst

@@ -0,0 +1 @@
+Add Sliceable `~ndcube.meta.NDMeta` class for axis-associated metadata.

docs/explaining_ndcube/index.rst

@@ -8,6 +8,7 @@ Explaining ``ndcube``
    :maxdepth: 2
 
    data_classes
+   metadata
    slicing
    coordinates
    tabular_coordinates

docs/explaining_ndcube/metadata.rst

@@ -0,0 +1,268 @@
+.. _metadata:
+
+*****************
+Handling Metadata
+*****************
+
+`ndcube`'s data objects do not enforce any requirements on the object assigned to their ``.meta`` attributes.
+However, it does provide an optional object for handling metadata, `~ndcube.NDMeta`.
+This class inherits from `dict`, but provides additional functionalities, chief among which is the ability to associate metadata with data array axes.
+This enables `~ndcube.NDMeta` to update itself through certain operations, e.g. slicing, so that the metadata remains consistent with the associated ND object.
+In this section, we explain the needs that `~ndcube.NDMeta` serves, the concepts underpinning it, and the functionalities it provides.
+
+.. _meta_concepts:
+
+Key Concepts
+============
+
+.. _coords_vs_meta:
+
+Coordinates vs. Axis-aware Metadata
+-----------------------------------
+
+The difference between coordinates and axis-aware metadata is a subtle but important one.
+Formally, a coordinate is a physical space sampled by one or more data axis, whereas axis-aware metadata is information describing the data that can alter along one or more physical dimension.
+An informative example is the difference between time and exposure time.
+The temporal axis of a 3-D image cube samples the physical dimension of time in a strictly increasing monotonic way.
+The times along the temporal axis are therefore coordinate values, not metadata.
+Additionally, a scalar timestamp of a 2-D image is also considered a coordinate in the `ndcube` framework.
+This is because it describes where in the physical dimension of time the data has been sampled, even though it does not correspond to a data axis.
+Because of this, such scalar coordinates are stored in `~ndcube.GlobalCoords`, while coordinates associated with array/pixel axes are stored in the WCS or `~ndcube.ExtraCoords`.
+(See the :ref:`global_coords` section for more discussion on the difference between global and other coordinates.)
+
+By contrast, exposure time describes the interval over which each image was accumulated.
+Exposure time can remain constant, increase or decrease with time, and may switch between these regimes during the time extent of the image cube.
+Like a coordinate, it should be associated with the image cube's temporal axis.
+However, exposure time is reflective of the telescope's operational mode, not a sampling of a physical dimension.
+Exposure time is therefore metadata, not a coordinate.
+
+One reason why it is important to distinguish between coordinates and axis-aware metadata is `ndcube`'s dependence on WCS.
+Most WCS implementations require that there be a unique invertible mapping between pixel and world coordinates, i.e., there is only one pixel value that corresponds to a specific real world value (or combination of such if the coordinate is multi-dimensionsal), and vice versa.
+Therefore, while there may be exceptions for rare and exotic WCS implementations, a good rule of thumb for deciding whether something is a coordinate is:
+coordinates are numeric and strictly monotonic.  Otherwise you have metadata.
+
+The keen-eyed reader may have realised of the above framework that, while not all axis-aligned metadata can be treated as coordinates, all coordinates can be treated like axis-aware metadata.
+This raises the question of why not dispense with coordinates altogether and only have axis-aligned metadata?
+The reason is that the stricter requirements on coordinates have led to a host of powerful coordinate infrastructure that are not valid for generalised axis-aware metadata.
+These include functional WCS implementations which save memory as well as saving compute time through operations such as interpolation, and `~astropy.visualization.wcsaxes.WCSAxes`, which make complicated coordinate-aware plotting easy.
+Therefore, where appropriate, it is beneficial to store coordinates separately from axis-aware metadata.
+
+.. _axis_and_grid_aligned_metadata:
+
+Types of Axis-aware Metadata: Axis-aligned vs. Grid-aligned
+-----------------------------------------------------------
+
+There are two types of axis-aware metadata: axis-aligned and grid-aligned.
+Axis-aligned metadata associates a scalar or string with an array axis.
+It can also assign an array of scalars or strings to multiple array axes, so long as there is one value per associated axis.
+For example, the data produced by a scanning slit spectrograph is associated with real world values.
+But each axis also corresponds to features of the instrument: dispersion (spectral), pixels along the slit (spatial), position of the slit in the rastering sequence (spatial and short timescales), and the raster number (longer timescales).
+The axis-aligned metadata concept allows us to avoid ambiguity by assigning each axis with a label (e.g. ``("dispersion", "slit", "slit step", "raster")``).
+
+By contrast, grid aligned metadata assigns a value to each pixel along axes.
+The exposure time discussion above is an example of 1-D grid-aligned metadata.
+However, grid-aligned metadata can also be multi-dimensional.
+For example, a pixel-dependent response function could be represented as grid-aligned metadata associated with 2 spatial axes.
+
+`~ndcube.NDMeta` supports both axis-aligned and grid-aligned metadata with the same API, which will be discussed in the next section.
+
+.. _ndmeta:
+
+
+NDMeta
+======
+
+.. _initializing_ndmeta:
+
+Initializing an NDMeta
+----------------------
+
+To initialize an `~ndcube.NDMeta`, simply provide it with a `~collections.abc.Mapping` object, e.g. a `dict` or `astropy.io.fits.header.Header`.
+
+.. code-block:: python
+
+  >>> from ndcube import NDMeta
+  >>> raw_meta = {"salutation": "hello", "name": "world"}
+  >>> meta = NDMeta(raw_meta)
+
+We can now access each piece of metadata by indexing ``meta`` as if it were a `dict`:
+
+.. code-block:: python
+
+  >>> meta["name"]
+  'world'
+
+In this example we have provided a very simple set of metadata.
+In fact, it is so simple that there is no practical difference between ``meta`` and a simple `dict`.
+To demonstrate one of the additional functionalities of `~ndcube.NDMeta`, let reinstantiate ``meta``, adding some comments to the metadata.
+To do this, we provide another `~collections.abc.Mapping`, e.g. a `dict`, with the same keys as the main metadata keys, or a subset of them, to the ``key_comments`` kwarg.
+
+.. code-block:: python
+
+
+  >>> key_comments = {"name": "Each planet in the solar system has a name."}
+  >>> meta = NDMeta(raw_meta, key_comments=key_comments)
+
+We can now access the comments by indexing the `~ndcube.NDMeta.key_comments` property:
+
+.. code-block:: python
+
+  >>> meta.key_comments["name"]
+  'Each planet in the solar system has a name.'
+
+Now let's discuss how to initialize how to `~ndcube.NDMeta` with axis-aware metadata.
+(Here, we will specifically consider grid-aligned metadata.  Axis-aligned metadata is assigned in the same way.  But see the :ref:`assigning_axis_aligned_metadata` section for more details.)
+Similar to ``key_comments``, we assign metadata to axes by providing a `~collections.abc.Mapping`, e.g. a `dict`, via its ``axes`` kwarg.
+And like with ``key_comments``, the keys of ``axes`` must be the same, or a subset of, the main metadata keys.
+The axis value must be an `int` or `tuple` of `int` giving the array axes of the data that correspond to the axes of the metadata.
+Note that this means that metadata can be multidimensional.
+Let's say we want to add exposure time that varies with the 1st (temporal) axis of that data, and a pixel response that varies with time and pixel column (1st and 3rd axes).
+
+.. code-block:: python
+
+  >>> import astropy.units as u
+  >>> import numpy as np
+  >>> raw_meta["exposure time"] = [1.9, 2.1, 5, 2, 2] * u.s
+  >>> raw_meta["pixel response"] = np.array([[100., 100., 100., 90., 100.], [85., 100., 90., 100., 100.]]) * u.percent
+  >>> axes = {"exposure time": 0, "pixel response": (0, 2)}
+  >>> meta = NDMeta(raw_meta, axes=axes)
+
+It is easy to see which axes a piece of metadata corresponds to by indexing the `~ndcube.NDMeta.axes` property:
+
+.. code-block:: python
+
+  >>> meta.axes["exposure time"]
+  array([0])
+  >>> meta.axes["pixel response"]
+  array([0, 2])
+
+Finally, it is possible to attach the shape of the associated data to the `~ndcube.NDMeta` instance via the ``data_shape`` kwarg:
+
+.. code-block:: python
+
+  >>> meta = NDMeta(raw_meta, axes=axes, key_comments=key_comments, data_shape=(5, 1, 2))
+
+Or by directly setting the ``~ndcube.NDMeta.data_shape`` property after instantiation:
+
+.. code-block:: python
+
+  >>> meta = NDMeta(raw_meta, axes=axes, key_comments=key_comments)
+  >>> meta.data_shape = (5, 1, 2)
+
+Note that the ``data_shape`` must be compatible with the shapes and associated axes of any axis-aware metadata.
+For example, we couldn't set the length of the first axis to ``6``, because ``meta["exposure time"]`` is associated with the first axis and has a length of ``5``.
+If no ``data_shape`` is provided, it is determined from the axis-aware metadata, if any is provided.
+See the :ref:`data_shape` section for more details.
+
+.. _adding_removing_metadata:
+
+Adding and Removing Metadata
+----------------------------
+
+Because `~ndcube.NDMeta` is a subclass of `dict`, it is possible to add new metadata via the simple ``__setitem__`` API, e.g ``meta[new_key] = new_value``.
+However, this API is not sufficient if we want to add axis-aware or commented metadata.
+This is why `~ndcube.NDMeta` provides an `~ndcube.NDMeta.add` method.
+This method requires the key and value of the new metadata, an optionally accepts a comment and/or axes.
+Let's use this method to add a voltage that varies with time, i.e. the first data axis.
+
+.. code-block:: python
+
+  >>> meta.add("voltage", u.Quantity([1.]*5, unit=u.V), key_comment="detector bias voltage can vary with time and pixel column.", axes=(0,))
+  >>> meta["voltage"]
+  <Quantity [1., 1., 1., 1., 1.] V>
+
+If you try to add metadata with a pre-existing key, `~ndcube.NDMeta.add` will error.
+To replace the value, comment, or axes values of pre-existing metadata, set the ``overwrite`` kwarg to ``True``.
+
+.. code-block:: python
+
+  >>> meta.add("voltage", u.Quantity([-300.]*5, unit=u.V), key_comment="detector bias voltage", axes=(0,), overwrite=True)
+  >>> meta["voltage"]
+  <Quantity [-300., -300., -300., -300., -300.] V>
+
+Unwanted metadata can be removing by employing the ``del`` operator.
+
+.. code-block:: python
+
+  >>> del meta["voltage"]
+  >>> meta.get("voltage", "deleted")
+  'deleted'
+
+Note that the ``del`` operator also removes associated comments and axes.
+
+.. code-block:: python
+
+  >>> meta.key_comments.get("voltage", "deleted")
+  'deleted'
+  >>> meta.axes.get("voltage", "deleted")
+  'deleted'
+
+.. _data_shape:
+
+Data Shape
+----------
+
+The `~ndcube.NDMeta.data_shape` property tracks the shape of the data with which the metadata is associated.
+We have already seen in the :ref:`initializing_ndmeta` section, that it can be assigned during initialization or by subsequently setting the `~ndcube.NDMeta.data_shape` property directly.
+However, if the ``data_shape`` is not provided, it is inferred from the shapes of axis-aware metadata.
+If no axis-aware metadata is present, `~ndcube.NDMeta.data_shape` is empty:
+
+.. code-block:: python
+
+  >>> from ndcube import NDMeta
+  >>> raw_meta = {"salutation": "hello", "name": "world"}
+  >>> meta = NDMeta(raw_meta)
+  >>> meta.data_shape
+  array([], dtype=int64)
+
+If we now add the ``"pixel response"`` metadata that we used, earlier the `~ndcube.NDMeta.data_shape` will be updated.
+
+.. code-block:: python
+
+  >>> meta.add("pixel response", np.array([[100., 85], [100., 100], [100., 90], [90., 100.], [100., 100.]]) * u.percent, axes=(0, 2))
+  >>> meta.data_shape
+  array([5, 0, 2])
+
+Note that since ``"pixel response"`` is associated with the 1st and 3rd axes, those axes now have the same shape as ``"pixel response"``.
+The existence of a 3rd axis, implies the presence of a 2nd.
+However, we have no metadata associated with it, and hence no knowledge of its length.
+It has therefore been assigned a length of ``0``.
+
+Now that the shape has been set for the 1st and 3rd axes, subsequently added grid-aligned metadata associated with those axes must be compatible with those axis lengths.
+For example, if we add a 1-D ``"exposure time"`` and associate it with the 1st axis, it must have a length of of ``5``, otherwise an error will be raised:
+
+.. code-block:: python
+
+  >>> meta.add("exposure time", [1.9, 2.1, 5, 2, 2] * u.s, axes=0)
+
+Moreover, if we now directly set the `~ndcube.NDMeta.data_shape` via ``meta.data_shape = new_shape``, we cannot change the length of axes already associated with grid-aligned metadata, without first removing or altering that metadata.
+However, these restrictions do not apply if we want to change the shape of the 2nd axis, or add new metadata to it, because its length is ``0``, and hence considered undefined.
+
+.. code-block:: python
+
+  >>> meta.add("row temperature", [-10, -11, -12] * u.deg_C, axes=1)
+  >>> meta.data_shape
+  array([5, 3, 2])
+
+.. _assigning_axis_aligned_metadata:
+
+Assigning Axis-aligned Metadata
+-------------------------------
+
+So far, we have only dealt with grid-aligned metadata, i.e. axis-aware metadata which provides a value for each pixel.
+To provide axis-aligned metadata, i.e. where each axis has a single value (see :ref:`axis_and_grid_aligned_metadata`), provide a scalar or string for a single axis, or a 1-D array-like with the same length as the number of associated axes for multi-axis-aligned metadata.
+
+.. code-block:: python
+
+  >>> meta.add("axis name", np.array(["a", "b", "c", "d"]), axes=(0, 1, 2, 3))
+
+Note that the length of ``"axis name"`` is the same as the number of its associated axes.
+Also note that we have now indicated that there is 4th axis.
+``meta.data_shape`` has therefore been automatically updated accordingly.
+
+.. code-block:: python
+
+  >>> meta.data_shape
+  array([5, 3, 2, 0])
+
+However, because axis-aligned metadata does not tell us about the length of the axes, the 4th axis has been assigned a length of zero.

docs/explaining_ndcube/slicing.rst

@@ -416,3 +416,109 @@ Let's say we axis order of the ``linewidths`` cube was reversed.
   array([2, 1], dtype=object)
 
 The same result is obtained.
+
+.. _meta_slicing:
+
+Slicing NDMeta
+==============
+
+Slicing is arguably the most important of `~ndcube.NDMeta`'s functionalities that sets it apart from a simple `dict`.
+It allows axis-aware metadata to be kept consistent with its associated data.
+
+Because `~ndcube.NDMeta` is a subclass of `dict`, the standard Python slicing API is reserved for accessing the values of keys.
+Therefore, slicing is achieved by applying Python's slicing API to `~ndcube.NDMeta`'s `~ndcube.NDMeta.slice` attribute.
+
+.. code-block:: python
+
+  >>> # Instantiate an NDMeta object with axis-aware and axis-agnostic metadata.
+  >>> import astropy.units as u
+  >>> import numpy as np
+  >>> from ndcube import NDMeta
+  >>> raw_meta = {"salutation": "hello", "name": "world",
+  ...             "exposure time": u.Quantity([2] * 4, unit=u.s),
+  ...             "pixel response": np.ones((4, 5))}
+  >>> axes = {"exposure time": 0, "pixel response": (1, 2)}
+  >>> meta = NDMeta(raw_meta, axes=axes, data_shape=(4, 4, 5))
+
+  >>> # Slice NDMeta object.
+  >>> sliced_meta = meta.slice[0, 1:3]
+  >>> sliced_meta.data_shape
+  array([2, 5])
+
+Note that by applying the slice item ``[0, 1:3]`` to ``meta``, the shape of the ``sliced_meta`` has been altered accordingly.
+The first axis has been sliced away, the second has been truncated to a length of 2, and the third remains unchanged.
+The shape of ``"pixel response"`` has been altered:
+
+.. code-block:: python
+
+  >>> sliced_meta["pixel response"].shape
+  (2, 5)
+
+while ``"exposure time"`` has been reduced to a scalar:
+
+.. code-block:: python
+
+  >>> sliced_meta["exposure time"]
+  <Quantity 2. s>
+
+Moreover, because the first axis has been sliced away, ``"exposure time"`` is no longer associated with a data array axis, and so is no longer present in the ``axes`` property:
+
+.. code-block:: python
+
+  >>> list(sliced_meta.axes.keys())
+  ['pixel response']
+
+Finally, note that axis-agnostic metadata is unaltered by the slicing process.
+
+.. code-block:: python
+
+  >>> sliced_meta["salutation"]
+  'hello'
+  >>> sliced_meta["name"]
+  'world'
+
+
+Automatically Slicing NDMeta Attached to Other ND Objects
+---------------------------------------------------------
+
+`~ndcube.NDMeta` has a dunder property called ``__ndcube_can_slice__``, which, by default, is set to ``True``.
+When attached to `~ndcube.NDCube`, `~ndcube.NDCubeSequence`, and `~ndcube.NDCollection`, this causes the `~ndcube.NDMeta` to be sliced as part of the overall slicing of the associated data class.
+To demonstrate this, let's reinstantiate the same metadata object as in the above section and attach it to a new instantiation of ``my_cube``.
+
+.. code-block:: python
+
+  >>> meta = NDMeta(raw_meta, axes=axes)
+  >>> my_cube = NDCube(data, wcs=wcs, meta=meta)
+  >>> my_cube.shape
+  (4, 4, 5)
+  >>> my_cube.meta.data_shape
+  array([4, 4, 5])
+
+Now let's apply the same slice item to the cube as we applied to ``meta`` in the above section.
+Note that shape of the resultant `~ndcube.NDCube` and its associated `~ndcube.NDMeta` object now have the same new shape consistent with the slice item.
+
+.. code-block:: python
+
+  >>> sliced_cube = my_cube[0, 1:3]
+  >>> sliced_cube.shape
+  (2, 5)
+  >>> sliced_cube.meta.data_shape
+  array([2, 5])
+
+Furthermore, the metadata's values, axis-awareness, etc., have also been altered in line with the slice item.
+In fact, ``sliced_cube.meta`` is equivalent to ``sliced_meta`` from the previous section, because we have applied the same slice item to two equivalent `~ndcube.NDMeta` objects.
+
+.. code-block:: python
+
+  >>> sliced_cube.meta.data_shape
+  array([2, 5])
+  >>> sliced_cube.meta["pixel response"].shape
+  (2, 5)
+  >>> sliced_cube.meta["exposure time"]
+  <Quantity 2. s>
+  >>> list(sliced_cube.meta.axes.keys())
+  ['pixel response']
+  >>> sliced_cube.meta["salutation"]
+  'hello'
+  >>> sliced_cube.meta["name"]
+  'world'

docs/nitpick-exceptions

@@ -16,3 +16,5 @@ py:class a set-like object providing a view on D's keys
 py:class an object providing a view on D's values
 py:class None.  Remove all items from D.
 py:class a shallow copy of D
+py:class v, remove specified key and return the corresponding value.
+py:class None.  Update D from dict/iterable E and F.