Utilities for handling UV data products for low-frequency aperture array telescopes.
ska-ost-low-uv provides a UVX data format and Python class for storing and handling interferometric data.
These codes use the UVData class from pyuvdata to convert the raw HDF5 correlator output files to science-ready data formats like UVFITS, MIRIAD, and CASA MeasurementSets.
Additionally, data can be loaded into the Visibility data model from ska-sdp-datamodels, which is based on xarray.
Some simple calibration and imaging utilities are provided in the postx submodule. This is an optional extra,
that requires the matvis package for simulations.
Download this repository, then install via pip install .. To install optional extras:
pip install .[postx] # Post-correlation imaging and QA tools
pip install .[sdp] # Support for SDP Visibility
pip install .[casa] # Installs python-casacore for MS support
Or run pip install .[all] to install all optional extras.
To install from scratch using conda, download this repository, cd into the directory, and run
conda env create -f environment.yml
conda activate ska_ost_low_uv
You can then use conda activate ska_ost_low_uv to start up the environment, and conda deactivate to leave it.
Then run pip install . and any extras (e.g. pip install .[postx]).
Once installed, a command-line utility, aa_uv, will be available:
> aa_uv -h
usage: aa_uv [-h] -o OUTPUT_FORMAT [-c ARRAY_CONFIG] [-n TELESCOPE_NAME] [-s] [-j] [-b] [-B] [-x FILE_EXT] [-i CONTEXT_YAML] [-w NUM_WORKERS] [-v] [-p PARALLEL_BACKEND] [-N N_INT_PER_FILE] [-z] infile outfile
AAVS UV file conversion utility
positional arguments:
infile Input filename
outfile Output filename
options:
-h, --help show this help message and exit
-o OUTPUT_FORMAT, --output_format OUTPUT_FORMAT
Output file format (uvx, uvfits, miriad, ms, uvh5, sdp). Can be comma separated for multiple formats.
-c ARRAY_CONFIG, --array_config ARRAY_CONFIG
Array configuration YAML file. If supplied, will override ska_ost_low_uv internal array configs.
-n TELESCOPE_NAME, --telescope_name TELESCOPE_NAME
Telescope name, e.g. 'aavs3'. If supplied, will attempt to use ska_ost_low_uv internal array config.
-s, --phase-to-sun Re-phase to point toward Sun (the sun must be visible!). If flag not set, data will be phased toward zenith.
-b, --batch Batch mode. Input and output are treated as directories, and all subfiles are converted.
-B, --megabatch MEGA batch mode. Runs on subdirectories too, e.g. eb-aavs3/2023_12_12/*.hdf5.
-x FILE_EXT, --file_ext FILE_EXT
File extension to search for in batch mode
-i CONTEXT_YAML, --context_yaml CONTEXT_YAML
Path to observation context YAML (for SDP / UVX formats)
-w NUM_WORKERS, --num-workers NUM_WORKERS
Number of parallel processors (i.e. number of files to read in parallel).
-v, --verbose Run with verbose output.
-p PARALLEL_BACKEND, --parallel_backend PARALLEL_BACKEND
Joblib backend to use: 'loky' (default) or 'dask'
-N N_INT_PER_FILE, --n_int_per_file N_INT_PER_FILE
Set number of integrations to write per file. Only valid for MS, Miriad, UVFITS, uvh5 output.
-z, --zipit Zip up a MS or Miriad file after conversion (flag ignored for other files)
The converter needs a yaml configuration file, which can be supplied with the -c argument, or internal defaults can be used instead via the -n argument (for '-n aavs2' and '-n aavs3'):
# Convert AAVS3 HDF5 data into a MeasurementSet
> aa_uv -n aavs3 -o ms correlator_data.hdf5 my_new_measurement_set.ms
from ska_ost_low_uv.io import hdf5_to_pyuvdata, hdf5_to_sdp_vis
def hdf5_to_pyuvdata(filename: str, yaml_config: str) -> pyuvdata.UVData:
""" Convert AAVS2/3 HDF5 correlator output to UVData object
Args:
filename (str): Name of file to open
yaml_config (str): YAML configuration file with basic telescope info.
See README for more information
Returns:
uv (pyuvdata.UVData): A UVData object that can be used to create
UVFITS/MIRIAD/UVH5/etc files
"""
def hdf5_to_sdp_vis(fn_raw: str, yaml_raw: str) -> Visibility:
""" Generate a SDP Visibility object from a AAVS2 HDF5 file
Args:
fn_raw (str): Filename of raw HDF5 data to load.
yaml_raw (str): YAML config data with telescope information
See https://github.com/ska-low/aa_uv/tree/main/config#uv_configyaml
Notes:
The HDF5 files generated by AAVS2/3 are NOT the same format as that found in
ska-sdp-datamodels HDF5 visibilty specification.
The AAVS DAQ receiver code in aavs-system has some info on the HDF5 format, here:
https://gitlab.com/ska-telescope/aavs-system/-/blob/master/python/pydaq/persisters/corr.py
"""To help install into a fresh conda environment, a environment.yml is provided. To create a new environment, download this repo then run:
conda env create -f environment.yml
This will create an environment called aavs, which you enter by typing conda activate aavs. You can then activate this and install via:
conda activate aavs
pip install .
Pip will then install ska_ost_low_uv and the few final packages that are not available in the conda-forge package manager (e.g. pygdsm, pyuvdata, ska-sdp-datamodels).
If you have an existing Python 3 installation, you can install with pip via:
pip install git+https://github.com/ska-sci-ops/aa_uv/edit/main/README.md
Alternatively, download this repository and install using pip install .. A list of required packages can be found in the pyproject.toml.
ska_ost_low_uv is built upon the following astronomy packages:
- astropy for coordinate calculations.
- pyuvdata for interferometric data format conversion.
- matvis for visibility simulation.
- pygdsm for diffuse sky model generation.# ska-ost-low-uv
SKA Ost Low UV provides utilities for handling UV data products for SKA Low.
The documentation for this project, including how to get started with it, can be found in the docs folder, or browsed in the SKA development portal: