Connectomics data often involves a combination of microscopy imagery and segmentation, labels of distinct objects applied to this imagery. While exploring the data in tools like Neuroglancer is great, a common task is often to make figures overlaying 2d images and segmentation sliced from the larger data. ImageryClient is designed to make it easy to generate aligned cutouts from imagery and segmentation, and make it efficient to produce attractive, publication-ready overlay images.
Because of the size of these volumes, cloud-based serverless n-d array file storage systems are often used to host this data. CloudVolume has become an excellent general purpose tool for accessing such data. However, imagery and segmentation for the same data are hosted at distinct cloud locations and can differ in basic properties like base resolution. Moreover, imagery and segmentation have data that means intrensically different things. Values in imagery indicate pixel intensity in order to produce a picture, while values in segmentation indicate the object id at a given location. ImageryClient acts as a front end for making aligned cutouts from multiple cloudvolume sources, splitting segmentations into masks for each object, and more.
We make use of Numpy arrays and Pillow Images to represent data. Both are extremely rich tools, and to learn more about them please see the appropriate documentation for information about saving data to image files and more.
The standard installation:
pip install imageryclient
If you have installation issues due to Cloudvolume, which has a fairly complex set of requirements, we recommend looking at their github issues page for help.
Here, we will use the ImageryClient to get some data from the Kasthuri et al. 2014 dataset hosted by Google. In its simplest form, we just intialize an ImageryClient object with an image cloudpath and a segmentation cloudpath. Values are taken from the layers in the linked neuroglancer state.
import imageryclient as ic
img_src = 'precomputed://gs://neuroglancer-public-data/kasthuri2011/image_color_corrected'
seg_src = 'precomputed://gs://neuroglancer-public-data/kasthuri2011/ground_truth'
img_client = ic.ImageryClient(image_source=img_src, segmentation_source=seg_src)Cutouts are defined by their bounds, which can be specified by providing bounds. The most direct form is a pair of points representing the upper and lower corners of the bounding box. By default, coordinates are in the default resolution of the imagery as you would see in neuroglancer.
bounds = [
[5119, 8477, 1201],
[5519, 8877, 1202]
]
image = img_client.image_cutout(bounds)
# Use PIL to visualize
from PIL import Image
Image.fromarray(image.T)
Since often we are using analysis points to center an image on, we can alternatively define a center point and the width/height/depth of the bounding box (in voxels). The same image could be achived from this specification.
ctr = [5319, 8677, 1201]
img_width = 400
image = img_client.image_cutout(ctr, bbox_size=(img_width, img_width))You can also generate bounds from a center and size.
bounds = ic.bounds_from_center(ctr, width=img_width, height=img_width, depth=1)A very important element in ImageryClient is the resolution, which specifies the units that you are using when providing bounds.
You can check the resolution that the client is expecting with img_client.resolution.
The resolution will default to the highest resolution available for the imagery, but you can specify another resolution manually.
For example, to say that you are going to provide bounds in [8,8,30] voxel units, you would add the resolution argument:
img_client = ic.ImageryClient(..., resolution=[8,8,30])You can also explicitly set the resolution to "image" or "segmentation" to use the highest available resolution available for either.
Resolution can also be specified in each of the functions for image or segmentation cutouts, but will default to the client values.
Note that the volumetric data itself is not necessarily at the resolution specified, but rather this parameter determines how to interpret the coordinates.
The resolution is set by the image and segmentation mip levels, which can be added either when creating the ImageryClient instance or when doing any cutout download.
By default, ImageryClient will use the highest resolution mip level that is not labeled as a "placeholder" in CloudVolume.
When upper and lower bounds are specified or a bbox_size is used, the resolution will change with mip level but the field of view that is downloaded will remain the same.
Alternatively, one might want to download an image with a specific size in pixels and a specific mip level without having to calculate what bounding box would get you that..
This can be done in image_cutout by specifying the center point in the place of bounds and also specify image_size as a 2- or 3-element array.
In this case, the center point will be adjusted according to the resolutions specified, while the field of view will change with image size.
In practice, this only is needed for non-default mip levels. If you specify mip level, this approach will always yield an image with the same size while a bounds-based approach will get smaller with increasing mips as the effective resolution gets coarser.
For example, using bounds:
image = img_client.image_cutout(bounds, mip=3)
Image.fromarray(image.T)
And using specified pixel dimensions:
img_size=(400, 400)
image = img_client.image_cutout(ctr, mip=3, image_size=img_size)
Image.fromarray(image.T)
You can also use the scale_to_bounds=True argument to upscale an image to the size specified in the bounding box, equivalent to having one pixel for each voxel as measured by the resolution parameter.
An aligned segmentation cutout is retrieved similarly. Note that segmentations show segment ids, and are not directly visualizable. However, in this case we can convert to a uint8 greyscale and see the gist, although there are many better approaches to coloring segmentations that will be shown later. Note that for dynamic segmentations, you can use the timestamp parameter to (optionally) set the time at which segmentation will e looked up.
seg = img_client.segmentation_cutout(bounds)
import numpy as np
Image.fromarray( (seg.T / np.max(seg) * 255).astype('uint8') )
Specific root ids can also be specified. All pixels outside those root ids have a value of 0.
root_ids = [2282, 4845]
seg = img_client.segmentation_cutout(bounds, root_ids=root_ids)
Image.fromarray( (seg.T / np.max(seg) * 255).astype('uint8') )
It's often convenient to split out the segmentation for each root id as a distinct mask. These "split segmentations" come back as a dictionary with root id as key and binary mask as value.
split_seg = img_client.split_segmentation_cutout(bounds, root_ids=root_ids)
Image.fromarray((split_seg[ root_ids[0] ].T * 255).astype('uint8'))
Aligned image and segmentations can be downloaded in one call, as well. If the lowest mip data in each differs in resolution, the lower resolution data will be optionally upsampled to the higher resolution in order to produce aligned overlays. Root ids and split segmentations can be optionally specified. This is the best option if your primary goal is overlay images.
image, segs = img_client.image_and_segmentation_cutout(bounds,
split_segmentations=True,
root_ids=root_ids)Note that image_size is not an option for joint image and segmentation calls, because it's not clear which bounds to use.
If this is needed, you can use the img_client.segmentation_bbox_size_from_dimensions or img_client.image_bbox_size_from_dimensions to get the appropriate bbox_size argument for a segmentation-based image dimension (or image-based, respectively).
Now let produce an overlay of segmentation and imagery to highlight a particular synapse.
Overlays are returned as a PIL Image, which has convenient saving options but can also be converted to RGBa via a simple np.array call.
Note that if imagery isn't specified, the segmentations are colored but not put over another image.
Segmentations must be either a list or a dict, such as comes out of split segmentation cutouts.
ic.composite_overlay(segs, imagery=image)
Colors are chosen by default from the perceptually uniform discrete HUSL Palette as implemented in Seaborn, and any color scheme available through Seaborn's color_palette function is similarly easy to specify. Alpha is similarly easy to set.
ic.composite_overlay(segs, imagery=image, palette='tab10', alpha=0.4)
Colors can also be specified in the same form as the segmentations, e.g. a dictionary of root id to RGB tuple.
colors = {2282: (0,1,1), # cyan
4845: (1,0,0)} # red
ic.composite_overlay(segs, imagery=image, colors=colors)
While the overlay guides the eye, it can also obscure the imagery. Because of that, one can also use highly configurable outlines instead of solid overlays. The default option puts the outlines along the outside of the segmentations, but omits lines where two segmentations touch.
ic.composite_overlay(segs, imagery=image, outline=True, alpha=0.5, width=15, colors=colors)
Outlines can also be put inside of the segmentation and width can be specified.
Additionally, setting merge_outline to False will not omit outlines in places where segmentations touch.
Lots of different effects are possible!
ic.composite_overlay(segs,
imagery=image,
outline=True,
alpha=1,
width=3,
merge_outline=False,
side='in',
colors=colors)
All of the functions are designed to also work for 3d image stacks.
Image and segmentation cutouts will return 3d arrays instead of 2d ones.
However, note that composite images will come back as a list of PIL images.
An optional dim argument will perform the slicing on axes other than the z-axis, although anisotropy in voxel resolution will not be accounted for.
ctr = [5019, 8677, 1211]
width = 100
z_slices = 3
bounds_3d = ic.bounds_from_center(ctr, width=width, height=width, depth=z_slices)
image, segs = img_client.image_and_segmentation_cutout(bounds_3d, split_segmentations=True)
overlays = ic.composite_overlay(segs, imagery=image, alpha=0.3, width=3,
merge_outline=False, side='in')
overlays[0]
In order to quickly assemble sequential images into a series, we can stack them.
A direction argument will let you specify vertical instead of the default, and spacing can be adjusted as well.
ic.stack_images(overlays)
While the ImageryClient was designed to work specifically with the CAVEclient and its associated suite of services, it should work with any cloudvolume project.
However, if you are working within an CAVEclient-compatible project, a CAVEclient can be used to help configure the ImageryClient, filling in the imagery source, the segmentation source, authentication information, and the default resolution used in Neuroglancer.
For example, we can download the data around this view of the MICRoNs mouse visual cortex data from Neuroglancer.
from caveclient import CAVEclient
client = CAVEclient('minnie65_public_v343') . # Note that you have to set up a token for this to work, see below.
img_client = ic.ImageryClient(client=client)
ctr = [240640, 207872, 21360]
image, segs = img_client.image_and_segmentation_cutout(ctr,
split_segmentations=True,
bbox_size=(1024, 1024),
scale_to_bounds=True,
)
ic.composite_overlay(segs, imagery=image, palette='husl')
Note that the following code requires setting up a CAVE token to access the server. See here for details.