Mnist dataset: where is the image "normalized"?

[hmf]

I am trying to adapt the ai.djl.basicdataset.Mnist so that both the images and labels are the same (same digit image). I have taken the class and created a new one where the labels = data; has been changed. It seems to work. However, the label contains an image that is not normalized (float values between 0..1). I am assuming this is done when we do a RandomAccessDataset.getData.

So my questions are:

1. Where is the normalization of the image (data) done?
2. How can I force this normalization on the labels?

TIA

[hmf]

I have come up with a solution and am recording here in case it is useful for someone else. And I am still unsure that this is the way to do it.

The ai.djl.basicdataset.Mnist extends the ai.djl.training.dataset.ArrayDataset, which in turn extends ai.djl.training.dataset.RandomAccessDataset. All these datasets have or inherit the following members:

    protected Pipeline pipeline;
    protected Pipeline targetPipeline;

which are set by:

        public T optPipeline(Pipeline pipeline) 
        public T optTargetPipeline(Pipeline targetPipeline) 

We can set the targetPipeline to have the same pipeline as the data pipeline. When are these transformations used? They are activated when one calls:

    AEMnist mnist  =
      AEMnist.builder()
        .optUsage(usage)
        .setSampling(batchSize, randomize) 
        .optLimit( maxBatches)
        .optTargetPipeline( Pipeline(new ToTensor()) )
        .build()

 DataIterable iter = mnist.getData

from the ai.djl.training.dataset.RandomAccessDataset abstract class. The iter is in effect an Iterable<Batch>. The next() method, which is used to access the batches, calls the method:

    private Batch fetch(List<Long> indices, int progress) throws IOException 

that checks if any pipeline with one or more transforms exists. If they do exist, then the transform method is applied respectively to the data and the labels. I still haven't grasped the use of Builder or BaseBuilder to set the pipelines, among other parameters. The method build is used to instantiate the actual data classes, wether Mnist or any other class that extends ArrayDataset or even RandomAccessDataset.

Note that ai.djl.modality.cv.transform.ToTensor is a Transform that simply calls the ai.djl.modality.cv.util.NDImageUtils.toTensor method. Side note here: the DJL documentation uses the word normalization, but I think the term usually referred to this operation is standardization.

So now that leaves me with one more issue: how to replace the current numeric labels with the original image? What I did was copy and paste Mnist to a new class AEMnist and did the following changes:

Assign the data to the labels so:

       //labels = new NDArray[] {readLabel(labelItem)};
       aelabels = new NDArray[] {readIntLabel(labelItem)};
       data = new NDArray[] {readData(imageItem, aelabels[0].size())};
       labels = data;
       prepared = true;

and then hijacked the readLabel method so:

    private NDArray readLabel(Artifact.Item item) throws IOException {
        return readData(item, aelabels[0].size());
    }

    private NDArray readIntLabel(Artifact.Item item) throws IOException {
        try (InputStream is = resource.getRepository().openStream(item, null)) {
            if (is.skip(8) != 8) {
                throw new AssertionError("Failed skip data.");
            }

            byte[] buf = Utils.toByteArray(is);
            try (NDArray array = manager.create(new Shape(buf.length), DataType.UINT8)) {
                array.set(buf);
                return array.toType(DataType.FLOAT32, false);
            }
        }
    }

Now, I looked at the underlying logic, which is quite complex, and am thinking that:

1. Maybe there is an easier/better way of doing this
2. Am unsure that using the same reference as data for the label will not mess up some of the logic below

Can anyone tell me if the changes I made are ok (point (2) above)?

TIA

[zachgk]

I mentioned another option for how you could create an Mnist variant in #601 (reply in thread), but this should also work fine.

For the normalization, you really just need a division by 255. So, you can add that as a transform like you are thinking with new Pipeline().addTransform(a -> a.div(255f)).addTransform(new ToTensor()). You can also add a LambdaBlock to your model as well, but I think the transform is a slightly nicer approach.

[hmf]

but this should also work fine.

Great. I think I will still try your example to see if it is simpler.

Quick note. I don't think we need to add the div(255f) because in the ToTensor calls:

NDImageUtils.toTensor(array)

which uses this method:

ai.djl.ndarray.internal.NDArrayEx 
public ai.djl.ndarray.NDArray toTensor()

has this line:

result = result.div(255.0).transpose(0, 3, 1, 2);

But your code serves as a good examples of the use of pipes and transforms.

Thank you.

[hmf]

@zachgk Cannot seem to mark your reply as an answer. If you copy your reply as an "answer", I will tag it as the answer. Sorry.

[zachgk]

I mentioned another option for how you could create an Mnist variant in #601 (reply in thread), but this should also work fine.

For the normalization, you really just need a division by 255. So, you can add that as a transform like you are thinking with new Pipeline().addTransform(a -> a.div(255f)).addTransform(new ToTensor()). You can also add a LambdaBlock to your model as well, but I think the transform is a slightly nicer approach.

[hmf]

Thank you