Is there an easy way to know what input/output types a zoo model supports?

[tadayosi]

Firstly, thank you for this great project!

I'm now developing Apache Camel to support DJL applications more comprehensively.

I'm trying to run as many provided zoo models as possible with Apache Camel, but one thing I'm struggling with is knowing which input/output types each zoo model supports. Some are indicated in the documentation, error messages, and TranslatorFactory source code, but each time I need to figure it out in an ad hoc way through trial and error.

Is there any way to know the information on the supported input/output types in a more uniform and comprehensive way, e.g., as in this example?
https://github.com/deepjavalibrary/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/ListModels.java

Thank you!

[frankfliu]

@tadayosi

Here is an example:

        Map<Application, List<Artifact>> models = ModelZoo.listModels();
        models.forEach(
                (app, list) -> {
                    String appName = app.toString();
                    list.forEach(artifact ->  {
                        logger.info("{} {}", appName, artifact);
                        Map<String, ?> arguments = artifact.getArguments();
                        String factoryName = ArgumentsUtil.stringValue(arguments, "translatorFactory");
                        ClassLoader cl = ClassLoaderUtils.getContextClassLoader();
                        TranslatorFactory factory = ClassLoaderUtils.initClass(cl, TranslatorFactory.class, factoryName);
                        Set<Pair<Type, Type>> types = factory.getSupportedTypes();
                    });
                });

[tadayosi]

@frankfliu Thanks, that's very helpful! The only thing still not clear to me is that there are artifacts that don't have the translatorFactory in their arguments and for those the above code throws NPE.

What's the best way to inspect the in/out types for those artifacts? Do we still need to find them in an ad hoc way?

[frankfliu]

@tadayosi

Most of the model zoo model has translatorFactory, the only miss one is cv/image_classification, you should be able to just use: ImageClassificationTranslatorFactory.

Let me if you see any other application has missing translatorFactory

[tadayosi]

Here is the list of models that miss translatorFactory:

CV.ACTION_RECOGNITION :: ai.djl.mxnet/action_recognition/0.0.1/vgg16_ucf101 {"backbone":"vgg16","dataset":"ucf101"}
CV.ACTION_RECOGNITION :: ai.djl.mxnet/action_recognition/0.0.1/inceptionv3_ucf101 {"backbone":"inceptionv3","dataset":"ucf101"}
CV.IMAGE_CLASSIFICATION :: ai.djl.pytorch/resnet/0.0.1/traced_resnet50 {"layers":"50","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.pytorch/resnet/0.0.1/traced_resnet18 {"layers":"18","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.pytorch/resnet/0.0.1/resnet101_v1 {"layers":"101","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.tensorflow/resnet/0.0.1/resnet50 {"flavor":"v1","layers":"50","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.tensorflow/mobilenet/0.0.1/mobilenet {"flavor":"v2","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.zoo/mlp/0.0.3/mlp {"dataset":"mnist"}
CV.IMAGE_CLASSIFICATION :: ai.djl.zoo/resnet/0.0.2/resnetv1 {"layers":"50","flavor":"v1","dataset":"cifar10"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/googlenet/0.0.1/googlenet {"dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/darknet/0.0.1/darknet53 {"layers":"53","flavor":"v3","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/squeezenet/0.0.1/squeezenet1.0 {"flavor":"1.0","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/inceptionv3/0.0.1/inceptionv3 {"dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/mobilenet/0.0.1/mobilenet0.25 {"flavor":"v1","multiplier":"0.25","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/mobilenet/0.0.1/mobilenet0.5 {"flavor":"v1","multiplier":"0.5","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/mobilenet/0.0.1/mobilenet0.75 {"flavor":"v1","multiplier":"0.75","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/mobilenet/0.0.1/mobilenet1.0 {"flavor":"v1","multiplier":"1.0","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/mobilenet/0.0.1/mobilenetv2_0.25 {"flavor":"v2","multiplier":"0.25","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/mobilenet/0.0.1/mobilenetv2_0.5 {"flavor":"v2","multiplier":"0.5","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/mobilenet/0.0.1/mobilenetv2_0.75 {"flavor":"v2","multiplier":"0.75","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/mobilenet/0.0.1/mobilenetv2_1.0 {"flavor":"v2","multiplier":"1.0","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/mobilenet/0.0.1/mobilenetv3_small {"flavor":"v3_small","multiplier":"1.0","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/mobilenet/0.0.1/mobilenetv3_large {"flavor":"v3_large","multiplier":"1.0","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/senet/0.0.1/senet_154 {"layers":"154","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/alexnet/0.0.1/alexnet {"dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/mlp/0.0.1/mlp {"dataset":"mnist"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/resnet/0.0.1/resnet18_v1 {"layers":"18","flavor":"v1","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/resnet/0.0.1/resnet50_v2 {"layers":"50","flavor":"v2","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/resnet/0.0.1/resnet101_v1 {"layers":"101","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/resnet/0.0.1/resnet152_v1d {"layers":"152","flavor":"v1d","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/resnet/0.0.1/resnet50_v1_cifar10 {"layers":"50","flavor":"v1","dataset":"cifar10"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/se_resnext/0.0.1/se_resnext101_32x4d {"layers":"101","flavor":"32x4d","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/se_resnext/0.0.1/se_resnext101_64x4d {"layers":"101","flavor":"64x4d","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/vgg/0.0.1/vgg11 {"layers":"11","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/vgg/0.0.1/vgg13 {"layers":"13","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/vgg/0.0.1/vgg16 {"layers":"16","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/vgg/0.0.1/vgg19 {"layers":"19","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/vgg/0.0.1/vgg11_bn {"flavor":"batch_norm","layers":"11","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/vgg/0.0.1/vgg13_bn {"flavor":"batch_norm","layers":"13","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/vgg/0.0.1/vgg16_bn {"flavor":"batch_norm","layers":"16","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/vgg/0.0.1/vgg19_bn {"flavor":"batch_norm","layers":"19","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/densenet/0.0.1/densenet121 {"layers":"121","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/densenet/0.0.1/densenet161 {"layers":"161","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/densenet/0.0.1/densenet169 {"layers":"169","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/densenet/0.0.1/densenet201 {"layers":"201","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/xception/0.0.1/xception {"flavor":"65","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/resnest/0.0.1/resnest14 {"layers":"14","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/resnest/0.0.1/resnest26 {"layers":"26","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/resnest/0.0.1/resnest50 {"layers":"50","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/resnest/0.0.1/resnest101 {"layers":"101","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/resnest/0.0.1/resnest200 {"layers":"200","dataset":"imagenet"}
CV.IMAGE_CLASSIFICATION :: ai.djl.mxnet/resnest/0.0.1/resnest269 {"layers":"269","dataset":"imagenet"}
CV.INSTANCE_SEGMENTATION :: ai.djl.mxnet/mask_rcnn/0.0.1/mask_rcnn_resnet101_v1d_coco {"backbone":"resnet101","flavor":"v1d","dataset":"coco"}

So, yes, mostly they are image classification but there are also a few from action recognition and instance segmentation.

[frankfliu]

@tadayosi

ACTION_RECOGNITION is also considered as image classification, you can use ImageClassificationTranslatorFactory as well. Will take a look mask_rcnn_resnet101_v1d_coco model

[frankfliu]

@tadayosi

I fixed mask_rcnn model issue. You can try it again.

Please remove the cache:

rm ~/.djl.ai/cache/repo/model/cv/instance_segmentation/ai/djl/mxnet/mask_rcnn/metadata.json

[tadayosi]

Awesome, thanks! Confirmed now it works with ai.djl.mxnet/mask_rcnn.