[RFC] ModelZoo Server solution

[lanking520]

DJL currently offers a complete ModelZoo design for different engines. It contains a complete model description, model files and model artifacts. We would like to extend its capability by offering a ModelZoo Server. The goal of the ModelZoo Server is to provide a solution to generalize the standard for model loading and help it smoothly used into most of the Java applications. The ModelZoo Server we are trying to provide are listed below:

• A Standard model format
• A frontend/cli to upload all DJL supported models (e.g TensorFlow, PyTorch, Apache MXNet, PaddlePaddle...)
• A frontend/cli to download the DJL supported models

Model Format

Comparing to the major model hub (e.g TorchHub, TensorFlowHub, HuggingFace), DJL can offer but not limited to:

• Version control
• Checksum validation
• Model descriptions
• Model license

These information will be stored into a metadata.json (https://github.com/awslabs/djl/blob/master/mxnet/mxnet-model-zoo/src/test/resources/mlrepo/model/cv/image_classification/ai/djl/mxnet/resnest/metadata.json) file. Apart from that, a complete DJL model should also have a definition of Translator and all artifacts. This design is aimed to be compatible with most of the model format.

Model uploading

Developers will use “DJL-upload-template” jar to build a local version of the DJL model. After that, They can submit the generated template model into the Model Server through CLI or frontend interface.

public abstract class ModelZooTest<I, O> {

  abstract ZooModel<I, O> getZooModel()
  
  abstract I getSampleInput();
  
  abstract O getSampleOutput();

  @Test
  public void test() {
    ZooModel<I, O> model = getZooModel();
    Predictor<I, O> predictor = model.newPredictor();
    O output = predictor.predict(getSampleInput());
    Assert.asserEquals(output, getSampleOutput());
  }
}

On the server end, the server will run the packaged model with a test provided by the user. After that, the model will be verified manually/automatically to check if there is any issues. Finally the model will generate a metadata.json and uploaded and stored into user designated location. if additional Java packages being used, they can also mention them in the metadata.json.

In addition, user can also leverage the ModelZoo webUI to manually upload the model through providing model files and fill in the required information for metadata.json.

Model downloading

To download the model, they can leverage the existing ModelZoo API to define a criteria and load the model directly into their Java application. All model store, versioning control are being done by the DJL ModelZoo API. They can also manually download the model along with a starter template through the web browser, such as the #525. User have the ModelZoo access can use CLI/interface to automatically remove/update the ModelZoo models. All other users can find what models are available in the ModelZoo server.

Call for action

There are several PoCs developing in this project , we will share the updates/progress in this RFC. If you are interested in participating, please feel free to leave a message here or email us: [email protected].

[roywei]

Some existing solutions we can reference:

• HuggingFace
• TensorFlow Hub
• Gluon Model Zoo
• PyTorch Model Zoo