ModelZoo artifacts: switch to git-lfs

[ambroise-arm]

ModelZoo hosts neural networks useful to Autoware (yolov3, apollo_lidar_segmentation, …). They are meant to be compiled with TVM ahead of time, and the resulting artifacts are to be consumed by the relevant packages in Autoware at build and run time. The ModelZoo CI does the compilation step for all supported architectures and backends, and uploads the results to an Amazon S3 bucket.

In Autoware.Auto we have a neural_networks package that handles downloading the artifacts and making the raw files available, and we have a tvm_utility package that provides an interface for other packages to access and use those files.

Currently, the download step of neural_networks gets the files from the S3 bucket. But it may be better to instead have the artifacts as git-lfs files.

The benefits of switching to git-lfs are:

• Integrated versioning. No need to manually check if a new version exists because the artifacts will already be tied to a commit, and no need for other extra work to ensure synchronization between the bucket and the repository.
• Finer-grained updates. If only the networks that had a modification on the ModelZoo side are updated in Autoware, then it will be a smaller delta to be downloaded by the user compared to the full archive currently held in the bucket.

And the drawbacks are:

• Losing the full-automation. Someone will need to manually commit the artifacts to Autoware (preferably only those that were actually changed in ModelZoo). We (Arm) have an internal validation process to go through before pushing binaries externally, which is time consuming and not assured to have a favorable outcome. We want to avoid doing that, so someone else will need to do it. The good news is that the bucket can still be useful for this, as the commiter can simply download the compiled networks from there, and avoid having to compile them locally.

Notes:

• There will probably be a need for a setup script to be run by the user either way (possibly along with the current vcs import command and possible future other git lfs pull usage; or the script could include those).
  • With the current S3 hosting, the download step happens as part of the build of neural_networks, which creates a bandwidth concern that can be solved by moving the download to that script.
  • Switching to git-lfs means that using a simple git-lfs pull will download the files of all supported architectures (currently aarch64 and x86_64), doubling the amount of bandwidth and user storage. We could indicate in the documentation a custom --include ... argument to use, but it may become confusing for the user. So having that script could also be useful in that case.
• My understanding is that a custom lfs hosting platform can be configured for the repository. Looking at the GitHub pricing for lfs, it feels like it would be the best economic decision to indeed have a custom hosting, because both of storage and bandwidth. I don’t know what is currently done and it will need to be discussed with the relevant people first.
• With git-lfs, my understanding is that bandwidth will also be consumed when forks are created. Although I expect there will be far less downloads by forks than by users.
• The transition from S3 bucket to git-lfs will only require minimal changes to the current implementation. Specifically, only in neural_networks, basically removing the download-related steps, without changing the interface to dependent packages (currently only tvm_utility).

To achieve the transition, I think we will need:

• Confirmation that the consensus is still on using git-lfs.
• An adapted lfs hosting solution for the repository.
• A list of named individuals responsible for creating the PRs to update the lfs files when necessary. I think that would prevent having to struggle to find someone every time. Possibly simply the Autoware maintainers.
• More details on how to handle the git lfs pull step, where it should appear in the documentation, if it should download the networks by default, if a script should be created.

I think it will be cleaner to sort this first and then port the neural_networks and tvm_utility packages from Autoware.Auto with the necessary modifications. Except if other development efforts depend on those packages, in which case they probably should be ported first "as is" and with the modifications done afterwards.
Issue for porting the packages: autowarefoundation/autoware.universe#628

[xmfcx]

Pricing

My understanding is that a custom lfs hosting platform can be configured for the repository. Looking at the GitHub pricing for lfs, it feels like it would be the best economic decision to indeed have a custom hosting, because both of storage and bandwidth. I don’t know what is currently done and it will need to be discussed with the relevant people first.

First for current pricing information:

• https://docs.github.com/en/billing/managing-billing-for-git-large-file-storage/about-billing-for-git-large-file-storage#about-billing-for-git-large-file-storage
  • One data pack costs $5 per month, and provides a monthly quota of 50 GB for bandwidth and 50 GB for storage.
• https://aws.amazon.com/s3/pricing/
  • First 50 TB / Month : $0.023 per GB (50 GB: $1.15)
    So S3 is definitely on the cheaper side.

Here I've tried to include transfers too and very similar pricing.

Also it is possible to host Gitlab on custom dedicated servers and have custom LFS (I tried this before). But it is not possible with Github, we have to use their solutions.

Functionality

git-lfs enables to preserve full history. But it prevents us from being able to do finer control over our data.

We cannot remove some old useless files without overriding those commits and ruining the git history.
And for this reason it only grows in size.

@ambroise-arm said:

Finer-grained updates. If only the networks that had a modification on the ModelZoo side are updated in Autoware, then it will be a smaller delta to be downloaded by the user compared to the full archive currently held in the bucket.

I don't get this part, git lfs doesn't have binary delta capability, it hosts all large binary files as they are, you still download full files even if binaries are slightly different.

Also there is this thing with git-lfs that it doesn't work with ssh protocol. which make it very annoying when moving the repos to a different configuration like switching https to ssh for keys.

For online rant on git-lfs: https://news.ycombinator.com/item?id=27134972

I would like to use S3 for hosting large files and their interface for keeping the portability. And not be tied to 'git-lfs'.

[xmfcx]

@esteve would like to try https://git-annex.branchable.com/special_remotes/S3/ on this.

[ambroise-arm]

Also it is possible to host Gitlab on custom dedicated servers and have custom LFS (I tried this before). But it is not possible with Github, we have to use their solutions.

I see, that's not great.

We cannot remove some old useless files without overriding those commits and ruining the git history.
And for this reason it only grows in size.

The S3 solution will also grow in size, because we probably want to keep some older versions of the artifacts (at least the ones used in previous tagged versions of autoware), but it may grow slower (because we may allow to remove artifacts only used in non-tagged autoware commits, at the cost of braking builds on those). So S3 can indeed be better for that, but not miraculous either.

I don't get this part, git lfs doesn't have binary delta capability, it hosts all large binary files as they are, you still download full files even if binaries are slightly different.

Right. So this point is more about the current implementation/use we have of the S3 solution than S3 itself. Right now there are 2 files in the bucket: networks-aarch64.tar.gz and networks-x86_64.tar.gz that contain all the networks/backends combinations for each architecture. So a change such as "adding a new network" in ModelZoo leads to an effective re-download of every file.
We could make the archives finer grained, such as one per network, containing all backends. Or even finer grained with one archive per file. But each step requires some more work to implement, and the full granularity sounds a lot like re-implementing the main git-lfs functionalities.
So it could be done as follow-up work, but our S3 solution doesn't currently have that same granularity as git-lfs.

[xmfcx]

So this point is more about the current implementation/use we have of the S3 solution than S3 itself. Right now there are 2 files in the bucket: networks-aarch64.tar.gz and networks-x86_64.tar.gz that contain all the networks/backends combinations for each architecture.

I didn't realize this. The models should be compressed separately and kept in a folder like structure. Then download script could download the folder like in https://stackoverflow.com/a/46680575

Should be fairly simple. And for keeping older models and such, we can follow what TF Model Zoo does and keep folders organized for versioning.

[xmfcx]

More on S3 folders: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html

[ambroise-arm]

cc @JWhitleyWork as you were part of earlier discussions

[xmfcx]

Why not do what https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md does?

Have every model hosted compressed individually and versioned like that. And hosted not in git-lfs.

We could do the same on S3 Bucket using folders.

Model name                                                                                                                                                                  | Speed (ms) | COCO mAP | Outputs
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------: | :----------: | :-----:
[CenterNet HourGlass104 512x512](http://download.tensorflow.org/models/object_detection/tf2/20200713/centernet_hg104_512x512_coco17_tpu-8.tar.gz)                    | 70         | 41.9           | Boxes
[CenterNet HourGlass104 Keypoints 512x512](http://download.tensorflow.org/models/object_detection/tf2/20200711/centernet_hg104_512x512_kpts_coco17_tpu-32.tar.gz)                    | 76         | 40.0/61.4           | Boxes/Keypoints
[CenterNet HourGlass104 1024x1024](http://download.tensorflow.org/models/object_detection/tf2/20200713/centernet_hg104_1024x1024_coco17_tpu-32.tar.gz)               | 197       | 44.5           | Boxes
[CenterNet HourGlass104 Keypoints 1024x1024](http://download.tensorflow.org/models/object_detection/tf2/20200711/centernet_hg104_1024x1024_kpts_coco17_tpu-32.tar.gz)               | 211       | 42.8/64.5          | Boxes/Keypoints
[CenterNet Resnet50 V1 FPN 512x512](http://download.tensorflow.org/models/object_detection/tf2/20200711/centernet_resnet50_v1_fpn_512x512_coco17_tpu-8.tar.gz)     | 27         | 31.2           | Boxes
[CenterNet Resnet50 V1 FPN Keypoints 512x512](http://download.tensorflow.org/models/object_detection/tf2/20200711/centernet_resnet50_v1_fpn_512x512_kpts_coco17_tpu-8.tar.gz)     | 30         | 29.3/50.7         | Boxes/Keypoints
[CenterNet Resnet101 V1 FPN 512x512](http://download.tensorflow.org/models/object_detection/tf2/20200711/centernet_resnet101_v1_fpn_512x512_coco17_tpu-8.tar.gz)     | 34         | 34.2           | Boxes
[CenterNet Resnet50 V2 512x512](http://download.tensorflow.org/models/object_detection/tf2/20200711/centernet_resnet50_v2_512x512_coco17_tpu-8.tar.gz)     | 27         | 29.5           | Boxes
[CenterNet Resnet50 V2 Keypoints 512x512](http://download.tensorflow.org/models/object_detection/tf2/20200711/centernet_resnet50_v2_512x512_kpts_coco17_tpu-8.tar.gz)     | 30         | 27.6/48.2           | Boxes/Keypoints
[CenterNet MobileNetV2 FPN 512x512](http://download.tensorflow.org/models/object_detection/tf2/20210210/centernet_mobilenetv2fpn_512x512_coco17_od.tar.gz)     | 6         | 23.4           | Boxes
[CenterNet MobileNetV2 FPN Keypoints 512x512](http://download.tensorflow.org/models/object_detection/tf2/20210210/centernet_mobilenetv2fpn_512x512_coco17_kpts.tar.gz)     | 6         | 41.7           | Keypoints

[ambroise-arm]

(from #150 (comment)):

The models should be compressed separately and kept in a folder like structure. Then download script could download the folder like in https://stackoverflow.com/a/46680575

Downloading through the aws cli requires credentials, even when trying to download public files. Maybe it can be acceptable if a dedicated “public” role is created on the S3 side that can be publicly shared, and whose credentials can be hardcoded in the repo’s file. Or maybe something like this: https://selfscrum.medium.com/how-to-use-aws-cli-without-user-credentials-on-ec2-d686874a2a81, although I didn’t look into it in detail. And if it is not acceptable (or doable) then we can probably mimic the directory discovery by keeping an index file in each directory, listing the files of that directory. Handling the parsing of that index file would be something else to add to the download script to be written for autoware.

Why not do what https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md does?

Absolutely. We are planning on formalizing the versioning of the ModelZoo repository (described in autowarefoundation/modelzoo#63; I still need to edit it to reflect the transition to .universe and create a discussion for it in this repo, but I feel like that should happen after the present discussion is settled and the tvm_utility package ported to .universe). And that already plans for a folder-like structure in the bucket that would look like what is done for the tensorflow models, except with versions instead of dates as the folder name.

Have every model hosted compressed individually and versioned like that.

That sounds good. We would need to think about the implementation. If we only update some of the files from one version to the next, how does a fresh download of the latest model know what to download? The file structure would be something like this (where version 2.0 only updated fileB):

/1.0/fileA
/1.0/fileB
/2.0/fileB

If autoware targets 2.0 files, we will want /1.0/fileA and /2.0/fileB to be downloaded.
Maybe the index file described above can be the list of files that correspond to a targeted version instead of the files in a directory.
/1.0/idx.txt would list /1.0/fileA and /1.0/fileB. And /2.0/idx.txt would list /1.0/fileA and /2.0/fileB.

[esteve]

@xmfcx the problem in my opinion is not about the storage medium, but it's more about how to distribute them when we have pieces of software that need them, hence why storing them somehow with Git (be it git-lfs, git-annex, etc.) solves this because we don't have to worry about integrating those files with how we distribute Autoware.

If we end up using some kind of script to download large files (instead of a git-based solution), we must make sure that it's consistent across all repos and files, otherwise we at risk of using different storage solutions, with different scripts, etc.

[xmfcx]

@esteve I think for the repository it should be clear what kind of big files we are planning to store. And have them on a organized centralized place. And use git for anything else. For example I wouldn't be against git-lfs for usage for image files. Organizing them is always a hassle, they are relatively small and many in numbers and perfect for git-lfs.

But for point clouds and nn-models lets organize them somewhere else and provide links for those files or downloading one-liner scripts.

I am not opposed to using git-lfs with s3 on the same repository. This is not about inconsistency. It's about using the right tool for the right problem.

[JWhitleyWork]

I don't quite understand how using git-lfs is not "organized and centralized." It keeps the files with the code, they are version-controlled, and you can download them (optionally) with just a git lfs fetch <file-name>. It isn't like they're all heaped in a big folder somewhere. They live in the same structure as the rest of the files. This is honestly the best part about git-lfs.

[xmfcx]

@JWhitleyWork You are right that the files can be organized with git-lfs too. In fact ONNX model zoo uses it. Although they seem to be having second opinions because of the cost.

This is also about balancing on how much history we want to preserve on these big files. If we are going to keep even the oldest versions available, then yes git-lfs will act same as storing them elsewhere.

But if we want to delete certain files, it is easy to do in traditional hosts. The file will just not be available anymore. But in git-lfs, if further commits were made, without rewriting the history, you cannot remove that file. I prefer having an option to just remove large files in expense of having some older git history state not functioning, while still keeping the commit sha's intact.

I think first we should look into what we are trying to store as big files. .PCD files and neural net models are the only instances I know so far.

For neural nets, i think holding them in folders and just providing readable links is the best solution. Similar to neural nets, we should also have a .PCD map database. We could handle it similarly to the neural nets and in the main repository, we would just link to these.

And for the small in size but large in count files like images, git-lfs can still be used.

https://github.com/pytorch/hub is the most advanced example I could find for how large models can be hosted while being able to get contributions.

And tensorflow zoo just does traditional hosting.

[esteve]

I gave git-annex a try, it looks promising, but not without its own caveats.

Pros:

• Supports different types of backends (e.g. S3)
• We can arbitrarily point to existing files on the web (e.g. https://github.com/esteve/git-annex-test contains a link to a Debian ISO)
• Integrated into the Git workflow, no need for special scripts for every new file we download
• Crossplatform (Linux, macOS, Windows)
• We can download specific files, without downloading all of them. Useful when we have multiple architectures and we are only interested on a handful (more for info https://git-annex.branchable.com/git-annex-get/)

Cons:

• Setup on GitHub requires more work because git-annex is not natively supported by GitHub
• Documentation is a bit hard to follow and sometimes confusing
• Making files available locally requires a separate command (git annex get, similar to git lfs pull)

In one of my tests, I have pushed the git-annex data to an S3 bucket, but I can't seem to be able to configure the region nor the name of the object in the bucket, not sure if it's because I didn't follow the documentation correctly or if it's a bug. Anyway, I've removed the bucket and will try again.

Overall, my first impressions are good, and seems to fix the issues that git-lfs has, while at the same time having more features (native support for S3 is huge plus in my opinion). I prefer this solution much more than every package having its own script for downloading files or having different storage mechanisms for different packages.

[LucaFos]

I don't want to talk for other people, but after the messages on this page and discussing with @xmfcx and @ambroise-arm I think we're in a good position to decide that we want to go on with S3.
Can I ask if there's anyone who has any objection?

[esteve]

I agree with using S3 folders.