Segment Anything - Next Steps

[tirthasheshpatel]

Segment Anything model in KerasCV

#1987 added the Segment Anything model to KerasCV but it needs a guide to demonstrate training and prompting the trained model. It would also be nice to have some benchmarks for it. Following is a list of tasks that still need to be addressed:

• Writing a guide to use the model (similar to the predictor demo on the original repo)
• Writing a guide to generate masks for the entire image (similar to the automatic mask generation demo on the original repo)
• (@IMvision12) A guide for end-to-end model training. See the appendix "A. Segment Anything Model and Task Details" from the paper that explains the training method used for the existing presets.
• Adding support for text prompts (e.g. CLIP). The paper mentions that a CLIP model was used to encode text prompts.
• (Optionally) It'd also be nice to have benchmarks for all the backends and document it somewhere.

[IMvision12]

@tirthasheshpatel i can take up end - to - end model training guide

[AnimeshMaheshwari22]

@tirthasheshpatel I can take the Writing a guide to use the model

[cosmo3769]

@tirthasheshpatel I can take up "Writing a guide to generate masks for the entire image". Thank you.

[cosmo3769]

@tirthasheshpatel I can take up "Writing a guide to generate masks for the entire image". Thank you.

@tirthasheshpatel Since this guide requires implementation of predictor and maskgenerator, I am working on this.

[cosmo3769]

@tirthasheshpatel Since the predictor and automaticmaskgenerator work is in wip and almost done by you as I checked your repo here: https://github.com/tirthasheshpatel/segment_anything_keras/tree/keras-cv-update. If there is any help required, I would be happy to help. Thank you.