Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Instance Segmentation Experiment #1169

Open
lichtefeld opened this issue Sep 7, 2022 · 0 comments
Open

Instance Segmentation Experiment #1169

lichtefeld opened this issue Sep 7, 2022 · 0 comments

Comments

@lichtefeld
Copy link
Collaborator

The goal of this experiment is to test various ways of performing instance segmentation and the impact this has on stroke extraction and then training the GNN. While a complete experiment with this may desire a complex grid search to optimize parameters this could be tuned later in the process. For now we're mostly interested in the broad impacts of changing our segmentation process (Additional point of experimentation: Color / internal object segmentation). This issue aims to outline the experiment design & necessary technical changes to support it.

Experiment Layout

Curriculum

This experiment will use two different base sets of curriculum enabling three easy-to-produce variants. A set of ~400 images from the month 4 object demonstration rendered in Unreal Engine 4, a set of ~400 images from the Unreal Engine 5 rendering, and a final curriculum which combines both of the above source data (maybe this is worth a variant with double the data and a variant with a 50/50 split for each object but the same number of training instances?)

Instance Segmentation

The month 4 curriculum already has instance segmentation provided from the legacy AirSim integration. This will be the only curriculum for which results over this segmentation method will be available.

All curriculums will undergo instance segmentation on a base-model Mask R-CNN via a server API as well as a STEGO segmentation model. It's not quite in the spirit of adam's no labeled training data but given our minimal set of labeled data we could use that to fine-tune the top layer of the Mask R-CNN to improve it's performance (Expected outcome by discussions with Blake)

Stroke Extraction

We'll perform stroke extraction using the default parameters provided to us by Sheng. If we discover issues with these extractions we may consider additional tuning metrics to train over but I would like to keep this as only one configuration if possible. That said, I may consider a few of the improvements we've discussed internally including:

  • Maximum stroke-length
  • short-stroke combination
  • others?

GNN Training

I'll use the provided GNN script to train the model. There may be parameters of the GNN training which we can tailor but this would require coordinating with Sheng in some capacity.

Technical Changes

STEGO & Mask R-CNN API

Right now the scripts only can support running one of the two of these models at a time, I'd like to adapt Blake's scripts to support running both in either two different servers (this seems simpler) or a single node with two different endpoints.

Curriculum generation script

Right now we have some source images in a desired curriculum layout. My goal is to write a unified preprocessing script which is designed to invoke a series of jobs to preprocess the required elements as well as manage the placement of files into the correct directories.

This requires determining how to integrate scripts from ASU into our processing steps. This currently requires utilizing different python environments. It would be nice to design this system in a way that minimizes the amount of multi-conda setup each of us needs to do to experiment on the cluster. I may borrow from other configuration of using Local python venvs for each of these modules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant