Optical flow dataset loader #12

fral92 · 2017-06-05T13:24:10Z

The optical flow is now loaded directly from the parallel loader
This makes the OF loading no dataset-specific
Add parameters to decide if the OF has to be computed or loaded from path
Add parameter to select the OF type

fvisin · 2017-09-25T22:20:09Z

dataset_loaders/data_augmentation.py

@@ -365,6 +372,10 @@ def random_transform(x, y=None,
        An image.
    y: array of int
        An array with labels.
+    sequence_names: list of strings
+        A list of prefix and names for the current sequence


What does it mean?

fvisin · 2017-09-25T22:23:12Z

dataset_loaders/data_augmentation.py

+    sequence_names: list of strings
+        A list of prefix and names for the current sequence
+    data_path: string
+        Current dataset path


You always pass self.path here:

call it path

change the docstring into: "The local path of the dataset. Hardcoded to self.path"

fvisin · 2017-09-26T21:35:17Z

dataset_loaders/data_augmentation.py

+    if return_optical_flow:
+        from skimage import io
+        flow = []
+        if compute_optical_flow:


Rather than having yet another flag, can you check if the optical flow exists on disk and compute it (and save it) only if it's missing?

Note: add a shared_path parameter as well so that if the OF is missing in the local path you can look for it in the shared path. If it's missing in both path you compute it and save it in both paths.

fvisin · 2017-09-26T21:36:09Z

dataset_loaders/data_augmentation.py

+        flow = []
+        if compute_optical_flow:
+            flow = optical_flow(x, rows_idx, cols_idx, chan_idx,
+                                return_rgb=return_optical_flow == 'rgb')


Why is optical_flow_type not being used here? I think it should.

fvisin · 2017-09-26T21:37:08Z

dataset_loaders/data_augmentation.py

+                                return_rgb=return_optical_flow == 'rgb')
+        else:
+            if optical_flow_type not in ['Brox', 'Farn', 'LK', 'TVL1']:
+                raise RuntimeError('Unknown optical flow type')


As I said, I think optical_flow_type should be used also in the other branch. Therefore this check should be done at the beginning of this new section (L588)

fvisin · 2017-09-26T22:01:53Z

dataset_loaders/data_augmentation.py

-        flow = optical_flow(x, rows_idx, cols_idx, chan_idx,
-                            return_rgb=return_optical_flow=='rgb')
-        x = np.concatenate((x, flow), axis=chan_idx)
+        x = np.concatenate((x, np.array(flow)), axis=chan_idx)


ditto (move before)

fvisin · 2017-09-26T22:05:13Z

dataset_loaders/data_augmentation.py

+    optical_flow_type: string
+        Indicates the method used to generate the optical flow. The
+        optical flow is loaded from a specific directory based on this
+        type.


Not sure what all of these mean, but I'll comment on the docstrings once you fix the code.

fvisin · 2017-09-26T22:05:35Z

dataset_loaders/data_augmentation.py

@@ -562,6 +585,43 @@ def random_transform(x, y=None,
                                    fill_mode=fill_mode,
                                    fill_constant=cvalMask,
                                    rows_idx=rows_idx, cols_idx=cols_idx))
+    flow = None


No need for this (see suggested modifications below)

fvisin · 2017-09-26T22:14:21Z

dataset_loaders/data_augmentation.py

+                    of = of.astype(x.dtype) / 255.
+                else:
+                    raise RuntimeError('Optical flow not found for this '
+                                       'file: %s' % of_path)


I am having a really hard time understanding this code. can you please answer these questions when you have time?

What does sequence_names stand for?

This is my understanding of L606-622. Can you confirm it is correct and intended?
for _ in sequence_names: if file_index == 0 and not repeat_1st_opt_flow: of = np.zeros(x.shape[1:], x.dtype) else: if repeat_1st_opt_flow: frame = filenames[0] else: frame = filenames[-1] of_path = os.path.join(optical_flow_path, frame + 'jpg') of = io.imread(of_path).astype(x.dtype) / 255.

If what I wrote is correct, why do you set the first OF to zero when repeat_1st_opt_flow is False?

If what I wrote is correct, why do you return the OF of the last frame for every frame after the first, when repeat_1st_opt_flow is False?

fvisin · 2017-09-26T22:21:16Z

dataset_loaders/parallel_loader.py

+                self.path + '/')[1].split('/', 1)[1]
+            # Get all the filenames for the current batch to load
+            current_filenames = [fname for fname in self.filenames if
+                                 el[0][0] in fname]


I am not sure if this is right. My understanding is that it's filtering the filenames with the same prefix as the one of the current batch. Is that right? If so, do you really need to do it for each batch? It seems expensive.

If this makes sense, since we would end up passing self.path, self.shared_path and current_filenames which is also computed out of self, it's probably more convenient to pass self directly to the random_transform function (e.g., as a dataset argument) and access all those attributes from the function itself.

* Do not create pointer seq_x, seq_y. It is easy to introduce bugs when operations on them are not reflected in the original dictioary. * Pass the dataset object rather than all its parameteres. * NOTE: This commit breaks the optical flow. Will be fixed in the next commit.

* Allow to load OF from disk from .npy files * Compute the OF at run time if missing (only Farneback available ATM) * Add parameter to select the OF type * Add parameter to select whether to return OF as RGB or displacement

* Better OF visualization, adapted from TransFlow

fvisin · 2017-11-17T19:05:27Z

@marcociccone can you review this when you have time?

marcociccone mentioned this pull request Jun 7, 2017

Optical flow loader for Davis dataset #10

Closed

fvisin added the needs review label Jun 12, 2017

fvisin force-pushed the master branch 2 times, most recently from c6a8d70 to ff0bbfe Compare June 22, 2017 18:30

fvisin suggested changes Sep 26, 2017

View reviewed changes

fvisin added changes requested and removed needs review labels Sep 26, 2017

fvisin force-pushed the optical_flow branch 2 times, most recently from ed8f880 to ca55ff1 Compare October 26, 2017 21:22

fvisin and others added 4 commits October 27, 2017 17:57

Improve optical flow to load/store from disk

cd2f428

* Allow to load OF from disk from .npy files * Compute the OF at run time if missing (only Farneback available ATM) * Add parameter to select the OF type * Add parameter to select whether to return OF as RGB or displacement

Improved OF visualization

1a5e620

* Better OF visualization, adapted from TransFlow

Cleanup and comment the new OF visualization code

439eda9

Replace numpy OF visualization with advanced one

863e8ec

fvisin force-pushed the optical_flow branch from ca55ff1 to 863e8ec Compare October 27, 2017 15:58

Replace gist with new OF code repo

032e737

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optical flow dataset loader #12

Optical flow dataset loader #12

fral92 commented Jun 5, 2017

fvisin Sep 25, 2017

fvisin Sep 25, 2017

fvisin Sep 26, 2017

fvisin Sep 26, 2017

fvisin Sep 26, 2017

fvisin Sep 26, 2017

fvisin Sep 26, 2017

fvisin Sep 26, 2017

fvisin Sep 26, 2017

fvisin Sep 26, 2017

fvisin Sep 26, 2017

fvisin commented Nov 17, 2017

Optical flow dataset loader #12

Are you sure you want to change the base?

Optical flow dataset loader #12

Conversation

fral92 commented Jun 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fvisin commented Nov 17, 2017