Video Noise Contrastive Estimation (VINCE)

This is a repository containing code used to implement the models in the paper
Watching the World Go By: Representation Learning from Unlabeled Videos (https://arxiv.org/abs/2003.07990).

Environment Setup

We recommend using Anaconda to manage your environment setup and run our code. The following commands will create an environment similar to ours with minimal requirements.

Conda

conda create -n video-env python=3.6.8
conda deactivate
conda env update -n video-env -f env.yml
conda activate video-env
pip install git+https://github.com/danielgordon10/dg_util.git -U

Virtualenv

If you instead prefer virtualenv or similar, we have also provided a requirements.txt.

virtualenv --python=python3.6 video-env
source video-env/bin/activate
pip install -r requirements.txt

Downlaod Random Related Video Views (R2V2)

1. To download via the command line, run python download_scripts/download_r2v2.py.
2. This may fail to download some of the files due to Google drive rate limiting, but you may still be able to download them via the web browser. You will have to manually download the links in r2v2_drive_urls.txt.

Notes

	Size (GB)	Number of Files	Number of Images	Number of Folders	Number of Source Videos
Train	110	2,788,424	2,784,328	4096	696,082
Val	8.8	226,620	222,524	4096	55,631

• Some folders have many more images than others. This is expected.
• The video and frame ids are also provided in datasets/info_files/r2v2_ids_train.txt and datasets/info_files/r2v2_ids_val.txt

Downloading your own set of YouTube videos

If you would like to download a different set of YouTube videos, you may still find our code helpful. Here is a basic workflow for downloading many YouTube videos.

1. Create cookies.txt
2. Create a list of many YouTube URLs to download.
   1. One option would be to use youtube_scrape/search_youtube_for_urls.py
   2. Another would be YouTube-8m URLs (https://github.com/danielgordon10/youtube8m-data)
3. Run python run_cache_video_dataset.py --title cache --description caching --num-workers 100 after appropriately formatting the files.
   • Note - You can often use more workers than your CPU has threads because YouTube downloading tends to be the bottleneck.
4. youtube_scrape/download_kinetics.py is a convenient file for downloading Kinetics videos.

Create cookies.txt

1. Install this extension.
2. Go to any youtube video: https://www.youtube.com/watch?v=AKQE9RyOIMY
3. Click the cookie icon and save the data into youtube_scrape/cookies.txt or adjust the COOKIE_PATH variable in constants.py

Training

Train VINCE

1. Download R2V2 training data or create your own dataset to train on.
2. Read over the arguments list in arg_parser.py.
3. Train the model. We have provided an example train script as well as a debug script to check everything is working. Edit the paths in the file to point to your data/output locations.

Train baselines

1. The official MoCo baseline is available at https://github.com/facebookresearch/moco, but for our work, we wrote our own version.
2. We have provided an example train script to train this model.
3. We additionally include MoCoV2 baseline scripts for ResNet50 at vince/train_moco_v2.sh. Pretrained weights and results are currently not provided.
4. We additionally include the Jigsaw method from PIRL and an accompanying script vince/train_vince_jigsaw.sh. Pretrained weights and results are currently not provided.

Train End Task

1. We include various end tasks and an interface for easily adding more. Training scripts for each task are available at:
   1. end_tasks/train_imagenet.sh
   2. end_tasks/train_sun_scene.sh
   3. end_tasks/train_kinetics_400.sh
   4. end_tasks/train_tracking.sh
2. New end tasks can be added by creating a new solver which inherits from EndTaskBaseSolver and an accompanying dataset which inherits from BaseDataset.

Evaluation

1. While training each end task, evaluation is done after every epoch on a val set.
2. If more evaluation is needed, it can be added by implementing run_eval for that solver. For an example, see solvers/end_task_tracking_solver.py and end_tasks/eval_tracking.sh.

Download Pretrained Weights

Pretrained weights are available for VINCE as well as all baselines mentioned in the paper. All models are trained on a ResNet18 backbone.

To download the weights, from the root directory, run sh download_scripts/download_pretrained_weights.sh Alternatively, download them directly from https://drive.google.com/uc?id=11TAz8w9xts7Tsqgn0UPVo617gqz5N_xL

Citation

@misc{gordon2020watching,
    title={Watching the World Go By: Representation Learning from Unlabeled Videos},
    author={Gordon, Daniel and Ehsani, Kiana and Fox, Dieter and Farhadi, Ali},
    year={2020},
    eprint={2003.07990},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}