Implementation of I-JEPA from (Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture)
Uses @lucidrains x-transfromers (https://github.com/lucidrains/x-transformers)
Basic Schematic of Architecture:
In order to run, just run: python pretrain_IJEPA.py or `python finetune_IJEPA.py' on your command line
TODO:
- linear probing setup
Citation:
@article{assran2023self,
title={Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture},
author={Assran, Mahmoud and Duval, Quentin and Misra, Ishan and Bojanowski, Piotr and Vincent, Pascal and Rabbat, Michael and LeCun, Yann and Ballas, Nicolas},
journal={arXiv preprint arXiv:2301.08243},
year={2023}
}