ControlGAN

Pytorch implementation for Controllable Text-to-Image Generation. The goal is to generate images from text, and also allow the user to manipulate synthetic images using natural language descriptions, in one framework.

Overview

Controllable Text-to-Image Generation.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, Philip H. S. Torr.
University of Oxford
In Neural Information Processing Systems, 2019.

Data

1. Download the preprocessed metadata for bird and coco, and save both into data/
2. Download bird dataset and extract the images to data/birds/
3. Download coco dataset and extract the images to data/coco/

Training

All code was developed and tested on CentOS 7 with Python 3.7 (Anaconda) and PyTorch 1.1.

DAMSM model includes text encoder and image encoder

• Pre-train DAMSM model for bird dataset:

python pretrain_DAMSM.py --cfg cfg/DAMSM/bird.yml --gpu 0

• Pre-train DAMSM model for coco dataset:

python pretrain_DAMSM.py --cfg cfg/DAMSM/coco.yml --gpu 1

ControlGAN model

• Train ControlGAN model for bird dataset:

python main.py --cfg cfg/train_bird.yml --gpu 2

• Train ControlGAN model for coco dataset:

python main.py --cfg cfg/train_coco.yml --gpu 3

*.yml files include configuration for training and testing.

Pretrained DAMSM Model

• DAMSM for bird. Download and save it to DAMSMencoders/
• DAMSM for coco. Download and save it to DAMSMencoders/

Pretrained ControlGAN Model

• ControlGAN for bird. Download and save it to models/

Testing

• Test ControlGAN model for bird dataset:

python main.py --cfg cfg/eval_bird.yml --gpu 4

• Test ControlGAN model for coco dataset:

python main.py --cfg cfg/eval_coco.yml --gpu 5

Evaluation

• To generate images for all captions in the testing dataset, change B_VALIDATION to True in the eval_*.yml.
• Inception Score for bird dataset: StackGAN-inception-model.
• Inception Score for coco dataset: improved-gan/inception_score.

Code Structure

• code/main.py: the entry point for training and testing.
• code/trainer.py: creates the main networks, harnesses and reports the progress of training.
• code/model.py: defines the architecture of ControlGAN.
• code/attention.py: defines the spatial and channel-wise attentions.
• code/VGGFeatureLoss.py: defines the architecture of the VGG-16.
• code/datasets.py: defines the class for loading images and captions.
• code/pretrain_DAMSM.py: creates the text and image encoders, harnesses and reports the progress of training.
• code/miscc/losses.py: defines and computes the losses.
• code/miscc/config.py: creates the option list.
• code/miscc/utils.py: additional functions.

Citation

If you find this useful for your research, please use the following.

@article{li2019control,
  title={Controllable text-to-image generation},
  author={Li, Bowen and Qi, Xiaojuan and Lukasiewicz, Thomas and H.~S.~Torr, Philip},
  journal={arXiv preprint arXiv:1909.07083},
  year={2019}
}

Acknowledgements

This code borrows heavily from AttnGAN repository. Many thanks.