Object-centric vs. Scene-centric Image-Text Cross-modal Retrieval

This repository contains the code used for the experiments in "Object-centric vs. Scene-centric Image-Text Cross-modal Retrieval: A Reproducibility Study" published at ECIR 2023.

License

The contents of this repository are licensed under the MIT license. If you modify its contents in any way, please link back to this repository.

Reproducing Experiments

First off, install the dependencies:

pip install -r requirements.txt

Download the data

Download the data from this repository. The repository contains two zip files: CLIP_data.zip and X-VLM_data.zip

After unzipping CLIP_data put the resulting data folder in the CLIP folder:

CLIP/
    data/
        datasets/
        results/

For evaluating X-VLM, we need to have access to the original images from CUB-200 (CUB), Amazon Berkley Objects (ABO), Fashion200k, MS COCO, and Flickr30k. We cannot redistribute the images, therefore, we ask you to download the images yourself. The images should be added to the X-VLM/image directory, each dataset in its own subfolder folder. Overall the X-VLM folder should organized as follows:

X-VLM/
    data/
        json_splits/
        models/
    images/
        cub/
            001.Black_footed_Albatross/*.jpg
            ...
            174.Palm_Warbler/*.jpg
        fashion200k/
            women/
                dresses/*.jpg
                jackets/*.jpg
                pants/*.jpg
                skirts/*.jpg
                tops/*.jpg
        abo/
            00/*.jpg
            ...
            fe/*.jpg
        mscoco/*.jpg
        flickr30k/*.jpg

Evaluate the models

Evaluate each model on each of the five datasets.

# CLIP evaluation
sh CLIP/jobs/evaluation/evaluate_cub.job
sh CLIP/jobs/evaluation/evaluate_abo.job
sh CLIP/jobs/evaluation/evaluate_fashion200k.job
sh CLIP/jobs/evaluation/evaluate_mscoco.job
sh CLIP/jobs/evaluation/evaluate_flickr30k.job
# printing the results for CLIP in one file
sh CLIP/jobs/postprocessing/results_printer.job

# X-VLM evaluation
sh X-VLM/jobs/evaluation/evaluate_flickr30k.job 
sh X-VLM/jobs/evaluation/evaluate_cub.job 
sh X-VLM/jobs/evaluation/evaluate_abo.job 
sh X-VLM/jobs/evaluation/evaluate_fashion200k.job 
sh X-VLM/jobs/evaluation/evaluate_mscoco.job 
# printing the results for X-VLM in one file
sh X-VLM/jobs/postprocessing/results_printer.job

Citing and Authors

If you find this repository helpful, feel free to cite our paper "Object-centric vs. Scene-centric Cross-modal Retrieval: A Reproducibility Study":

@inproceedings{hendriksen-2023-object-centric,
author = {Hendriksen, Mariya and Vakulenko, Svitlana and Kuiper, Ernst and de Rijke, Maarten},
booktitle = {ECIR 2023: 45th European Conference on Information Retrieval},
month = {April},
publisher = {Springer},
title = {Scene-centric vs. Object-centric Image-Text Cross-modal Retrieval: A Reproducibility Study},
year = {2023}}