about ImageNet

[daihu-ye]

Hi, yaoyao.
I directly followed the instructions and ran the code
"python main.py --nb_cl_fg=50 --nb_cl=10 --gpu=0 --random_seed=1993 --baseline=lucir --branch_mode=dual --branch_1=ss --branch_2=free --dataset=imagenet".
I just changed the dataset option and download the imagenet data and changed the "data_dir " option to the path i stored the data.It works and showed the results.Does it mean i successfully run the code for training the model on imagenet?.
I remembered that you said the code for ImageNet is not included in the current GitHub repository in
https://github.com/yaoyao-liu/class-incremental-learning/issues/12.Or you have already uploaded the code for imagenet?

[yaoyao-liu]

The code also works for ImageNet, but the default hyperparameters are not suitable for ImageNet. Besides, there might be some small bugs.

To run experiments on ImageNet, you need to adjust some hyperparameters according to LUCIR and modify some functions.

I'll check this project and update it later.

[daihu-ye]

Thanks for your response.
I set the epoch to 1 to test if there are some bugs for training ImageNet,it finished successfully.If you have update the project,please let me know.
Thanks in advance!

[yaoyao-liu]

No problem.

If you have any further questions, feel free to email me or add comments to this issue.

Best,
Yaoyao

[daihu-ye]

A small question,the accuracy reported in your code has 3 types(FC,Proto,Proto-UB ).FC is CNN accuracy and Proto is NME like iCaRL?And what does Proto-UB mean?The final average accuracy has 2 types,which is the actual accuracy you reported in your paper since FC and Proto have a lilttle diffreence(may be 0.5~1),i want to figure out whether i got the same result as your paper.

[yaoyao-liu]

FC denotes using a fully connected classifier. Proto denotes using iCaRL’s classifier. Proto-UB denotes using iCaRL’s classifier, and the prototypes are computed on all data instead of the exemplars. The third setting is an oracle setting.

In most of the experiments, we use FC classifiers.

[daihu-ye]

Thanks,it seems i got the same result as your paper and even better.But i just run for one time,maybe i should run for 3 times
and compute averages ± standard deviations.

[daihu-ye]

Hi,i have run the code for training the model on imagenet and changed the default hyperparameters (epochs,custom_weight_decay),but i only got 63%(64.94% in your paper) when N=5.I remember you said we should modify some functions to run the code for ImageNet.Could you tell me which function to modify ,or could you give some tips on running the code for ImageNet to get the same results sa your paper.Thanks.

[yaoyao-liu]

The additional function I mentioned refers to class-balance finetuning. You may see the code here. All the parameters are available in this file.

You may first try to run the experiments for the baseline (LUCIR) by setting --baseline=lucir --branch_mode=single. The accuracy should be 64.54%.

If you have further questions, feel free to email me or add comments to this issue.

[JosephKJ]

Hi, yaoyao. I directly followed the instructions and ran the code "python main.py --nb_cl_fg=50 --nb_cl=10 --gpu=0 --random_seed=1993 --baseline=lucir --branch_mode=dual --branch_1=ss --branch_2=free --dataset=imagenet". I just changed the dataset option and download the imagenet data and changed the "data_dir " option to the path i stored the data.It works and showed the results.Does it mean i successfully run the code for training the model on imagenet?. I remembered that you said the code for ImageNet is not included in the current GitHub repository in https://github.com/yaoyao-liu/class-incremental-learning/issues/12.Or you have already uploaded the code for imagenet?

Hi @daihu-ye : The config reads --nb_cl_fg=50 --nb_cl=10, which should have been --nb_cl_fg=500 --nb_cl=100 for ImageNet experiments rt? Was that what you had? Wondering how you could replicate the results.

[JosephKJ]

Hi @yaoyao-liu :

Thanks again for your work.

I tried to replicate ImageNet results, but the numbers for the baselines are far from what we have on paper (~10%) or so.

# ImageNet - 5 task - iCaRL
python main.py --nb_cl_fg=500 --nb_cl=100 --gpu=0 --random_seed=1993 --baseline=icarl --branch_mode=single --branch_1=free --dataset=imagenet --num_workers 16 --epochs 90 --custom_weight_decay 1e-4 --num_classes 1000 --ckpt_dir_fg=./base_500_imagenet.pt --data_dir=~/joseph/imagenet | tee ebm_imagenet_icarl_5_tasks.log

# ImageNet - 10 task - iCaRL
python main.py --nb_cl_fg=500 --nb_cl=50 --gpu=0 --random_seed=1993 --baseline=icarl --branch_mode=single --branch_1=free --dataset=imagenet --num_workers 16 --epochs 90 --custom_weight_decay 1e-4 --num_classes 1000 --ckpt_dir_fg=./base_500_imagenet.pt --resume_fg --data_dir=~/joseph/imagenet | tee ebm_imagenet_icarl_10_tasks.log

# ImageNet - 25 task - iCaRL
python main.py --nb_cl_fg=500 --nb_cl=20 --gpu=0 --random_seed=1993 --baseline=icarl --branch_mode=single --branch_1=free --dataset=imagenet --num_workers 16 --epochs 90 --custom_weight_decay 1e-4 --num_classes 1000 --ckpt_dir_fg=./base_500_imagenet.pt --resume_fg --data_dir=~/joseph/imagenet | tee ebm_imagenet_icarl_20_tasks.log


# LUCIR
# ImageNet - 5 task - LUCIR
python main.py --nb_cl_fg=500 --nb_cl=100 --gpu=0 --random_seed=1993 --baseline=lucir --branch_mode=single --branch_1=free --dataset=imagenet --num_workers 16 --epochs 90 --custom_weight_decay 1e-4 --num_classes 1000 --ckpt_dir_fg=./base_500_imagenet.pt --resume_fg --data_dir=~/joseph/imagenet | tee ebm_imagenet_lucir_5_tasks.log

# ImageNet - 10 task - LUCIR
python main.py --nb_cl_fg=500 --nb_cl=50 --gpu=0 --random_seed=1993 --baseline=lucir --branch_mode=single --branch_1=free --dataset=imagenet --num_workers 16 --epochs 90 --custom_weight_decay 1e-4 --num_classes 1000 --ckpt_dir_fg=./base_500_imagenet.pt --resume_fg --data_dir=~/joseph/imagenet | tee ebm_imagenet_lucir_10_tasks.log

# ImageNet - 25 task - LUCIR
python main.py --nb_cl_fg=500 --nb_cl=20 --gpu=0 --random_seed=1993 --baseline=lucir --branch_mode=single --branch_1=free --dataset=imagenet --num_workers 16 --epochs 90 --custom_weight_decay 1e-4 --num_classes 1000 --ckpt_dir_fg=./base_500_imagenet.pt --resume_fg --data_dir=~/joseph/imagenet | tee ebm_imagenet_lucir_20_tasks.log

Here are the numbers that I got:

I am sure that I am missing something critical, that is causing this. I have-not done class-balance finetuning. Would that be the root cause for this drop in accuracy?

[yaoyao-liu]

Hi @JosephKJ,

I will re-run the experiments on ImageNet after the CVPR deadline and keep you informed when I find the issue.

According to the command lines you use, it seems you are running the baseline experiments, i.e., iCaRL and LUCIR w/o AANets. If so, you may directly run the code in this repository because my implementation of the baselines is based on this repository.

If you have further questions, feel free to let me know.

Best,
Yaoyao

[JosephKJ]

HI @yaoyao-liu,

Many thanks for your swift response. You are correct, I am indeed trying to replicate the baseline results.

I will also try to fix this by CVPR, I will let you know incase I succeed. I am quite sure that we are missing something obvious.

I should say that I am able to replicate ImageNet subset results very well. Your code base in amazing. It allowed me to modify to my setting.

Thanks,
Joseph

[daihu-ye]

Hi, yaoyao. I directly followed the instructions and ran the code "python main.py --nb_cl_fg=50 --nb_cl=10 --gpu=0 --random_seed=1993 --baseline=lucir --branch_mode=dual --branch_1=ss --branch_2=free --dataset=imagenet". I just changed the dataset option and download the imagenet data and changed the "data_dir " option to the path i stored the data.It works and showed the results.Does it mean i successfully run the code for training the model on imagenet?. I remembered that you said the code for ImageNet is not included in the current GitHub repository in https://github.com/yaoyao-liu/class-incremental-learning/issues/12.Or you have already uploaded the code for imagenet?

Hi @daihu-ye : The config reads --nb_cl_fg=50 --nb_cl=10, which should have been --nb_cl_fg=500 --nb_cl=100 for ImageNet experiments rt? Was that what you had? Wondering how you could replicate the results.

@JosephKJ You are right , i just run the code,it works but i did't get the resuslt as the paper.I just run the LUCIR with AANets(5 tasks),the result i got was 63.02%

[yaoyao-liu]

Hi @daihu-ye and @JosephKJ,

Both of you have issues running the experiment on ImageNet-Full, so I will re-run and check the code after the CVPR deadline. I will keep you informed when I have the results. Thanks for reporting this issue.

Best,
Yaoyao