Bugs in AutoAttack implementation

[SignedQiu]

Describe the bug
There is a data type error in the auto pgd implementaion of autoattack.

To Reproduce
Simply run the AutoAttack using the default parameters in Mnist dataset and PyTorchClassifier (where i met the bug), the errors will occurs with "RuntimeError: Expected object of scalar type Long but got scalar type Byte for argument #2 'target' in call to _thnn_nll_loss_forward"

Expected behavior
I attempted to fix this bug, but i think the code "grad = self.estimator.loss_gradient(x_k, y_batch) * (1 - 2 * int(self.targeted))" in the 455th line of the file auto_projected_gradient_descent.py is the key to solving this problem.

Screenshots

System information (please complete the following information):

• OS = Linux
• Python version = 3.7.9
• ART version or commit number = 1.14.0
• PyTorch = 1.8.2+cu111

[SignedQiu]

In the process of attempting to fix the vulnerability, I initially thought it was a simple data type error, and I completed the data type conversion at the corresponding position. However, I discovered a new bug. The input of the "target" parameter in the function "torch._C._nn.nll_loss" does not conform to the dimensional specification. The function "self.reduce_labels(y_preprocessed)" clearly checks the dimensions, but it appears that the check was bypassed due to some conditional settings. Therefore, I believe this is a system-level bug in the auto pgd implementation. I don't have any more time to fix it, so I am temporarily reporting the bug here.

[beat-buesser]

Hi @SignedQiu Thank you for using ART! What is the dtype of variable y_batch?

[SignedQiu]

(batch_size, 10)

[beat-buesser]

Thank you. I have noticed that your example does not provide true labels in line adv_inputs = attack.generate(x=x_test). In that case the attack uses your model's output as true label to attack. What is the dtype of the output of your model`?

[SignedQiu]

On GPU, the dtype of the output of the model is torch.float32.

[beat-buesser]

Hi @SignedQiu I think you have found a bug! Actually two. It looks like the order of prediction and label arguments is wrong in the custom loss class and the PyTorchClassifier does not recognize the custom loss class to set internal attributes correctly. I have opened issue #2116 and we'll include the solution in ART 1.14.1. Thank you for report this issue!

[beat-buesser]

I'm converting this issue into a discussion on the Discussion tab to continue.