Patch attacks on PyTorch classifiers

[rrmhearts]

It seems that either the patch attacks either do no update the patch itself, or I receive an error. The error itself is the following and occurs during the DPatch or the RobustDPatch. It seems that my PyTorch classifier does not have the ObjectDetectorMixin. Is that solvable with the given model?

  File ..., line 86, in <module>
    patch_attack = DPatch(classifier, max_iter=1000)
...
  File "...venv\lib\site-packages\art\attacks\attack.py", line 125, in 
__init__
    raise EstimatorError(self.__class__, self.estimator_requirements, estimator)
art.exceptions.EstimatorError: DPatch requires an estimator derived from <class 'art.estimators.estimator.BaseEstimator'> and <class 'art.estimators.estimator.LossGradientsMixin'> and <class 'art.estimators.object_detection.object_detector.ObjectDetectorMixin'>, the provided classifier is an instance of <class 'art.estimators.classification.pytorch.PyTorchClassifier'> and is derived from (<class 'art.estimators.classification.classifier.ClassGradientsMixin'>, <class 'art.estimators.classification.classifier.ClassifierMixin'>, <class 'art.estimators.pytorch.PyTorchEstimator'>).

For the more standard AdversarialPatchPyTorch, it appears to work, the code runs, however the patch itself is not adversarial and does not decrease the accuracy. It seems as if it the gradients are not being applied to the patch. I've tried both optimizer options as well as many other optional parameters. My model is a standard ResNet50 and the setup code is below. Again, patches are applied to images but they are not adversarial.

    model = torchvision.models.resnet50(weights=ResNet50_Weights.DEFAULT) #pretrained=True)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.01)
    classifier = PyTorchClassifier(
        model=model,
        clip_values=(0, 1),
        loss=criterion,
        optimizer=optimizer,
        input_shape=(3, 320, 320),
        nb_classes=1000, # ResNet has 1000 classes
    )
    patch_attack = AdversarialPatchPyTorch(classifier, max_iter=3000, optimizer="pgd")

Further down in the code, I convert my labels to one hot encoding and imagenet size, (I am using Imagenette).

        nb_classes = 1000
        x_data = next(iter(train_loader))[0].numpy() # 512 images
        y_data = next(iter(train_loader))[1]               # 512 labels
        one_hot = torch.nn.functional.one_hot(get_imagenet_labels(y_data), nb_classes)
        patch_itself = patch_attack.generate(x_data, 
                                             y = np.array(one_hot ) )

Please let me know what I am doing wrong, or if there is a bug in the implementation. In addition, is 512 images enough to generate a patch? I think it should be and as I scaled up the number of images, the accuracy was not affected.

Thank you for your help!

[rrmhearts]

In the test code at,
https://github.com/Trusted-AI/adversarial-robustness-toolbox/blob/main/tests/attacks/test_adversarial_patch.py

Why is it that in the test_4_pytorch function, AdversarialPatchPyTorch is not being used or tested?

[Foxglove144]

It seems that either the patch attacks either do no update the patch itself, or I receive an error. The error itself is the following and occurs during the DPatch or the RobustDPatch. It seems that my PyTorch classifier does not have the ObjectDetectorMixin. Is that solvable with the given model?

DPatch and RobustDPatch use estimator: OBJECT_DETECTOR_TYPE where estimator is a trained object detector like Faster RCNN or YOLOv5. Refer to this example.

Further down in the code, I convert my labels to one hot encoding and imagenet size, (I am using Imagenette).

nb_classes = 1000
x_data = next(iter(train_loader))[0].numpy()       # 512 images
y_data = next(iter(train_loader))[1]               # 512 labels
one_hot = torch.nn.functional.one_hot(get_imagenet_labels(y_data), nb_classes)
patch_itself = patch_attack.generate(x_data, y = np.array(one_hot))

It seems that the next(iter(train_loader)) is called twice, which means it fetches the next batch of data from the train_loader iterator twice. This will lead to different values of x_data and y_data in subsequent calls.

To fix this issue, you should call next(iter(train_loader)) only once and store its result in a variable to use it later:

nb_classes = 1000
data = next(iter(train_loader))
x_data = data[0].numpy()  # 512 images
y_data = data[1]          # 512 labels
one_hot = torch.nn.functional.one_hot(get_imagenet_labels(y_data), nb_classes)
patch_itself = patch_attack.generate(x_data, y=np.array(one_hot))