How to increase the drop in mAP cause by Robust DPatch?

[dominic-simon]

Hello,

I'm trying to use the Robust DPatch to attack images from COCO. I am able to attack images such that many hallucinations appear, but all hallucinations have a very low confidence (< 20%, usually ranging from 5% - 10%). These low confidence hallucinations do not have much of an effect on the Mean Average Precision (mAP), which is the main metric I am using to measure the success of the attack. Is there any way I can increase the confidence of the hallucinated objects so that the mAP of the image, and whole dataset, will decrease?

For reference, here is the code I'm using to generate the patches:

import torch
from torchvision import utils
import torchvision.transforms as transforms
import torchvision.models.detection as models
from torchvision.datasets import CocoDetection

from art.attacks.evasion import RobustDPatch
from art.estimators.object_detection import PyTorchFasterRCNN

import os
import numpy as np
from tqdm import tqdm
from PIL import Image, ImageDraw

PATCH_SIZE = 75
NUM_ITERATIONS = 500

resnet = models.fasterrcnn_resnet50_fpn_v2(weights='FasterRCNN_ResNet50_FPN_V2_Weights.DEFAULT')
model = PyTorchFasterRCNN(model=resnet,
                          clip_values=(0, 1.0),
                          channels_first=False,
                          preprocessing_defences=None,
                          postprocessing_defences=None,
                          preprocessing=None,
                          attack_losses=('loss_classifier',
                                         'loss_box_reg',
                                         'loss_objectness',
                                         'loss_rpn_box_reg'),
                          device_type='gpu')

ms_coco = CocoDetection('../datasets/MSCOCO/val2017', '../datasets/MSCOCO/annotations/instances_val2017.json')

for i in tqdm(range(len(ms_coco))):
    image = (np.array(ms_coco[i][0]) / 255).astype(np.float32)
    image = np.expand_dims(image, axis=0)

    h_loc = torch.randint(0, image.shape[1]-PATCH_SIZE-1, (1,)).item()
    w_loc = torch.randint(0, image.shape[2]-PATCH_SIZE-1, (1,)).item()
    dpatch = RobustDPatch(model,
                          patch_shape=(PATCH_SIZE, PATCH_SIZE, 3),
                          patch_location=(h_loc, w_loc),
                          learning_rate=0.01,
                          max_iter=NUM_ITERATIONS,
                          batch_size=1,
                          targeted=False,
                          verbose=True)
    _ = dpatch.generate(image)
    adv_image = dpatch.apply_patch(image)

    adv_image = np.transpose(adv_image[0], (2, 0, 1))
    adv_image = torch.from_numpy(adv_image)
    utils.save_image(adv_image, f'./robust_dpatch/{PATCH_SIZE}/{i}.png')

I use pycocotools to measure the mAP. I can provide the script where I do that if necessary. Any recommendations are appreciated. Also, if there is a different object detection patch I can try, I am open to that as well.

Finally, if someone working on this repo knows, are there any plans to add an object detection patch that hides objects in the image from the detector (a "disappearing" patch)? That would be very useful to have.

Thank you for your consideration.

[Vaycold]

did you solve this problem? I have a same problem

[beat-buesser]

Hi @Vaycold and @dominic-simon Have you tried attacking different losses of the model? The argument attack_losses allows you to attack different combinations of the term of the total loss of the Faster-RCNN model and through that focus the attack of different failure modes.