Question about DPatch implementation

[yuki2222]

Hi,

I have some question regarding the DPatch code on here: /adversarial-robustness-toolbox/tree/main/art/attacks/evasion)/dpatch.py.

From the line 163 through 201, it goes:

    if self.target_label:

        for i_image in range(patched_images.shape[0]):
            if isinstance(self.target_label, int):
                t_l = self.target_label
            else:
                t_l = self.target_label[i_image]

            i_x_1 = transforms[i_image]["i_x_1"]
            i_x_2 = transforms[i_image]["i_x_2"]
            i_y_1 = transforms[i_image]["i_y_1"]
            i_y_2 = transforms[i_image]["i_y_2"]

            target_dict = {}
            target_dict["boxes"] = np.asarray([[i_x_1, i_y_1, i_x_2, i_y_2]])
            target_dict["labels"] = np.asarray(
                [
                    t_l,
                ]
            )
            target_dict["scores"] = np.asarray(
                [
                    1.0,
                ]
            )

            patch_target.append(target_dict)

    else:

        predictions = self.estimator.predict(x=patched_images, standardise_output=True)

        for i_image in range(patched_images.shape[0]):
            target_dict = {}
            target_dict["boxes"] = predictions[i_image]["boxes"]
            target_dict["labels"] = predictions[i_image]["labels"]
            target_dict["scores"] = predictions[i_image]["scores"]

            patch_target.append(target_dict)

For the untargeted attack it is taking the estimator predictions as the patch target, but shouldn’t this take the true labels (annotated labels) instead? As far as I understand, the original paper says to maximize the loss of the estimator to the true labels (page 3: https://arxiv.org/abs/1806.02299v4).

Sorry if I’m missing some point, but I am quite confused with this one. I’d really appreciate if someone could help me out. Thanks you!

[beat-buesser]

Hi @yuki2222 You are right, the current implementation of DPatch in the untargeted case attacks the object detector's predictions on the patched images and does not accept true labels. I think the reason was that the predictions to be attacked are made on the images with the untrained patch applied in an approach to automatically remove the influence of the patch on the object detection accuracy under the assumption that the predicted and true labels are similar.
I think it would be easily possible to add support for true labels, would you need it for an application or to exactly reproduce DPatch?

[yuki2222]

Hi @beat-buesser Thank you for the details! I am almost done with the changes. Here is the code I made: 2be96ee

I changed line 193 to below. Does this look okay? I feel like this could throw an error as y may not always have the keys like "boxes" or "scores" inside the for loop, depending on the format of y.

    else:

        predictions = y if y is not None else self.estimator.predict(x=patched_images, standardise_output=True)

        for i_image in range(patched_images.shape[0]):
            target_dict = {}
            target_dict["boxes"] = predictions[i_image]["boxes"]
            target_dict["labels"] = predictions[i_image]["labels"]
            target_dict["scores"] = predictions[i_image]["scores"]

            patch_target.append(target_dict)

I had seen that paper just a bit before, but didn't know it was also implemented here. Thanks for sharing!

[beat-buesser]

Hi @yuki2222 I think your code looks good. I agree on the possibility of an error if the format of y is not as expected. To better inform the user on the expected format of y, could you please add the following change to the docstring of method generate of DPatch by replacing line 110 with:

        :param y: True labels of type `List[Dict[np.ndarray]]` for untargeted attack, one dictionary per input image. The
                  keys and values of the dictionary are:

                 - boxes [N, 4]: the boxes in [x1, y1, x2, y2] format, with 0 <= x1 < x2 <= W and 0 <= y1 < y2 <= H.
                 - labels [N]: the labels for each image
                 - scores [N]: the scores or each prediction.

[yuki2222]

Hi @beat-buesser Thank you! I added your comment to the code and just sent the pull request. I ran some test with it, so it should work :)

[beat-buesser]

Hi @yuki2222 That's great! Thank you very much. I'll start the review of the PR.

[beat-buesser]

Hi @yuki2222 I have merged #1780 Congratulations to your first PR to ART!