implement PreprocessorTensorFlowV2

[le000043]

Hello,
I am trying to implement the PreprocessorTensorFlowV2. Below is my implementation. For some reason, I had this error, did the gradient just vanish?
My implementation:

class MyImplementation(PreprocessorTensorFlowV2):
  def __init__(self, params):
    super().__init__(is_fitted=True, apply_fit=False, apply_predict=True)
    self.params = params
    self.intermediary_size = params['intermediary_size']
    self.resizeMethod = params['resizeMethod']
    self.resolution = params['resolution']
    self._check_params()
  
  def _check_params(self):
    if self.intermediary_size < 0 or not type(self.intermediary_size) == int:
      raise ValueError("intermediary_size must be a positive integer")

  # override forward method
  def forward(self, x: "tf.Tensor", y: Optional["tf.Tensor"] = None) -> Tuple["tf.Tensor", Optional["tf.Tensor"]]:
      # simply return the input
      output = tf.convert_to_tensor(x.numpy(), dtype = x.dtype)
      return output, y

The error:

2022-03-11 09:18:14.958804: I tensorflow/stream_executor/cuda/cuda_dnn.cc:381] Loaded cuDNN version 8301
Traceback (most recent call last):
  File "Main.py", line 30, in <module>
    main(sys.argv[1:])
  File "Main.py", line 24, in main
    attacker.runAttack()
  File "/gscratch/src/Attacker.py", line 199, in runAttack
    x_test_adv = attacker.generate(x = x_copy, y = y.numpy())
  File "/mmfs1/home/.local/lib/python3.8/site-packages/art/attacks/evasion/fast_gradient.py", line 245, in generate
    adv_x_best = self._minimal_perturbation(x, y, mask)
  File "/mmfs1/home/.local/lib/python3.8/site-packages/art/attacks/evasion/fast_gradient.py", line 157, in _minimal_perturbation
    perturbation = self._compute_perturbation(batch, batch_labels, mask_batch)
  File "/mmfs1/home/.local/lib/python3.8/site-packages/art/attacks/evasion/fast_gradient.py", line 387, in _compute_perturbation
    grad = self.estimator.loss_gradient(batch, batch_labels) * (1 - 2 * int(self.targeted))
  File "/mmfs1/home/.local/lib/python3.8/site-packages/art/estimators/classification/tensorflow.py", line 1220, in loss_gradient
    gradients = gradients.numpy()
AttributeError: 'NoneType' object has no attribute 'numpy'

The above forward function simply returns the input image by first extracting the NumPy array and returning a new Tensor with the extracted data. I suspect this is a mismatch of the data structure. Everything works fine if I just simply return x from the input.

[beat-buesser]

Hi @le000043 I think this operation is breaking your gradient flow: tf.convert_to_tensor(x.numpy(). Why do you convert to numpy and immediately back to tensor?

[le000043]

Thank you for your reply. I plan to transform the input and it requires in numpy form. For example, I plan to convert the Jpeg Compression into an instance of PreprocessorTensorFlowV2. Is there a way I can do this ?

[beat-buesser]

There are multiple possible approaches. If the compression is done in non-framework code an instance of Preprocessor is the easiest approach. ART has recently received a new pattern for PreprocessorPyTorch, applied for example in VideoCompressionPyTorch which uses a non-framework ffmpeg compression in a framework-specific preprocessor. You would have to find out how to create a similar approach for TensorFlow and you have to provide the calculation/approximation of the gradients but it can then be integrated into framework-specific attacks. Third you can try to implement the differentiable approximation to JPEG, let me know if you implement this approach, it would be great to add it to ART.

[le000043]

thank you for you reply. If a preprocessing method outputs a NumPy array, how can I convert it to a tensor output in PreprocessorTensorFlowV2.forward() without breaking the gradient flow? I have tried tf.convert_to_tensor() but it failed.

[beat-buesser]

I think you would have to find a similar pattern as in VideoCompressionPyTorch but for TensorFlow by wrapping the operation into a TensorFlow layer and define the gradient calculation. I think it should be possible, but I have not tried it myself so far.