How to set the distillation temperature - Defensive Distillation

[MiguelCosta94]

Hi,

I am experimenting with defensive distillation using the ART framework. For this purpose, I am using the DefensiveDistillation class. However, I cannot figure out how to set the distillation temperature. I do not see this configuration parameter in the implementation of DefensiveDistillation class.

Best regards,
Miguel

[beat-buesser]

Hi @MiguelCosta94 Thank you for your interest in ART! Do you have a short code example that shows your experiment?

[MiguelCosta94]

Hi @beat-buesser Below you can find my code:

loss_object = tf.keras.losses.CategoricalCrossentropy()
optimizer = tf.keras.optimizers.SGD()

def train_step(model, data, labels):
    with tf.GradientTape() as tape:
        predictions = model(data, training=True)
        loss = loss_object(labels, predictions)
    gradients = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))

def main(args):
    x_train_f32, y_train, x_test_f32, y_test = ds.get_cifar10_full_ds_f32()
    ann_f32 = tf.keras.models.load_model(args['ann_f32_dir'])
    art_classifier = TensorFlowV2Classifier(model=ann_f32, clip_values=(0, 1),
                            nb_classes=10, input_shape=(32,32,3), train_step= train_step,
                            loss_object=tf.keras.losses.CategoricalCrossentropy())

    distiller = DefensiveDistillation(art_classifier, batch_size=128, nb_epochs=50)
    distiller(x_train_f32, art_classifier)

[beat-buesser]

Hi @MiguelCosta94 Ok, now I got it. You are right, the DefensiveDistillation tool itself does not provide an option to set the temperature, it assumes that the temperature has been defined inside of the model's softmax layer.