How did you produce the heatmap

[ThomasHenckel]

Hi Utayao

Great work making this work with Keras, i got the code running with the Colin dataset, and it outputs a mean accuracy of 0.92 :-)

My question is how you produced the heatmap, i dont see any code for this, here or in the original tensorflow implementation https://github.com/AMLab-Amsterdam/AttentionDeepMIL

[utayao]

Hi, you can get model output ak for each patch in the bag by using the following function:
get_alpha_layer_output = K.function([model.layers[0].input], [model.layers[10].output])
Then you need to rescal the ak to get the attention weight as described in the paper (Fig.5). The heatmap is produced by multiplying the patch and its corresponding attention weight.

[ThomasHenckel]

Thanks a lot for the quick reply, i will se if i can make that work :-)

[ThomasHenckel]

Hi Utayao

Worked great thanks, i get the weights for each patch.

I get that i can multiply this value with the image patch, and use the cordinates in the filename to combine the paches into a final result, but befor i try it out i just want to ask if you would consider sharing this part of the code as well ?

Best Regards
Thomas

[b02902032]

This is how I get the heatmap. I hope I've done it right.

Get one image from training data

    ak_x = model_train_set[bag_id][0]

    # print("ak_x.shape {}".format(ak_x.shape))   
    # e.g. (n, 27, 27, 3)
    # print("ak_x img_name {}".format(model_train_set[bag_id][2][0]))    
    # model_train_set stores (images, bag_label, image_name). 
    # Therefore, model_train_set[bag_id][0] is the image data
    # model_train_set[bag_id][1] is the label
    # model_train_set[bag_id][2] is the image name.

Get alpha layer input to output function

    ak = K.function([model.layers[0].input], [model.layers[10].output])

Feed the training data in to the function to get the result

    ak_output = ak([ak_x])  
    ak_output = np.array(ak_output[0]).reshape((63))    
    # For my dataset, there are 63 images in a bag.

    # rescale the weight as described in the paper
    minimum = ak_output.min()
    maximum = ak_output.max()
    ak_output = ( ak_output - minimum ) / ( maximum - minimum )

Get the n largest patch in the bag. In my case, n=30

    n_largest_idx = np.argpartition(ak_output, -30)[-30:]
    # print("10_largest_idx {}".format(n_largest_idx))
    # print("ak_output[n_largest_idx] {}".format(ak_output[n_largest_idx]))

    # Then you can draw the image by multiply ak_x and ak_output 
    # or draw the n_largest rectangle(ROIs) on the origin image

I tried to draw the heatmap by multiply ak_x and ak_output, though, the output is a little bit weird. Therefore, I just draw the ROIs. If anyone has suggestions, please reply! Thanks!
Here are some example images.
Origin MRI on img_254

Heatmap(n_largest) on img_254

ROIs on img_012

[SuzaKrish]

Hi, I'm unable to get these results. I get only black patches when I multiply ak_output with the image patch. Any suggestions?