Subsampling 256 fg/bg anchors

[brkageo]

I am trying to understand training process of RPN. I have problem with creating mini batches of 256 anchors. If features map has shape 18x25=450 and every position has 9 anchors it is 4050 potential anchors. Output class shape will be 18x25x18 and output regression shape will be 18x25x72. How to select only 256 anchors? I read that we have to select 128 fg and 128 bg randomly. If we label fg anchors with [1,0] and bg anchors with [0,1], how to label anchors which have to be ignore? With [0,0]?

For example, if we have 9 anchors for same position on features map and only 1 of 9 anchors is fg, while others are bg. My question is how to discard 7 other anchors, ie. how to make 0.5/05 fg/bf ratio (in this example 1fg, 1bg and 7 discarded anchors)? if we cannot change the output shape, we have to somehow discard redundant bg anchors. We don't want RPN to backpropagate loss over all of 9 anchors. It will be biased to bg anchors.