[metrics] speed up polygon iou (for --rotation) by keeping balanced memory footprint

[metrics] speed up polygon iou (for --rotation) by keeping balanced memory footprint #3010