[metrics] speed up polygon iou (for --rotation) by keeping balanced memory footprint

[metrics] speed up polygon iou (for --rotation) by keeping balanced memory footprint #2580