Weird downscaling factor behavior on selfie photos

[DebZakari]

Leaving the default downscale value (1.0) on this code:

!python test.py --cfg_path="./configs/retinaface_mbv2.yaml" --img_path="./data/out_becky-test.jpg" --down_scale_factor=1.0

The following result is shown here:

Changing this value to 0.25 makes it way better:

I also noticed that downscaling it to 0.25 only works on selfie photos. In group photos, you're better off setting it to 1.0

[DebZakari]

Is there a reliable way to set the downscale value dynamically?