vo_gpu_next: raise LUT file max size and report an error if exceeded

vo_gpu_next: raise LUT file max size and report an error if exceeded #7401