All pixels are abnormal [0, 135, 0] after H265 MP4 is decoded into images

[chenc29xpeng]

Describe the question.

All images are green after the video is decoded.

When I use fn.readers.video to decode my own h265 video, the decoded image size is normal, but the pixels are abnormal. The pixels are all [0, 135, 0], I think there is something wrong with my mp4, because the official video is decoded normally. But my mp4 can be played normally and can be decoded normally based on the cpu through opencv. Can anyone help me take a look?

video file

google cloud drive: https://drive.google.com/file/d/1VwnAq2sunrGkZIGgJgF1d-97CCZj51aO/view?usp=sharing
baidu cloud drive: https://pan.baidu.com/s/1AM-OSo2bjWg1pE6GRSAdSQ?pwd=1ram

Here is my code:

import cv2
import tempfile
import numpy as np
from PIL import Image
from typing import List, Optional
from nvidia.dali import pipeline_def
import nvidia.dali.fn as fn
from nvidia.dali.plugin.pytorch import DALIGenericIterator

def mp4_to_image_opencv(input_file: str) -> List[np.ndarray]:
    """
    Return: list
    """
    frame_list = []
    cap = cv2.VideoCapture(input_file)
    while(cap.isOpened()):
        ret, frame = cap.read()
        if ret == True:
            frame_list.append(frame)
        else:
            break
    cap.release()
    return frame_list

if __name__ == "__main__":
    video_filename = "cut.mp4"
    frame_list = mp4_to_image_opencv(video_filename)
    cv2.imwrite("debug_opencv.png", frame_list[10])
    save_path = "debug_dali.png"
    sequence_length = 64

    @pipeline_def
    def video_pipe(file_list):
        video, label = fn.readers.video(device="gpu", file_list=file_list, sequence_length=sequence_length, file_list_frame_num=True, name="my_reader", pad_sequences=True)
        return video, label

    my_file_list_str = f"{video_filename} 0 0 20\n" # label start_frame_num end_frame_num
    tf = tempfile.NamedTemporaryFile()
    tf.write(str.encode(my_file_list_str))
    tf.flush()
    pipe = video_pipe(batch_size=1, file_list=tf.name, num_threads=1, device_id=0)
    pipe.build()

    dali_iter = DALIGenericIterator([pipe], ["image", "label"], reader_name="my_reader")
    for data in dali_iter:
        label = data[0]['label'].cpu().numpy()[0][0]
        image = data[0]['image'][0]
        pil_image = Image.fromarray(image[10].cpu().numpy())
        pil_image.save(save_path)
        

dali decode

opencv decode

Check for duplicates

• I have searched the open bugs/issues and have found no duplicates for this bug report

[chenc29xpeng]

By the way, ffmpeg can decode this mp4 file normally using ffmpeg -i cut.mp4 '%04d.png'

[JanuszL]

Hi @chenc29xpeng,

Indeed, DALI returns not the output we may expect. Let us investigate this and get back to you soon.

[chenc29xpeng]

Hi @chenc29xpeng,

Indeed, DALI returns not the output we may expect. Let us investigate this and get back to you soon.

@JanuszL, Thank you very much. This problem has troubled me for a long time. I am looking forward to your reply!

[chenc29xpeng]

@JanuszL @awolant Is there any progress on this issue? I would be very grateful if you could fix it.

[JanuszL]

Hi @chenc29xpeng,

I'm sorry to say that there is no progress. The issue seems to be not related to DLAI itself but the GPU video decoding capability. I reached out to a relevant team and await their response.

[bpleshakov]

Is it related to #4135?

[JanuszL]

@bpleshakov it could be but I cannot tell.

[chenc29xpeng]

Hi @chenc29xpeng,

I'm sorry to say that there is no progress. The issue seems to be not related to DLAI itself but the GPU video decoding capability. I reached out to a relevant team and await their response.

@JanuszL Got it. If there is any progress, please contact me as soon as possible. Thank you.

[chenc29xpeng]

Is it related to #4135?

@bpleshakov I think it's irrelevant because I only have one mp4 video spliced ​​from h265 files, and the fps is 12.