Using x265 --csv-log-level 2 information for RPU generation

[ghost]

Note firstly I'm not asking you to write this, I'd likely do it (slowly) if it seems reasonable.

I only did a first pass over the MadVR parsing code so I may be off-base on what it's using here, but (without --use-custom-targets) it appears to only be using the average scene PQ and max scene PQ values (and the MaxCLL / MaxFALL if not specified).

When re-encoding a file, x265 can output various per-frame statistics in CSV format including, importantly:

• Scenecut 1 if the frame is a scenecut, 0 otherwise.
• Luma/Chroma Values minimum, maximum and average(averaged by area) luma and chroma values of source for each frame.

It would seem as though the scenecut + luma average + max could be used to calculate the average and max PQ per scene across the entire file, and used without the scenecuts to calculate the MaxCLL and MaxFALL values. Do you see any particular issues with using this data to calculate these items?

Scene cut detection is on by default in x265 but I'm not sure how well the default version works. I have to force --no-open-gop and keep the max I-frame distance very low to keep seeking from being broken in DV files so I have concerns about that interfering.

I wouldn't be against writing the code for this, I just wanted to get your opinion on whether it seemed doable before diving in and finding that the data wasn't usable for some reason. The full histogram seems necessary to compensate for the letterbox black areas so doing it this way might require specifying their dimensions to adjust the average upwards regardless of whether a crop is required in the DV data but I can't think of any other problems since CSV is easy to parse and the luma averages are all there.

[quietvoid]

From a quick glance at the x265 code, it seems feasible if you can calculate a good target for the aggregate of the scene frames.
However the scene detection in x265 is not as precise as madVR's.

x265 also provides MaxCLL/MaxFALL values in the same log.

[quietvoid]

Also, it doesn't really need to be implemented in the code.
You could simply make a Python script to make use of the generate config, which should be plenty flexible enough to implement this.

[ghost]

I'd write it in pure assembly (pick the architecture and generation) before I caused another line of Python to exist in the world, but I get the point. :D

Rust is a clean enough language coming from C++ that it'd be easier to do in that even if it only stays local... if I ever get around to it. I'll have to do some checking and see if the histogram based scenecut detection in x265 is more accurate than the default and whether or not it outputs detected cuts regardless of whether they're used for encoding. Right now I have to many other projects away from the computer to bother with a workflow that's pretty fast on the 1080p conversions that are most of what I'm dealing with. Madmeasure only gets slower on 4k stuff.

[quietvoid]

I'll have to do some checking and see if the histogram based scenecut detection in x265 is more accurate than the default

It's very broken and detects a lot of false cuts.
Save you some time.

[ghost]

Thanks :-)