Rethinking our approach to waveform generation

[zackkrida]

Our current approach to waveform generation in the API extracts dependencies and the audiowaveform binary from the realies/audiowaveform:latest docker container. I see several problems with this:

• This package is only ever tagged latest, so we're automatically getting all changes every time the package is updated and lack the ability to lock the version.
• Extracting the dependencies is brittle, as identified in the recent critical issue API Docker Images are failing to build #2411 which caused our API builds to fail

I plan to write a larger discussion about how we handle waveforms and thumbnail generally, and how they should likely be completely decoupled from Django, but in the meantime, are there any smaller changes we could make here to improve the stability of bundling this code into our Django application?

CC @dhruvkb for relevant experience with the original implementation.

[sarayourfriend]

This package is only ever tagged latest, so we're automatically getting all changes every time the package is updated and lack the ability to lock the version.

That's unfortunate. I wonder if we could contribute version-specific tagging to the underlying Dockerfile repository: https://github.com/realies/audiowaveform-docker

In lieu of that or in the meantime, we could build the base image ourselves in CI, checking out a specific commit to pin ourselves to a known audiowaveform version.

[realies]

realies/audiowaveform now tags images when commits in the upstream repository are tagged (e.g. versioned)

[dhruvkb]

That is very awesome, we've not had problems with using latest for a long time since this discussion was created but now we'll surely pin it to a fixed version.

Created issue #3782.

[dhruvkb]

I won't mark this comment as an answer and close the discussion, because we haven't found a solution for the "brittle"-ness in extracting the dependencies.

[sarayourfriend]

@dhruvkb copying the dependencies is mostly brittle due to the possibility of a sudden change in the image, right? It's certainly annoying to have to do it, but if we pin the image that shouldn't be as much an issue.

Rather than using the docker image and copying the dependencies, what if we pulled the .deb for the audiowaveform directly? https://github.com/bbc/audiowaveform/releases/tag/1.10.1

That way we just have to apt install ./audiowaveform...deb. Only issue is whether it's easy to make a decision about which arch to download based on the image 🤔