FFmpeg 9.0 — Jean-Baptiste Kempf

FFmpeg 9.0: swscale, Vulkan, WebP, asm and a big cleanup

FFmpeg 9.0 “Lei” is out.

FFmpeg needs no introduction: it is the engine room of everything multimedia, from every player to every transcoding workflow, and VLC uses libavcodec and libavformat heavily. The FFmpeg and VideoLAN communities have always been close and share quite a few developers, often meeting at FOSDEM and VDD.

FFmpeg 9.0 is a major release. It comes four and a half months after 8.1 “Hoare”, with more than 2200 commits from over 160 authors, touching 1781 files, adding almost 85000 lines and deleting more than 33000. It bumps the major version of all seven libraries (libavutil 61, libavcodec 63, libavformat 63, libavdevice 63, libavfilter 12, libswscale 10, libswresample 7), which means an ABI break across the board, and it means the big cleanup that FFmpeg only allows itself at major bumps.

Let me go through it, subsystem by subsystem.

swscale, rewritten

The most important long-term work in this release is invisible on a feature list: the multi-year rewrite of swscale, led by Niklas Haas, who landed more than 300 commits in libswscale alone this cycle, with Ramiro Polla close behind on the ARM side.

The old swscale is twenty years of bespoke, format-specific conversion code. The new architecture is completely different: every conversion is decomposed into a list of elementary operations (read, swizzle, linear transform, scale, pack…), an optimizer simplifies and splits that list, and a backend then compiles it into a chain of kernels.

In 9.0 this becomes user-visible: the public API gains an SwsBackend selector, and the backends now include the template-based C reference, a fast memcpy path, chained x86 SIMD kernels, chained AArch64 NEON kernels, and a Vulkan SPIR-V backend that compiles the same operation lists into compute shaders, so the exact same conversion graph can run on the CPU or on the GPU. There is also a new SwsScaler enum to pick the scaling algorithm explicitly, instead of the old flags soup.

The correctness engineering underneath is the part I like. The constant math driving every conversion moved to a new 64-bit rational type, computed exactly instead of in floating point, which let a whole series of overflow checks simply disappear. Ask for SWS_BITEXACT and even the Vulkan backend complies, down to decorating its linear arithmetic with SPIR-V NoContraction so the GPU is not allowed to fuse multiply-adds; leave it off and you get a faster matrix-multiply variant instead. And the optimizer learned to split operation lists per plane and into linked subpasses, so a planar conversion compiles into independent kernels per plane instead of one monolithic loop.

The new paths are still gated behind SWS_UNSTABLE, with the legacy code as the stable default. This cycle also brought palette (PAL8) support to the new architecture, with an AVX2 path for palette reads, and proper modelling of interlaced content in the format negotiation. The architecture is now in place.

You should look at Niklas’ talk at the last VDD about this topic.

libavcodec

More than 700 commits landed in libavcodec. The most important ones:

FFmpeg can finally decode animated WebP, natively, with a demuxer to go with it. This closes ticket #4907, opened in 2015, a regular of “why does FFmpeg only decode the first frame?” threads for a decade. The patches were originally started by Josef Zlomek, and Ramiro Polla finally finished it: it was more complex than anticipated, notably with RGB and YUV cases.
The AAC decoder now handles 960-sample frames, the variant used by DAB+ digital radio.
NVENC gains AV1 hierarchical B-frame reference mode, compatibility with Video Codec SDK 13.1, and accepts 12-bit input formats (truncated to 10 bits).
We have a video encoder and muxer for the Playdate, the little yellow console with a crank. It produces 1-bit, 400×240 video using delta coding and zlib compression. It is not important, but it is cool.
A new decoding API, avcodec_receive_frame_flags(), with an AV_CODEC_RECEIVE_FRAME_FLAG_SYNCHRONOUS flag that bypasses frame-threading delay and returns the next frame as soon as possible. This is useful for low-latency consumers that still want threaded decoding the rest of the time.

Assembly and SIMD: x86, NEON and RISC-V

FFmpeg still writes assembly, and this cycle was busy: nearly 200 commits touched the x86 directories, and about a hundred the ARM ones.

On x86, a large part of the work is Andreas Rheinhardt dragging the oldest DSP code in the tree into the present: the half-pel motion compensation code (hpeldsp and fpel, which date back to the MMX era) was ported to SSE2, the H.264 intra prediction gained AVX2 horizontal predictors, the pp7 postprocessing DCT left MMX behind, and the last MMX remnants were removed from mpegvideoenc. The motion-estimation compare functions gained SSSE3 versions of the median SAD, and the new swscale x86 backend generates its SIMD kernels from NASM macros, including the AVX2 palette-read path mentioned above.

On ARM, the NEON yuv2rgb paths were reworked to process two lines at a time, across packed RGB, planar GBR, 16-bit RGB and yuva420p outputs, including big-endian 16-bit formats. HEVC intra angular modes 10 and 26 got NEON implementations, and the AAC SBR and float DSP loops were re-unrolled to better fill modern pipelines. But the most interesting ARM work hides in swscale. Ramiro Polla built the new AArch64 backend on top of rasm, a small assembler framework with an instruction-level IR and builder API. It generates the chained NEON kernels as assembler text at build time, and is designed to allow runtime code generation later.

RISC-V was quieter in FFmpeg this cycle, with RVV optimizations for hevc_add_res and the pixelutils SAD, plus portability fixes. Most of the RISC-V vector energy in the multimedia ecosystem is currently going into dav1d, where FFmpeg gets it back for free through its dav1d-based AV1 decoding.

Camera raw and FFV1

One of the themes of this release is camera raw video in open source.

The native ProRes RAW decoder, introduced in 8.0, was synchronized with the reference implementation this cycle: it now parses the linearization curve from the bitstream, sets frame cropping, and exports the raw Bayer color parameters through a new AV_FRAME_DATA_RAW_COLOR_PARAMS side data, designed to be codec-specific because ARRIRAW is explicitly on the roadmap. A GPU-accelerated Vulkan decoder is available, as is new VideoToolbox decoding on Apple platforms.

On the other side of the pipeline, FFV1 learned to encode Bayer pixel formats: raw sensor mosaics, compressed losslessly, without debayering first. Lynne implemented it in the native codec, with a per-slice search for the best reversible colour transform coefficients, and in a separate GPU-accelerated Vulkan FFV1 encoder and decoder (yes, FFmpeg encodes and decodes FFV1 on the GPU now). The search runs as its own compute pass. Part of the work was sponsored by the Sovereign Tech Fund. The Vulkan encoder also gained a 32-bit float RGB path with a Rice-plus-remap coding mode, and takes care to preserve denormals when reading raw 16-bit floats. FFV1 is widely used for preservation masters, and losslessly compressing camera sensor data at GPU speed is a useful new capability for digital preservation.

Vulkan and GPUs

The Vulkan work keeps expanding across the codec and filter layers:

A new v360_vulkan filter does 360° projection remapping in compute shaders, GPU-accelerating one of the most expensive filters in the tree.
APV, Samsung’s professional production codec added in 8.0, gets a Vulkan GPU-accelerated decoder.
A significant infrastructure change: runtime shader compilation is gone. All Vulkan shaders are now compiled to SPIR-V at build time, and the remaining filters (scale, nlmeans, blackdetect) were ported accordingly. No more shipping a GLSL compiler at runtime to run a scale filter; the swscale GLSL backend was removed in favour of the SPIR-V one.
The Vulkan hwcontext API was modernized for the major bump: VkAccessFlagBits2, publicly exposed queue creation flags, queryable optional instance/device extensions, and new pixel format support (X2RGB10, X2BGR10, XV30, RGBAF16), plus a series of CUDA interoperability fixes.

Vulkan is not the only GPU API in town. AMD’s AMF gains a frame rate converter filter (frc_amf), HDR capabilities in its colour converter (vpp_amf, with new public helpers to translate HDR metadata to and from the AMF representation), and hardware memory mapping support. And the CUDA pipeline gains a new transpose_cuda filter, a generic filtering path in scale_cuda, and end-to-end handling of the 10- and 12-bit 4:2:2 and 4:4:4 formats that recent NVIDIA hardware can finally decode. These professional formats are mapped through NVDEC, the CUDA filters and NVENC alike.

HDR and Dolby Vision

Ten days ago, I wrote about libbluray 1.5.0 exposing Dolby Vision tracks on UHD Blu-rays. FFmpeg 9.0 picks up the other end of that pipeline, and this time the story is complete.

Dolby Vision Profile 7, the flavour used on UHD Blu-ray discs, is a strange beast: a full enhancement-layer HEVC bitstream and the RPU metadata are interleaved inside the base-layer stream, hidden in unspecified NAL unit types 62 and 63. The new dovi_split bitstream filter takes such a stream apart: keep the base layer only and get a plain, standard HEVC stream, or properly separate the layers. Around it, libavformat gained a real model for the thing: a Dolby Vision stream group (a generalization of the LCEVC stream groups into AVStreamGroupLayeredVideo), detected when demuxing MP4, MPEG-TS and Matroska, and written out again, including the hvcE enhancement-layer configuration, when muxing to MP4 or Matroska. This makes it possible to demux, split and remux these streams while keeping the signaling intact.

On the metadata front, 9.0 adds parsing, writing and passthrough for SMPTE 2094-50 dynamic HDR metadata, another SMPTE 2094 format alongside HDR10+, down to encoding with libaom and libvpx, Matroska block additions, and ffprobe output. The spec is being finalized in the open, on GitHub of all places. Yes, SMPTE drafts standards on GitHub now. The whole ITU-T T.35 dynamic metadata zoo, including HDR10+, China’s HDR Vivid, AOM film grain and Active Format Description, is now parsed through one unified set of helpers instead of being reimplemented codec by codec. And if your file carries wrong static HDR metadata, the ffmpeg CLI gains per-stream -mastering_display and -content_light input overrides to correct the metadata on input, which anyone who has handled badly-mastered HDR files will appreciate.

libavformat

Close to 400 commits landed in libavformat. Beyond the Dolby Vision work above:

A new demuxer command API (avformat_send_command() / avformat_receive_command_reply()) lets applications talk to a running demuxer. The first user is RTSP, exposing SET_PARAMETER, so interactive camera control no longer needs an out-of-band connection. The design is deliberately asynchronous: you keep reading packets between the command and its reply.
LCEVC enhancement layers can now be muxed as proper tracks in MP4, and as ITU-T T.35 block additions in Matroska, completing the LCEVC chain that started two releases ago.
MP4/MOV gained TREF stream groups to model track references, so timed metadata streams (think EXIF or camera telemetry) stay attached to the media they describe, version 1 chnl boxes, stsd version 2 for more than two audio channels, and export of initial padding and trailing-sample trimming for gapless audio.
A new shared: protocol implements a concurrent block cache: thread-safe and cross-process, so several FFmpeg instances reading the same remote stream can share a single cache file. It is memory-mapped, with a nice side effect: once the stream has been fully read, the cache file is a working copy of the streamed file. By Niklas Haas, when he is not rewriting swscale.
IAMF, the Alliance for Open Media’s immersive audio format (the one underneath Eclipsa Audio), got its dynamic parameters, including mix gain, demixing information and recon gain, exposed as proper frame side data, printable by ffprobe and the showinfo filter; by Marvin Scholz. The IAMF muxer now enforces the format’s fixed frame size through a new AVFMT_FIXED_FRAMESIZE capability flag, instead of silently producing broken files.
On the network side: the WHIP muxer now requires and verifies the remote DTLS fingerprint from the SDP answer, mbedTLS got a fix to actually check the certificate verification result on external sockets, and RTMP, FTP and RTSP all received input-validation hardening.

The big cleanup

Everything deprecated during the 8.x cycle is gone, and the list is long.

On the command line, -vsync is finally removed (use -fps_mode), along with -top, -qphist, -filter_complex_script and -adrift_threshold. The OpenMAX encoders are removed. The NPP filters (scale_npp, scale2ref_npp, sharpen_npp, transpose_npp) are gone with the rest of the libnpp support, deprecated last September; the transpose_cuda filter mentioned above picks up where transpose_npp left off, without the external dependency. The packed-YUV oddities v308/v408/v410 lose their dedicated codecs. And NVENC drops the old preset aliases, the legacy rate-control modes and support for SDKs older than 11.1.

One behaviour change deserves everyone’s attention: FFmpeg now verifies TLS peer certificates by default. This was announced with 8.0 and applied at this bump, exactly as promised. If your scripts talk to servers with broken certificates, they were living on borrowed time anyway.

My two favourite removals, though, are pure FFmpeg lore:

The standalone CELT decoder is gone (this does not affect Opus, which contains its own CELT). The commit message notes it had actually been broken since 2014, “and no one noticed and complained”. Eleven years of silent bitrot, removed by VideoLAN’s own Marvin Scholz.
The Sonic codecs, Michael Niedermayer’s experimental codec family from the early 2000s, are removed too. The removal had been scheduled for the previous major bump and missed the deadline. If you have Sonic files somewhere, and statistically you do not, transcode them with an 8.x release while you can.

The command line

The -re/-readrate machinery was reworked to throttle on the slowest stream instead of the fastest, skipping finished and not-yet-started streams. If you have ever seen a live restream starve because of a sparse subtitle track, this is your fix. ffprobe learned to print the new side data types (SMPTE 2094-50, IAMF mix gain and demixing parameters, downmix info), and the ffmpeg CLI can now create LCEVC stream groups when muxing.

And the DNN filters gain an ONNX Runtime backend with CUDA and DirectML execution providers, so you can run ONNX models directly in a filter graph for super-resolution, denoising or background removal, without converting them for TensorFlow or OpenVINO first. Combined with the Whisper filter from 8.0, the filter graph is becoming a decent place to run inference next to your decode loop.

Under the hood

Some less visible changes are worth mentioning too:

FFmpeg’s assembly tests now use the standalone checkasm library, the project I mentioned in the dav1d 1.5.4 post: born in x264, rewritten in FFmpeg, relicensed for dav1d, and now maintained once, for everyone.
Development now runs on FFmpeg’s own Forgejo instance at code.ffmpeg.org, alongside the mailing list; GitHub pull requests “will be ignored”.
Among the 160+ contributors of this cycle sits an automated one: a bug-finding bot landed nine accepted hardening patches in the demuxers and protocols, alongside the continuous stream of fuzzing fixes from OSS-Fuzz.

Conclusion

FFmpeg 9.0 is a substantial release, with new features as well as a lot of cleanup. The ABI break means distributions and applications will take some time to migrate the seven new sonames, and VLC will follow.

The full Changelog is worth a read, and so is the git log; congratulations to the 160+ people who wrote this release, and in particular to Niklas Haas, Andreas Rheinhardt, Michael Niedermayer, James Almer, Lynne, Marvin Scholz and Ramiro Polla, who top a very long list, several of them doing part of this work at FFlabs and/or VideoLabs.

The code is, as always, open source. Patches welcome.