A declarative video engine

Context

The AI Lab at Folha de S.Paulo, Brazil’s largest newspaper, produces news videos automatically. An upstream service writes the script from an article, and something has to turn that script into an actual MP4. That rendering step used to run on the MoviePy-based engine of Mosaico, the lab’s open-source video framework. Kubrick is its replacement: it takes a spec and renders the file. It’s still a work in progress (the rendering step is what I’m building right now), and this page describes the design and the parts already in place.

My role

I co-designed Kubrick with the lab’s tech lead and wrote the implementation. Every implementation commit in the repo is mine. I specced the JSON format and wrote the compiler and the renderer.

There was an earlier attempt. I wiped it and rebuilt the engine from scratch in February 2026.

The input is a ScriptSpec, a JSON format I designed (a small domain-specific language). It describes shots, audio, subtitles and effects. Time ranges are either absolute or relative to the media itself, so a clip can say “start two seconds into this audio” without anyone precomputing offsets. A scene can be reused by key with overrides on top of it. An effect targets a single clip or a whole track.

1. 01 LLM scripting service upstream, not mine: writes the script from an article
2. 02 ScriptSpec a JSON DSL, the canonical format
   • time ranges absolute or relative to the media
   • a scene reused by key, with overrides
   • effects targeted per clip or per track
3. 03 Compiler spec to OpenTimelineIO timeline
   • video, audio, subtitle and overlay tracks
   • OTIO never leaves the backend
4. 04 Renderer written directly on PyAV, NumPy and Pillow; no MoviePy
   • a frame generator per track
   • alpha compositing of overlays and subtitles
   • stereo resampling and FIFO audio mixing
   • interleaved mux; frame and audio effects in the same loop
5. 05 MP4 frame-accurate

= my work

Kubrick compiles the spec into an OpenTimelineIO (OTIO) timeline, the interchange format professional editing tools speak, then renders that timeline with code written directly on PyAV, with NumPy and Pillow for the raster work. No MoviePy. Each track becomes a generator that yields frames. Overlay and subtitle tracks are alpha-composited onto the video underneath. Audio tracks are resampled to a common stereo format and mixed through a FIFO. Video and audio get interleaved into the MP4 as they come out. Frame effects (blur, crop, fade, flip, rotate) run in the same loop, and audio has its own set.

The hard part

Two things.

The first was the DSL. Other tools have to write and edit these specs (the scripting service upstream, and the editing tooling that adjusts a video before it renders), so the format has to survive being touched by something that isn’t Kubrick and still compile. I kept the ScriptSpec as the canonical format and pushed OTIO entirely into the backend (nothing upstream ever sees a timeline object). That constraint drove most of the design. Media-relative time ranges exist so an edit doesn’t force a recalculation of every offset after it. The key mechanism exists so a repeated scene is one declaration plus overrides, instead of a copy-pasted block that drifts out of sync with the original.

The second is writing the compositor and the mixer by hand, and this is the rendering work still in progress. ffmpeg bindings hand you decoded frames and packets, not a timeline. Everything between those two things is code you write: frame timing, which track sits on top at a given instant, what to do when an audio stream is shorter than the clip pointing at it. Frame accuracy is the requirement. One frame of drift between the picture and the narration and the video looks broken.

Outcome

Kubrick is a work in progress. The spec format and the compiler are in place, and I’m currently building out the rendering step; once that lands, it takes over as the rendering stage of the lab’s video pipeline. It’s internal, so there’s no link. The part I’d point at today is the spec format. It’s plain JSON, and the tools that generate it don’t need to know anything about video.