An open-source framework for AI-composed video

Context

Mosaico is the open-source Python framework of the AI Lab at Folha de S.Paulo, Brazil’s largest newspaper, for building news videos. You give it media and an article; it gives you back a finished video. An LLM writes the script scene by scene with structured output through instructor and litellm, a TTS engine narrates it, a transcriber syncs the captions to the audio, and MoviePy renders everything. Each stage sits behind a protocol-based plugin, so the model or the voice can be swapped without touching the pipeline. It’s published under GPL-3.0.

My role

I contributed two pieces. The first is the caption typography engine (the code that decides where a line of on-screen caption text breaks). I rewrote it with tests, around 410 lines. The second is the news script generator. I reworked it for newsroom use, mainly to handle the case where the same photo or clip shows up more than once in a story. Around 280 lines there. Both are in the public repo.

Before: greedy wrap

Heavy rain expected acrossthe state throughout the weekend

one word stranded on the last line

wrap and rebalance

After: measured and rebalanced

Heavy rain expectedacross the statethroughout the weekend

three lines of similar length, nothing hanging

• measure each line in pixels, stroke width included
• split a word wider than the box instead of overflowing
• move words back across the breaks so no line is left alone

= my work

The hard part

Caption wrapping looks like a trivial problem until you look at the rendered frames. Videos were coming out with bad line breaks: a word stranded alone on the last line, or a line running past the edge of its box.

The reason is that the old wrapping logic counted characters. Characters aren’t width. A rendered caption has a stroke around the glyphs so the text stays readable over video, and that stroke adds pixels on both sides of every letter. So I measure instead. Each candidate line is measured in real pixels, stroke width included, against the box it has to fit in.

That stops the overflow. It doesn’t stop the orphan words. Two more passes handle the rest. A word that is wider than the box on its own gets split rather than pushing the line out. Long compound names and URLs, mostly. Then, once the lines fit, the wrapper rebalances them, moving words backwards across the breaks so the last line isn’t a single orphan word. The result is a caption whose lines come out close to even with nothing hanging off the end.

A caption either fits its box or it doesn’t, and you can write a test for that.

Outcome

Mosaico is public, and my typography work ships with it. Anyone rendering captions with the framework gets the measured wrap by default.

I picked up the video side of the stack here. I later co-designed the lab’s new rendering engine and wrote it from scratch (that’s the Kubrick case study). What I knew about frame-level text and media handling going in came from Mosaico.