AI news monitoring

Context

The newsroom of Folha de S.Paulo, Brazil’s largest newspaper, used to track competitors and social feeds by hand. Someone opened the tabs, read everything, and told the desk what was moving. Chase, the lab’s news monitoring system, does that on a schedule instead. It ingests RSS feeds, scraped pages, YouTube channels, Instagram profiles and JSON APIs. Duplicates get dropped. What survives gets an LLM summary, a desk classification and an embedding, and is matched against the topics editors follow. Editors run all of it from a Vue 3 dashboard, hear back on Discord or by e-mail, and can ask it questions in a chat.

My role

I designed and implemented Chase’s current architecture. It started in a much simpler format. Between March and September 2026 I revamped it into the four-stage pipeline this page describes and built most of what runs on it. It has been in daily use since, and I keep shipping on it.

The user side is mine end to end: SSO login (backend, Vue screens and deploy secrets), then user tiers, ownership of topics and rounds, follows, pins and personal notifications. So are the monitoring rounds, which the newsroom organizes itself around. An editor groups topics and sources into a round (a “ronda”) and gets reports from it on a schedule, as PDF or spreadsheet, on Discord and by e-mail. I added YouTube (transcripts, with a Whisper fallback for videos without captions), Instagram and JSON APIs as source connectors, plus source filters and report timing and retention settings. YouTube monitoring grew out of a standalone Whisper-and-Claude pipeline I had built before Chase existed.

The newest piece is the chat, shipped in October 2026. An editor picks a scope (a topic, a round, some sources or everything) and asks questions in Portuguese. The model answers only from the monitored content: it has to call tools to search passages and count mentions before it answers, every claim carries a numbered citation, and the citation keeps the item’s title and URL because retention will purge the item itself. Tool calls and opened items are capped per question, conversations and daily questions are capped per user, and a model failure returns an error with nothing saved.

Editors’ dashboard Vue 3

• SSO login
• Monitoring rounds topics and sources grouped, reports on a schedule
• Follows, pins, user tiers
• Chat questions answered only from the monitored content, with citations
• Topics what editors follow, matched by embedding
• Reports briefings by topic, round or digest
• Sources, trends

configures the pipeline and reads its store

1. 01 Sources one connector per type, each on its own schedule
   • RSS feeds
   • Web pages Playwright, LLM-assisted extraction
   • YouTube transcripts, with Whisper when a video has no captions
   • Instagram
   • JSON APIs
2. 02 Article store PostgreSQL + pgvector
   • deduplicated by normalised URL
   • kept for the retention window
3. 03 LLM enrichment a child workflow per article
   • summary in Portuguese
   • desk classification
   • embedding
   • matched against followed topics
4. 04 Trend detection every ten minutes
   • DBSCAN over the last 24 hours
   • clusters merged into live trends
   • named by an LLM
5. 05 Notify, clean up
   • Alerts Discord, e-mail
   • Reports PDF, XLSX, e-mail, on a schedule
   • Cleanup hourly: expired content, sessions, dead sources, stuck reports

Temporal durable workflows and schedules under every stage: retries, timeouts and partial failure are the framework’s problem, and an editor’s round is a schedule

= my work

The hard part

Durability. The pipeline has four stages (ingest, enrich, detect trends, notify), and on plain cron jobs that breaks constantly. A feed times out, an LLM call gets rate-limited, the job dies halfway and you hear about it from an editor. That was the heart of the revamp: I rebuilt the stages as workflows and schedules on Temporal, a workflow engine that makes retries and partial failure the framework’s problem instead of defensive code scattered through every service. An editor’s round is a schedule.

Then trend detection. Each article gets an embedding (a numeric representation of its meaning) stored in pgvector. Every ten minutes a job clusters the last 24 hours with DBSCAN, a clustering algorithm that doesn’t need to be told the number of clusters up front, and an LLM names each cluster in Portuguese. Nobody knows how many stories are running at a given moment, and an article that belongs to no cluster is labelled as noise.

Then keeping it alive. Once the newsroom used it every day, the bugs were the operational kind: an LLM call that never returned and held a worker slot, a half-open database connection doing the same, an enrichment step firing twice for one article, polling schedules stampeding after a deploy. Each fix is small (timeouts on every model call, bounded statements, capped worker concurrency, jitter on the schedules). Together they are the difference between a demo and a system. An hourly cleanup workflow purges expired content, stale sessions, dead sources and stuck reports.

Last, the test suite: hermetic unit tests with fakes for the session, Temporal and the LLM, and integration tests that run the real migrations against Postgres and a Temporal dev server in containers.

Outcome

In daily use in the newsroom, where it replaced manual monitoring: editors set up their own rounds, the reports land on Discord and in their inbox, and the chat answers the questions the reports don’t.