vmarker

Video is the worst-indexed media type most people own. Lectures, meetings, gameplay, walkthroughs, family footage: the retrieval story is usually scrubbing a timeline. Full-text search solved this for documents decades ago. Video never got the equivalent, because the content is not text until you spend compute to make it text.

vmarker spends that compute once, at ingest, then treats the video like a document collection. Every frame becomes a caption. Every stretch of audio becomes a transcript chunk. Optionally, every frame's on-screen text becomes another string. Once all of that is embedded into one store, you get semantic search and grounded question answering over footage that previously had zero index.

Drop a video in. Ask it a question. Get an answer with citations that jump the player to the exact frame.

The two user-facing operations stay apart on purpose. They cost wildly different amounts and answer different questions.

If I had merged them behind one endpoint with flags, the cheap operation would have inherited the expensive one's latency and cost model. Then I would have had to rate-limit browsing. Keeping the cheap operation cheap is the point.

Three deployable units. The split is driven entirely by how long ingest takes.

ffmpeg, Whisper, and a vision call per frame take minutes and block. Splitting the worker out buys three things: the API stays responsive no matter how deep the queue gets, processing scales horizontally by adding worker containers with nothing about the API changing, and the job is durable because it lives in Redis, so an API restart does not lose queued work.

I chose RQ over Celery and Arq for the smallest dependency surface. No broker beyond the Redis I already run for pubsub. Its built-in scheduling gives me q.enqueue_in, which implements the 30-second soft-delete purge with no cron and no extra process. SimpleWorker specifically, because it does not fork and several of the model SDKs in the pipeline misbehave across fork(). The cost is one job at a time per worker and a crash taking the worker down, which is fine when the unit of scale is a container.

• React
• Python
• FastAPI
• PostgreSQL
• RAG
• Video

One detail worth noticing: GPT-4o-mini does four jobs here (vision captioning, query planning, reranking, and answer generation). One deployment, four call sites, four prompts. That keeps the vendor surface small and the cost model easy to reason about. ChromaDB survives in the dependency list purely as the source of that default sentence-transformer. I dropped it as the vector store and kept it as the encoder.

The entry point is process_video_frames in video/frame_processing.py, roughly a 450-line orchestrator over the video/pipeline/ package. It builds a frozen PipelineContext once (ids, paths, interval and fps, max frames, language, database handles) and passes it down. The leaf stages are pure functions, so the orchestrator holds all the mutable state and everything below it is testable in isolation.

1. 1

   Download and transcode

   Pull the source from R2 to a temp path, then normalize to a web-playable MP4. Uploaded MOV and AVI files will not reliably play in a browser <video> element, and the UI has to seek to a citation timestamp.

2. 2

   Extract frames

   ffprobe reads fps and duration, then uniform sampling at a default interval=1.0s, capped at max_frames=1000. Timestamps are computed as i/fps so they align to the real timeline rather than to frame ordinal.

3. 3

   Transcribe

   Audio out to mp3, Azure Whisper, chunked into ~30-second windows (_TRANSCRIPT_WINDOW_SEC = 30.0), each embedded into video_transcript_chunks. Audio under 1 KB is treated as "no real audio", the cheap guard against a silent video generating a garbage transcript.

4. 4

   Caption, check, OCR, embed

   A ThreadPoolExecutor fans process_frame across frames. Caption with Azure vision, three retries at 0.5 * 2^attempt backoff with automatic Gemini fallback. Read the vendor's own safety metadata. Optionally OCR. Embed and persist.

Threaded, not async. That was a change from the original plan. The workload is blocking vendor SDK calls and ffmpeg subprocesses, and the vision and Whisper SDKs had no proper async client, so I would have been wrapping sync calls in executors anyway: async's complexity with threads' performance.

The 1:1 guarantee matters. On terminal caption failure the pipeline writes a placeholder caption, and _safe_add_frame falls back to a constant embedding if embedding fails outright. Either way the row count always equals the frame count, because a lot of downstream logic (progress percentages, frame navigation, citation indexes) assumes frame i has a row.

Progress flows worker to Redis pubsub to API to WebSocket to browser at WS /api/ws/progress/{video_id}. The pubsub hop exists because the worker and the API are separate containers. The worker has no idea which API container, if any, is holding that user's socket. Alongside it, a heartbeat thread updates last_heartbeat_at on the video row, which solves a different problem: pubsub tells you what is happening, the heartbeat tells you whether anything is happening at all, so the UI can distinguish a stalled worker from a slow one and offer Retry.

Naive retrieval merges all channels and sorts by raw cosine. That quietly favours whichever channel produces bigger numbers, independent of relevance. Captions, ASR windows, and OCR strings have different lengths and spreads, so a 0.62 from OCR and a 0.62 from ASR do not mean the same thing.

The product path calibrates within each channel first, adds a lexical overlap score for proper nouns and short identifiers the sentence encoder tends to squash, then expands strong hits by a few seconds of temporal neighbours. Fusion method switches at runtime through VMARKER_FUSION_METHOD (calibrated, rrf, or raw), so a bad fusion incident rolls back with an env change and a restart, not a deploy.

plan_to_weights maps planner flags to channel weights: enabled gets 1.0, disabled gets _CHANNEL_FLOOR = 0.25. Soft down-weight, not a hard drop, because the planner is an LLM and will sometimes be wrong. A mis-planned channel can still surface a strong hit and climb back into the candidate set.

Five stages (quota pre-flight, plan, hybrid retrieve, LLM rerank, stream) with a best-effort persist at the end. The interesting half is what happens when each one fails.

The content-filter case was a real decision, not an afterthought. Azure's filter false-positives regularly on benign footage. The retrieval was correct and the vendor refused only the synthesis, so throwing away the retrieval too would punish the user for someone else's false positive.

Citations arrive first. The stream emits a citations SSE event before a single token of text, then delta chunks at temperature 0.2 and max 800 tokens, then done. Perceptually, the gap before the first token is where the product feels slow, and retrieval and reranking are already finished by then, so the UI can render citation chips immediately. Architecturally, it fixes the citation set before generation instead of parsing it out of model output afterwards. The inline [N] markers index into a list the server already committed to, so a citation can never point at something that was not retrieved.

• processing_stage is separate from processing_state. The fine-grained phase (extracting, captioning, cancel_requested, and the rest) drives a specific UI chip without overloading one column with two meanings. Cancel works by the UI setting cancel_requested and the worker polling it between frames, so it stops cleanly rather than being killed mid-write.
• trashed_cascade_video_ids is JSONB on projects. When a project is trashed, the cascaded video IDs are recorded, so undo restores exactly the set that was removed rather than everything currently in the project.
• ask_quota uses lazy reset. An expired window is computed as used=0 at read time and only written when the row actually rotates. No cron, and cost proportional to active users rather than registered ones.
• video_frame_ocr is a nullable-embedding sidecar to video_frames with ON DELETE CASCADE, because a frame can legitimately contain no text.
• HNSW runs at defaults, m=16 and ef_construction=64, sufficient up to a few million rows per user. The revisit trigger is tens of millions of frames per user, at which point I shard or move the hot vectors to Qdrant.

Why put vectors in Postgres instead of a dedicated vector database?

Because I get one operable store instead of two, and the joins are real. Every vector row sits next to user_id, video_id, and timestamp in the same table, so scoping a search to one user's videos inside one project is a single SQL statement, transactionally consistent with the metadata. With a separate vector store I would filter in two places and every soft delete would become a distributed consistency problem. Trash a project in Postgres and the vector store has orphaned rows until some reconciliation job catches up. pgvector also came free with the Supabase project I was already running for auth.

Two extra LLM calls per ask, one to plan and one to rerank. Is that not expensive?

Yes, and it is the clearest cost-versus-quality tradeoff in the system: three round trips where a naive RAG pays one, with the first two serial and sitting in front of time-to-first-token. The path out, in order: cache plans, since similar queries want the same routing; replace the LLM reranker with a cross-encoder, which is faster, cheaper, and deterministic. Until then, I bought quality with latency and I know the price.

How do you control cost for users, and what is wrong with how you do it?

A 50 MB upload cap, a 1000-frame cap per video, and a rolling 24-hour ask quota defaulting to 10. What is wrong is that a daily ask count is blunt: it treats a one-frame lookup and a full-library question over a thousand frames as the same unit of cost, which they are not by an order of magnitude. Pricing by channel-tokens consumed would be fairer. I have not built that yet.

Two more in the same spirit. The shipped lexical channel is substring overlap, not BM25. Same intent, less sophistication. And the R2 bucket is public, so object keys are guessable; namespacing by user is obscurity, not authorization. Acceptable for a free tier where users upload things they are willing to have a public URL for, and unacceptable the moment private projects ship, at which point the fix is signed URLs.