# rea-video: reverse-engineering a finished video, and what its shot detector actually sees
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.satyajit.si/articles/rea-video
> date: 2026-11-09
> tags: video, agents, computer-vision, multimodal
A post from @CY160909 introduces REA-Video as an open-source project for reverse-engineering video. Install it, hand a video to an agent, and it splits out every shot, judges whether each one was filmed, AI-generated or made as code motion graphics, and works back to each shot's prompt, camera move, edit rhythm, captions and sound effects, ending in a full remake plan with keyframe stills. "Free and open source, supports local video analysis," the post says (reported, my translation from Chinese).
That is a lot of claims for one tool. Some of them are measurement problems a CPU can settle offline. Others are judgement calls that need a model which can see. This article sorts the two, explains the parts that are code, and tests the first step every other step depends on: where the shots start and end.
Every claim below carries a label: *measured* (I computed it from files or from clips I made), *reported* (the repo or the post says so) or *reasoned*.
## What is in the repo
I cloned [JuneYaooo/rea-video](https://github.com/JuneYaooo/rea-video) at commit `356815c` (9 November 2026). It is Apache-2.0, with a `NOTICE` file naming JuneYaooo as copyright holder (measured). The Python package `rea_video/` is 42 modules and 12,695 lines, with 174 test functions under `tests/` (measured). It credits [REA (Reverse Engineer Anything)](https://github.com/morluto/rea) for the stance: back every claim with evidence and verify by measuring (reported).
There are three layers, and the split matters for everything that follows.
| Layer | What it is | Runs where |
|---|---|---|
| `rea_video/` | the CLI: shot detection, forensic features, audio, OCR, fusion, report, remake plan | your CPU: NumPy, OpenCV, ffmpeg, onnxruntime |
| `skills/rea-video/SKILL.md` | instructions for an agent: run the CLI, review uncertain shots by looking at the frames, merge the review | inside Claude Code, Codex or similar |
| optional VLM | any OpenAI-compatible multimodal endpoint, set by `REA_VLM_BASE_URL`, `REA_VLM_API_KEY`, `REA_VLM_MODEL` | someone's API |
The hard dependencies are `numpy`, `opencv-python-headless`, `Pillow`, `PyYAML` and `onnxruntime` (measured, `pyproject.toml`). The ONNX runtime is there for one model: UVR MDX-Net "Inst HQ 3", about 67 MB, downloaded on first run, which separates the voice from the music by predicting the instrumental and subtracting it from the mix (reported, `stems.py`). OCR is an optional extra through `rapidocr-onnxruntime`. Nothing else is a neural network. Shot detection, frame-cadence checks, noise estimates, camera motion and tempo are classical signal processing (measured, by reading the modules).
The VLM client is 426 lines of `urllib` against `/chat/completions`, with a comma-separated model list as fallbacks and a refusal to send a key over plain `http://` to anything but localhost (measured). Keys are read from an env file, never from the repo.
## The pipeline, in four steps
The README states the order (reported); the code follows it (measured):
1. **Split.** `shots.py` cuts the video into shots. `audio.py` and `stems.py` split the soundtrack into voice, music and effects.
2. **Gather evidence.** `features.py` and `probe.py` measure each shot: duplicate-frame cadence (a 24 fps generator output converted to 30 fps repeats one frame in five), sensor-like noise, whether the whole frame moves as one rigid plane (a still image being panned), palette size and flat blocks (rendered graphics), and container metadata such as AIGC labels, encoder signatures and C2PA.
3. **Decide.** `classify.py` turns the evidence into a probability over 11 base labels, fuses it with a VLM's or the agent's opinion if there is one, and flags conflicts for review.
4. **Plan the remake.** `plan.py` routes each label to a recipe: AI shots go to first-and-last-frame generation, live shots get a shooting plan, code shots get a Remotion or HTML spec, still-image pans are rebuilt locally with ffmpeg from the measured zoom and pan.
The 11 labels are live action, AI video, avatar, 3D render, code render, screen capture, still image with pan or zoom, static hold, 2D animation, stop motion and film footage (measured, `classify.LABELS`). The module's own docstring is candid about the last three: "anim_2d / stop_motion have no pixel test yet", and film footage "is camera footage like any other, so pixels cannot tell who shot it". Only an image reader can call those (reported).
The "keyframes" in the post are plain ffmpeg stills. For each shot the pipeline saves a three-frame contact sheet plus full-resolution `_first.jpg` and `_last.jpg`, which the remake recipe hands to a video model as the first and last frame (measured, `pipeline.py`).
## What needs a model that can see
This is the line the post blurs, so here it is from the code.
**Offline, no model:** shot boundaries and transition types, frame cadence, noise, rigid-plane motion, whether anything moves at all, camera support and push or pull from global motion, AIGC metadata, caption timing from OCR, tempo in BPM, whether cuts land on beats, voice and music separation (measured, by module).
**Needs a VLM or the agent looking at frames:** the reverse-engineered prompt for each shot, live action versus AI video when there is no metadata or cadence evidence, 2D animation, stop motion, film footage, lens and lighting descriptions, and the full sound-effect list. The prompts in particular come from one place: the `replication` field the VLM is asked to return (`image_prompt`, `video_prompt`, `video_prompt_en`), or the description an agent writes during review (measured, `vlm.py` and `plan.py`). With neither, the plan has no prompt to give.
The repo's own eval log says the same about sound effects: on a voice-plus-music mix with 110 labelled effects, the signal features could not tell effects from speech or music onsets, at about 0.2 precision, "no better than chance", so the full effect list is left to the VLM (reported, `eval/RESULTS.md`).
So "supports local video analysis" is true for the forensics and the edit structure. The prompts, which are the headline of the post, need a model.
## How it fuses the tools with a model
The tools produce a distribution over labels. A VLM, or the agent reviewing frames, produces another. `classify.fuse` pools them per label as a weighted product:
$$
s_k = \exp\big(0.6\,w_k \log(p^{\text{tools}}_k + 10^{-4}) + 0.7\,\log(p^{\text{reader}}_k + 10^{-3})\big)
$$
then normalises. $w_k$ is a per-label weight on the tools: 1.6 for static hold, 1.4 for still image with motion, 1.0 for code render, 0.6 for live action and AI video, 0.5 for 3D, 0.4 for avatar, 0.3 for the three labels pixels cannot see (measured). The reader's weight is 0.7. A code comment says 0.7 was chosen after the first real Gemini run, where 1.0 "let a confident but wrong VLM override 24fps/transcode evidence on AI videos" (reported).
A larger $w_k$ is a sharper exponent on the tools' probability. Its effect is less "trust the tools when they say yes" than "trust them when they say no": a label the tools rate at 5 % with $w_k = 1.6$ enters at $0.05^{0.96} \approx 0.056$, against $0.05^{0.36} \approx 0.34$ at $w_k = 0.6$ (reasoned). That matches the docstring: the tools are decisive for temporal questions and weak for live action versus AI versus 3D.
The widget shows only that pool for two labels. The real `fuse` runs it twice, once over label groups and once inside the winning group, multiplies every moving label by 0.05 when the shot is measured as frozen, requires the answer to be in some source's top two, and clamps the result when the tools agree with exactly one of the two image readers (measured). Read it as the shape of the weighting, not as the tool.
## The first step: finding the shots
Everything downstream is per shot. A missed cut merges two shots into one, and the classifier then averages two production methods into one verdict. A false cut splits one shot into two, and the remake plan asks for an extra generation. So I tested the boundary detector.
`shots.py` works on gray thumbnails 96 pixels wide, one per decoded frame, plus a 256-bin HSV colour histogram per frame (16 hue, 4 saturation, 4 value bins). For each frame it computes three signals against the previous frame (measured):
- `hist`, the Bhattacharyya distance between the two histograms;
- `pix`, the mean absolute gray difference, over 255;
- `changed`, the share of thumbnail pixels that moved by more than $\max(6,\ 0.35\,\sigma)$, where $\sigma$ is the smaller of the two frames' standard deviations.
The cut score is $0.6\cdot\text{hist} + 0.4\cdot\min(1,\ 4\cdot\text{pix})$. A hard cut fires when the score passes 0.30, passes $2.5\times$ its local median over half a second either side plus 0.06, and more than a quarter of the frame changed. A second "relative" test catches cuts where most of the frame changes even if the colours barely do. Then come four more rules: fades through black or white, dissolves, wipes, and jump cuts. Two filters throw out flashes: if brightness collapses within two frames, or the picture two or three frames back reappears, nothing was cut. Shots under 0.35 s are merged into their neighbour, keeping the stronger boundary.
The dissolve test is the neat one. For a frame $M$ with frames $A$ and $B$ at $k$ frames either side, it fits $M \approx A + \alpha (B - A)$ by least squares. A cross-dissolve leaves a small residual with $\alpha$ between 0.2 and 0.8. A camera move does not, because the middle of a pan is not a blend of its ends. A Farneback optical-flow check then rejects pairs where $B$ is a warped copy of $A$ (measured, `shots.py`).
### What I ran, and what I could not
I could not run `rea_video` itself. Importing the cloned package was refused by the sandbox's permission layer as executing downloaded code, and I did not route around it. What I did instead:
1. Made test clips with ffmpeg 6.1.1 from its built-in sources: the test pattern, a Mandelbrot zoom, Conway's Game of Life, colour bars, and a 1,600-pixel cellular-automaton strip panned through a 640-pixel window. Every transition is placed to the frame with `concat` and `xfade`.
2. Re-typed the rules from `shots.py` in my own 150-line script: the score, both hard-cut tests, the jump-cut test, both flash filters, fades, dissolves at all three window sizes with the flow check, wipes, and the minimum-length merge. I left out the motion-graphics "beat" rule, which marks changes inside a shot and does not add boundaries. Python 3.13, NumPy 2.5.3, OpenCV 4.10.0 headless, on 2 CPU threads.
3. Ran ffmpeg's own `scdet` filter at its default threshold of 10 as a baseline.
So these are measurements of the rules as I read them, not of the shipped tool. Where my re-typing differs from theirs, the numbers would too.
The three clips:
- **Fair clip**, 575 frames at 30 fps: two hard cuts, a 0.25 s cross-dissolve (the length used in the repo's own eval) into a 10-pixel-per-frame pan, a hard cut, a two-frame flash inside the colour-bars shot, a 1 s dip to black, and a 0.6 s wipe, encoded with x264 at CRF 18.
- **Fair clip, re-encoded**: the same, scaled to 480 pixels wide at CRF 38, 3.9 MB down to 0.21 MB. A stand-in for a platform download.
- **Stress clip**, 567 frames: a 0.5 s dissolve out of a dark, busy Game of Life shot, a three-frame flash, and the wipe made out of a field of moving noise.
A detection counts when it lands inside a transition's span, give or take three frames. Each transition can be hit once.
| Clip | Detector | Detections | Hits | False | Missed | F1 |
|---|---|---|---|---|---|---|
| fair | rea rules (re-typed) | 7 | 6 | 1 | 0 | 0.92 |
| fair | ffmpeg `scdet` | 31 | 6 | 25 | 0 | 0.32 |
| fair, re-encoded | rea rules (re-typed) | 7 | 6 | 1 | 0 | 0.92 |
| fair, re-encoded | ffmpeg `scdet` | 30 | 4 | 26 | 2 | 0.22 |
| stress | rea rules (re-typed) | 8 | 5 | 3 | 1 | 0.71 |
| stress | ffmpeg `scdet` | 18 | 4 | 14 | 2 | 0.33 |
(measured). The whole re-typed detector, decode included, took 0.85 s of wall time on the 19-second clip (measured).
### What held
**Motion does not fire cuts.** This is where the design earns its keep. Inside the fast pan, 52 % of the thumbnail's pixels change from frame to frame, and the score never rises above 0.083 across the whole shot, because the colour histogram of a pan barely moves (measured). `scdet`, which looks at pixel difference, fired every sixth frame through that shot of the fair clip, 13 times before the real cut (measured). The Mandelbrot zoom and the Game of Life produced no false cuts either.
**Two-frame flashes are rejected.** The flash at frame 398 passes both hard-cut tests with a score of 0.904, and the flash filter removes it because brightness is back two frames later (measured). The docstring promises exactly this for one- and two-frame flashes (reported).
**Hard cuts land on the frame, and survive re-encoding.** All three cuts in both fair clips were found on the exact frame (measured). Dropping the bitrate by a factor of about 18 changed nothing about the cuts, moved the dissolve's boundary by four frames, and moved the second half of the dip to black by eight.
**The wipe was found and named.** At frame 493 in both fair clips, labelled `wipe` (measured).
### What did not
**A dip to black becomes two boundaries and a 0.4 s black "shot".** The colour bars fall to black in about five frames, crossing the histogram's value bins as they go, so the histogram half of the score jumps above 0.5 on consecutive frames as if a new shot had started (measured, frames 414 to 418). The hard-cut rule claims frame 416. The dedicated fade rule then never adds its own boundary, because it skips any fade within half a second of an existing boundary. The fade-in fires the cut rule again at frame 428. Result: two boundaries 12 frames apart, both labelled `cut` (measured). That is the one false boundary in both fair clips.
**Dissolves come out as cuts.** The 0.25 s dissolve was found, on its last frame in the fair clip and at frame 266 in the re-encoded one, but labelled `cut` both times (measured). The 0.5 s dissolve in the stress clip fired the hard-cut rule at both ends, frames 256 and 270, giving another short spurious shot; the dissolve rule only appeared, mislabelled, inside the dip to black (measured). The colour histogram of a dark frame is sensitive to the first frames of a blend toward a bright one, which is the same mechanism as the fade (reasoned).
**A three-frame flash is a cut.** At frame 390 in the stress clip: the flash filter checks two frames ahead, and a three-frame flash is still bright there (measured). The code says one or two frames, so this is as documented, but camera flashes and muzzle flashes at 30 fps often last three (reasoned).
**A wipe out of moving noise is missed.** The wipe rule requires that the two ends of the window differ far more than the picture changes in an equal stretch just before or after. In a shot of moving noise, they do not (measured). Real footage is rarely that noisy; heavy grain or rain might be (reasoned).
None of these are disasters. The transition-type label is wrong more often than the boundary, and the boundary errors are short extra shots, which the agent review step is designed to catch. But they are worth knowing before trusting a per-shot "AI video 41 %".
## What the repo claims about accuracy
The README's accuracy table, weighted by shot duration, for the "how was it made" call (reported):
| | Tuning set (15) | Holdout (9 / 7) | Re-encoded WeChat videos (10) |
|---|---|---|---|
| Tools only, offline | 98.3% | 99.6% / 98.9% | 88.6% |
| + vision model (Gemini) | — | 99.6% / 99.8% | 99.6% |
| + agent visual review | — | 99.6% / 99.8% | 100% |
It also says "shot boundaries are all correct". The eval log behind that sentence is narrower: cut F1 is 1.00 on the two tuning-set videos that have ground-truth cuts, 23 cuts in all, including 14 dissolves of 0.25 s (reported, `eval/RESULTS.md` and `eval/labels.yaml`). The holdout sets have no cut ground truth. My numbers above are not a contradiction. They are a different kind of test: synthetic, adversarial on purpose, and of my re-typing.
The eval log is unusually honest, and worth reading in full. Three entries stood out (all reported):
- **The VLM alone made things worse on held-out data.** In the 30 October run on the two holdout sets, adding Gemini without agent review moved holdout2 from 0.998 to 0.965, because Gemini called text cards stills and plush AI characters live action or 3D, at 0.85 to 1.0 confidence. Tools plus VLM plus agent review came back to 0.998 only after a rule that sides with the tools when exactly one image reader agrees with them.
- **The rules were tuned on the data they report.** The tuning set is labelled as such. Holdout 1 "is no longer fully independent" after two misses there led to a frame-rate rule. Only holdout2 was first run after the rules were frozen.
- **The live-action sample is small.** The known-limitations list says there are three live-action videos, all 60 fps phone clips from Douyin. The same list says the VLM path was only tested end to end against a fake server, which the dated entries further down have since overtaken.
A non-vision agent reviewing the tools' output overturned them four times, two right and two wrong, both wrong ones by re-reading pixel statistics. The skill now forbids an agent that cannot see images from overturning the tools' production-method call (reported). That is a rule I would keep in any pipeline like this.
## Using it
The intended path is to tell an agent: install the repo, then break down a given video. The agent runs `bash scripts/install_skill.sh`, which installs the CLI and links the skill into `~/.claude`, `~/.codex` and `~/.agents`, then (reported):
```bash
rea-video decompose ref.mp4 -o out/ref --lang en # tools only, offline
rea-video --env-file .env decompose ref.mp4 -o out/ref --vlm --lang en
rea-video summary out/ref/breakdown.json # the compact view
rea-video review out/ref # review_tasks.json for an agent that can see
rea-video apply-review out/ref --by
```
The outputs are `breakdown.json`, `report.html`, `report.md`, a storyboard image, per-shot contact sheets and first and last frames, and separated stems (reported). The skill tells the agent to read `rea-video summary` rather than the full `breakdown.json`, which it describes as hundreds of thousands of characters.
Remakes go through a sibling project, capsule-cinema, which generates the AI shots and places them back on the original timeline after a free dry run and an explicit go before paid calls (reported). The README's showcase remakes used a self-hosted MiniMax H3, the model behind [FreeVideo](/articles/freevideo-minimax-h3).
For the opposite direction, writing a motion-graphics film from a brief rather than taking one apart, see [Scenario's kinetic-film skill](/articles/scenario-kinetic-film-skill). rea-video's `code_render` route ends where that skill starts: text tracks, layout and motion timing, rebuilt as code.
The README has a section on rights that I would underline: the tool is for learning a video's structure and pacing, then remaking it with material you have the rights to, not for reusing someone else's footage, music, characters or likeness (reported).
## Verdict
rea-video is a careful piece of classical video forensics with an agent workflow wrapped around it. The offline half (shots, cadence, noise, motion, metadata, tempo, stems) runs on a laptop and is the reason the tool is more than a prompt. Its shot detector ignores fast pans, zooms and short flashes that a pixel-difference detector trips on. Its weak spots on my clips were dips to black and dissolves out of dark shots, which split into short extra shots labelled as cuts.
The reverse-engineered prompts, which the post leads with, are a VLM's or an agent's description, not a measurement. The repo says so. The fusion weights and the eval log show the authors learned the hard way that a confident vision model is not the same as a right one.
What I would do before relying on it: run `rea-video decompose` on two or three of my own videos where I know the cuts, check `breakdown.json`'s shot list against them, and look at every shot under about half a second. That takes ten minutes and tells you whether the per-shot verdicts are about real shots.