# GEM-4D: a video world model taught geometry it never has to output
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.satyajit.si/articles/gem-4d
> date: 2026-11-13
> tags: robotics, world-models, video-generation, flow-matching, distillation, depth-estimation, 3d
A video model can show a robot arm picking up a cup and still be no use for picking up the cup. The frames look
right one at a time. Between them, the gripper's fingers bend a little, the cup's rim slides a few pixels against
the table, and the depth an estimator reads off frame 12 does not agree with frame 13. A person watching does not
notice. A program that turns the video into joint angles does, because it has to commit to one 3D position per
frame.
[GEM-4D](https://arxiv.org/abs/2605.22882), from Kaichen Zhou, Yuzhen Chen and colleagues at Harvard, MIT and
Princeton (ECCV 2026), attacks that gap during training only. A second, small transformer is bolted onto the
video model's middle layer and made to predict what a frozen geometry model sees in the same clip. Its loss can
only fall if that middle layer encodes depth, camera motion and object motion. After training, the second
transformer is thrown away. The launch post by [@alex_kai2020](https://x.com/alex_kai2020/status/2120923983155470806)
puts it as "Train with 3D geometry. No extra video-generation overhead", and quotes seven simulated tasks going from
22.7% to 75.1% success with the same action extractor.
I read the paper (v4, 3 August 2026), the [project page](https://gem-4d.github.io/), the released code at
[`kaichen-z/GEM-4d`](https://github.com/kaichen-z/GEM-4d) and the index of the
[released checkpoint](https://huggingface.co/datasets/kaichen-z/Gem-4D). I did not train or run anything. The
training trick is clean and the code matches the paper's description of it. The headline numbers need more care
than the post gives them, and the part that turns video into robot actions is not in the release.
## A video model as a robot policy
The recipe GEM-4D builds on goes back to [UniPi](https://arxiv.org/abs/2302.00111): do not learn actions directly.
Learn to predict the future as video, then read the actions off that video. The paper writes it as two functions.
A world model $M$ takes the first frame $\mathbf{I}_0$ and an instruction $c$ and returns frames
$\{\mathbf{I}_t\}_{t=1}^{N}$. A policy extractor $P$ takes those frames and returns actions
$\{\mathbf{a}_t\}_{t=0}^{N-1}$.
The attraction is data. Robot demonstrations with actions are scarce and tied to one arm. Video of things being
manipulated is plentiful, and a pretrained video generator already knows a lot about how objects move. The
action-specific part, $P$, can be a fixed pipeline of off-the-shelf vision models with no training per task.
That differs from the world action models this site has covered recently.
[Praxis-1](/articles/praxis-1) and [FLUX 3 Action](/articles/flux-3-action) put an action head inside the video
model and denoise actions in the same pass, trained on robot data, and [Flex-π](/articles/flex-pi) denoises 3D
pointmaps and actions in one shared latent. Vision-language-action models skip the video and map pixels and
text straight to motor commands. GEM-4D keeps the two stages apart, so the
video has to be good enough for a separate program to measure. Whether it is depends on a property the training
loss of a video model never asks for.
## Why pixels cannot pin down geometry
The paper's Section 3.2 starts from where a scene point lands in the next frame. A point $\mathbf{X}_t$ seen at
pixel $\mathbf{p}_t$, at depth $\mathbf{D}$, with the camera moving by rotation $\mathbf{R}$ and translation
$\mathbf{T}$ and the point itself moving by scene flow $\Delta\mathbf{X}$, projects to
$$
\mathbf{p}_{t+1} \sim \mathbf{K}\Big[\mathbf{R}_{t\to t+1}\,\mathbf{D}(\mathbf{p}_t)\,\mathbf{K}^{-1}\mathbf{p}_t + \mathbf{T}_{t\to t+1} + \Delta\mathbf{X}_t\Big]
$$
where $\mathbf{K}$ is the camera's intrinsic matrix. Four unknowns, $(\mathbf{D}, \mathbf{R}, \mathbf{T}, \Delta\mathbf{X})$,
decide every correspondence between frames. The paper draws two conclusions from it.
The first is that a pixel loss cannot enforce correspondence. Many settings of the four unknowns produce the same
pixels: double the depth and double the motion and nothing on screen changes; pan the camera left while the
object stays still, and it looks like the object moved right. The paper's sentence is blunt: "A pixel loss can
reach zero while the underlying correspondences are entirely wrong." A video diffusion model trained on pixels or
VAE latents is graded on what is on screen, so nothing stops it from implying one depth in one frame and another
in the next.
The second is the reverse. A representation that does encode the four unknowns has no freedom left, because the
equation then fixes where every point goes.
The widget makes the first point concrete in one dimension. The left panel is fixed: five frames in which the
gripper moves the same number of pixels each step. The right panel is what those frames mean in 3D under
different choices of depth and camera motion, lifted the way an action extractor lifts them, with depth times
the ray through the pixel.
Scaling the depth stretches the reach, and shifting motion onto the camera shrinks how far the gripper itself
travels; the frames never change. The drift toggle is the failure that matters for a robot. Each frame is lifted
with its own depth, as happens when a generator's geometry is not consistent across frames, and a straight reach
becomes a zigzag. An extractor that follows it will drive the arm along the zigzag.
## Teaching the backbone through a branch it will lose
Earlier fixes for this changed what the video model outputs. [TesserAct](https://arxiv.org/abs/2504.20995)
generates RGB, depth and surface normals together; 3DFlowAction predicts 3D flow. GEM-4D's argument against that
route is that it needs geometric labels at scale and constrains the output space of a pretrained backbone.
GEM-4D leaves the output alone and works on the inside. Feed-forward geometry models in the
[DUSt3R and VGGT line](/articles/deja-view), and [Depth Anything 3](/articles/da3-streaming-reconstruction), are
trained to estimate per-frame depth and camera pose across a clip. Those are the dominant factors in the equation
above. The paper's claim is that their intermediate features therefore encode the correspondence structure,
including object motion, since a point whose depth changes in a way the camera motion does not explain must be
moving. Making the video model predict those features should push the same structure into it.
Both branches are flow matching. The video branch is a standard latent video DiT: noise $\mathbf{z}_1$ and a
clean latent $\mathbf{z}_0$, a velocity field trained against the analytic target,
$$
\mathcal{L}^{\text{vid}}_{\text{FM}} = \mathbb{E}\big[\lVert \mathbf{v}^{\text{vid}}_\theta(\mathbf{z}_t, t, c) - \mathbf{v}^*(\mathbf{z}_t, t)\rVert_2^2\big].
$$
The geometry branch runs the same objective on a different latent. A frozen geometry model $G$ turns the clip into
$\mathbf{g}_0$, which is noised to $\mathbf{g}_t$, half its tokens are masked, and a Geometry DiT predicts the
velocity, conditioned on $\mathbf{m}_t$, the video DiT's features at one middle block:
$$
\mathcal{L} = \mathcal{L}^{\text{vid}}_{\text{FM}} + \alpha\,\mathcal{L}^{\text{geo}}_{\text{FM}}.
$$
The design choice that makes this work is in what the Geometry DiT can see. It gets the masked, noised geometry
latent and $\mathbf{m}_t$, and nothing else from the scene: no pixels, no depth, no camera. To predict how the
geometry latent should be denoised, the information has to come through $\mathbf{m}_t$, and the gradient of the
geometry loss flows back into the video backbone through it. The coupling runs one way. The Geometry DiT reads the
video features and never writes back, so the video DiT runs alone at inference with the same weights layout and
the same cost as before.
This is close in spirit to [REPA](https://arxiv.org/abs/2410.06940), which aligns a diffusion model's hidden
states with a frozen encoder's features through a projection. The difference is that GEM-4D's target is a whole
generative problem rather than a cosine similarity: the branch has to denoise the teacher's features, not just
point in the same direction. Geometry Forcing, the closest baseline in the paper, is the REPA-style version with a
geometry teacher.
### What the code actually trains
The paper does not give $\alpha$, which block $\mathbf{m}_t$ comes from, or which geometry model is the teacher
for the main results. It lists PAGE-4D, Depth Anything 3, VGGT and the DUSt3R family as examples. The released
code fixes all of it, in `training/config/default.yaml` and `model/gem4d/`:
```yaml
geometry:
teacher: weights/da3 # DepthAnything-3 (local dir or depth-anything/DA3-LARGE-1.1)
feat_layer: 12 # DA3 ViT-L layer used as the geometry representation g0
depth: 4 # Geometry DiT blocks
mask_ratio: 0.5 # fraction of geometry tokens masked (mask mechanism M)
alpha: 0.03 # weight of L_geo
coupled_noise: true # geometry and video share one noise sample
```
A few details in the code are not in the paper and change how I read the method.
- **The teacher is DA3-Large, layer 12, through a frozen random projection.** `GeometryProjector` compresses the
1,024-channel features to 256 with a Xavier-initialised 1x1 convolution, box-averages every 4 frames (49 frames
become 13, matching the VAE's temporal stride), resizes to the latent grid, and projects to the latent's channel
count (16 on Wan-14B). All of it has `requires_grad_(False)`. The geometry latent is a fixed random view of DA3's
features, shaped like a video latent.
- **The noise is shared.** With `coupled_noise: true` the geometry latent is noised with the same sample as the
video latent, so both branches denoise the same random draw at the same timestep.
- **The Geometry DiT reads $\mathbf{m}_t$ only through cross-attention.** Each of its 4 blocks is self-attention
over geometry tokens, cross-attention to the video features, and an MLP, with zero-initialised gates so the
branch starts as the identity. On the Wan-14B config `hook_layer: 20` of 40 blocks; on Wan-5B it is block 15
of 30. The hook does not detach the features, so the gradient reaches the backbone.
- **The weight is small.** $\alpha$ is 0.03, ramped in linearly over the first 100 steps.
### What "no extra overhead" covers
At inference, the claim holds exactly. `inference/infer.py` loads only the video DiT. The released checkpoint's
index lists 1,303 tensors and a total of 32,789,757,568 bytes; in bf16 that is 16.39B parameters, and none of
the tensor names contain `geo`. The deployed model is a plain Wan2.1 image-to-video DiT.
Training is another matter. Counting the Geometry DiT from `heads/geo_dit.py` with the Wan-14B width of 5,120 and
40 heads, I get 1.73B parameters: four blocks at about 419M each, plus the timestep MLP and the in and out
projections. Added to the 16.39B backbone that is 18.1B, which matches the README's "18.1 B trainable parameters"
for full fine-tuning with the branch. So training carries about 10.6% more trainable weights, plus a DA3-Large
forward pass on every 49-frame clip. The README asks for eight GPUs with at least 141 GB each for the 14B run, or
optimizer state offloaded to CPU on eight 80 GB cards. On Wan-5B the branch is 0.62B by the same count. That is
the right trade for a model you train once and sample many times, and it is still a cost.
## From frames to a gripper pose
The second half of the paper is the extractor, the Adaptive Inverse Dynamic System. Its job is to turn a
generated clip into a 6-DoF end-effector trajectory and then joint commands. It uses no learned parameters of its
own and no per-task training; it chains existing models.
1. **3D grounding.** The geometry model supplies depth and intrinsics for the generated frames. Qwen3.5-VL and
SAM 2 segment the target object and the end effector from the instruction. FoundationPose aligns a CAD model
of the gripper to its point cloud to get the starting pose.
2. **Tracking with two triggers.** CoTracker3 follows keypoints sampled on the gripper mask. The system watches
the fraction of anchors still tracked, $s_t$, and its frame-to-frame change $\Delta s_t$. A slow decay below a
threshold means drift: re-sample keypoints from the last good mask. A sudden drop means a generation artifact:
ask the vision-language model to find the gripper again.
3. **Pose fallback.** FoundationPose gives a pose and a confidence per frame. Where the confidence is low, or the
pose jumps more than a threshold in translation or rotation, the frame is rejected; translation is taken from
the centroid of the gripper's depth pixels and rotation is slerped between the nearest good poses.
4. **Grasp and inverse kinematics.** GraspGen proposes grasps on the object's point cloud. The one closest, in a
weighted sum of translation and rotation error, to the trajectory's pose nearest the object is inserted as a
waypoint, the path is smoothed, and inverse kinematics produces the actions.
Read as engineering, this is a robust tracker written to survive bad video. The two-trigger rule and the
depth-centroid fallback exist because generated frames have exactly the artifacts the first half of the paper
tries to train out. Two things follow. "No extra overhead" is a statement about the generator; the full path from
image to motion runs a VLM, SAM 2, CoTracker3, FoundationPose with a CAD model of the gripper, GraspGen and a
geometry model on top of 30 sampling steps of a 14B DiT with classifier-free guidance. And the extractor is
open-loop: it reads a whole imagined clip and plans from it. The README is explicit that this module is "Not in
this repository", along with the VGGT and depth-target ablations.
## What the video numbers say
Table 1 scores generated clips on two held-out sets: 400 unseen Droid samples (real) and 780 unseen RLBench
samples (simulated), each generated 20 times. Appearance is FVD, SSIM and PSNR. Geometry is AbsRel, $\delta_1$
and $\delta_2$ of depth estimated from the generated frames by Depth Anything 3, a Chamfer distance between point
clouds, and $\delta^{vis}_{avg}$, the fraction of visible points tracked within 1 to 16 pixels.
| Droid (real) | FVD ↓ | SSIM ↑ | AbsRel ↓ | δ1 ↑ | Chamfer ↓ | Tracking ↑ |
|---|---|---|---|---|---|---|
| CogVideoX | 35.56 | 75.91 | 22.33 | 68.32 | 0.2670 | 66.22 |
| Wan 2.2-14B | 33.43 | 76.24 | 21.39 | 71.18 | 0.2349 | 68.18 |
| TesserAct | 33.28 | 75.66 | 22.07 | 66.80 | 0.2630 | 67.14 |
| Geometry-Forcing | 33.17 | 76.12 | 21.96 | 69.74 | 0.2443 | 67.97 |
| GEM-4D | 31.82 | 82.05 | 20.13 | 78.19 | 0.2001 | 71.23 |
GEM-4D leads every column on both sets. The largest margins are geometric: on RLBench, Chamfer falls from 0.1813
(TesserAct) and 0.1488 (Geometry-Forcing) to 0.0702, and $\delta_1$ on Droid rises from 71.18 (Wan) to 78.19. The paper says the
standard deviation across the 20 generations is 1 to 2%, which is smaller than most of these gaps.
Two caveats sit under the geometry columns. On Droid there is no recorded depth, so "ground truth" depth is
Depth Anything 3 run on the real frames, and the code's teacher is the same model. A generator trained to encode
DA3's features being scored by how DA3 reads its output is not circular in a way that invalidates the result,
but it favours this method by construction. RLBench, where depth is the simulator's, does not have that problem,
and the gaps there are as large. GEM-4D's RLBench Chamfer is less than half the best baseline's.
The second is the backbone. The paper does not name GEM-4D's own backbone in Table 1; the README says those rows
are "GEM-4D on a CogVideoX-5B backbone, not part of this release". So the cleanest comparison in the table is
GEM-4D against CogVideoX, same family, fine-tuned on the same training set without the branch, and the gap there
(AbsRel 22.33 to 20.13, Chamfer 0.2670 to 0.2001 on Droid) is the effect of the geometry loss plus whatever else
differed in the recipe.
The ablation in Table 3 is real-domain only. Predicting depth velocity instead of geometry features
(GEM-4D (Dep)) recovers part of the gain: $\delta_1$ 74.60 against 78.19. Swapping in VGGT as the teacher lands
below the fine-tuned Wan baseline on SSIM (75.89 against 76.24) and AbsRel (21.73 against 21.39), roughly where
plain CogVideoX sits, which the authors attribute to VGGT being trained on static scenes. That result is the most interesting one in the paper to me: the teacher matters, and a weak
teacher is not free.
## Where 22.7% to 75.1% comes from
The post's headline is not printed in the paper. It is the unweighted mean of seven RLBench columns in Table 2.
TesserAct's cells sum to 159, so 159 / 7 = 22.7%; GEM-4D's sum to 526, so 526 / 7 = 75.1%; the difference is
52.4 points. That arithmetic checks out. What the numbers measure needs reading carefully.
**The extractor is GEM-4D's own.** The paper says it plainly: "As the inverse dynamics module of
TesserAct is not open-source, we use our AIDS to process its generated videos." "Same action extractor" is true,
and it is the right choice when the alternative is no comparison. The paper does not say how the extractor's
thresholds were set, but it was built by the same authors alongside their own model's videos, and I would not
assume it is neutral. TesserAct's own pipeline might do
better or worse on its videos; nobody has measured it.
**One baseline, and it is not the same model minus the branch.** CogVideoX would have been the ablation that
isolates the geometry loss, and it has no RLBench row: "most videos generated by CogVideoX cannot be reliably
processed by our AIDS". That is itself evidence for the paper's thesis, since the extractor fails outright on a
plain fine-tune. But it leaves no success number for "same backbone, same data, no geometry branch". The ablation
table has no success column either.
**The training data is not stated to match.** CogVideoX and Wan are described as fine-tuned on the authors'
training set (ManiSkill3, RLBench, Bridge, RT-1). TesserAct is described as a CogVideoX model that "further
fine-tunes CogVideoX on robotics datasets", with no mention of retraining it on the same set.
**No trial counts or spread.** Table 2 gives one percentage per task and no number of episodes or variance.
Two TesserAct cells, Put Rubbish In Bin at 0 and Reach Target at 2, carry a lot of the mean; leave them out and
the gap is 42.4 points over five tasks. That is still large. It is also a sign of how much rides on whether
the extractor can lock onto a gripper at all.
The real-world number needs the most correction. The abstract says GEM-4D "improves real-world manipulation
success from 61% to 81%". The Metric paragraph of Section 4.2 says the Droid columns come from "a human study to
assess the quality of generated videos, focusing on task success, object and robot-arm deformation, and
instruction following", averaged over 15 participants. No robot executed those tasks. The three columns average
60.7 and 81.7, so "81" is rounded down. The project page calls it "Real-robot success", which is the wrong label
for a rating of videos.
There are real-robot runs: the paper's Figures 4 and 6 and the post's video show a UF arm executing trajectories
extracted from generated rollouts. They are qualitative, with no count of attempts.
## What was released, and what was not
The repository is more useful than most paper code. It trains GEM-4D on two backbones, Wan2.2-TI2V-5B and
Wan2.1-I2V-14B, with RGB-only baseline configs beside them (`wan14b_rgb.yaml`, `wan5b_rgb.yaml`), so the missing
ablation can at least be run by someone with the GPUs. It ships a 14B checkpoint under Apache-2.0 on Hugging Face,
a post-processing tool that estimates depth and scene flow from any generated clip, and CPU smoke tests with tiny
random models. The repository itself has no licence file.
It does not reproduce the paper. The checkpoint is a Wan-14B model, while Table 1's rows are a CogVideoX-5B model
that is not released. The release's training mix is RLBench, Bridge, RT-1 and Droid at 0.40, 0.25, 0.25 and 0.10;
the paper's training list does not include Droid and evaluates on "unseen" Droid samples. And without the action
extractor, nothing in the release turns a video into a robot motion, so the RLBench success table cannot be
checked from it. Inference alone needs about 48 GB of GPU memory at 240 by 416, 49 frames, per the README.
## The idea is worth more than the headline
I like the method. "Make the hidden state predict what a frozen expert sees, through a branch you will delete" is
a general move, and here it targets the right failure: video that is plausible frame by frame and incoherent as
geometry. It costs nothing at sampling time and about a tenth more parameters at training time, and on RLBench,
with simulator depth, the geometry metrics improve by more than the noise. The VGGT ablation adds a useful
warning that the choice of teacher can make things worse.
I do not buy "+52.4 pp" as a measure of what geometry supervision does for manipulation. It compares GEM-4D
against a different model trained on possibly different data, through an extractor its authors built for their
own videos, on seven tasks with no trial counts. The "61% to 81%" real-world claim is a rating of videos by 15
people. The experiment that would settle it is in the repo's configs already: train `wan14b_rgb` and
`wan14b_gem4d` on the same manifest, run both through one extractor, report episodes. Until that exists, I would
cite GEM-4D for Table 1 and the training trick, and treat the success numbers as encouraging.
For a related angle on the same problem, the [Cosmos 3](/articles/cosmos-world-model) and
[ABot-World-0](/articles/abot-world) pieces cover world models scored on generation, and
[SAIL](/articles/sail-robot-trajectories) is a policy that searches imagined trajectories in a simulator rather
than in video.
How I checked: I read the paper's v4 HTML, the project page and its video captions, the README, configs and model
code of `kaichen-z/GEM-4d` at commit `2f2b374`, and the released checkpoint's `config.json` and safetensors index.
The 16.39B, 1.73B and 0.62B parameter counts are mine, from the index's byte total and from the module shapes in
`geo_dit.py`. The RLBench and Droid means are my arithmetic on Table 2. I ran no model and no part of the
training code.