MetricTrail: a single-image depth prior turns DPVO into a long-video pose estimator

mdjsonmcpcopy↗ai

2026-11-09 · 18 min · 3d · slam · visual-odometry · depth · state-estimation · nvidia

Why read this

Notable· top 60%

How a metric-depth Huber factor stops DPVO's scale drift, read from NVIDIA's code, measured in the checkpoint and rebuilt as a CPU toy.

• Interactive explanations
• Original analysis
• A lasting reference

·3D & spatial·Needs a workstation GPU·Non-commercial licence·Research paper

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

1 of 3: API-only, gated or restrictive licence

Will I understand it?

3 of 3: Mechanism carried by interactives built from real code or data

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

2 of 3: A reference for a year or more

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 63 of 100, ranked 317 of 701 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Long-video pose estimation has spent the last year going the transformer way. LingBot-Map carries geometric context across frames in structured attention memory; HorizonStream couples local attention with a bounded recurrent state. Both need training on long sequences to learn to remember. MetricTrail, from NVIDIA with Shanghai Jiao Tong, Toronto and Stanford, goes the other way. Its first author's thread puts it plainly: "revisit good old DPVO, add MoGe-2 depth priors and DINOv3 features." The paper is MetricTrail: Revisiting the Power of Metric Depth and Local Optimization for Long-Sequence Streaming 3D Reconstruction, the code lives inside NVIDIA's ViPE repository under vipe/metrictrail/, and the trained matcher is on Hugging Face as nvidia/MetricTrail.

The claim worth taking apart is narrow and checkable: a sparse visual odometry with a fixed, small optimisation window stops drifting over thousands of frames once each patch's depth is tied to a single-image metric depth prediction. No bigger window, no long-sequence training, no correspondence network retraining for that part. This piece explains why that works from the residuals up, shows what the released code actually does, and reproduces the effect in a toy I could run on a CPU. I could not run the real pipeline: it needs CUDA kernels, and this box has no GPU.

expand

Why a small window drifts

DPVO (Teed, Lipson and Deng, NeurIPS 2023) is the estimator underneath. Every frame contributes a few dozen 3x3 patches. A patch $k$ has a fixed pixel centre $\mathbf{u}_k$ in its source frame and one free number, its inverse depth $\rho_k$. An edge connects a patch to a nearby target frame $j$, and the projection of the patch into that frame is

A recurrent network looks at feature correlations around $\mathbf{q}_{kj}$ and predicts where the patch really is, $\hat{\mathbf{q}}_{kj}$, plus a confidence $\mathbf{w}_{kj}$. Bundle adjustment then moves the poses and the inverse depths to agree with those targets:

The loop runs inside a bounded window. In the shipped metrictrail_local preset that is OPTIMIZATION_WINDOW: 10 poses and PATCH_LIFETIME: 13 frames, with 96 patches per frame. Older poses leave the optimiser and are frozen into the output trajectory.

Here is the problem. Every term in $E_{\text{DPVO}}$ is a pixel residual, and pixel residuals are blind to scale: multiply every translation and every depth by the same factor and nothing changes. Inside one window the scale is pinned only by the frozen poses at its old end. Each window inherits the scale of the last one, plus a little error, and over 4,541 frames of KITTI 00 those little errors compound into a random walk. DPVO makes it worse at the seam where new geometry enters: a new patch starts at the median inverse depth of the patches from the preceding three frames. That seed carries no information about where this particular patch is, and it copies forward whatever scale the estimator has already drifted to. The paper's own phrasing is that the median "carries forward the current monocular scale rather than correcting it."

The result, in the paper's Table 1: the official DPVO checkpoint with calibrated intrinsics averages 52.69 m of trajectory error across KITTI 00-10, and 113.11 m on sequence 00 alone. That is the number MetricTrail starts from.

Change one: metric depth that stays in the optimiser

Modern single-image depth models predict depth in metres. MoGe-2 does it with intrinsics estimated from the same image. MetricTrail uses that prediction twice.

1. Initialisation. A new patch starts at $\rho_k^{(0)} = 1 / \hat{D}_{s(k)}(\mathbf{u}_k)$, the predicted depth at its own pixel, in metres.
2. A persistent factor. The same observation stays in bundle adjustment as a Huber-weighted unary on log inverse depth:

The second is what does the work over long sequences. A seed is forgotten the moment BA moves the patch; a factor keeps pulling. Because every patch in the window now has a term that measures its depth in metres, the window's scale is anchored to the depth network's notion of a metre, not to the frozen poses behind it. The scale random walk loses its integrator. DPVO's own scale normalisation is switched off in metric runs, since it would wash that anchor out.

The code is short enough to read in full. In vipe/metrictrail/dpvo/ba.py, _metric_unary_terms() turns each patch's prior into one diagonal entry and one right-hand-side entry of the depth block of DPVO's Schur system, as an iteratively reweighted Huber:

# vipe/metrictrail/dpvo/ba.py, _metric_unary_terms (abridged)
residual = safe_prior.log() - safe_current.log()          # log(rho_hat) - log(rho)
robust = torch.where(abs_residual <= huber_delta,
                     torch.ones_like(abs_residual),
                     huber_delta / abs_residual.clamp_min(1e-6))
weight = prior_lambda * safe_confidence * robust
jacobian = safe_current.clamp_min(1e-4).reciprocal()      # d log(rho) / d rho
C = weight * jacobian.square()                            # added to the depth Hessian diagonal
rhs = weight * residual * jacobian                        # added to the depth gradient

The constants are in configs/slam/metrictrail_local.yaml and match the paper's appendix: METRIC_DEPTH_PRIOR_WEIGHT: 1.0 ($\lambda_m$), METRIC_DEPTH_PRIOR_HUBER: 0.3 ($\delta$), and METRIC_DEPTH_PRIOR_WARMUP: 12, so the factor is off for the first 12 accepted frames and the first regular BA update with it lands on frame 13. Depths outside 0.2-500 m count as invalid: such a patch is seeded with the frame's median and gets zero factor weight.

Two details matter. The residual is in log inverse depth, so the factor measures ratios: a 10% error costs the same at 3 m and at 80 m, which is what you want from a depth network whose errors scale with depth. And $\delta = 0.3$ in log units is a depth ratio of about 1.35. Inside that band the prior pulls in proportion to the disagreement; past it, the pull is capped. A wrong prediction on a window or a sky patch cannot drag the geometry further than a merely imprecise one, and the reprojection terms from the other frames get the final say. The widget evaluates the exact formula above.

Change two: DINOv3 features in the matcher, through 41,536 parameters

The prior fixes where the matcher starts looking. It does not help the matcher choose between two similar-looking candidates in a row of identical windows. For that MetricTrail adds features from a frozen DINOv3 ViT-S/16. A 1x1 convolution projects the 384-dimensional DINO tokens to 64 channels, split into two 32-channel branches: one is projected to 128 channels and added to DPVO's matching feature map, the other is sampled at each patch centre, projected to 384 channels and added to the recurrent context. The final projections are zero-initialised, so the fused model starts out exactly reproducing the pretrained DPVO predictions.

I downloaded the released checkpoint (dino_dpvo.pth, 13,739,570 bytes) and loaded it with torch.load(..., weights_only=True) in PyTorch on CPU, which reads tensors and runs no pickled code. Measured:

So 41,536 of 3,425,860 parameters, 1.2%, are new; the rest is DPVO's architecture, fine-tuned. The DINOv3 backbone itself is not in the file: the checkpoint's metadata lists patchify.dino.backbone. under omitted_external_state_prefixes and records the SHA-256 of the dinov3_vits16_pretrain_lvd1689m weights it expects, which you fetch separately under Meta's DINOv3 licence. The zero-initialised projections did move in training: match_fusion.weight ended at a Frobenius norm of 0.52 and context_fusion.weight at 3.40, so the context branch is the one the network learned to lean on more.

Training is short and local. Every sample is a 15-frame clip from TartanAir, TartanGround, CarlaOcc, LightwheelOcc or OmniWorld-Game; 15,000 steps on 16 A100s, 240,000 clips, about seven hours. MoGe-2 never appears in training. The metric prior enters only at inference, which is why the paper can bolt it onto the original DPVO weights and still get most of the gain.

What the depth prior does, rebuilt on a CPU

The real pipeline needs three CUDA extensions (cuda_corr, cuda_ba, lietorch_backends), the 1.3 GB MoGe-2 ViT-L checkpoint and the gated DINOv3 weights, and this machine has no GPU or nvcc. So I rebuilt the one idea I wanted to test, in about 170 lines of numpy: a DPVO-shaped estimator in the plane, small enough to run thousands of frames on one core.

The toy keeps DPVO's shape and drops everything learned. A camera drives 3,000 frames along a winding path of about 3 km at roughly 1 m per frame, like a car in KITTI. Every frame gets 16 patches: a pixel on a 1-D image (focal 300 px) and one inverse depth, with true depths drawn between 4 and 80 m. The "matcher" is the true projection plus 1 px of Gaussian noise. The window is 12 frames: patches from all 12, edges to every other frame in it, the 10 newest poses free and the 2 oldest frozen, 4 Gauss-Newton steps per new frame. The "single-image depth network" is the true depth times a log-normal error: 10% per patch, 5% shared by the whole frame, and 5% of patches off by a factor of 1.6 to 4.5. Three rules for a new patch's depth:

• median: DPVO's rule, the median inverse depth of the previous three frames' patches.
• init: the noisy metric prediction as the seed, then nothing.
• prior: the same seed, plus the factor exactly as _metric_unary_terms() builds it ($\lambda = 1$, $\delta = 0.3$, off for the first 12 frames).

The factor is ported by reading the code: I meant to call the repository's own _metric_unary_terms() on the same inputs as a cross-check, and the sandbox refused to execute the downloaded code, so that comparison is not done. Same trajectory, same scene and same measurement noise for all three. ATE is the RMSE after one Sim(2) alignment of the prefix, so a constant scale error is forgiven and only drift counts. Measured, mean of three seeds, on one CPU core at about 34 frames per second:

Three things fall out, and they line up with the paper's argument.

The median seed drifts without bound. By frame 3,000 the estimate has shrunk to about a quarter of true size (the alignment needs a scale of 3.80), and the error is still growing. That is the scale random walk, with the median rule copying each step forward.

A metric seed alone only delays it. Seeding from the depth network keeps the first few hundred frames honest, because every new patch arrives at roughly the right depth. But the BA is free to move patches once they are in, the frozen poses still pass scale from window to window, and by 3,000 frames the error is 122 m on average, with one seed at 198 m. This is the paper's point that "initialization alone cannot prevent later BA updates from drifting away from the predicted metric scale", reproduced.

Keeping it as a factor stops the drift. With the factor the error at 3,000 frames is 6.52 m on average (4.23 to 8.14 m across seeds) and the scale stays within 4% of true the whole way. What is left grows slowly, which is heading drift: the depth prior pins scale, not rotation.

Two variations, one seed or three as marked, also measured:

• Huber versus plain least squares. With $\delta$ set to infinity, so every prior pulls in proportion however wrong it is, the 3,000-frame error rises from 6.52 m to 9.16 m (mean of three seeds: 5.70, 10.72, 11.07 m). The cap is doing its job on the 5% of gross outliers.
• A biased depth network. If every prediction is 10% too far, the factor still stops the drift (4.58 m, seed 1) but the trajectory comes out about 10% too large: the alignment needs a scale of 0.94 against 1.04 unbiased. The metric scale is exactly as good as the depth network's. Init alone with the same bias: 152.05 m.

Tripling the per-patch noise to 30% costs the factor little (7.06 m, seed 1) and does nothing for init alone (167.74 m). What the toy does not have: a learned matcher, outliers in the matches, keyframing, rotation drift of any realistic size, dynamic objects. It tests one mechanism, and that mechanism behaves the way the paper says it does.

The paper's numbers

All of this section is reported: the paper's tables, which I checked for internal consistency but did not reproduce.

expand

The ablation is where the claim lives. Table 5, calibrated intrinsics, no loop closure:

The metric prior is worth more than everything else combined, on every dataset. The other two ingredients are not monotone: multi-dataset training makes VBR worse, and DINOv3 earns it back. Table 6 repeats the ladder with five other depth providers in place of MoGe-2, which is the honest test of "the depth model is swappable":

The swap mostly holds. On KITTI and VBR every provider takes DPVO far below its no-prior error. On Oxford Spires, where DPVO alone is already at 2.59 m, two providers make it worse (Pi3X at 5.57 m, UniDepthV2 at 3.89 m), which is what a confident, wrong prior does to a working estimator. DA3Metric and Metric3Dv2 beat MoGe-2 in this calibrated setting; the paper keeps MoGe-2 as the default because it also estimates intrinsics from RGB alone, and the other two need a focal length to produce metric depth at all. If you have calibrated cameras, the paper's own table says to try DA3Metric first.

Against the rest of the field, under the RGB-only protocol (intrinsics estimated from the video), Table 1 and Table 2:

The thread's "cuts ViPE's average trajectory error by 51% on KITTI and 75% on Oxford Spires" checks out against these rows: 28.68 to 14.05 m is 51.0%, and 7.06 to 1.77 m is 74.9%. With calibrated stereo on KITTI the error falls to 3.66 m, and 3.00 m with loop closure. The loop closure is simple: SALAD retrieves near-duplicate frames, and each pair adds a weak identity relative-pose edge to an SE(3) pose graph, with no geometric verification of the pair.

expand

Three things the headline rows do not say.

VBR depends on an offline focal refinement. The 16.36 m uses a "revisit-based focal refinement" from the appendix: it takes the base trajectory, finds temporally distant revisit pairs, matches them with RoMa, fits fundamental matrices with MAGSAC and re-estimates the focal length. That step reads the whole trajectory, so it is not streaming, and it averages 7.1 minutes per routed sequence. The paper says it "was motivated by our analysis of VBR", where the base calibration overestimated focal length by 7-9% on six of seven sequences. Without it, Table 16 reports 33.51 m on VBR, which is worse than LoGeR* (25.24 m) and HorizonStream (25.30 m). The paper discloses all of this, and the thread's percentages are for KITTI and Oxford Spires only.

Dynamic scenes are not a win. On TUM-Dynamic dense reconstruction, MetricTrail's 0.13 m Chamfer distance trails TTT3R's 0.06 m and LingBot-Map's 0.08 m. The paper says so itself: "the gains on static indoor and large outdoor scenes do not extend uniformly to scenes with dynamic content."

The runtime number is calibrated. 12.70 FPS on KITTI-02 with 3.21 GB peak GPU memory and about 0.7 s to first output, on an A100, assumes supplied intrinsics. Estimating them from RGB adds about 175 s of start-up and raises peak memory to 6.81 GB; throughput afterwards is the same 12.70 FPS. For comparison the paper measures LingBot-Map at 11.37 FPS and HorizonStream at 9.92 FPS on the same sequence.

What you can run, and under which licence

The licensing is in three layers, and the strictest one governs.

• Code: NVIDIA-authored source in nv-tlabs/vipe is Apache-2.0. The vendored DPVO under vipe/metrictrail/dpvo/ keeps its MIT licence, LieTorch is BSD-3-Clause, and the repository README notes incorporated Unik3D code under CC BY-NC-SA 4.0.
• Weights: the model card says "Use of the model weights is governed by the NVIDIA License, which limits use to noncommercial research or evaluation", linking NVIDIA's One-Way Non-Commercial licence. The card's metadata field reads license_name: nvidia-open-model-license, a different NVIDIA licence that does allow commercial use, but its link and its prose both point at the non-commercial one. Read it as non-commercial.
• Dependencies: DINOv3 under Meta's DINOv3 licence (gated download), MoGe-2 (Ruicheng/moge-2-vitl, 1,305,030,700 bytes) under MIT, and SALAD for loop closure under its own terms.

To run it you need an NVIDIA GPU, the ViPE install, then python -m vipe.metrictrail.build to compile the DPVO kernels, then python run.py pipeline=metrictrail streams.base_path=<video>. The evaluation scripts live on a separate metrictrail_eval branch. I did not run any of it.

Where this leaves long-video pose

The interesting result is not that MetricTrail beats LingBot-Map. It is how little it took. The trajectory-level problem everyone has been throwing memory at is, for a classical sparse estimator, mostly one missing term: an absolute reference for scale at every patch, which a single-image depth network now supplies for free. The window stays at 10 poses. The matcher never sees a sequence longer than 15 frames. What changed is that the window can no longer lose track of what a metre is.

The costs are real. You inherit the depth network's biases, and the provider table shows that a confident wrong prior hurts a working estimator. You inherit its calibration needs, which is where VBR's offline focal fix comes from. And dynamic content still breaks the static-scene assumption that both DPVO and the prior make. For the transformer side of this same problem, see R³ and AMB3R-SLAM on a frozen Depth Anything 3 and the second 3D reconstruction roundup; for what a metric depth model needs from its wrapper, Depth Anything 3 in ROS 2; and for odometry with a sensor that measures metres directly, FAST-LIO2 from scratch.