HiP3R: one scene from thousands of frames, and what each of its two graphs buys

mdjsonmcpcopy↗ai

2026-10-27 · 24 min · 3d · slam · pose-graph · vggt · feed-forward-reconstruction · long-video

Why read this

Niche· last 15%

Rebuilds the two-graph idea on a synthetic loop with known truth: progressive loops cut ATE 5.68 to 1.58 m, shared cameras mostly fix RPE; false loops hurt.

• Original analysis
• Explained from first principles

·3D & spatial·Runs on a laptop CPU·Research paper

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

1 of 3: Spot-checks a few numbers

Can I run it?

1 of 3: API-only, gated or restrictive licence

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

1 of 3: General advice

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 45 of 100, ranked 621 of 701 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Chen Change Loy posted HiP3R, a training-free way to keep one coherent scene across thousands of video frames. The paper is arXiv 2610.31635, by Zunnan Xu, Shangchen Zhou, Yihang Luo and Chen Change Loy (S-Lab, NTU), with a project page. The post names a code repository, HiP3Rs/HiP3R. My first pass could not open it, so the first half of this article rests on the paper's HTML and all of its tables, plus a small stand-in for the method that I wrote and ran. In a second pass the repository turned out to be cloneable (a plain git clone over the sandbox proxy works; it was the GitHub API tool that was refused), so I read its pose-graph code, compared it with my stand-in, and rebuilt three pieces of my stand-in to match what I read. The new material is in the two sections after "A harder setting, and a bad loop".

Every claim below is labelled. Reported is a number from the paper that I did not re-run. Read is something I saw in the authors' source and did not execute. Measured is a number from my own script, on a synthetic backend I built, not on video and not with the authors' pipeline. Reasoned is my arithmetic or inference on the other three.

The problem: local maps in different coordinates

Feed-forward geometry models such as VGGT read a short window of frames and output camera poses and pointmaps in one go. Attention is quadratic in the window, so a video of thousands of frames has to be cut into overlapping chunks. HiP3R's setting is a chunk of 60 frames with an overlap of 30 for VGGT-Omega, and 120 with 32 for LingBot-MAP (reported).

The chunks do not agree with each other. Each is reconstructed in its own coordinate frame, at its own scale, and the frames in an overlap get two predictions, one from each chunk. The paper names three things that go wrong:

1. Drift along the chain. Align chunk 2 to chunk 1, chunk 3 to chunk 2, and so on, and scale and orientation errors accumulate.
2. Loops that do not close. When the camera comes back to a place it has been, the chain disagrees with itself. Adding every long loop constraint at once to an unaligned chain makes the optimisation non-convex and can diverge.
3. Two cameras for one frame. A rigid similarity per chunk preserves the chunk's inside up to scale. If the inside is bent, two chunks disagree about the same frame no matter how well each is placed.

HiP3R answers with two graphs, solved one after the other.

expand

Stage I: a graph of chunks

Each chunk k is a node with a similarity transform A_k = (s_k, R_k, t_k) that maps its local coordinates to the world: x_world = s_k R_k x_local + t_k. Seven numbers a node. Chunk 1 is fixed as the world frame and scale reference.

Edges are measured relative transforms. Overlap edges join consecutive chunks. The two chunks predict the same pixels in the shared frames, which gives paired 3D points; the paper fits the relative similarity with confidence-weighted, Huber-reweighted Umeyama alignment on a bounded number of points, not on every pixel. Loop edges join chunks that revisit a place: visual place recognition proposes frame pairs, the temporal neighbourhoods of both ends go into one joint backend call, and the two endpoints are registered back to their original chunks (reported).

The residual of an edge is seven numbers, rotation (3), translation (3) and log-scale (1), under a Huber loss, with a bounded weight per loop based on its registration residual. It is solved with trust-region reflective least squares over a sparse Jacobian.

The part I found most useful to know is the schedule. Adjacent edges stay on throughout. Loops come in by increasing temporal span, each stage starting from the last solution. The reason given is the one in the problem list: far loops on an unaligned chain are a bad start for a non-convex solve.

Stage II: one camera per frame

Fix the chunk layout. For each frame t there is one camera F_t in SE(3), rigid, no scale. Each chunk that saw the frame contributes two kinds of factor:

• a placement prior: the chunk's own prediction, moved to the world by its stage-I transform, pulls on F_t;
• a motion edge between consecutive frames of a chunk: the relative pose the chunk predicted should hold between the shared cameras.

Priors keep the trajectory on the recovered layout; motion edges keep it smooth across chunk seams. For long sequences only a keyframe subgraph is optimised and the corrections are interpolated to the rest. The pointmaps are then moved with the cameras: placed by A_k, then corrected by F_t Q^-1, so depth and chunk scale are kept.

That is the whole method. Everything is on top of frozen predictions: no fine-tuning, and it runs behind a chunked backend or a streaming one.

What the paper reports

The headline, macro-averaged over seven datasets and 61 sequences with VGGT-Omega as the backend (reported, Table 1): ATE 18.555 to 4.906 m, a 74% cut; rotation RPE 1.820 to 0.590 degrees; translation RPE 0.9878 to 0.0622 m, a 94% cut. With LingBot-MAP, ATE 8.288 to 4.017 m. On the compact indoor sets there is little to fix: native VGGT-Omega already has 0.037 m on 7-Scenes and 0.019 m on Neural RGB-D.

The ablation I find most informative is the stage-wise one, Table 6, on identical VGGT-Omega predictions. Drag through it:

Read it as the paper's own split of labour (reported). The chunk layout does nearly all of the ATE work: KITTI goes from 77.42 to 9.75 m, Oxford from 9.861 to 1.599 m, VBR from 41.91 to 22.44 m. The shared camera then moves ATE by 0.2% on KITTI, 0.0% on VBR and 1% on Oxford. What it does change is the relative error: rotation RPE down 24% on KITTI, 30% on Oxford, 5% on VBR, and translation RPE down 29%, 40% and 22%. That is the job the paper assigns it, resolving cross-window conflicts, and the table shows it doing that and little else.

Two more tables matter for how to read the headline.

• Against VGGT-Long (Table 2, same VGGT-Omega backend): with loops, VGGT-Long gets KITTI to 10.19 m and VBR to 41.27 m; HiP3R gets 9.730 and 22.44 m. On KITTI that is 0.46 m of difference, and on VBR it is nearly half. The part of the headline that comes from simply having loop closure at all is the 110.3 to 10.19 m step, which VGGT-Long's own loop version already takes.
• The shared-camera design (Table 5): Layout Only, Single, Unary, Motion and Shared. Motion-only (no placement priors) gets slightly lower relative errors but its ATE jumps to 11.678 m on VGGT-Omega and 6.501 m on LingBot-MAP, against about 4.9 and 4.0 for the others. Priors anchor the layout, motion edges smooth it, and each is bad alone.

The cost side (Table 4, extended sequences): bounded point sampling cuts graph build and solve from 12,232.18 to 2,004.20 s with ATE 21.7751 to 21.7569 m, and the complete solver reaches 17.4708 m in 753.43 s. The paper's own conclusion lists what is missing: online updates, and robustness to unreliable loop closures.

expand

What I built: the same two graphs on a backend I control

Real numbers need real video, a GPU and the weights. I have none of those here, so I did the next most useful thing: wrote the two-graph idea from the paper's section 3 in about 300 lines of NumPy and SciPy, and ran it on a simulated chunked backend where the true trajectory is known. This is our simplified re-implementation, not the authors' pipeline. The file is chunkgraph.py; one seed takes about two minutes on one CPU thread.

The world. 800 frames, 1 m apart, two laps of a closed loop about 64 m in radius (so the second lap lies on the first and every chunk in lap one has a revisit in lap two). Chunks of 40 frames with an overlap of 20: 39 chunks. I used a shorter chunk than the paper's 60 and 30 to keep the graph small.

The backend. For each chunk I draw an arbitrary local frame (a random rotation, a random translation, and a random scale about exp(N(0, 0.4))). Predictions are the ground truth moved into that frame, plus errors that a rigid transform cannot undo:

• a rotation random walk inside the chunk, 0.12 degrees a frame;
• per-frame rotation noise of 0.08 degrees and translation noise of 0.04 m;
• a scale that drifts linearly across the chunk, with a slope drawn from N(0, 0.03);
• twelve scene points per frame, with 1% depth noise, so each frame's pointmap is tied to its own (wrong) camera.

All numbers are my choices. The point is that each chunk is bent inside, which is the situation the paper argues for.

Stage I. Overlap edges come from a Huber-reweighted Umeyama fit of up to 200 shared points. The chain is the initialisation. Loop edges: for every chunk pair at least 150 frames apart whose middle frames are within 8 m, one per chunk (19 loops), I give the graph the relative pose between the two revisit frames (truth plus 0.3 degrees and 0.5 m of noise) and solve for the similarity that makes the two chunks' own predicted cameras obey it, which is the role of the paper's joint-window registration. That is generous to loops: place recognition is assumed to be right. The solve is scipy.optimize.least_squares with a Huber loss, a sparse Jacobian pattern and the first chunk fixed. Progressive loops: three stages, shortest span first, a third each, adjacent edges always on.

Stage II. One SE(3) variable per frame (800 frames, 4,800 parameters), unary priors from every chunk that saw the frame, and relative-motion edges inside each chunk, the same factors as the paper's equations 4 to 6. No keyframe subgraph.

Metrics. ATE after a Sim(3) fit to ground truth, rotation RPE in degrees and translation RPE in metres at a one-frame gap, as in the paper. Six seeds. Each number below is the mean, with the range over seeds in brackets (measured).

Here is one of the six runs, drawn top-down. Both laps trace the same curve, so a trajectory that closes its loop shows one curve and one that does not shows two:

What the numbers say

• Loops are where the ATE goes. The chain gives 5.68 m on a 400 m loop; loops cut that to 1.58 m, 72%. That is the paper's pattern (stage I does almost all the ATE) reproduced in miniature, which is a check on my implementation rather than on theirs.
• Progressive loops beat all-at-once here, 1.58 against 2.63 m, and in every one of the six seeds the progressive run was at least as good as the all-at-once run. The paper's Table 4 reports its progressive scheduling mainly saving time on its data, with ATE 21.7770 to 21.5241 m, so the size of the accuracy effect differs between the two settings. I cannot say which parts of mine are due to the simulation.
• Loops hurt the relative error before stage II repairs it. Translation RPE goes 0.137 (chain) to 0.172 (all loops): the loops drag chunks into place and open small steps at the seams. The camera graph brings it to 0.108, below the chain, and rotation RPE falls from 0.31 to 0.21 degrees. Stage II leaves ATE essentially where stage I put it: 1.58 to 1.43 m. The same division of labour as Table 6, again in miniature.
• Shared against the one-sided variants. Shared has the best relative errors on translation and rotation by small margins and ties priors-only on ATE. Motion-only did not blow up. The paper's Table 5 has motion-only ATE going from 4.9 to 11.678 m, and mine goes from 1.58 to 1.54. I did not reproduce that effect. A likely reason (reasoned) is that my solver starts at the layout and is capped at 15 function evaluations, so a gauge-free problem has little time to wander; I did not test that.

One overlap, up close

Why does one camera per frame help at all? Because the two chunks disagree about the same frame. The widget plots one overlap from the run above: the distance from ground truth of each chunk's placed prediction, and of the shared camera, over the 20 shared frames.

In that overlap the two placed predictions are 0.45 m apart on average (measured, one overlap of one run), and their errors against ground truth average 2.59 and 1.86 m. The shared camera sits between them at 2.22 m: it does not find the better chunk, it removes the disagreement. That is what one pose per frame can promise. If both chunks lean the same way, the shared camera leans with them, and only loops can say otherwise. It is the reason ATE barely moves in the second stage while RPE does.

A harder setting, and a bad loop

Two more runs of the same script, same measuring (measured, means over seeds).

Twice the in-chunk drift (--drift 2.5: rotation walk and scale slope times 2.5, six seeds). The chain is now 13.77 m [8.30, 21.65]. All loops at once: 6.99 m. Progressive: 6.17 m. Shared cameras: 6.00 m [2.07, 8.18]. The ordering is the same. Progressive was at least as good as all-at-once in all six seeds. The relative errors tell a less comfortable story than in the base run: rotation RPE goes 0.646 to 0.438 degrees with the camera graph, but translation RPE ends at 0.263 m, against 0.248 m for the plain chain. The loops opened seam steps that one camera per frame closed only to the chain's level.

Wrong loops. Each candidate loop becomes a false match with probability 0.15, with a 20 degree and 20 m error in its relative pose, which is what a bad place-recognition hit would hand the graph. Four seeds, 2 to 4 false loops of 19 each. Chain 5.54 m, progressive loops 6.96 m, shared cameras 6.82 m. Two seeds are better than the chain (4.00 to 2.68 m and 5.37 to 3.38 m), two are much worse (6.49 to 9.39 m and 6.30 to 11.85 m). My robust loss, Huber with a scale of 3 normalised units, did not contain a 20 m error. In my setup, three bad loops outweigh sixteen good ones. The paper gives each loop a bounded weight from its registration residuals and names unreliable loops as future work; I did not implement the weights, so this run shows the exposure and nothing about how well their weights would hold. The second pass below implements a version of the weights and repeats the experiment with a controlled false-positive rate.

What the authors' code does differently (read, not run)

The repository is HiP3Rs/HiP3R (Apache-2.0), commit a46f185. About 46,700 lines of Python under src/, most of it the vendored backbones; the method is src/long_reconstruction/, of which hierarchical_pose_graph.py alone is 15,945 lines. The demo (demo.py) builds its configuration with method_config("keyframe", ...), chunk 60 / overlap 30 for VGGT-Omega and 120 / 32 for LingBot-MAP, matching the sizes in the paper. Everything in this section is read: I tried to import the package and drive its edge estimator and solver with my synthetic chunks, and this sandbox's permission layer refused to execute the downloaded code. I did not work around that, so no function of theirs ran.

Where it differs from chunkgraph.py:

Two more observations from the source. First, adaptive_loop.py holds a scene-level gate: it scores a no-loop and a loop solution on the measurements they share, against frozen thresholds, and decides whether to keep the loops. I grepped the repository and nothing else imports it, including the demo, so I cannot tell whether the paper's numbers used it. Second, the preset the demo uses solves the chunk graph first and then freezes it, which is the variant my stand-in has always been; whether the paper's tables are from frozen, keyframe or joint is not something I could settle from the code.

Three pieces rebuilt after reading the code

The extension is chunkgraph_ext.py, which imports the original script unchanged apart from one added field (the true world points of each frame). It is again our re-implementation, not their code, written from the source I read. Same world, same six seeds (0 to 5), same simulated chunks: the chain-only ATE comes out at 5.68 m [3.13, 8.76], identical to the first table, so only the new pieces differ.

Loop edges the way the authors build them. For each revisit, a simulated joint model returns the true points of the two 20-frame windows in a frame of its own with 0.2 m of point noise (my choice); I register that to each chunk's own points (12 per frame) with the same Huber fit as the overlaps, and compose the two fits. The residual becomes the weight. Perfect retrieval, the same 19 loops, progressive schedule (measured, mean [min, max] over six seeds):

This is lower than the 1.58 m of the first version, but that is mostly a property of my noise settings: a registered window pair is more accurate than a relative pose I perturbed by 0.5 m and 0.3 degrees. It is not evidence about their pipeline. The residual weights came out at 0.17 to 0.39 (mean 0.325) for true loops, because my window residuals are about 11% of the cloud radius; so every loop is weighed at about a third of an adjacent edge, and that moved ATE by 0.03 m.

Keyframes. Every 5th frame, chunk ends, at least 3 per overlap: 199 of the 800 frames. The same priors and motion edges are solved on those only, and the correction at the other frames is interpolated (slerp for rotation, linear for translation). ATE is no worse (0.87 against 0.90 m, and lower in all six seeds, by 0.02 to 0.04 m); relative rotation error is 26% worse (0.266 against 0.211 degrees) and translation 15% worse (0.108 against 0.094 m). The solve takes about 1.0 s against 3.4 s on one thread (15 evaluations, measured). Reasoned: the in-between frames inherit a smooth correction, so keyframes keep the layout and give up the frame-to-frame cleanup, which is what RPE measures. At 800 frames the saving is a few seconds; the authors' threshold of 2,000 frames is where I would expect it to matter, and I did not test sizes that large.

A place-recognition model with a dial

The retrieval step is now a model, not an oracle. Each true revisit (19 of them) is returned with probability 0.85; then false positives are added at a set number per returned true loop, drawn from chunk pairs at least 25 m and 150 frames apart. The joint model has to say something about a false pair, and I do not know what a real one says, so I tried two stories (both mine). Coherent: it believes the places overlap and lays window B over window A, with a 15 degree yaw error and 2 m of slack; the points stay rigid, so both endpoint fits are good. Incoherent: the same, plus 1.5 m of jitter on B's points, as if the model could not make the views agree. Four conditions, then weights off and on. ATE in metres over six seeds, mean [min, max] (measured):

The chain with no loops is 5.68 m. With the weights off, loops with false positives beat the chain in one seed of 24 across the four conditions (0.25 coherent, seed 2); at precision 0.81 the mean is already worse than having no loops. Two or three false loops in twenty are enough.

What the weights do depends entirely on the story. Under the coherent one, false loops get weights of 0.26 to 0.38, indistinguishable from the true ones (0.17 to 0.39), because a rigid, wrong overlay registers just as well as a right one. The weighted rows still improve, and the control below shows why. Under the incoherent one, false loops get 0.07 to 0.10, well separated, and the solver recovers: 3.16 m at precision 0.81 beats the chain in five of six seeds, though 7.18 m at precision 0.66 beats it in two.

A control for the coherent rows. If the weights cannot tell true from false there, why do they help at all? Because every loop gets weighed at about a third of an adjacent edge, and less pull from a bad set of loops is less damage. To check, a separate run (new random loops, six seeds, 0.5 false per true, coherent) solved the same loops three ways: weight 1, one uniform weight equal to that run's mean residual weight (0.31 to 0.34), and the residual-based weights. ATE: 28.11 m [12.53, 38.65], 16.51 m [6.42, 25.88] and 16.85 m [6.32, 26.26]. The uniform weight does as well as the residual weights (it is lower in three seeds of six), and both beat the chain in one seed of six. So in the coherent case the gain from the weights is a gain from down-weighting, and the discrimination only shows in the incoherent case, which is the case I made up.

Reasoned, from both tables: a residual weight can only catch a false loop that the joint model itself finds inconsistent. A false loop that the model resolves into a clean, wrong geometry passes it. That is why a gate on the whole solution, such as the one in adaptive_loop.py, is a separate defence rather than a duplicate, and why I would want the authors' own false-positive numbers (the paper's conclusion lists unreliable loops as open) before trusting a loop-heavy setting on a scene with repeated structure.

What I did not run

• Nothing from the authors' code was executed. I read it (commit a46f185) and compared it with my stand-in, but the sandbox refused to run the downloaded code, so no function of theirs ran on my synthetic chunks, and I did not run HiP3R, VGGT-Omega or LingBot-MAP or reproduce any number from the paper. Every measured table is from my own re-implementation, on a simulation with errors I chose. The obvious next step is to drive their estimate_loop_edge and optimize_hierarchical_pose_graph with these same chunks, which would test my reading of the code directly.
• Place recognition is simulated. The retrieval model has a recall and a false-positive count I set, and the joint model's behaviour on a false pair is one of two stories of mine. No SALAD descriptor, no image, no real aliasing. The paper's own conclusion names robustness to unreliable loops as open work, and these runs show the exposure, not the rate at which a real retriever errs.
• Keyframes were tested at 800 frames, below the authors' 2,000-frame threshold, and there is still no point-sampling bound or solver restarts. The paper's timings (Table 4) come from sequences of tens of thousands of frames; one seed of mine takes about 20 minutes for the whole false-positive sweep and two minutes for the first table.
• The joint preset and the 8-stage schedule are not implemented. I stayed with the frozen variant and three stages.
• The error model is mine. A random walk, a linear scale drift and iid noise. Real feed-forward errors are structured by the scene, and that is what the paper's benchmarks have and mine do not.

What to take from it

1. Treat chunk placement and per-frame consistency as two problems. The reported tables and my own run agree on this one: the Sim(3) graph decides where things are, and the shared-camera graph decides how smoothly the camera moves through them. If you only need a map, the first stage is the one to build: on KITTI it is the 77.42 to 9.75 m step.
2. Order your loop closures. Short spans first costs nothing and, in my simulation, was never worse than all at once. The paper's own justification is the non-convexity of far loops on an unaligned chain.
3. Keep placement priors in the camera graph. The paper's Table 5 shows motion-only ATE going to 11.678 m. My version did not reproduce that, so I would not drop the priors on the strength of my run.
4. Plan for bad loops. The method's weak point, by the authors' own account, is a loop that should not exist. In my simulation precision of 0.81 was enough to lose to having no loops, and a residual weight helped only where the false loop was also inconsistent in its own registration. Check the retriever's precision on your footage before trusting loop closure.

The code for my experiments is chunkgraph.py and chunkgraph_ext.py; the paper's own repository, HiP3Rs/HiP3R, is the place for anything you want to trust.