Iris-Z: the depth map is just another image in the sequence

Speridlabs' Iris-Z is a text-to-image model that also draws depth. Prompt in, picture and depth map out. Photo in, depth out. Depth map in, picture out. One set of weights does all of it, with no depth head and no second network. The announcement says it is "the first general image model to combine appearance and geometry", and that the 32B version reaches 2.92 mean AbsRel against 3.12 for MoGe-3 on Marigold's benchmarks while beating MoGe-3 on six of seven of MoGe-3's own datasets.

MoGe-3 is the current task-specific leader in monocular geometry: a ViT built for this one job. A general image generator passing it would be a real result. So I read the technical report (a 14-page PDF linked from the project page), the SDK at commit 6c8925f, the Space's source, and the weights, whose safetensors headers I range-requested without downloading a byte of tensor data.

expand

Why a single photo only gives you relative depth

A single image does not fix scale. A doll's house photographed from 30 cm and a real house photographed from 30 m can produce the same pixels. Without the camera's focal length the shift is not fixed either: the network can see that the sofa is in front of the wall, and roughly by how much in proportion, but not where zero is.

So most of the field predicts depth up to an unknown transform and lets the evaluation remove it. MiDaS made this the default: train on a mixture of datasets whose depths have incompatible scales, with a loss that does not care about scale and shift, and the model transfers to new domains far better than any single-dataset model. A prediction defined up to an affine map $\hat d \mapsto s\hat d + t$ is called affine-invariant depth.

The price is that every score needs an alignment step. For a prediction $\hat d$ and ground truth $d$ over the valid pixels $\mathcal{M}$, the benchmark first solves

for some penalty $\rho$, and only then scores the aligned map $\tilde d_i = s^*\hat d_i + t^*$. The headline metric is the absolute relative error:

It is a mean of per-pixel relative errors, so 0.03 means the aligned depth is off by 3% on an average pixel. Papers in this area print $100\times$ AbsRel, which is why the numbers below look like 2.92 rather than 0.0292. The companion metric is $\delta_1$, the share of pixels whose ratio $\max(\tilde d/d,\, d/\tilde d)$ is under 1.25.

Two choices hide inside "after alignment". The first is the penalty: least squares, an L1 on relative error, or a robust RANSAC fit that ignores outliers. The second is the space: you can fit $s, t$ on depth itself, or fit them on $\log d$. A fit in log space removes a power as well, since $a\log d + b = \log(e^b d^a)$, so it forgives a prediction whose depths are right up to an exponent. The two benchmark suites Iris-Z reports on make different choices, and that matters for a model whose output is log depth.

The lab above builds a toy room in metres, makes a "model output" that is log depth with an unknown scale $a$ and shift $b$ (the form Iris-Z's output takes, as the next sections show), and scores it four ways. With $a = 1.6$ and a little noise, the toy gives these scores: a least-squares fit in metres gives AbsRel×100 of 10.92, the relative-L1 fit in metres gives 7.95, and a least-squares fit in log space gives 1.95. Set $a$ back to 1 and all three agree at about 1.9: a depth-space fit can stretch and slide $e^x \propto d^a$, but it cannot straighten $d^{1.6}$. Turn on the "glass" toggle, where the model reads a window as a pane at 3 m instead of a garden at 9 m, and least squares gets dragged by the wrong pixels while the relative-L1 fit holds its ground.

The two families Iris-Z is measured against

The specialists. After MiDaS came DPT (the same recipe on a ViT), Depth Anything (pseudo-labelling about 62M unlabelled images with a teacher), Depth Pro (sharp edges and focal length), and the MoGe line from Microsoft. MoGe predicts an affine-invariant 3D point map rather than a depth map, with a robust alignment inside its training loss; MoGe-2 adds metric scale; MoGe-3 refines the point map in 3D with sparse convolutions. Its ViT-G model is the reference here: one pass, plus three refinement steps, and it predicts the camera too.

The repurposed generators. Marigold showed in 2023 that a Stable Diffusion U-Net, fine-tuned on a small amount of synthetic data, becomes a strong zero-shot depth estimator. The generator already knows what scenes look like; depth is a short step from there. The image goes in as a condition, the depth comes out of the denoiser. Follow-ups made it faster (DepthFM, Lotus, single-step variants), and Marigold V2 moved it to a 20.43B Qwen-Image diffusion transformer. In every one of these, the report points out, "the generator becomes an image-to-depth model". Text-to-image generation is gone after fine-tuning.

There is a smaller third line that generates RGB and depth together: LDM3D, JointNet, Wonder3D for normals across views, Aether for video. The report cites them and says none "were shown to reach specialist accuracy on established depth benchmarks, which is the gap Iris-Z addresses". In the report itself the claim is narrower: Iris-Z-big is, "to our knowledge, the first open general-purpose generator to beat MoGe-3 on Marigold's benchmarks". The word "open" is there for a reason. The report also cites Vision Banana, an instruction-tuned Nano Banana Pro that reports a lower metric-depth error than MoGe-2 but, in the report's words, "predates MoGe-3 and is not available to test". Joint image-and-depth generation is not new. Doing it at specialist accuracy, if it holds, is.

Depth, encoded as an image

The trick that makes everything else possible is boring on purpose. Depth is turned into something that looks like a picture, so the existing autoencoder can encode it unchanged.

The SDK's encode_log_depth (in iris_z/depth.py) does it in four moves:

1. Take log ray distance, the distance along each pixel's ray rather than the z-coordinate. The report says every training source was converted to metric ray distance.
2. Find the 2nd and 98th percentile of the non-sky log distances in this image, and map that range linearly to $[-1, c]$, where $c$ is a "sky ceiling": 0.95 for Iris-Z-big and 1.0 for Iris-Z-small, per each variant's config.json.
3. Set sky to $+1$. On Iris-Z-big this leaves a margin between the farthest real depth (0.95) and the sky; at inference, pixels above 0.975, halfway between, are called sky.
4. Copy the single channel into three, encode it with the frozen FLUX.2 autoencoder like any RGB image, and on the way out decode and average the three channels.

In symbols, with $p_2$ and $p_{98}$ the percentiles of $\log r$ for ray distance $r$:

Because $p_2$ and $p_{98}$ are per image, the output really is defined only "up to a scale and shift in log space", as the report's limitations section says. Iris-Z gives neither metres nor a camera, and the depth is affine-invariant in log depth, not in depth. And because the range is set by percentiles, the 2% of nearest and farthest pixels are clipped.

One sequence, one backbone

expand

Iris-Z starts from Black Forest Labs' FLUX.2: [dev] (32B, with a Mistral Small 24B text encoder) for Iris-Z-big, and [klein] Base 9B (with a Qwen3 8B text encoder) for Iris-Z-small. The transformer is fully fine-tuned; the autoencoder and text encoder stay frozen.

The image and the depth map are each encoded into latents of 128 channels on a grid one sixteenth of the picture's side: shape = (1, 128, height // 16, width // 16) in iris_z/rgbd.py. Each grid cell becomes one token. A 512² picture is therefore 32×32 = 1,024 image tokens and another 1,024 depth tokens; a 1024² picture is 4,096 plus 4,096. The sequence is text tokens, then all the image tokens, then all the depth tokens.

Each token carries three labels, and they are the whole mechanism:

• Where it is. FLUX.2 uses rotary position embeddings over four coordinates. Row and column are identical for an image token and the depth token of the same patch, so attention knows they describe the same point of the scene.
• Which grid it is. The first rotary coordinate is a "time" axis the backbone already uses to tell several reference images apart. Iris-Z sets it to 0 for image tokens and 1 for depth tokens (DEPTH_TIME_COORDINATE = 1 in the SDK). No new parameters: the backbone already knew how to keep two images apart.
• What role it plays. A learned modality embedding, added to every token after the input projection, says "generated image", "generated depth", "given image" or "given depth". In the code these are ids 0 to 3 and the table is nn.Embedding(4, hidden_size).

Image and depth also get their own flow-matching time. The SDK's forward pass takes rgb_timesteps and depth_timesteps and builds a separate adaptive-norm modulation for each part of the sequence. A grid being generated sits at the sampler's current $t$; a grid that is given goes in clean, at $t = 0$. Because attention runs over the whole sequence, the picture and its depth condition each other at every layer and every step.

# iris_z/rgbd.py (SDK commit 6c8925f)
RGB, DEPTH, GIVEN_RGB, GIVEN_DEPTH = 0, 1, 2, 3
DEPTH_TIME_COORDINATE = 1
 
def rgbd_sequence(rgb_latent, depth_latent):
    """[RGB tokens, depth tokens] on the same (h, w) positions."""
    rgb_tokens, rgb_ids = grid_tokens(rgb_latent, 0)
    depth_tokens, depth_ids = grid_tokens(depth_latent, DEPTH_TIME_COORDINATE)
    ...

With that sequence, the four tasks differ only in what is noisy, what is clean and which ids are set. Pick one below and drag the sampler from $t = 1$ to $t = 0$.

Spelled out from the SDK's four samplers:

The last row surprised me. Plain text-to-image in Iris-Z still carries the depth grid, as noise, through every pass. That doubles the image tokens the model attends over compared with the base FLUX.2. The README's timing table shows the effect indirectly: on one H200 at 512², Iris-Z-big takes 33.2 s for prompt → image and 34.6 s for prompt → image + depth. They cost nearly the same because the first already paid for the second grid. I could not time base FLUX.2 [dev] for comparison.

The first row has a footnote too. The SDK's text_to_rgbd generates image and depth jointly, then by default throws the joint depth away and re-estimates it from the finished image with the image → depth sampler; joint_depth=True returns the joint one. The report says why: the re-estimate "is sharper". The 1.4 s gap between the two timings above matches the 1.44 s the README gives for one image → depth run. So the demo's "image + depth" is, by default, a two-stage pipeline built from one model, not the depth that was denoised alongside the picture.

Training: five tasks, one loss

According to the report, each training sample draws one task:

• Joint: image and depth noised to the same $t$, both scored.
• Image given: only depth is denoised.
• Depth given: only the image is denoised.
• Independent: image and depth noised to independent times.
• Image only: depth stays pure noise and only the image is scored, which "keeps training plain text-to-image generation and prevents the model from drifting away from the base model's images".

The loss is the base model's rectified-flow objective. With clean latent $x_0$, noise $\varepsilon$ and time $t \in [0,1]$, the noised latent is $x_t = (1-t)\,x_0 + t\,\varepsilon$ and the network predicts the velocity $v = \varepsilon - x_0$. A second loss decodes the depth the model currently implies,

runs it through the frozen decoder, and takes an L1 to the target depth over valid pixels, weighted by $(1 - t)$ so near-clean steps count most. On LiDAR, stereo and other holed sources, that decoded loss is the only depth supervision, applied on measured pixels. Captions are dropped for a share of samples so image → depth works with an empty prompt.

FLUX.2 [dev] is guidance-distilled: it takes the guidance scale as an input embedding. Iris-Z-big removes that embedding and trains classifier-free guidance back in through caption dropout. The checkpoint agrees: Iris-Z-big's tensors include no guidance_in, and its config.json says "guidance_mode": "remove". The modality embedding starts at zero, so step one of training is the unchanged base model.

Data: about 3.8M unique RGBD frames for Iris-Z-small and 3.6M for Iris-Z-big, from indoor captures, driving scenes with LiDAR and stereo, multi-view reconstructions, rendered scenes and generated images, each with a dense caption. The report says "We do not train on the benchmark datasets". It does not list its sources by name.

What the checkpoints contain

The release is two folders of sharded bf16 safetensors. I read every shard's header by HTTP range request and summed the tensor shapes:

That matches the card's 32.2B and 9.1B. The only tensor with "modality" in its name is the 4-row embedding, and the output layer projects back to the same 128 latent channels it takes in. "No depth head" is literal: the depth comes out of the same final_layer as the picture. The new parameters are 24,576 out of 32.18 billion, under one millionth. Everything else Iris-Z learned lives in weights FLUX.2 already had.

The SDK loads with strict=True, so this tensor list is exactly what runs.

Sampling depth costs 40 passes

Image → depth, as benchmarked, runs 4 steps from each of 10 seeds:

1. For each seed, start the depth grid from noise and take 4 steps with the photo given clean.
2. Average the clean-depth estimates $\hat x_0 = x_t - t\hat v$ from the last two steps.
3. Align every seed to the first with a least-squares scale and shift in log space (align_affine in the SDK). This matters: two seeds can land on different log scales, and a per-pixel median of unaligned maps would mix them.
4. Take the per-pixel median.

That is 40 transformer passes per picture. The report times one Iris-Z-big pass at 0.35 s at 512² and 1.4 s at 1024² on one H200, so one prediction is about 15 s at 512²; Iris-Z-small is 0.10 s a pass and 4.8 s a prediction. MoGe-3 takes 0.41 s at 512² on the same GPU. The report's own ratio is "about 37 times less", and 15 / 0.41 does come to about 37. Pictures under 0.5 MP run at about 512², larger ones at about 1024².

expand

Panel c is where the claim has to be read carefully. Reading the plot by eye (these are not printed numbers): at 4 passes, which is one seed, Iris-Z-big sits at about 3.15 and Iris-Z-small at about 3.3; MoGe-3 is 3.12 at one pass. At 10 passes Iris-Z-big is 3.13, just behind. Only the 40-pass setting gets it to 2.92. The report notes that Iris-Z-big's 10-pass points come from an earlier checkpoint.

That matters because the SDK's default is one seed: image_to_depth(..., steps=4, seeds=(0,)) in pipeline.py. The README says so ("Ten seeds, the setting behind the benchmarks, run only in the image → depth pipeline") and shows how to ask for ten. The Space's image → depth examples for Iris-Z-big are precomputed with 10 seeds, labelled "as in the report", while the Space's live depth call passes seeds=(seed,), one seed. If you call the default API you get a depth model that is, on this benchmark, slightly behind MoGe-3, not ahead of it, and still slower: the README times that 4-step default at 1.44 s at 512² against MoGe-3's 0.41 s, about 3.5 times.

Panel b is the report's argument for scale. A LoRA on the 9B backbone that saw 0.2M frames scored 3.83 at 40 passes, already ahead of Marigold V2. The full fine-tune, Iris-Z-small, brought it to 3.11. Moving to a backbone 3.5× larger brought it to 2.92, with slightly fewer unique frames. At 10 passes the same three runs score 3.99, 3.27 and 3.13. Three runs that change data and model size together are a direction, not a scaling law.

The benchmark, checked

The report runs two tables, each "exactly as its paper defines it":

• Marigold's protocol: NYUv2, KITTI, ETH3D, ScanNet, DIODE with Marigold V2's split lists and masks, 3,331 images, scored after RANSAC scale-and-shift alignment in log depth.
• MoGe-3's protocol: NYUv2, KITTI, ETH3D, iBims-1, Sintel, DIODE and HAMMER with MoGe's masks, 4,470 images, scored after MoGe's relative-error L1 alignment. MoGe-3 reports this as "affine-invariant depth". I did not find the evaluation's alignment formula in what I read; MoGe-3's training loss fits its scale and shift with an L1 residual in depth weighted by $1/Z$, which is the relative-error L1 the lab above implements, and I assume the evaluation is the same family.

The same dataset name gives different numbers in the two tables because each protocol prepares the data differently: Marigold's uses native images and cameras with fixed depth ranges and crops; MoGe's re-renders every sample into a centred, undistorted pinhole view.

Marigold's sets, AbsRel×100, lower is better, from the report's Table 4 (Speridlabs' own runs):

Iris-Z-big wins three of five: NYUv2, ETH3D and ScanNet. MoGe-3 keeps KITTI and DIODE. By my arithmetic, Iris-Z-big is 28% better on NYUv2 and 12% worse on DIODE. The means are plain per-dataset averages; I recomputed them from the columns and they match to rounding. "Iris-Z-small is level with MoGe-3" is fair: 3.11 against 3.12, which is 3.110 against 3.116 unrounded.

The MoGe-3 row here is Speridlabs' own run (ViT-G, resolution level 9, 3 refinement steps, "run by our code on the same pixels"). MoGe-3 did not publish numbers under Marigold's protocol, and Marigold V2's Table 1 lists MoGe and MoGe-2 but not MoGe-3, so 3.12 has no independent source. The run still looks trustworthy. Their Marigold V2 rerun gives 4.10 against the 4.14 Marigold V2 published. And on MoGe-3's own protocol, below, their MoGe-3 rerun reproduces MoGe-3's published Table C.1 within 0.04 on every set (3.70 mean against 3.72); I checked that against the table in MoGe-3's arXiv HTML.

MoGe-3's sets, AbsRel×100, from the report's Table 3:

expand

"Six of seven" holds on AbsRel. It comes with caveats.

Two of MoGe-3's nine sets are missing. MoGe-3's Table C.1 has nine sets; Iris-Z drops Spring (2 MP) and Synth4K (8 MP) because "both models were trained at up to about 1024² pixels, and these sets test resolutions we did not train for". That is a stated limitation, not a hidden one. But the seven-set mean is not MoGe-3's benchmark mean. On all nine, MoGe-3's Table C.1 gives it 4.80, and there is no Iris-Z number to compare it with.

δ1 tells a different story. The report's appendix scores the same runs with $\delta_1$. On MoGe-3's protocol MoGe-3 has the higher mean, 97.15% against 97.03% for Iris-Z-big, and wins four of the seven sets on $\delta_1$ (ETH3D, Sintel, HAMMER, DIODE). On Marigold's protocol Iris-Z-big leads, 98.78% against 98.46%. So on MoGe-3's own suite, Iris-Z-big is closer on the average pixel but has slightly more pixels off by over 25%. That is a split decision, not a sweep. HAMMER, where Iris-Z-big has its biggest AbsRel win (−16%), is one where it loses on $\delta_1$ (99.38% against 99.58%).

The margins are small where it matters. Sintel is −2%; iBims-1 is −4%. Most of the seven-set mean's lead (3.56 against 3.70) comes from HAMMER, NYUv2 and KITTI. On MoGe-3's KITTI, Iris-Z runs at about 1024² because the LiDAR ground truth is sparse, an exception to its 0.5 MP rule that the report states.

The SDK ships no evaluation code, so a couple of questions stay open.

• How log depth meets a depth-space fit. Iris-Z's output is affine in log depth. Marigold's protocol aligns in log depth, so it fits naturally. MoGe's protocol fits a scale and shift in depth. As the lab shows, a depth-space fit cannot undo an unknown exponent, so the report must convert the log output to depth somehow before MoGe's alignment. It does not say how. If that step uses the ground truth, say a log-space fit first, it is a stronger alignment than MoGe-3 gets. I am not claiming it does; the report does not say.
• Ray distance against z-depth. Iris-Z predicts distance along the ray; most ground truth is planar z-depth. They differ by a per-pixel factor set by the camera, which no global scale and shift removes. The report does not say whether it converts with the benchmark's intrinsics. The effect is small near the image centre and grows towards the corners on wide lenses.

The report says it does not train on the benchmark datasets, and names no sources, so leakage is not something I can check either way. For what it is worth, DIODE is the one set where Iris-Z-big loses on both protocols.

Generation is preserved, mostly

"A depth estimator that forgets how to draw would be just another specialist," the report says, and its Table 1 checks, on the same held-out prompts and seeds at 50 steps with guidance:

Prompt adherence (CLIP) moves by under half a point. Images keep the base model's subject and composition, at about 0.88 CLIP similarity to the base model's own picture for the same seed. Human preference drops: Iris-Z-big falls from 26.71 to 24.30 on HPSv2.1, as the model drifts towards the look of its RGBD training data. Iris-Z-big's images get more photographic (0.48 against 0.38); Iris-Z-small's get less (0.34 against 0.41). The report calls this "generation drift" in its limitations. "Keeps the generative abilities" is true in the sense that it still follows prompts; it is not true that the images are as good as FLUX.2 [dev]'s by the preference metric the report itself chose.

What conditioning on depth buys

expand

Depth → image is the task that the Marigold family gave up and that ControlNet-style adapters bolt on. Here it is the same weights with the roles swapped: the depth grid goes in clean with the "given depth" id, and the picture is denoised onto it. The SDK reads depth maps three ways: bright-is-far, bright-is-near (disparity), or Iris-Z's own colouring, inverted back through the colour map with a nearest-neighbour lookup.

What that enables, concretely:

• Repaint a captured scene. Estimate depth from a photo, then paint a new picture onto the same geometry: "the same street in winter" is the README's own example, and the Space has a "reimagine" mode that chains exactly those two calls.
• Synthetic RGBD. Prompt → RGBD gives a picture and an aligned depth map from one model; the report never tests whether that depth is good enough to train on.
• Layout control from a depth sketch. A rough depth map pins where the walls, floor and objects go, and the prompt fills in the rest.

The report frames this as the first modality of several: normals, segmentation, motion and the camera would each be "one more group of tokens in the same sequence". The next one it names is the camera's intrinsics, which would turn the relative log depth into a point cloud and leave a single global scale.

"Fully open"

The announcement says "Fully open: weights, code, demo (Apache 2.0)". The code is Apache 2.0. The weights are not. The model card's licence is other / flux-non-commercial-license, and the SDK's NOTICE says the weights "are derived from FLUX.2 [dev] and FLUX.2 [klein] Base 9B and are distributed under the FLUX Non-Commercial License". Speridlabs cannot relicense FLUX.2 [dev]'s weights, so this is the only licence they could ship under. It is still not what "Apache 2.0" in a launch post leads people to expect.

Running it also needs Black Forest Labs' gated FLUX.2 autoencoder: accept the terms on the FLUX.2 [dev] repo, then log in. The PyTorch build in the lockfile targets CUDA 13.0 and needs an NVIDIA driver 580 or newer, per the README.

The SDK is about 2,800 lines of Python, mostly FLUX.2's own code plus the modality embedding and per-grid noise levels. The Hub API lists the Space on a ZeroGPU A10G, and its app.py and depth_backend.py show what that buys: it runs Iris-Z-small locally for image → depth and depth → image, sends Iris-Z-big requests to private GPU workers, and shows only precomputed examples for Iris-Z-small's prompt modes.

On hardware: the card's 33 GB for Iris-Z-small covers all tasks, and image → depth can load without the text encoder. The transformer alone is 18.16 GB in bf16, so I would expect a 24 GB card to be tight and a 48 GB workstation card to be comfortable. I have not tried either. Iris-Z-big's transformer is 64.37 GB on its own, which is datacenter territory.

In the specialists' band, not yet a depth model to ship

The architecture is the part worth remembering. Depth becomes a picture, the picture becomes tokens, the tokens join the sequence on the same rotary positions as the image with one coordinate flipped, and a 4-row embedding says who is who. That is the whole change to a 32B model, and the header count confirms it. It is a clean demonstration that a big image generator's attention can carry geometry with almost no new machinery, and the scaling run (LoRA, full fine-tune, larger backbone) points the right way.

The headline needs its conditions attached. 2.92 against 3.12 is AbsRel on Marigold's five sets, with log-space alignment, at 40 passes and about 15 s a picture, against a MoGe-3 that Speridlabs ran themselves. On MoGe-3's own seven sets Iris-Z-big wins the AbsRel mean and loses the $\delta_1$ mean. At the SDK's default single seed it is, by the report's own plot, slightly behind MoGe-3 and about 3.5 times slower. KITTI and DIODE on Marigold's protocol still go to MoGe-3. None of that makes the result small. A generator that also draws is now in the same accuracy band as the best specialist, which a year ago no one had shown. It is not yet a better depth model to deploy.

For a practitioner: if you need depth in production, MoGe-3 is still the tool. It is about 37 times faster, gives you metric scale and a camera, and carries no non-commercial licence. If you need aligned image and depth from a prompt, or want to paint new appearance onto captured geometry, Iris-Z does that in one model, and nothing in the specialist line does at all. Within a non-commercial licence, that second use is the one to try. Related reading: how Marigold V2 repurposes a 20B DiT for depth, what Depth Anything 3 does and does not give you in metres, PixelUMM's case for dropping the VAE, and the diffusion transformer from first principles.