schopra909

They’re not trained on “next pixel prediction”, hence they don’t have likelihoods on tap

Empirically they learn an approximation to the derivative of the likelihood (Ie the score function). So we have a bound on the true likelihood distribution, but don’t have exact likelihoods (which we do have in LLMs).

In practice this bound is too loose to be used for train with GRPO effectively (though folks have tried).

When it comes to reasoning specifically, it's a far simpler explanation IMO. Modern RLVR techniques like GRPO rely on the model understanding the likelihood of the generated phrase. This is really easy to compute for LLMs that are trained with next-word prediction. Diffusion models have a harder time accessing this likelihood value (i.e., you can approximate it but the approximation isn't good). Hence, reasoning doesn't work (yet) for diffusion models. Though, alternative approaches are currently being tried out in academia; so someone might crack it!

It’s a good question; it’s not something that worries us too much to be honest.

The linear patchificaion is per patch, so if it works on 360p 2 seconds (really short low res) it should adapt naturally (given ROPE3D). The final layers of the DiT start moving us from latent space to pixel space.

Net net the compression is as much as H3 (32x32xTime) so it should be comparable to the best VAEs in terms of expense

Thanks for links! I hadn’t seen this yet.

Very related; our models are quite literally cousins, as we’re both interesting on the work of JiT from last fall.

Architecture is roughly the same up to some minor differences (they use MM-DiT from FLUX, we use single stream from Z-Image).

The biggest delta is that we’re going with 32x32 token reduction and they go for 16x16. Our goal here was to cut attention windows 4x to speed up training inference. And have been publishing work (this included) on how to match models with less compression to than us.

In this blog specifically there are 3 things we stack on top of each other to get good results:

1) transformer blocks per modality before they enter shared single stream transformer blocks

2) very different noise schedules during training than those proposed in prior work

3) predicting the output at k different resolutions along the depth of the diffusion transformer to provide gradient signal earlier in the network to learn better (and faster)

The other work does a great job scaling up the JiT to 3B parameter; this is purely an experimental release highlighting how we make up the gap.

Also i failed to clarify that the VAE for Linum v2 was the Wan2 VAE

This is a totally fair point and definitely worth exploring!

The jumping point for this no-VAE work was three-fold:

1) Our goal here is to get 32x32 token reduction to make video training and inference downstream cheaper. To date, the best open-weight Image VAEs like Flux-2 seem to cap out at 16x16 token reduction (8x8 VAE + 2x2 linear patchification). Others like H3 have pushed to 32x32 reduction but requires them swapping out the small VAE decoder with a 2B parameter decoder. So, this is a foray to get 32x32 compression without compromising quality.

2) We believe that end-to-end trained networks will tend to perform better than modularly trained networks (e.g. VAE + DiT). This hypothesis comes from work like REPA-E, where authors are able to get much better results by backpropagating through the pretrained VAE. The latent space for perception / reconstruction seems to have a different "optimal" configuration than a latent space specifically built for generation. That's why we liked the idea of trained E2E here.

3) There's work with VAEs that show that providing additional modalities (e.g. text captions) can help the VAEs improve as well. That's natural to this construction, so we thought language might actually help with the compression, not hurt. To be honest, this this is the most hypothetical of three ideas; and definitely warrants specific ablations.

We haven't built a ComfyUI integration; but all the weights and inference code are available to download. So, it should be pretty easy to point Codex/Claude to it and have it re-arrange stuff to be compatible.

From our POV, this quite an experimental checkpoint. That's why we've dubbed it a research release. Our goal is to share our findings (and checkpoints) along the way to a legit image and video release for Linum v3.

Image release should be out in the coming weeks, so stay tuned for that. Video will take a little longer (we're doing on-going research atm on pixel-space video training).

I appreciate the clarifications!

If we have extra compute lying around in the coming weeks, we’ll try this out and report back. It’s a good idea :)

That sounds like an interesting idea!

Can you confirm I'm understanding correctly?

1) Linear unpatchify as usual to go from hidden states to pixel space

2) Attention within a local window (e.g. 3x3, 5x5) to "blend" pixel space data and come up with a better image (as an alternative to MLP or Convolution)

And follow up questions:

1) How do you handle boundaries between your "attention windows"? Do you move the window just like a convolution does or are the "attention windows" all mutually exclusive from one another?

2) How much faster/slower is this operation vs. a linear layer + MLP?

One of the reasons we’re focused on training efficiencies is because those are upstream of end user costs. We achieved these specific gains while cutting context windows down 4x. That will have major implications for inference speed and expense.

For us specifically, we want to target prosumers who aren’t gonna pay $1 for a few seconds of footage. It has to be 1/50th of the cost of the big guys. It helps that our ultimate focus/niche is animation, so we can get away with smaller models.

But to answer question more broadly, I think you have to ask the questions:

1) How will this lab front run the hyperscalers? Is there someone thing they’re doing (or if executed correctly) could break right such that they could have a hook for customers to use the product over hypscaler (eg crazy low cost of inference)?

2) If this lab lands the hook and gets small lead how could they maintain and grow it? This doesn’t have to be technical per say. It could be a clear land and expand sales motion at the enterprise level, for example.

Startup history is littered with examples of companies that seemed too close to a big guy to sprout up in the first place (eg stripes adjacency to PayPal).

From the outside looking in it might seem likes there no space for the lab to bloom; so you’ll have to bring the specifics of the lab and your on the ground expertise of the space / company’s situation to come to an answer for yourself.

Thanks for the references! I’ll have to take a deeper look, haven’t read these before.