Building Visual Intelligence — Andreas Blattmann & Dustin Podell, Black Forest Labs

This episode is sponsored by Deepgram. Try Deepgram for voice agents that truly understand your context! #contextiseverything Not many people know that Stable Diffusion is why I went fulltime into AI (I aptly titled Multiverse, not Metaverse to call out Zuck’s mistaken investment). It turns out that Dustin Podell also had the same aha moment - quitting his day job when the model came out. But we took very different paths… I stuck to what I knew in human content creation, and Dustin immersed deep into AI content creation with Andreas Blattmann, co-inventor of Latent Diffusion and the rest of the world-leading imagegen crew at Stability AI. Stability fell apart, but Anj Midha helped the core group spin out to BFL, which immediately built an incredible new imagegen model in the Flux series. But why are they now going into video and world models? Andreas gives an example you can try with one hand. If somebody tosses you a ball, your hand starts moving before the ball reaches you. You can do that without being able to write down the trajectory. When he talks about building visual intelligence, that’s the kind of knowledge he means. A still can show you where the ball is. A video shows you how it got there. The sound of impact can tell you when it hit something, even if the camera missed the contact. That’s BFL’s argument for training FLUX 3 on the three signals together. BFL’s FLUX 3 releaseThe horse in motion is one of the FLUX 3 examples we show in the intro. The audio examples are worth playing with the sound on. FLUX 3 VideoWhat a moving camera teaches you Suppose the camera starts in front of a person and pans around them. At some point the model owes you the back of their head. Andreas thinks ordinary two-dimensional video contains a great deal of evidence about what objects look like from other angles. That doesn’t settle whether a model “understands physics,” a point we go back and forth on. It does give you a much better test than asking whether one frame looks convincing. There’s a good engineering example late in the episode. Sharpen each frame of a clip independently and each frame might look terrific. Hit play and the extra detail can shimmer, because the upscaler made a slightly different decision every time. BFL derives its video upscaler from the same FLUX 3 backbone. Temporal consistency matters even when all the user asked for was a higher-resolution file. What’s worth spending training compute on? One of the questions I had for them was whether autoregression was taking over image generation. Andreas gives the other side its due. An autoregressive model can learn from all the tokens in a training example at once. On the diffusion side, the team can distill a long sampling process down to a handful of steps. Which advantage matters depends on whether you’re paying to train the model or waiting for an image to finish. Dustin’s early Kontext experiment is a nice example of how they decide where to invest. He suspected an image model that could draw a dog wearing a hat already knew something about where hats go. He and a colleague collected a small set of photos labeled with relationships. A training run of a few hours got the model to put a logo on a shirt. It was rough, but it was enough to take to the next offsite. The team then spent months on the data and the model. FLUX Kontext examples My favorite user story comes near the end. People were feeding FLUX old, degraded family photos, restoring them, then showing the pictures to their families. Dustin says he hadn’t predicted that use. After talking about preserving somebody’s identity across edits and across frames, it’s a lovely example of who actually notices if you get it right. Show notes BFL explains what a still image, a moving camera and a soundtrack each contribute during training. FLUX 3 launch and research overview Watch and listen to the results of generating video and sound together. FLUX 3 Video demos See how the lab teaches a model to make a requested change while preserving the rest of an image. FLUX Kontext examplesAndreas Blattmann co-created Stable Diffusion and co-invented the latent diffusion method behind it. Guest bio · Original researchDustin Podell led FLUX Kontext and was the lead author of SDXL. Dustin’s project credits · FLUX Kontext · SDXL The first Latent Space essay grew out of the Stable Diffusion release. Multiverse, not MetaverseChapters0:00 Host introduction3:39 Cold open4:23 Black Forest roots9:23 Dustin’s path17:04 Visual intelligence27:38 Inside SDXL33:59 Learning motion39:57 Flow matching46:57 Starting BFL50:36 Deepgram sponsor break51:43 Early use cases for FLUX54:58 Editing with Kontext1:03:15 Training data1:07:32 Testing an edit1:13:12 Autoregression1:20:50 World action models1:30:08 Robotics in 20271:32:16 Who uses FLUX?
Read Original

Related