Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

World models are moving beyond offline generation towards interactive, real-time experiences. Introducing ⚡FlashDreams⚡: an open-source high-performance inference and serving library built for autoregressive world models: 🔥 Up to 3.10× faster LingBot-World inference 🔥 Up to 2.12× faster Self-Forcing inference 🔥 Up to 1.40× faster Wan2.1 inference 🔥 8 integrated...

ruilong_li's profile picture

Ruilong Li

5,300 subscribers

100,490 Aufrufe • vor 5 Monaten •via X (Twitter)

BildungWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

24 Kommentare

Profilbild von Ruilong Li
Ruilong Livor 5 Monaten

World models are quickly advancing, but for most users the experience is still passive: watching impressive promo videos, not running the models, probing them, or building applications on top of them. FlashDreams aims to change that. Spin up a server, select a model, and interact with it live. 2/n

Profilbild von Ruilong Li
Ruilong Livor 5 Monaten

A world model learns to generate and evolve an environment over time: video, actions, state, audio, sensors, control signals, and more. Serving it means keeping a live session running while input, model state, GPU inference, and output evolve together. 3/n

Profilbild von Ruilong Li
Ruilong Livor 5 Monaten

Serving a world model is different from offline video generation. The goal is not just prompt → wait → video. A live world-model runtime must stay responsive while the world keeps advancing: low latency, high throughput, steady streaming, and rolling state. 4/n

Profilbild von Ruilong Li
Ruilong Livor 5 Monaten

Efficient serving infra like vLLM and SGLang helped move LLMs from research checkpoints into real apps. World models are reaching a similar moment, but with a different interactive pattern than LLM: initialize once, then keep evolving the world forward at a constant pace. 5/n

Profilbild von Ruilong Li
Ruilong Livor 5 Monaten

Similar to vLLM and SGLang, FlashDreams is built from the ground up for high-performance serving. FlashDreams delivers up to 2.12× speedup on Self-Forcing, 3.10× on LingBot-World, and 1.40× on Wan2.1. The system is designed specifically for autoregressive world models, where generation unfolds continuously over time. At the same time, the architecture naturally extends to bidirectional models by treating each generation task as a single autoregressive rollout. 6/n

Profilbild von Ruilong Li
Ruilong Livor 5 Monaten

FlashDreams is more than an inference engine. It is a serving stack purpose-built for interactive world-model applications, supporting persistent sessions, streaming inputs and outputs, rolling model state, and multi-GPU execution. FlashDreams already integrates eight world models, with many more on the way. 7/n

Profilbild von Ruilong Li
Ruilong Livor 5 Monaten

Ending this thread with a fun walk on w/ FlashDreams: Try it out and tag us with your exploration. #flashdreams

Profilbild von Ruilong Li
Ruilong Livor 5 Monaten

The team behind this is extremely talented. I can't be more happy to work with them! Listing in random order: @jatentaki @wilson_over @jmartinezesturo @JunchenLiu77 @csyonghe, Gangzheng Tong, Jesse Archer, @mccaffrey_j_b, William Lew, @heuristic42, Aidan Foster, Wenqing Wang, @AdityaMaha47495, @FidlerSanja @ZGojcic, @natdoesgraphics, Matt Cragun, @wang_jialiang, Anis Ahmad, @killerooo, Deep Shekhar, Michael Chock, @ATMGuille9

Profilbild von Xun Huang
Xun Huangvor 5 Monaten

Congrats! Fantastic work

Profilbild von Ruilong Li
Ruilong Livor 5 Monaten

Congrats to you too 😄

Profilbild von Andrew Carr 🤸
Andrew Carr 🤸vor 5 Monaten

oh this is super nice, well done!

Profilbild von Boyang Deng
Boyang Dengvor 5 Monaten

This is so cool and gonna be so useful! Congrats ... and thanks for the contribution! 🙂

Profilbild von Jiawei Yang
Jiawei Yangvor 5 Monaten

Legendary as always!

Profilbild von Jathushan Rajasegaran
Jathushan Rajasegaranvor 5 Monaten

Congratulations! Looks amazing!

Profilbild von Yossi Gandelsman
Yossi Gandelsmanvor 5 Monaten

congrats!

Profilbild von Ruilong Li
Ruilong Livor 5 Monaten

to you as well! big day for both of us : )

Profilbild von Richard Palethorpe
Richard Palethorpevor 5 Monaten

Amazing stuff, does RTX 6000 or Spark have enough compute to handle these models or is a data center GPU needed?

Profilbild von Ruilong Li
Ruilong Livor 5 Monaten

This is running 30 fps live on a single RTX 6000 pro

Profilbild von Richard Palethorpe
Richard Palethorpevor 5 Monaten

Wow

Profilbild von Prithvi Jadwani | AI SEO | GEO | REDDIT SEO | GMB
Prithvi Jadwani | AI SEO | GEO | REDDIT SEO | GMBvor 5 Monaten

That's an impressive benchmark, but how does FlashDreams handle model drift from continuous evolution?

Profilbild von Ruilong Li
Ruilong Livor 5 Monaten

There are some kv cache tricks can alleviate that during inference time. You can check out this configuration if interested: self-forcing-wan2.1-t2v-1.3b-sink5-window7-rerope Doc:

Profilbild von Maya Srinivasan
Maya Srinivasanvor 3 Monaten

real-time world models

Profilbild von Neha Patel
Neha Patelvor 4 Monaten

real-time world models finally

Profilbild von Mathias Heide
Mathias Heidevor 4 Monaten

Oh, i'd love to introduce this into Summer Engine for automated playtesting of games - possible?

Ähnliche Videos

Introducing NVIDIA Nemotron 3 Ultra. A frontier smart open model built for long-running agents that need to plan, reason, use tools and keep working across complex coding, research and enterprise workflows. Up to 5x faster inference and up to 30% lower cost for agentic tasks. Learn more:
0:24

Introducing NVIDIA Nemotron 3 Ultra. A frontier smart open model built for long-running agents that need to plan, reason, use tools and keep working across complex coding, research and enterprise workflows. Up to 5x faster inference and up to 30% lower cost for agentic tasks. Learn more:

NVIDIA

230,981 Aufrufe • vor 5 Monaten

We are innovating the full-stack AI infrastructure platform on a one-year rhythm — pushing the boundaries of inference performance across chips, systems, and software. NVIDIA Blackwell is the most advanced AI platform ever built — continuously optimized for record-breaking performance. Groundbreaking advancements include the NVIDIA-designed NVFP4 format, enabling high inference performance and accuracy. #GTCParis at #VivaTech
1:06

We are innovating the full-stack AI infrastructure platform on a one-year rhythm — pushing the boundaries of inference performance across chips, systems, and software. NVIDIA Blackwell is the most advanced AI platform ever built — continuously optimized for record-breaking performance. Groundbreaking advancements include the NVIDIA-designed NVFP4 format, enabling high inference performance and accuracy. #GTCParis at #VivaTech

NVIDIA

80,665 Aufrufe • vor 1 Jahr

Compute is the backbone of the AI-driven future. OptimAI Compute Engine would enable scalable, high-performance workloads across image and video models, supporting: • Text-to-image generation • Image editing and inpainting • 2K / 4K / 8K super-resolution • Brand-aligned visual synthesis • OCR and image intelligence • Video generation and motion synthesis • Frame interpolation and enhancement • Multimodal model inference Distributed compute built for faster inference, shorter training cycles, and production-grade AI execution.
0:24

Compute is the backbone of the AI-driven future. OptimAI Compute Engine would enable scalable, high-performance workloads across image and video models, supporting: • Text-to-image generation • Image editing and inpainting • 2K / 4K / 8K super-resolution • Brand-aligned visual synthesis • OCR and image intelligence • Video generation and motion synthesis • Frame interpolation and enhancement • Multimodal model inference Distributed compute built for faster inference, shorter training cycles, and production-grade AI execution.

OptimAI Network

18,909 Aufrufe • vor 8 Monaten

AI answers your questions in seconds, but behind that speed is something called inference—the compute-intensive process where trained models generate responses. At AWS, we've built custom chips like Trainium, intelligent routing systems, and unified infrastructure to make inference faster and more affordable. As AI agents handle complex multi-step tasks, inference accounts for 80-90% of AI computing power. We're engineering at planetary scale to keep those milliseconds reliable.
0:25

AI answers your questions in seconds, but behind that speed is something called inference—the compute-intensive process where trained models generate responses. At AWS, we've built custom chips like Trainium, intelligent routing systems, and unified infrastructure to make inference faster and more affordable. As AI agents handle complex multi-step tasks, inference accounts for 80-90% of AI computing power. We're engineering at planetary scale to keep those milliseconds reliable.

Amazon Web Services

27,021 Aufrufe • vor 8 Monaten

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering
27:58

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 Aufrufe • vor 7 Monaten

Playing with omnidreams ( inside flashdreams ( Fully interactive video generation at - 30FPS, - 720P resolution, - On a single RTX 6000 PRO GPU
1:23

Playing with omnidreams ( inside flashdreams ( Fully interactive video generation at - 30FPS, - 720P resolution, - On a single RTX 6000 PRO GPU

Ruilong Li

18,727 Aufrufe • vor 4 Monaten

⚡Faster models mean smarter AI. ⚡ With Nemotron's open-source technologies, we're enabling organizations to create AI agents that get more done in less time. Big or small, we are committed to helping companies all around the world build & deploy AI for their own needs. See how:
0:49

⚡Faster models mean smarter AI. ⚡ With Nemotron's open-source technologies, we're enabling organizations to create AI agents that get more done in less time. Big or small, we are committed to helping companies all around the world build & deploy AI for their own needs. See how:

NVIDIA

35,710 Aufrufe • vor 1 Jahr

SGLang-Diffusion is a high-performance serving framework for diffusion models, designed for both large-scale offline generation and latency-sensitive real-time inference. At PyTorch Conference North America 2026, Yihao Wang and Kevin Mi of RadixArk will present the design of SGLang-Diffusion and the PyTorch-based inference optimizations behind it, including efficient request scheduling, memory management, batching strategies, and execution-path optimizations for diffusion pipelines. Join us in San Jose, October 20-21: #PyTorchCon
0:35

SGLang-Diffusion is a high-performance serving framework for diffusion models, designed for both large-scale offline generation and latency-sensitive real-time inference. At PyTorch Conference North America 2026, Yihao Wang and Kevin Mi of RadixArk will present the design of SGLang-Diffusion and the PyTorch-based inference optimizations behind it, including efficient request scheduling, memory management, batching strategies, and execution-path optimizations for diffusion pipelines. Join us in San Jose, October 20-21: #PyTorchCon

PyTorch

17,471 Aufrufe • vor 24 Tagen

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.
1:24

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.

kwindla

274,975 Aufrufe • vor 10 Monaten

Open source powers every layer of the AI software stack—from agents and models to inference, training and cloud infrastructure. At #OSSummit Korea, Linux Foundation CEO Jim Zemlin explains how open collaboration is making AI faster, more affordable and more accessible. Watch:
1:01

Open source powers every layer of the AI software stack—from agents and models to inference, training and cloud infrastructure. At #OSSummit Korea, Linux Foundation CEO Jim Zemlin explains how open collaboration is making AI faster, more affordable and more accessible. Watch:

The Linux Foundation

13,685 Aufrufe • vor 2 Monaten

Today we're shipping Nemotron 3 Ultra. A 550B MoE frontier-intelligence open model built for long-running agents. It delivers 5x faster inference and lowers the cost of complex agentic tasks by up to 30% versus other open frontier models.
2:59

Today we're shipping Nemotron 3 Ultra. A 550B MoE frontier-intelligence open model built for long-running agents. It delivers 5x faster inference and lowers the cost of complex agentic tasks by up to 30% versus other open frontier models.

NVIDIA AI

1,331,256 Aufrufe • vor 5 Monaten

Cyber defense is entering the superintelligence era. ⚡ Introducing the CrowdStrike Cyber Superintelligence Lab, the first frontier AI research organization built for cyberdefense and AI safety. The Lab delivers CrowdStrike SafeMind, a family of purpose-built security models and harnesses. SafeMind runs as one system designed to deliver AI safety: an offensive model that finds the attack path, a defensive model that closes it, and the harnesses that operate both in the same loop. Built using NVIDIA Nemotron open models and using CoreWeave’s AI Cloud for training and inference, SafeMind goes beyond AI that simply identifies threats, it defeats them. Learn more:
0:27

Cyber defense is entering the superintelligence era. ⚡ Introducing the CrowdStrike Cyber Superintelligence Lab, the first frontier AI research organization built for cyberdefense and AI safety. The Lab delivers CrowdStrike SafeMind, a family of purpose-built security models and harnesses. SafeMind runs as one system designed to deliver AI safety: an offensive model that finds the attack path, a defensive model that closes it, and the harnesses that operate both in the same loop. Built using NVIDIA Nemotron open models and using CoreWeave’s AI Cloud for training and inference, SafeMind goes beyond AI that simply identifies threats, it defeats them. Learn more:

CrowdStrike

58,367 Aufrufe • vor 2 Monaten

Long video generation is a systems problem. Introducing LongLive-2.0 from NVIDIA Research: an end-to-end NVFP4 training and inference system for long video generation. Low-precision deployment often relies on post-training quantization, creating a gap between how models are trained and how they run. LongLive-2.0 aligns NVFP4-aware training, distillation, and W4A4 inference, maintaining strong benchmark quality while improving speed and memory efficiency.
0:59

Long video generation is a systems problem. Introducing LongLive-2.0 from NVIDIA Research: an end-to-end NVFP4 training and inference system for long video generation. Low-precision deployment often relies on post-training quantization, creating a gap between how models are trained and how they run. LongLive-2.0 aligns NVFP4-aware training, distillation, and W4A4 inference, maintaining strong benchmark quality while improving speed and memory efficiency.

NVIDIA AI

60,715 Aufrufe • vor 5 Monaten

world models are here! fal is the best platform for real-time audio and video inference. RT for access 😇 incredible work by Eloi Alonso to open source this model
0:25

world models are here! fal is the best platform for real-time audio and video inference. RT for access 😇 incredible work by Eloi Alonso to open source this model

fal

123,318 Aufrufe • vor 2 Jahren

Our research team just released Flex-Forcing: a video generation method that lets a single model switch between generation methods at inference time. Right now there are two main approaches to video generation. Bidirectional diffusion models attend to every frame at once, holding structure well at the cost of speed. Autoregressive models generate frame by frame, so they stream fast and scale to long clips, but accumulate error and drift over time. Flex-Forcing trains a single model to do both, letting you choose from the range at inference based on your compute budget.
0:22

Our research team just released Flex-Forcing: a video generation method that lets a single model switch between generation methods at inference time. Right now there are two main approaches to video generation. Bidirectional diffusion models attend to every frame at once, holding structure well at the cost of speed. Autoregressive models generate frame by frame, so they stream fast and scale to long clips, but accumulate error and drift over time. Flex-Forcing trains a single model to do both, letting you choose from the range at inference based on your compute budget.

NVIDIA AI

31,412 Aufrufe • vor 3 Monaten

Real-time inference is a big challenge for VLAs. We’ve been working on a way to amortize inference delays in π0.5. Our new Real-Time Chunking (RTC) method speeds up π0.5 by allowing the robot to “think” while it’s moving, which makes it quite a bit faster! 🧵👇
0:29

Real-time inference is a big challenge for VLAs. We’ve been working on a way to amortize inference delays in π0.5. Our new Real-Time Chunking (RTC) method speeds up π0.5 by allowing the robot to “think” while it’s moving, which makes it quite a bit faster! 🧵👇

Sergey Levine

73,796 Aufrufe • vor 1 Jahr

(1/n) Time to unify your favorite visual generative models, VLMs, and simulators for controllable visual generation—Introducing a Product of Experts (PoE) framework for inference-time knowledge composition from heterogeneous models.
0:55

(1/n) Time to unify your favorite visual generative models, VLMs, and simulators for controllable visual generation—Introducing a Product of Experts (PoE) framework for inference-time knowledge composition from heterogeneous models.

Yunzhi Zhang

49,225 Aufrufe • vor 1 Jahr

Introducing DeepProve—Lagrange’s zkML Library—a breakthrough in verifiable AI inference. We can now verify AI decisions instead of blindly trusting black-box models. And we can do it up to 158x faster than ever before. The future of AI is ZK. The future of humanity is Lagrange: 🧵
2:10

Introducing DeepProve—Lagrange’s zkML Library—a breakthrough in verifiable AI inference. We can now verify AI decisions instead of blindly trusting black-box models. And we can do it up to 158x faster than ever before. The future of AI is ZK. The future of humanity is Lagrange: 🧵

LAGRANGE

1,740,979 Aufrufe • vor 1 Jahr