When people talk about robotics, they usually talk about models, data, or hardware. Few people talk about the infrastructure that lets you iterate on all three quickly. Today we're publishing how we trained Dyna-2 on over 1,000,000 hours of egocentric video, repeatably. At this scale, most of what worked... at ten thousand hours did not hold up: • ingestion throughput was capped at 14,000 episode-hours per week — a million hours would have taken over a year • building a training manifest took 48 hours before a run could even start • reading a petabyte from cloud storage during training left GPUs exposed to latency and packet loss 🧵show more

Read our full technical blog for details: Our ingestion pipeline was once capped at 14,000 episode-hours per week — at that rate, a million hours would have taken over a year. We rewrote it around DAG decomposition, staggered starts, and bin-packed batches: • throughput scaled 14,000 → 440,000 episode-hours/week, a 31x improvement • one million hours now processes in under three weeks, down from ~16 months

At a million hours, building a training manifest took 48 hours before a run could even start. We moved manifest construction from a file-by-file walk to a single SQL query over a data warehouse: • manifest build time: ~48 hours → under 1 minute • manifest load time: 737s → 12.4s via memory-mapped columnar tables • memory footprint per node: 2,151 GB → 218 GB

Our training clusters don't always sit next to our data, and reading a petabyte-scale corpus straight from cloud storage exposes GPUs to egress latency and packet loss. We built an on-cluster NVMe caching layer on top of Alluxio: • time to read one petabyte (single thread): 57.9 days (cloud) → 5.8 days (cluster-local cache) • steady-state throughput: ~2 GB/s per node per read, keeping GPUs at 98% utilization

Feeding the GPUs is one problem; what runs on them is another. We rebuilt our Muon optimizer to shard state inside each node instead of across the job, avoiding the slower inter-node fabric as node count grows: • optimizer step time: ~3x faster at scale • job resilience: preflight node health gating + auto-restart from last checkpoint, new clusters stood up in days instead of weeks If working on these systems is interesting to you:

Infra ppl are the unsung heroes

🐐🐐

Interesting how the Dyna-2 world model architecture can sacrifice some data loader random sampling for improved throughput due to the prediction objective difference

Deep respect and gratitude for sharing these foundational infra designs. Model ideas are often the easy part now. Building fast, reliable infra for rapid iteration is the real bottleneck, and many of these challenges resonate deeply with what we are facing. Thanks for sharing!

Infra team for the win!

Love seeing this come together. being on the ground collecting this data and watching the infra scale to handle it has been such a cool ride.

Nobody talks about infrastructure. Nobody talks about who refills the cupboard either. The demo gets the applause; the plumbing only gets noticed the day it fails. You have described motherhood with extra cables.

most people focus on the models themselves but not the ops that let you spin up 10 different iterations a week. our team spent just as much time on the latter as the former.

These DYNA-2 writeups are exceptionally useful. 14K→440K episode-hours/week and 48h→<1m manifests make the lesson concrete: at this scale, data readiness is part of the training loop, not preprocessing. Closely aligned with what we’re building at Euler. Thanks for sharing.

Since the trainer dynamically picks between world sharding and intra node sharding based on the node count, where does that inflection point usually trigger for you guys? Does that threshold shift depending on which vendor's InfiniBand setup you're dealing with?

Infrastructure is everything!!

At 1M hours, robotics training becomes a data-engineering problem: 48h manifests and cloud I/O latency are the real constraints.

The 48 hour manifest build is the detail nobody warns you about.

🐐

At million-hour scale, “more data” becomes an infrastructure problem. The real challenge is keeping collection, ingestion, curation, validation, and training moving as one continuous pipeline.

@ch3njus Wow!

Repeatability is the part nobody screenshots but it's the difference between a demo and a pipeline you can run every Tuesday.

Insane improvements. Just comes to show how early we are, still finding new problems with scale. Imagine now digesting 1B+ hours.

At million-hour scale, Dyna’s bottleneck was not model architecture or GPU count—it was time-to-next-experiment. Their reported numbers: ingestion 14k → 440k episode-hours/week; first batch ~48h → <1m; 98% GPU utilization on a warm multi-node run. This is a systems lesson: scaling intelligence can become a data-movement and queueing problem before it is a compute problem. The compounding advantage is not one heroic training run; it is how many falsifiable experiments a team can complete per week. My question: as iteration speed becomes the moat, how do data lineage and training decisions remain auditable at the same pace?

@Rewkang 👆 Something feels different here

ETA on 1 billion hours?

1,000,000 hours makes data lineage the bottleneck, not training throughput

Let’s collaborate for FREE!🚀 Exciting project ahead! 💥

A million hours of egocentric video is scale. The question is whether any of it is held out, and whether the model fails on the slice that doesn't look like training.

@grok 这家公司直接卖机器人本体的，还是卖采集设备的，还是卖数据的？

Haha, did you see that insane! Completely outplayed them. Love this sport.

The infrastructure behind robotics training is often overlooked. Scaling from thousands to a million hours of video clearly requires an entirely different approach to data pipelines, storage, and training.