We’re introducing ToyBlocks, an open simulated tabletop for controlled experiments in robot learning. It enables robotics researchers to experimentally control training data & models, run targeted evals, & trace failures to their causes. 🧵

Young children learn to plan & reason about space by playing with toy blocks, rebuilding a stack after it falls or turning one to find its letters. ToyBlocks uses the same setting—a simulated robot must reason about how blocks interact as it pushes, flips, & stacks them.

In ToyBlocks, users define high-level tasks like stacking a block sequence. ToyBlocks plans how to complete them via low-level skills. Simulating each plan produces a demonstration to train models that act from sensor observations, including camera images.

ToyBlocks lets researchers build training mixtures that emphasize particular skills or fill gaps in existing datasets. Our initial release contains 85K long-horizon demonstrations, each comprising about 6–12 skill demonstrations depending on the set.

ToyBlocks lets you test things like: 📷 Does depth input improve robot control compared with color images? 🧱 Can models stack four blocks without training on four-block stacks? 🔤 How do models perform at planning long tasks? Or design your own evals.

ToyBlocks includes a tool interface for connecting a frontier model so users can instruct the robot & try tasks in natural language. Through camera images & skill calls, researchers can study how a model searches, plans, & handles mistakes.

In our initial ToyBlocks tests, ManiFlow, π0.5, & MolmoAct 2 generalized poorly to new spatial configurations. Controlled experiments let us test changes to training & model design, then measure which techniques improve generalization.

Trained only in ToyBlocks’ virtual world, Molmo 2 + ManiFlow stacked lettered blocks to spell a word in 15/30 attempts on a physical robot, versus 111/200 in simulation. Simulation-trained models can successfully transfer to a physical robot, though more trials are needed.

We’re targeting a larger dataset, more skills, & broader benchmarks in a follow-up release, as well as integrated reinforcement learning training. Longer-term research will include two-arm manipulation & hands with force & touch sensors.

We hope ToyBlocks is a step toward robots that can apply what they learn to unfamiliar tasks and environments. We’re releasing simulation assets, task & demonstration generators, 2 trained models, training code, evaluation code, & validation sets. 📝 Blog:

Auto demo generation in LeRobot format, zero teleoperation needed? OH YES. And tracing failures back to the training data is the part everyone else skips. Downloading tonight :) This changes how fast I can iterate on my own robot experiments.

Does the 8-env quick start fit in 16GB? My 4080 is exactly Isaac Sim's minimum card 😂

robot learning getting a controlled tabletop is science admitting the dining table was a terrible production environment

ToyBlocks enables a useful failure test: hold out a block arrangement and compare one policy in sim and hardware, recording contact failures separately from task success. If sim pass rate rises but physical placement does not, synthetic volume alone isn't the fix. #Robotics

85,336 long-horizon demos (~144 million frames), and ManiFlow atomic skills about 80% in sim and 81% on a real Franka… zero-shot from sim. Love that they instrumented failure tracing. Most open stacks still film a demo reel and call it science. Does the hierarchical spelling task hold up once letters start face-down more often?

Tracing failures to their cause is what most robot learning evals are missing. Can you hold the planner and scenes fixed and only swap the demos? That would tell you whether a failure came from the data or the model, which is usually impossible to untangle on real robots.

the ability to trace failures back to causes is exactly what sim to real needs

tracing a failure back to its cause is the part i'd love to see. is that tied to specific training samples?

Could the oracle-swap evaluation report all four planner/controller pairings on identical initial states? The interaction matters: a learned planner may request states the learned controller rarely saw. Separate oracle scores alone could miss that coupling.

Controlled experiments like this could really speed up robot learning