How can we create a single navigation policy that works for different robots in diverse environments AND can reach navigation goals with high precision? Happy to share our new paper, "VAMOS: A Hierarchical Vision-Language-Action Model for Capability-Modulated and Steerable Navigation"! 📜 Paper: 🌐 Website:

Problem: Even with massive datasets, navigation generalists still fail to both generalize and stay performant. We argue this stems from naively mixing data from robots with very different capabilities. E.g., data from a quadruped navigating stairs is of limited use to a rover.

Our solution is VAMOS, a hierarchical VLA that decouples high-level planning from the robot's specific physical abilities. The key insight? High-level heuristics (like "avoid big obstacles") are general, while low-level affordances ("can my wheels handle this terrain?") are embodiment-specific.

VAMOS features a two-part hierarchy: 1️⃣ A high-level "generalist" planner: This VLM is trained on a large real-world, heterogeneous, and diverse dataset. It understands semantics and proposes multiple candidate paths as 2D image coordinates given an input image and a position goal encoded as text.

2️⃣ A low-level "specialist" affordance model: This lightweight model is trained for a specific robot embodiment (e.g., wheeled vs. legged) in safe, low-cost simulation. It evaluates the paths from the generalist and asks, "Can I actually do that?" This allows us to ground the general plan in the robot's real-world capabilities.

And the results? VAMOS sets a new state-of-the-art! In real-world navigation tests, it achieved a 90% average success rate across challenging indoor and outdoor courses, outperforming both traditional modular systems and other end-to-end learning methods.

This hierarchical design enables easy cross-embodiment navigation. We used the exact same high-level planner on both a legged (Spot) and a wheeled (Hound) robot, simply by swapping the specialist affordance model. In the example below, the wheeled robot correctly chooses the ramp, not the stairs.

Because our high-level planner is a Vision-Language Model, VAMOS is also easily steerable with natural language! We can simply append a text preference like, "...and stay to the right of the people" to guide the robot's path, making it more intuitive to specify preferences. Try our demo!

VAMOS is a step towards general-purpose navigation agents that can reason about the world both semantically and geometrically. Big thanks to my amazing co-authors at the @uwcse @sidharth_0_r, Daniel Gorbatov, Matt Schmittle, @RohanBaijal, Octi Zhang, Rosario Scalise, @sidharthtalia, Emma Romig, @Celso_M_de_Melo, Byron Boots, and @abhishekunique7 Check it out and let us know what you think! 📜 Paper: 🌐 Website:

Man how you navigate behind obstacles when the path is visualized on obstacles

@ChongZitaZhang we were very careful during training to project the pose of the robot onto the ground plane, so that the model learns to draw paths on the ground plane, and during inference we project the 2d image trajectories onto the ground, which worked surprisingly well :)

@ChongZitaZhang so even if the paths look as if they go over the obstacles, when reprojected to 3d, the paths actually go behind obstacles, letting us deal with occluded goals

Nice! Wondering how would you solve more generalized 3d cases or you would throw away the image interface.

Like when it's not near flat ground or consistent elevation

the "data" answer imo is to try something similar but instead of 2d trajectories, you predict 3d trajectories directly, but this may lose out on some of the base model pre-training (PaliGemma 2 is pretrained on some spatial tasks, as are Molmo, Qwen, etc these days), see EMMA from Waymo for example:

the "systems" answer could be to estimate the ground plane (e.g. TerrainNet: and instead of having a flat ground assumption, during inference you project the 2D paths to 3D by projecting the ray that the 2D coordinate specifies onto the predicted ground plane.

Neat!

Nice work @mateoguaman! We also predicted in image space ( for long horizon planning in self-driving, works well. Q: You also predict occluded pathway points (e.g. while going up stairs), did that hinder learning? Culling those points seemed to help.

oh awesome! love the website :) A: we decided to keep occluded points in the dataset, with the assumption that the data contains points projected on the ground, so that the model could predict paths behind obstacles towards occluded goals

Interesting! I was training a tiny model so that version (with occlusions) didn’t work as well for me, but maybe it’s time to train a VLM. Q: Do you predict the trajectories as text coordinates? Or this is using a special head/discrete token vocabulary?

yep exactly! our base model (PaliGemma 2) was pre-trained on some spatial tasks (e.g. object detection) and uses 2D location tokens in the format <locXXXX>, where XXXX is int(x_coord/width * 1024). So in order to make the most out of pre-training, we use coords in this format both to specify the goal as input and in the output trajectories. Other models (e.g. Gemini Robotics, Molmo, Qwen) now have even more spatial pre-training with pointing, etc. with different tokenization schemes, but I found that aligning the format of coordinates in our data with the base model's improved performance quite a bit

Cool stuff, I didn’t know that about PaliGemma. So, predict K discrete tokens each amounting to a 2D location — autoregressively? Also maybe there is potential for more efficiency. On “format” alignment, this ( can be thought of as reusing circuits.

yup, for 5 waypoints, we predict 10 tokens autoregressively (consecutive x-y pairs). Thanks for the link, that's very interesting!

Thanks for explaining! Last Q: What do you mean when you encode goal as text? Why not provide goal in the image space too?

The input to the model is an fpv image and a prompt of the form: "Navigate to x=<locXXXX>, y=<locXXXX>. Optional Preference". The main reason for providing the goal as text was to match the base model's pre-training on how spatial coordinates are encoded, but we found that newer sota models, e.g. GPT5, are quite good at reasoning directly about annotations in image space. For instance, we drew paths and goals directly in image space and asked GPT5 to describe the paths and goals as a way to obtain data to make our model steerable, and this worked pretty well! So maybe nowadays it'd be possible to reason directly in image space if you were to finetune a sota VLM such as Qwen 3-VL, but we haven't tried that

Interesting approach with VAMOS. Decoupling high-level planning from specific robot capabilities could address the performance issues seen with generalist models. Curious about the training process for that specialist affordance model.