Загрузка видео...

Не удалось загрузить видео

На главную

Announcing Robostral Navigate, our first model for embodied navigation: an 8B robotics navigation model that guides robots to autonomously perform tasks specified with natural language. Single RGB camera. State-of-the-art on R2R-CE.

296,911 просмотров • 3 месяцев назад •via X (Twitter)

Комментарии: 35

Фото профиля Mistral AI
Mistral AI3 месяцев назад

No LiDAR. No depth sensors. No camera rig. Where leading systems lean on depth or multiple cameras, Robostral Navigate works from one ordinary RGB camera, and still comes out ahead.

Фото профиля Mistral AI
Mistral AI3 месяцев назад

76.6% success on R2R-CE validation unseen (79.4% on validation seen), the benchmark for following instructions in previously unseen environments. It beats the best single-camera approach by 9.7 points while using far less sensing.

Фото профиля Mistral AI
Mistral AI3 месяцев назад

Trained entirely in simulation: ~400,000 trajectories across 6,000 scenes. A prefix-caching recipe cuts training tokens by 22×, turning months-long runs into days. Online RL (CISPO) pushes success rates higher still.

Фото профиля Mistral AI
Mistral AI3 месяцев назад

It runs on wheeled, legged, and flying robots and generalizes across sizes, unlocking delivery, logistics, manufacturing, and hospitality. Read more:

Фото профиля Lucas Beyer (bl16)
Lucas Beyer (bl16)3 месяцев назад

@fouriergalois This is fun and all but why would you spread yourself thin like that?

Фото профиля Pinkman
Pinkman3 месяцев назад

single rgb camera doing the whole navigation job is the impressive part here

Фото профиля Mihailo Jovanovic
Mihailo Jovanovic3 месяцев назад

@Scobleizer we needed le chaton fat, but we got metal guy 😭

Фото профиля Ekstra.ai
Ekstra.ai3 месяцев назад

We can help robostral see around corners without adding any more hardware or cost.

Фото профиля anthod
anthod3 месяцев назад

Single RGB camera robot navigation is wild, my vacuum still loses to chair legs.

Фото профиля Andrea Del Bene 🇪🇺
Andrea Del Bene 🇪🇺3 месяцев назад

I thought Daft Punk split up few years ago! 😍

Фото профиля Anastasios-Antonios Toulkeridis
Anastasios-Antonios Toulkeridis3 месяцев назад

i swear Mistral AI is the company that makes Europe relevant and proud. Keep at it!

Фото профиля Apollo
Apollo3 месяцев назад

say what you have about Mistral, but their model names are top notch. Devstral, Magistral, Pixtral , Codestral and now Robstral!

Фото профиля ɴᴏᴠᴀ
ɴᴏᴠᴀ3 месяцев назад

this is so cool, most robotics demos require sensor arrays that costs more than the robot, this is changing the game

Фото профиля Pindo 🇪🇺
Pindo 🇪🇺3 месяцев назад

Sick

Фото профиля adit
adit3 месяцев назад

super sick

Фото профиля 🔗Don Okko
🔗Don Okko3 месяцев назад

Cool

Фото профиля Liz Zhang
Liz Zhang3 месяцев назад

8B and a single RGB camera is the headline: no depth sensor, no lidar, just learned priors. That puts navigation in reach of any robot with a webcam. Curious how R2R-CE transfers to messy real homes versus staged evals.

Фото профиля ExileAI
ExileAI3 месяцев назад

Mistral AI, does ‘Robostral Navigate’ have training data for Rov’s? If not i might know someone that can get you plenty. Getting ready to have Hermes Agent interface with this Trident with your model for total control. Let me know, DM’s are open. @Teknium @NousResearch @mr_r0b0t

Фото профиля clankr
clankr3 месяцев назад

It's exciting to see Mistral entering the Physical AI race. Starting from the ground up by solving navigation, rather than jumping straight to flashy manipulation tasks, might actually be the more important approach.

Фото профиля Back LLM Radar|Syl
Back LLM Radar|Syl3 месяцев назад

Embodied navigation is where agent reliability gets very concrete. For robots, I’d watch not only task success, but recovery after a bad step, latency variance, and the fallback path when natural-language intent is underspecified. Benchmarks matter; operational behavior matters more.

Фото профиля Evolution Plus AI
Evolution Plus AI3 месяцев назад

I wonder how much data Roomba collected to help build this logic. 😂

Фото профиля SuperFreshTT
SuperFreshTT3 месяцев назад

@droneforge will you guys be testing this model out, flying robots are listed as compatible.

Фото профиля Alex Freitas
Alex Freitas3 месяцев назад

Single-camera NL navigation is a clean product surface. The hard part after the paper: recovery evals under lighting shift, occlusion, and wrong-turn recovery — demos rarely show the failure loop. #Robotics #PhysicalAI

Фото профиля luis
luis3 месяцев назад

@Presidentlin

Фото профиля #INTERNETofAGENTS
#INTERNETofAGENTS3 месяцев назад

#zeitgeist

Фото профиля Gerard Sans | Axiom 🇬🇧
Gerard Sans | Axiom 🇬🇧3 месяцев назад

wait what

Фото профиля Mr.X
Mr.X3 месяцев назад

It's time to create your own comma

Фото профиля ᴘʀᴏᴍᴘᴛʏᴢᴇ 🤖
ᴘʀᴏᴍᴘᴛʏᴢᴇ 🤖3 месяцев назад

Impressive milestone! The ability to leverage natural language for navigation opens up new avenues for human-robot interaction. How do you see this impacting real-world applications in complex environments?

Фото профиля Felipe Barbosa
Felipe Barbosa3 месяцев назад

@GuillaumeLample 👏👏👏

Фото профиля Back LLM Radar|Syl
Back LLM Radar|Syl3 месяцев назад

Interesting because embodied navigation turns model reliability into physical-world reliability. For production robotics, I’d watch not just benchmark score, but route-failure recovery, p95 action latency, camera edge cases, and explicit stop/escalation rules.

Фото профиля Musedwell
Musedwell3 месяцев назад

Robostral Navigate听起来很有前景,但关键在于实际部署时的可靠性和稳定性。在复杂多变的真实环境中,模型能否保持高效导航是需要重点关注的问题。

Фото профиля Zohaib Ai
Zohaib Ai3 месяцев назад

Natural language control with a single RGB camera is impressive. Excited to see where this goes.

Фото профиля Dipankar Sarkar
Dipankar Sarkar3 месяцев назад

single camera gets the headline but 8B is the sleeper. small enough to actually run on the robot. a nav loop can't wait on a cloud round trip, one tail-latency spike and it's already into the chair leg. on device isn't the cheap path here, it's the only one that closes the loop in time.

Фото профиля Perceptron.cloud
Perceptron.cloud3 месяцев назад

waiting for the FAT version

Фото профиля Ptaah the שדי
Ptaah the שדי3 месяцев назад

Ok google let's try that hand

Похожие видео

Mistral AI Releases Robostral Navigate: An 8B Model Enabling Robots to Navigate Complex Environments Hitting 76.6% on R2R-CE With One RGB Camera. No LiDAR. No depth sensor. No multi-camera rig. Here's how it works. 👇 1. Pointing, not metric commands The model predicts the pixel coordinates of the next target in the camera view, plus the arrival orientation. Working in pixel space keeps it robust to camera intrinsics and world scale. When the target leaves the frame, it falls back to local displacements ("2m forward, 1.5m left, turn 25°"). 2. Grounding-first No open-source VLM base. It starts from Mistral's grounding model (pointing, counting, localization). Navigation emerges once the model knows where things are. → ~400,000 trajectories across 6,000 simulated scenes 3. Prefix-caching for training A tree-based attention mask packs a full episode into one sequence — all time steps in a single forward pass. → 22× fewer training tokens; months of training done in days 4. Online RL on top After supervised training, CISPO adds trial-and-error learning to fight distribution shift from behavior cloning. → +3.2% success rate from RL alone 5. The numbers (R2R-CE, Matterport3D) → 76.6% success on validation unseen → +9.7 pts over best single-camera approach → +4.5 pts over best depth/multi-camera system The key takeaway: state-of-the-art continuous VLN without a sensor stack — grounding-init, pixel-space actions, prefix-cached SFT, and online RL, on one RGB camera. Full analysis: Technical details: Mistral AI Mistral AI for Developers

Marktechpost AI

39,955 просмотров • 3 месяцев назад

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,798 просмотров • 3 лет назад