Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Introducing TraceVLA: a fully open-source Vision-Language-Action model reimagining spatial-temporal awareness: ✨ 3.5x gains on real robots, SOTA in simulation 💡 Fine-tunes on just 150K trajectories ⚡ Compact 4B model = 7B performance

cheryyun_l's profile picture

Yongyuan Liang

2,953 subscribers

39,603 görüntüleme • 1 yıl önce •via X (Twitter)

Bilim & Teknoloji
LIVE

Anya Rossi• Live Now

Private livecam show

11 Yorum

Yongyuan Liang profil fotoğrafı
Yongyuan Liang1 yıl önce

We introduce visual trace prompting:🔹Track robot's movement via point-tracking (Co-Tracker) 🔹Overlay traces on observations Model processes: 1️⃣ Original view (preserve full info) 2️⃣ View with traces as prompts A simple yet powerful technique to boost VLA's spatial understanding

Yongyuan Liang profil fotoğrafı
Yongyuan Liang1 yıl önce

TraceVLA in action: Watch it excel at diverse manipulation tasks on a real WidowX-250 robot! From soft-object handling to precision pick-and-place, TraceVLA consistently outperforms OpenVLA in both in-distribution and out-of-distribution tasks.

Yongyuan Liang profil fotoğrafı
Yongyuan Liang1 yıl önce

Superior simulation results: On Google’s SimplerEnv robot tasks, TraceVLA outshines OpenVLA across all metrics in both 7B and 4B versions! 🚀 20% boost in handling: ▪️ Camera changes ▪️ Distractors ▪️ Varied visual backgrounds

Yongyuan Liang profil fotoğrafı
Yongyuan Liang1 yıl önce

Efficient and lightweight: 🔸 TraceVLA requires <10GB memory on 8 H100 GPUs 🔸 Adds only 0.036s per timestep A powerful VLA upgrade with minimal overhead!

Yongyuan Liang profil fotoğrafı
Yongyuan Liang1 yıl önce

Available resources include: ▫️7B TraceVLA checkpoints ▫️Lightweight 4B Phi3V-OpenVLA & TraceVLA models ▫️Fine-tuned TraceVLA models 💻 Code: 🤗 Models: Try TraceVLA family models today!

Yongyuan Liang profil fotoğrafı
Yongyuan Liang1 yıl önce

Check out our project page: ArXiv: Joint work with @ruijie_zheng12 @ShuaiyiH @JianfengGao0217 @haldaume3 @Andrey__Kolobov @furongh @jw2yang4ai

Yang profil fotoğrafı
Yang2 yıl önce

Want to learn how practical AI skills and automations for your business and work? Check out our 50+ step-by-step video tutorials 100% FREE 20+ hours of Ai and Automation goodness absolutely free 🥳

Mu Cai @ Industry Job Market profil fotoğrafı
Mu Cai @ Industry Job Market1 yıl önce

Congratulations! Really interesting work on applying visual prompts on VLA tasks!

Yongyuan Liang profil fotoğrafı
Yongyuan Liang1 yıl önce

Thanks!!!

Dmytro Kuzmenko profil fotoğrafı
Dmytro Kuzmenko1 yıl önce

thank you very much for sharing, great idea and rather impressive results!

Ray | AI marketer - Social Media Assistant profil fotoğrafı
Ray | AI marketer - Social Media Assistant1 yıl önce

real-time engagement is key. we help brands connect with their audience 24/7, no burnout.

Benzer Videolar

✨ Introducing 𝐎𝐩𝐞𝐧𝐕𝐋𝐀 — an open-source vision-language-action model for robotics! 👐 - SOTA generalist policy - 7B params - outperforms Octo, RT-2-X on zero-shot evals 🦾 - trained on 970k episodes from OpenX dataset 🤖 - fully open: model/code/data all online 🤗 🧵👇
1:11

✨ Introducing 𝐎𝐩𝐞𝐧𝐕𝐋𝐀 — an open-source vision-language-action model for robotics! 👐 - SOTA generalist policy - 7B params - outperforms Octo, RT-2-X on zero-shot evals 🦾 - trained on 970k episodes from OpenX dataset 🤖 - fully open: model/code/data all online 🤗 🧵👇

Moo Jin Kim

227,218 görüntüleme • 2 yıl önce

🚀 MiniCPM enters the physical world — enabling robots to understand, remember, and act. We open-source MiniCPM-Robot, our first embodied AI model series, including: 🤖 MiniCPM-RobotManip — a 1.5B general-purpose Vision-Language-Action (VLA) model for robotic manipulation. 🐕 MiniCPM-RobotTrack — a compact model for real-world target tracking. ⚡ PhyAI — a high-performance inference framework built for embodied models. Together, they bring efficient, practical, and open embodied intelligence closer to real-world robots. ⭐ GitHub: 🤗 MiniCPM-RobotManip: 🤗 MiniCPM-RobotTrack:
1:17

🚀 MiniCPM enters the physical world — enabling robots to understand, remember, and act. We open-source MiniCPM-Robot, our first embodied AI model series, including: 🤖 MiniCPM-RobotManip — a 1.5B general-purpose Vision-Language-Action (VLA) model for robotic manipulation. 🐕 MiniCPM-RobotTrack — a compact model for real-world target tracking. ⚡ PhyAI — a high-performance inference framework built for embodied models. Together, they bring efficient, practical, and open embodied intelligence closer to real-world robots. ⭐ GitHub: 🤗 MiniCPM-RobotManip: 🤗 MiniCPM-RobotTrack:

OpenBMB

331,849 görüntüleme • 3 ay önce

DynamicVLA A compact 0.4B Vision-Language-Action model that finally lets robots manipulate *moving* objects in real-time, closing the perception-execution gap with Continuous Inference and Latent-aware Action Streaming.
0:23

DynamicVLA A compact 0.4B Vision-Language-Action model that finally lets robots manipulate *moving* objects in real-time, closing the perception-execution gap with Continuous Inference and Latent-aware Action Streaming.

DailyPapers

16,357 görüntüleme • 8 ay önce

Everyone is sleeping on this new OCR model! dots-ocr is a new 1.7B vision-language model that achieves SOTA performance on multilingual document parsing. - Supports 100+ languages - Works with both images and PDFs - Handles text, tables, formulas seamlessly 100% open-source.
0:33

Everyone is sleeping on this new OCR model! dots-ocr is a new 1.7B vision-language model that achieves SOTA performance on multilingual document parsing. - Supports 100+ languages - Works with both images and PDFs - Handles text, tables, formulas seamlessly 100% open-source.

Akshay 🚀

252,443 görüntüleme • 1 yıl önce

Everyone is sleeping on this new OCR model! - 85.9% (sota) on olmocr bench - 90+ language support w/benchmarks - 4B model (down from 9B) - Full layout information - Extracts + captions images and diagrams - Strong handwriting, math, form, table support 100% open-source.
0:36

Everyone is sleeping on this new OCR model! - 85.9% (sota) on olmocr bench - 90+ language support w/benchmarks - 4B model (down from 9B) - Full layout information - Extracts + captions images and diagrams - Strong handwriting, math, form, table support 100% open-source.

Akshay 🚀

169,351 görüntüleme • 7 ay önce

Apple just released and open-sourced FastVLM! FastVLM is a lightning-fast vision-language model that combines rapid image and text understanding with efficient on-device performance. 100% Open Source
0:21

Apple just released and open-sourced FastVLM! FastVLM is a lightning-fast vision-language model that combines rapid image and text understanding with efficient on-device performance. 100% Open Source

Sumanth

43,734 görüntüleme • 1 yıl önce

📢 First contact between a frontier model and robots! Gemini Robotics is a SOTA generalist Vision-Language-Action model bringing frontier model intelligence to the physical world. It's an extremely capable model enabling dexterous, steerable, and general robot control. 🧵⬇️
0:31

📢 First contact between a frontier model and robots! Gemini Robotics is a SOTA generalist Vision-Language-Action model bringing frontier model intelligence to the physical world. It's an extremely capable model enabling dexterous, steerable, and general robot control. 🧵⬇️

Ted Xiao

152,554 görüntüleme • 1 yıl önce

🚀 First step to unlocking Generalist Robots! Introducing 🤖LAPA🤖, a new SOTA open-sourced 7B VLA pretrained without using action labels. 💪SOTA VLA trained with Open X (outperforming OpenVLA on cross and multi embodiment) 😯LAPA enables learning from human videos, unlocking potential for robotic foundation model ❗Over 30x pretraining efficiency for VLA training 🤗Code and checkpoints are all open-sourced!
0:51

🚀 First step to unlocking Generalist Robots! Introducing 🤖LAPA🤖, a new SOTA open-sourced 7B VLA pretrained without using action labels. 💪SOTA VLA trained with Open X (outperforming OpenVLA on cross and multi embodiment) 😯LAPA enables learning from human videos, unlocking potential for robotic foundation model ❗Over 30x pretraining efficiency for VLA training 🤗Code and checkpoints are all open-sourced!

Seonghyeon Ye

33,356 görüntüleme • 2 yıl önce

Molmo by Ai2 - Open source SoTA Multimodal (Vision) Language model, beating Claude 3.5 Sonnet, GPT4V and comparable to GPT4o 🔥 They release four model checkpoints: 1. MolmoE-1B, a mixture of experts model with 1B (active) 7B (total) 2. Molmo-7B-O, most open 7B model 3. Molmo-7B-D, demo model 4. Molmo-72B, best model System Architecture > Input: Multi-scale, multi-crop images generated from the original image. > Vision Encoder: OpenAI's ViT-L/14 336px CLIP model, a powerful ViT, encodes images into vision tokens. > Connector: MLP projects tokens to LLM input space, followed by pooling for dimensionality reduction. > LLM: Decoder-only Transformer, various options (OLMo, OLMoE, Qwen2, Mistral, Gemma2, Phi) with diverse scales and openness. Model Variants > Vision Encoder: Consistent ViT-L/14 CLIP model across variants. > LLM: OLMo-7B-1024, OLMoE-1B-7B-0924, Qwen2 (7B, 72B), Mistral 7B, Gemma2 9B, Phi 3 Medium, offering different capacities and openness levels. Training Strategy > Stage 1: Multimodal pre-training for caption generation with new captioning data. > Stage 2: Supervised fine-tuning on a dataset mixture, updating all parameters. > No RLHF involved, Learning rates adjusted based on component types and pre-training status. > All the weights are available on Hugging Face Hub 🤗 > Compatible with Transformers (Remote Code) Kudos Ai2 for such a brilliant and open work! 🐐 Video credits: Allen AI YT Channel
2:15

Molmo by Ai2 - Open source SoTA Multimodal (Vision) Language model, beating Claude 3.5 Sonnet, GPT4V and comparable to GPT4o 🔥 They release four model checkpoints: 1. MolmoE-1B, a mixture of experts model with 1B (active) 7B (total) 2. Molmo-7B-O, most open 7B model 3. Molmo-7B-D, demo model 4. Molmo-72B, best model System Architecture > Input: Multi-scale, multi-crop images generated from the original image. > Vision Encoder: OpenAI's ViT-L/14 336px CLIP model, a powerful ViT, encodes images into vision tokens. > Connector: MLP projects tokens to LLM input space, followed by pooling for dimensionality reduction. > LLM: Decoder-only Transformer, various options (OLMo, OLMoE, Qwen2, Mistral, Gemma2, Phi) with diverse scales and openness. Model Variants > Vision Encoder: Consistent ViT-L/14 CLIP model across variants. > LLM: OLMo-7B-1024, OLMoE-1B-7B-0924, Qwen2 (7B, 72B), Mistral 7B, Gemma2 9B, Phi 3 Medium, offering different capacities and openness levels. Training Strategy > Stage 1: Multimodal pre-training for caption generation with new captioning data. > Stage 2: Supervised fine-tuning on a dataset mixture, updating all parameters. > No RLHF involved, Learning rates adjusted based on component types and pre-training status. > All the weights are available on Hugging Face Hub 🤗 > Compatible with Transformers (Remote Code) Kudos Ai2 for such a brilliant and open work! 🐐 Video credits: Allen AI YT Channel

Vaibhav (VB) Srivastav

80,474 görüntüleme • 2 yıl önce

Spatial Intelligence just went Open Source. 🌐 Introducing InSpatio-WorldFM — the world’s first open-source real-time interactive 3D world model. Last October, World Labs's RTFM showed the vision. Today, we’re bringing that idea into the open for everyone. ✅Efficient, multi-view consistent spatial reasoning ✅Real-time interactive exploration ✅Open-source weights & code Stop watching demos. Start building now. 👇
0:54

Spatial Intelligence just went Open Source. 🌐 Introducing InSpatio-WorldFM — the world’s first open-source real-time interactive 3D world model. Last October, World Labs's RTFM showed the vision. Today, we’re bringing that idea into the open for everyone. ✅Efficient, multi-view consistent spatial reasoning ✅Real-time interactive exploration ✅Open-source weights & code Stop watching demos. Start building now. 👇

InSpatio｜影溯

37,454 görüntüleme • 7 ay önce

CMU Vision-Language-Autonomy update: The team just released SORT3D, the first general spatial relation toolbox for autonomous vision-language navigation that is fully integrated into real-robot systems! 🤖👀 Simulation and real-robot data is provided!:
0:45

CMU Vision-Language-Autonomy update: The team just released SORT3D, the first general spatial relation toolbox for autonomous vision-language navigation that is fully integrated into real-robot systems! 🤖👀 Simulation and real-robot data is provided!:

CMU Robotics Institute

18,592 görüntüleme • 1 yıl önce

Introducing Meta Perception Language Model (PLM): an open & reproducible vision-language model tackling challenging visual tasks. Learn more about how PLM can help the open source community build more capable computer vision systems. Read the research paper, and download the code and dataset:
1:11

Introducing Meta Perception Language Model (PLM): an open & reproducible vision-language model tackling challenging visual tasks. Learn more about how PLM can help the open source community build more capable computer vision systems. Read the research paper, and download the code and dataset:

AI at Meta

94,534 görüntüleme • 1 yıl önce

SpatialLM just dropped on Hugging Face Large Language Model for Spatial Understanding
0:50

SpatialLM just dropped on Hugging Face Large Language Model for Spatial Understanding

AK

674,718 görüntüleme • 1 yıl önce

Cohere transcribe Sota open source transcription model running in the browser :) Weights on Hugging Face link below
0:36

Cohere transcribe Sota open source transcription model running in the browser :) Weights on Hugging Face link below

Nick Frosst

191,651 görüntüleme • 7 ay önce

No words. Just wow. LingBot-World: A playable open-source world model built on Wan2.2+Qwen3-VL-2B; - real-time interactive simulation at 720p@16fps, <1s latency; - minute-long contextual memory. - open source!
0:56

No words. Just wow. LingBot-World: A playable open-source world model built on Wan2.2+Qwen3-VL-2B; - real-time interactive simulation at 720p@16fps, <1s latency; - minute-long contextual memory. - open source!

Wildminder

36,918 görüntüleme • 8 ay önce

Introducing OFT—an Optimized Fine-Tuning recipe for VLAs! Fine-tuning OpenVLA w/ OFT, we see: -25-50x faster inference ⚡️ -SOTA 97.1% avg SR in LIBERO 💪 -high-freq control w/ 7B model on real bimanual robot -outperforms π₀, RDT-1B, DiT Policy, MDT, Diffusion Policy, ACT 🧵👇
0:25

Introducing OFT—an Optimized Fine-Tuning recipe for VLAs! Fine-tuning OpenVLA w/ OFT, we see: -25-50x faster inference ⚡️ -SOTA 97.1% avg SR in LIBERO 💪 -high-freq control w/ 7B model on real bimanual robot -outperforms π₀, RDT-1B, DiT Policy, MDT, Diffusion Policy, ACT 🧵👇

Moo Jin Kim

84,359 görüntüleme • 1 yıl önce

Sheepy-T: A fully open-source instruction-tuned language model based on GPT-J running locally on iPhone 14. Reply for beta access via TestFlight.
0:25

Sheepy-T: A fully open-source instruction-tuned language model based on GPT-J running locally on iPhone 14. Reply for beta access via TestFlight.

Kevin Kwok

102,376 görüntüleme • 3 yıl önce

Introducing Jan-Code-4B 💻 A compact coding model tuned for practical day-to-day tasks. Generation, refactors, debugging, tests — all runnable locally in Jan. Download Jan: Model:
0:20

Introducing Jan-Code-4B 💻 A compact coding model tuned for practical day-to-day tasks. Generation, refactors, debugging, tests — all runnable locally in Jan. Download Jan: Model:

👋 Jan

112,380 görüntüleme • 7 ay önce

Aging has a language. 🧬 We’re teaching AI to read it. On the cover of Cell: compact Longevity-LLMs, jointly trained by Insilico Medicine and Liquid AI, demonstrate SOTA performance on LongevityBench. Powered by these models, Longevity Claw nominated 328 candidate gene targets for aging research. Small model. Big questions. Open source. 🧵👇 #insilicoSOTAFM
0:32

Aging has a language. 🧬 We’re teaching AI to read it. On the cover of Cell: compact Longevity-LLMs, jointly trained by Insilico Medicine and Liquid AI, demonstrate SOTA performance on LongevityBench. Powered by these models, Longevity Claw nominated 328 candidate gene targets for aging research. Small model. Big questions. Open source. 🧵👇 #insilicoSOTAFM

Insilico Medicine

17,357 görüntüleme • 1 ay önce