Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

📢 [New Preprint]European Conference on Computer Vision #ECCV2026 #ECCV2024 paper AMEGO: Active Memory from long EGOcentric videos Semantic-free representation of all interacting objects and locations in a long egocentric video. w/ 20K VQA benchmark that uses object crops to query interactions.

Dima Damen ✈️ #ECCV2026

11,497 subscribers

10,010 просмотров • 2 лет назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

My publicity chair retirement didn't last long 😉 I just joined the #ECCV2026 European Conference on Computer Vision #ECCV2026 publicity team 😅 New mask, same task. We are looking to grow our team and coverage, including adding a social media account in China. Stay tuned.

My publicity chair retirement didn't last long 😉 I just joined the #ECCV2026 European Conference on Computer Vision #ECCV2026 publicity team 😅 New mask, same task. We are looking to grow our team and coverage, including adding a social media account in China. Stay tuned.

Kosta Derpanis (sabbatical in Zurich)

20,291 просмотров • 1 год назад

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄

NVIDIA Robotics

23,137 просмотров • 2 месяцев назад

Chinese robotics company Astribot released their latest World-Action Model (WAM), Lumo-2. Technical breakdown: - based on a frozen 🥶 Qwen-3.5 4B VLM - trained in 3 progressive stages: 1. Action is aligned with latent world dynamics (an abstract representation of action). Real-world actions are anchored to physical constraints, while the latent space is guided to focus on motion-relevant changes. This bidirectional relationship makes the model physically grounded -> critical for a world model. 2. Action is aligned with vision and language. Reusing the vision backbone and action encoder from the frozen VLM, the authors add a custom vocabulary (for new actions), a semantic module, an action decoder, and an action projector. This aligns the (new) action representations with the (existing) vision-language semantic space. Most importantly: it builds a direct mapping from natural-language instructions to motor execution. 3. End-to-end training on language, video, and robot data. Only the new modules (everything outside the frozen backbone) are trained end-to-end across temporal reasoning, physical understanding, long-horizon, and dexterous manipulation. At the end of the day, Lumo-2 is not the best on benchmarks, but that's not the point. What's genuinely new: - a way to combine latent world modeling and action generation through progressive alignment - a physically-grounded latent dynamics space - it lifts performance on unseen objects using un-annotated human egocentric video + Vision Pro captures, no special transfer algorithm needed Why it matters: - the whole model is thin trainable adapters (semantic module, action decoder/projector) on a frozen 4B backbone (cheap) - that scale is suited for real-time embedded inference (~2.71× decode speedup, no accuracy loss) - its real moat is long-horizon execution, where the added temporal memory pays off far more than on any other task As a result, this robot can now make your latte (5x sped up video):

Chinese robotics company Astribot released their latest World-Action Model (WAM), Lumo-2. Technical breakdown: - based on a frozen 🥶 Qwen-3.5 4B VLM - trained in 3 progressive stages: 1. Action is aligned with latent world dynamics (an abstract representation of action). Real-world actions are anchored to physical constraints, while the latent space is guided to focus on motion-relevant changes. This bidirectional relationship makes the model physically grounded -> critical for a world model. 2. Action is aligned with vision and language. Reusing the vision backbone and action encoder from the frozen VLM, the authors add a custom vocabulary (for new actions), a semantic module, an action decoder, and an action projector. This aligns the (new) action representations with the (existing) vision-language semantic space. Most importantly: it builds a direct mapping from natural-language instructions to motor execution. 3. End-to-end training on language, video, and robot data. Only the new modules (everything outside the frozen backbone) are trained end-to-end across temporal reasoning, physical understanding, long-horizon, and dexterous manipulation. At the end of the day, Lumo-2 is not the best on benchmarks, but that's not the point. What's genuinely new: - a way to combine latent world modeling and action generation through progressive alignment - a physically-grounded latent dynamics space - it lifts performance on unseen objects using un-annotated human egocentric video + Vision Pro captures, no special transfer algorithm needed Why it matters: - the whole model is thin trainable adapters (semantic module, action decoder/projector) on a frozen 4B backbone (cheap) - that scale is suited for real-time embedded inference (~2.71× decode speedup, no accuracy loss) - its real moat is long-horizon execution, where the added temporal memory pays off far more than on any other task As a result, this robot can now make your latte (5x sped up video):

Léo

32,331 просмотров • 1 месяц назад

1/ 🧠Humans are the best robot data source! 2/ 👓Human egocentric video is rich in quantity, but poor in quality. 3/ Beyond scaling data, smarter representation and architecture matter just as much. 4/ Want an open-source framework to train your own learn-from-human-data robot policy? 🚀We introduce HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos⬇️ ✦ Zero-Shot Human-to-Robot Transfer ✦ Robot-Data-Free ✦ Just 30 min of data per task ✦ Collect by Anyone, Anytime, Anywhere ✦ Deploy on Any Robot, Any Camera, Any Environment ✦ Open-Source & Easy-to-Implement Let's squeeze every bit of signal out of human data! 🌐 Website: 📄 Paper: 💻 Code: 📹 Video: 🧵 1/n

1/ 🧠Humans are the best robot data source! 2/ 👓Human egocentric video is rich in quantity, but poor in quality. 3/ Beyond scaling data, smarter representation and architecture matter just as much. 4/ Want an open-source framework to train your own learn-from-human-data robot policy? 🚀We introduce HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos⬇️ ✦ Zero-Shot Human-to-Robot Transfer ✦ Robot-Data-Free ✦ Just 30 min of data per task ✦ Collect by Anyone, Anytime, Anywhere ✦ Deploy on Any Robot, Any Camera, Any Environment ✦ Open-Source & Easy-to-Implement Let's squeeze every bit of signal out of human data! 🌐 Website: 📄 Paper: 💻 Code: 📹 Video: 🧵 1/n

Zhi (Leo) Wang

111,063 просмотров • 3 месяцев назад

Infinite Photorealistic Worlds using Procedural Generation paper page: introduce Infinigen, a procedural generator of photorealistic 3D scenes of the natural world. Infinigen is entirely procedural: every asset, from shape to texture, is generated from scratch via randomized mathematical rules, using no external source and allowing infinite variation and composition. Infinigen offers broad coverage of objects and scenes in the natural world including plants, animals, terrains, and natural phenomena such as fire, cloud, rain, and snow. Infinigen can be used to generate unlimited, diverse training data for a wide range of computer vision tasks including object detection, semantic segmentation, optical flow, and 3D reconstruction. We expect Infinigen to be a useful resource for computer vision research and beyond.

Infinite Photorealistic Worlds using Procedural Generation paper page: introduce Infinigen, a procedural generator of photorealistic 3D scenes of the natural world. Infinigen is entirely procedural: every asset, from shape to texture, is generated from scratch via randomized mathematical rules, using no external source and allowing infinite variation and composition. Infinigen offers broad coverage of objects and scenes in the natural world including plants, animals, terrains, and natural phenomena such as fire, cloud, rain, and snow. Infinigen can be used to generate unlimited, diverse training data for a wide range of computer vision tasks including object detection, semantic segmentation, optical flow, and 3D reconstruction. We expect Infinigen to be a useful resource for computer vision research and beyond.

AK

275,530 просмотров • 3 лет назад

Got five papers accepted by #ECCV2024 European Conference on Computer Vision #ECCV2026 ! Huge thanks to all my collaborators! 😃 See you in Milan 🇮🇹 Summary of Selected Works (I made a fast-forward for them 😄) - [Shape Generation] Surf-D: Generating High-Quality Surfaces of Arbitrary Topologies Using Diffusion Models, ECCV 2024. - [Efficient Motion Generation] EMDM: Efficient Motion Diffusion Model for Fast, High-Quality Human Motion Generation, ECCV 2024. - [Controllable Motion Generation] TLControl: Trajectory and Language Control for Human Motion Synthesis, ECCV 2024. - [Avatar Generation] Disentangled Clothed Avatar Generation from Text Descriptions, ECCV 2024. Project Page: Surf-D: EMDM: TLControl: SOSMPL:

Zhiyang (Frank) Dou

18,192 просмотров • 2 лет назад

Thank you AK for sharing! Introducing our newest CVPR 2025 paper EgoLife: Towards Egocentric Life Assistant Homepage: Blog: Code: EgoLife is a project focused on building AI-powered egocentric life assistants. We invited 6 volunteers to live together for 7 days, capturing daily activities Project Aria @Meta smart glasses, which inspire 3K+ life-oriented, long-context Q&A pairs, and the development of EgoButler. 🔍 Meet EgoButler—our AI system that seamlessly integrates: 🧠 EgoGPT: A state-of-the-art vision-language model 📚 EgoRAG: A retrieval-augmented system for ultra-long-context queries EgoButler unlocks powerful capabilities, including: 🔹 Recall forgotten details from past experiences 🔹 Suggest plans by analyzing past events holistically 🔹 Reveal hidden habits users may overlook 🔹 Enhance relationships by tracking social interactions Checkout our code, dataset, and demo: Code: Data: Demo: Together Let's pioneering the next era of AI companionship! #AI #EgocentricAI #EgoLife

Thank you AK for sharing! Introducing our newest CVPR 2025 paper EgoLife: Towards Egocentric Life Assistant Homepage: Blog: Code: EgoLife is a project focused on building AI-powered egocentric life assistants. We invited 6 volunteers to live together for 7 days, capturing daily activities Project Aria @Meta smart glasses, which inspire 3K+ life-oriented, long-context Q&A pairs, and the development of EgoButler. 🔍 Meet EgoButler—our AI system that seamlessly integrates: 🧠 EgoGPT: A state-of-the-art vision-language model 📚 EgoRAG: A retrieval-augmented system for ultra-long-context queries EgoButler unlocks powerful capabilities, including: 🔹 Recall forgotten details from past experiences 🔹 Suggest plans by analyzing past events holistically 🔹 Reveal hidden habits users may overlook 🔹 Enhance relationships by tracking social interactions Checkout our code, dataset, and demo: Code: Data: Demo: Together Let's pioneering the next era of AI companionship! #AI #EgocentricAI #EgoLife

Jingkang (Jake) Yang

35,452 просмотров • 1 год назад

Tracking Everything Everywhere All at Once paper page: present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively.

Tracking Everything Everywhere All at Once paper page: present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively.

AK

280,570 просмотров • 3 лет назад

Happy to share that my PhD paper "Scene2Hap: Generating Scene-Wide Haptics for VR from Scene Context with Multimodal LLMs" has received a Best Paper Award (top < 1%) out of 6,730 submissions at ACM CHI (ACM CHI Conference), the most prestigious conference in the human-computer interaction field🙌 Scene2Hap is an LLM-centered system that automatically designs object-level vibrotactile feedback for entire VR scenes based on objects' semantic attributes (e.g., whether and how the object vibrates) and physical context (e.g., the object's density, spatial relationships). It then renders real-time haptic feedback across the scene, calculating vibration propagation based on LLM-inferred material properties. To the best of our knowledge, this is the first paper to address the problem itself: "designing haptic characteristics of a whole VR scene with one click." Thanks a lot to my co-first-author Easa, Sara Safaee, Paul Strohmeier, and my advisor Jürgen Steimle!

Happy to share that my PhD paper "Scene2Hap: Generating Scene-Wide Haptics for VR from Scene Context with Multimodal LLMs" has received a Best Paper Award (top < 1%) out of 6,730 submissions at ACM CHI (ACM CHI Conference), the most prestigious conference in the human-computer interaction field🙌 Scene2Hap is an LLM-centered system that automatically designs object-level vibrotactile feedback for entire VR scenes based on objects' semantic attributes (e.g., whether and how the object vibrates) and physical context (e.g., the object's density, spatial relationships). It then renders real-time haptic feedback across the scene, calculating vibration propagation based on LLM-inferred material properties. To the best of our knowledge, this is the first paper to address the problem itself: "designing haptic characteristics of a whole VR scene with one click." Thanks a lot to my co-first-author Easa, Sara Safaee, Paul Strohmeier, and my advisor Jürgen Steimle!

Arata Jingu

32,022 просмотров • 6 месяцев назад

What’s the point of a “helpful assistant” if you have to always tell it what to do next? In a new paper, we introduce a reasoning model that predicts what you’ll do next over long contexts (LongNAP 💤). We trained it on 1,800 hours of computer use from 20 users. 🧵

What’s the point of a “helpful assistant” if you have to always tell it what to do next? In a new paper, we introduce a reasoning model that predicts what you’ll do next over long contexts (LongNAP 💤). We trained it on 1,800 hours of computer use from 20 users. 🧵

Omar Shaikh

125,974 просмотров • 6 месяцев назад

New short course: Long-Term Agentic Memory with LangGraph. Learn to build an agent with long-term memory in this course developed in collaboration with taught by its Co-Founder and CEO, Harrison Chase! Personal assistance and productivity tasks have become important use cases for agents. An important feature of an AI assistant, such as a coding or calendar assistant, is its ability to keep improving over time from its experience. Agent memory is the key capability that enables this. To add memory to an agent, you must first figure out what to store and what to retrieve when it is time to use the information. Additionally, you’ll have to decide when to update the stored information. For example, you might update in each iteration loop of the agent or perform updates in the background, with a helper agent. In this course, you will learn a mental framework to build agents with long-term memory. You'll create a useful email assistant that can respond, ignore, and notify using writing, scheduling, and memory-management tools. You’ll develop your agent's memory by adding facts to its memory store, provide examples to learn the user's preferences, and optimize system prompts to evolve instructions based on previous responses. In detail, you’ll: - Learn how the three types of memory--semantic, episodic, and procedural–and the two update mechanisms–via hot path and in the background–apply to your agents. - Build an email agent with writing, scheduling, and availability tools, along with a router that triages incoming email and handles it accordingly by ignoring, responding, or notifying the user. - Add tools to your email agent that allow it to operate on semantic memory by learning facts about the user, storing them in a long-term memory store, and searching over them in future interactions. - Incorporate episodic memory, in the form of few-shot examples, in the triage step of your agents to help them learn and update user preferences. - Add procedural memory as system prompts, optimized with feedback to improve the instructions the agent follows. Learn how to approach memory in agents, and start building agents with long-term memory with LangGraph! Please sign up here:

New short course: Long-Term Agentic Memory with LangGraph. Learn to build an agent with long-term memory in this course developed in collaboration with taught by its Co-Founder and CEO, Harrison Chase! Personal assistance and productivity tasks have become important use cases for agents. An important feature of an AI assistant, such as a coding or calendar assistant, is its ability to keep improving over time from its experience. Agent memory is the key capability that enables this. To add memory to an agent, you must first figure out what to store and what to retrieve when it is time to use the information. Additionally, you’ll have to decide when to update the stored information. For example, you might update in each iteration loop of the agent or perform updates in the background, with a helper agent. In this course, you will learn a mental framework to build agents with long-term memory. You'll create a useful email assistant that can respond, ignore, and notify using writing, scheduling, and memory-management tools. You’ll develop your agent's memory by adding facts to its memory store, provide examples to learn the user's preferences, and optimize system prompts to evolve instructions based on previous responses. In detail, you’ll: - Learn how the three types of memory--semantic, episodic, and procedural–and the two update mechanisms–via hot path and in the background–apply to your agents. - Build an email agent with writing, scheduling, and availability tools, along with a router that triages incoming email and handles it accordingly by ignoring, responding, or notifying the user. - Add tools to your email agent that allow it to operate on semantic memory by learning facts about the user, storing them in a long-term memory store, and searching over them in future interactions. - Incorporate episodic memory, in the form of few-shot examples, in the triage step of your agents to help them learn and update user preferences. - Add procedural memory as system prompts, optimized with feedback to improve the instructions the agent follows. Learn how to approach memory in agents, and start building agents with long-term memory with LangGraph! Please sign up here:

Andrew Ng

131,965 просмотров • 1 год назад

Introducing Skywork Videos — Your All-in-One Production Workspace - From Clips to Full Story: Start with a 5s/8s clip. Build a long video production in one workflow. - Multi-Track Editing: Fine-tune and edit with full control, beyond one-click output. - Start from Templates: Access a library of templates to jumpstart your vision. Try Skywork Videos, turn your idea into a full video. (Watch this with sound on!)

Introducing Skywork Videos — Your All-in-One Production Workspace - From Clips to Full Story: Start with a 5s/8s clip. Build a long video production in one workflow. - Multi-Track Editing: Fine-tune and edit with full control, beyond one-click output. - Start from Templates: Access a library of templates to jumpstart your vision. Try Skywork Videos, turn your idea into a full video. (Watch this with sound on!)

Skywork

43,695 просмотров • 8 месяцев назад

Robot AI brains, aka Vision-Language-Action models, cannot adapt to new tasks as easily as LLMs like Gemini, ChatGPT, or Grok. LLMs can adapt quickly with their in-context learning (ICL) capabilities. But can we inject ICL abilities into a pre-trained VLA like pi0? Yes! Introducing RICL (Retraining for In-Context Learning), our Conference on Robot Learning (CoRL) 2025 paper. Our RICL-pi0 model can adapt to unseen objects, novel motions, and new scenes with just ICL and RAG (retrieval-augmented generation). RICL-pi0 also boosts performance on the long-tail of tasks. A quick 1 minute video summary:

Robot AI brains, aka Vision-Language-Action models, cannot adapt to new tasks as easily as LLMs like Gemini, ChatGPT, or Grok. LLMs can adapt quickly with their in-context learning (ICL) capabilities. But can we inject ICL abilities into a pre-trained VLA like pi0? Yes! Introducing RICL (Retraining for In-Context Learning), our Conference on Robot Learning (CoRL) 2025 paper. Our RICL-pi0 model can adapt to unseen objects, novel motions, and new scenes with just ICL and RAG (retrieval-augmented generation). RICL-pi0 also boosts performance on the long-tail of tasks. A quick 1 minute video summary:

Kaustubh Sridhar

52,868 просмотров • 1 год назад

🚨To all Pi pioneers, please see 📢 Dr. Fan Chengdiao's speech at the conference in Singapore 🗣 Pi has a grand vision and has taken a big step forward in innovation, aiming to truly realize its ecosystem. Coins that rely solely on speculation will eventually go to zero ️⃣. The #PiNetwork is a long-term project that combines AI and blockchain to allow ordinary people to quickly participate in ecosystem building.🚀

🚨To all Pi pioneers, please see 📢 Dr. Fan Chengdiao's speech at the conference in Singapore 🗣 Pi has a grand vision and has taken a big step forward in innovation, aiming to truly realize its ecosystem. Coins that rely solely on speculation will eventually go to zero ️⃣. The #PiNetwork is a long-term project that combines AI and blockchain to allow ordinary people to quickly participate in ecosystem building.🚀

PiNetwork DEX⚡️阿龙

13,087 просмотров • 7 месяцев назад

ViPE: Video Pose Engine for 3D Geometric Perception Contributions: • A robust and efficient framework, ViPE, for estimating camera parameters and dense depth from diverse, in-the-wild videos. • A system design that integrates the strengths of classical SLAM (efficiency, scalability) and learned models (robustness), with key improvements in efficiency, dynamic object handling, and depth quality over prior work. • A large-scale dataset of annotated videos, created using ViPE, to facilitate future research in 3D computer vision.

ViPE: Video Pose Engine for 3D Geometric Perception Contributions: • A robust and efficient framework, ViPE, for estimating camera parameters and dense depth from diverse, in-the-wild videos. • A system design that integrates the strengths of classical SLAM (efficiency, scalability) and learned models (robustness), with key improvements in efficiency, dynamic object handling, and depth quality over prior work. • A large-scale dataset of annotated videos, created using ViPE, to facilitate future research in 3D computer vision.

MrNeRF

42,644 просмотров • 1 год назад

Introducing Open-TeleVision: with Fully Autonomous policy video👇. We can conduct a long-horizon task with inserting 12 cans nonstop without any interruptions. We offer: 🤖 Highly precise and smooth bimanual manipulation. 📺 Active egocentric vision (with a moving neck) feedback. It is achieved by imitation learning from teleoperation: We propose a VR-based REAL-TIME teleoperation that streams the stereo video observation from the robot camera to the VR device. The robot neck moves as the human head moves, the robot hands move as the human hands move, offering the operator an intuitive experience as the human herself becomes the robot. The devils are all in the details, and how to implement things right: ✅How to perform IK/retargeting for smooth and precise control. ✅How to do all these and also stream stereo video without no latency, all in real time. We released our code here: Active head hardware design: 1/n

Introducing Open-TeleVision: with Fully Autonomous policy video👇. We can conduct a long-horizon task with inserting 12 cans nonstop without any interruptions. We offer: 🤖 Highly precise and smooth bimanual manipulation. 📺 Active egocentric vision (with a moving neck) feedback. It is achieved by imitation learning from teleoperation: We propose a VR-based REAL-TIME teleoperation that streams the stereo video observation from the robot camera to the VR device. The robot neck moves as the human head moves, the robot hands move as the human hands move, offering the operator an intuitive experience as the human herself becomes the robot. The devils are all in the details, and how to implement things right: ✅How to perform IK/retargeting for smooth and precise control. ✅How to do all these and also stream stereo video without no latency, all in real time. We released our code here: Active head hardware design: 1/n

Xiaolong Wang

25,572 просмотров • 2 лет назад

New paper from Apple - Sharp Monocular View Synthesis in Less than a Second Mescheder et al. @ Apple just released a very impressive paper (congrats! 🎉🥳). You give it an image and it generates a really great looking 3d Gaussian representation. Uses depth pro. It's really good. The model is about 3GB, and takes ~5-10s on my M1 Max per image. Single feedforward pass network. The video is using "Metal Splatter" to view the .ply from ml-sharp on Apple Vision Pro. Some really wow moments today trying it on different scenes.

New paper from Apple - Sharp Monocular View Synthesis in Less than a Second Mescheder et al. @ Apple just released a very impressive paper (congrats! 🎉🥳). You give it an image and it generates a really great looking 3d Gaussian representation. Uses depth pro. It's really good. The model is about 3GB, and takes ~5-10s on my M1 Max per image. Single feedforward pass network. The video is using "Metal Splatter" to view the .ply from ml-sharp on Apple Vision Pro. Some really wow moments today trying it on different scenes.

Tim Davison ᯅ

133,044 просмотров • 9 месяцев назад

Tired of high token costs for video workflows? Agentic video understanding gives Gemini the tools to dynamically search, scan, and inspect target video segments to achieve its goal. Building on agentic vision, Gemini now actively scans visual frames, audio, and transcripts instead of ingesting video at a static 1 FPS. This reduces token consumption by up to 88%, improves accuracy, and unlocks new ways to process long-form video. Tackle complex video workflows like: ▸ Sub-second retrieval: Catch split-second cuts missed at 1 FPS ▸ Long-form search: Query multi-hour videos without overflowing context ▸ Anomaly detection: Resample interesting windows at higher FPS ▸ Counting: Accurately track repeated actions and objects over time See how agentic video understanding transforms token efficiency in Gemini 3.7 Flash when compared with static video processing:

Tired of high token costs for video workflows? Agentic video understanding gives Gemini the tools to dynamically search, scan, and inspect target video segments to achieve its goal. Building on agentic vision, Gemini now actively scans visual frames, audio, and transcripts instead of ingesting video at a static 1 FPS. This reduces token consumption by up to 88%, improves accuracy, and unlocks new ways to process long-form video. Tackle complex video workflows like: ▸ Sub-second retrieval: Catch split-second cuts missed at 1 FPS ▸ Long-form search: Query multi-hour videos without overflowing context ▸ Anomaly detection: Resample interesting windows at higher FPS ▸ Counting: Accurately track repeated actions and objects over time See how agentic video understanding transforms token efficiency in Gemini 3.7 Flash when compared with static video processing:

Google for Developers

38,369 просмотров • 12 дней назад

Better data for robot hands! 🤲🏼 Embodied reasoning is the bottleneck now. It’s the data we are missing. Specifically, understanding what the robot's hands are doing throughout a task. Perceptron AI new Egocentric offering tracks both hands through an entire video instead of guessing from sampled frames. That matters because a lot happens between frames. Hand positions shift. Contacts change. The robot adjusts grip. You miss all that if you're sampling. 0.280 semantic end-to-end F1 score on WGO-Bench vs 0.158 from the best pipelines built on Gemini Robotics-ER 1.6. Training a robot to manipulate objects requires understanding every moment of hand-object interaction. If your training data has gaps: missing frames, missed contacts, unclear hand positions, the policy learns from incomplete information. You end up with robots that work in demos but fail in real tasks. Continuous hand tracking gives you the full picture. The policy learns from cleaner, more complete data. That compounds into better real-world performance. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Better data for robot hands! 🤲🏼 Embodied reasoning is the bottleneck now. It’s the data we are missing. Specifically, understanding what the robot's hands are doing throughout a task. Perceptron AI new Egocentric offering tracks both hands through an entire video instead of guessing from sampled frames. That matters because a lot happens between frames. Hand positions shift. Contacts change. The robot adjusts grip. You miss all that if you're sampling. 0.280 semantic end-to-end F1 score on WGO-Bench vs 0.158 from the best pipelines built on Gemini Robotics-ER 1.6. Training a robot to manipulate objects requires understanding every moment of hand-object interaction. If your training data has gaps: missing frames, missed contacts, unclear hand positions, the policy learns from incomplete information. You end up with robots that work in demos but fail in real tasks. Continuous hand tracking gives you the full picture. The policy learns from cleaner, more complete data. That compounds into better real-world performance. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

18,430 просмотров • 2 месяцев назад