Загрузка видео...

Не удалось загрузить видео

На главную

Planning with the views: Can VLMs predict how each camera move changes the view, and plan many such moves ahead? We introduce ViewSuite with 6 DoF camera control and ~165K task instances, testing: Path-to-View View-to-Path Interactive View Planning A sharp Planning Gap emerges: + can roughly "track" how camera...

ManlingLi_'s profile picture

Manling Li

11,509 subscribers

62,612 просмотров • 2 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)
0:43

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)

Wenhu Chen

82,829 просмотров • 1 год назад

Split layout framework update. Since the last video: focus views animate, corner drags can move both axes, cross-window drag and drop with animation, all animations backed by CoreAnimation, and full accessibility framework integration (e.g. compat with VoiceOver). The focus view work was the hardest and most complicated. We allow caller-provided SwiftUI views for the focus style, but have to drop that back to a native AppKit/UIKit hosted view for smooth CoreAnimation integration with our other animations so they move in lock-step with our other stuff. Lot of work to get that to work, but I think it turned out great. The caller lives in easy SwiftUI land we handle that complexity. The animation timings and curves need to be tweaked. I think some animations are too long or don't feel right. But thats just parameterization at this point...
0:23

Split layout framework update. Since the last video: focus views animate, corner drags can move both axes, cross-window drag and drop with animation, all animations backed by CoreAnimation, and full accessibility framework integration (e.g. compat with VoiceOver). The focus view work was the hardest and most complicated. We allow caller-provided SwiftUI views for the focus style, but have to drop that back to a native AppKit/UIKit hosted view for smooth CoreAnimation integration with our other animations so they move in lock-step with our other stuff. Lot of work to get that to work, but I think it turned out great. The caller lives in easy SwiftUI land we handle that complexity. The animation timings and curves need to be tweaked. I think some animations are too long or don't feel right. But thats just parameterization at this point...

Mitchell Hashimoto

104,095 просмотров • 2 месяцев назад

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.
0:50

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,798 просмотров • 3 лет назад

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency
0:21

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency

AK

31,997 просмотров • 2 лет назад

You can now operate robots by just thinking about it. With your brain signals. WOW. This robot system from Stanford has so much sci-fi vibe and wild implications that I don't even know where to start. NOIR decodes the EEG signal from your head into a library of robot skills. It is demonstrated on 20 household activities, such as cooking Sukiyaki, ironing clothes, grating cheese, playing Tic-Tac-Toe, and even petting a robot dog! NOIR learns to predict your intended goals in advance, so that your thinking efforts (literally) can be reduced to a minimum. It works with both adults and children as young as five years old. Imagine scaling up this system to Neuralink + Humanoid robots like Tesla Optimus in the future! There's now a decent probability that we can see the movie Avatar become a reality in our lifetime. Website: Paper: This work is from Stanford Vision Lab, my PhD home. The project is led by Ruohan Zhang and advised by Fei-Fei Li Jiajun Wu. Congrats on the phenomenal idea!
0:33

You can now operate robots by just thinking about it. With your brain signals. WOW. This robot system from Stanford has so much sci-fi vibe and wild implications that I don't even know where to start. NOIR decodes the EEG signal from your head into a library of robot skills. It is demonstrated on 20 household activities, such as cooking Sukiyaki, ironing clothes, grating cheese, playing Tic-Tac-Toe, and even petting a robot dog! NOIR learns to predict your intended goals in advance, so that your thinking efforts (literally) can be reduced to a minimum. It works with both adults and children as young as five years old. Imagine scaling up this system to Neuralink + Humanoid robots like Tesla Optimus in the future! There's now a decent probability that we can see the movie Avatar become a reality in our lifetime. Website: Paper: This work is from Stanford Vision Lab, my PhD home. The project is led by Ruohan Zhang and advised by Fei-Fei Li Jiajun Wu. Congrats on the phenomenal idea!

Jim Fan

378,309 просмотров • 2 лет назад

Punchline: World models == VQA (about the future)! Planning with world models can be powerful for robotics/control. But most world models are video generators trained to predict everything, including irrelevant pixels and distractions. We ask - what if a world model only predicted the semantic information necessary for decision-making? Introducing Semantic World Models (SWM). Given an observation and an action sequence, SWMs cast modeling as answering textual questions about the future outcome resulting from the actions. Recasting world modeling as a VQA problem lets us directly leverage the pretrained knowledge and machinery of VLMs for generalizable modeling. We had a lot of fun thinking about how this work helps connect these two seemingly very different fields of study - VLMs and world models! 🧵(1/6) Paper: Fun demo:
0:49

Punchline: World models == VQA (about the future)! Planning with world models can be powerful for robotics/control. But most world models are video generators trained to predict everything, including irrelevant pixels and distractions. We ask - what if a world model only predicted the semantic information necessary for decision-making? Introducing Semantic World Models (SWM). Given an observation and an action sequence, SWMs cast modeling as answering textual questions about the future outcome resulting from the actions. Recasting world modeling as a VQA problem lets us directly leverage the pretrained knowledge and machinery of VLMs for generalizable modeling. We had a lot of fun thinking about how this work helps connect these two seemingly very different fields of study - VLMs and world models! 🧵(1/6) Paper: Fun demo:

Abhishek Gupta

61,413 просмотров • 10 месяцев назад

RL is a powerful mechanism for training company-specific models on their unique work and data. This is what we do at Applied Compute. A key challenge is how to make RL efficient, because we need runs to be fast (delivered in days), cheap (scalable unit economics), and predictable (not just fast, but reliably fast). Here are some takeaways: • Synchronous RL is wasteful with time and compute. • Asynchronous RL is more efficient but introduces staleness, which causes learning instabilities. • Modeling and simulations can help analytically solve for what configuration leads to optimal efficiency. This allows us to rapidly prototype training configurations, without burning expensive compute cycles on trial runs. Two of our co-founders, Rhythm Garg and Linden Li, discussed some of this research at AI Engineer recently, with a focus on the following subproblem: what is the highest throughput way to do RL given a maximum staleness and compute budget?
20:12

RL is a powerful mechanism for training company-specific models on their unique work and data. This is what we do at Applied Compute. A key challenge is how to make RL efficient, because we need runs to be fast (delivered in days), cheap (scalable unit economics), and predictable (not just fast, but reliably fast). Here are some takeaways: • Synchronous RL is wasteful with time and compute. • Asynchronous RL is more efficient but introduces staleness, which causes learning instabilities. • Modeling and simulations can help analytically solve for what configuration leads to optimal efficiency. This allows us to rapidly prototype training configurations, without burning expensive compute cycles on trial runs. Two of our co-founders, Rhythm Garg and Linden Li, discussed some of this research at AI Engineer recently, with a focus on the following subproblem: what is the highest throughput way to do RL given a maximum staleness and compute budget?

Applied Compute

45,652 просмотров • 8 месяцев назад

Glad that our work “Inference-Time Enhancement of Generative Robot Policies via Predictive World Modeling”, led by Han Qi, has been accepted to IEEE Robotics and Automation Letters! 🎉 We propose Generative Predictive Control (GPC): sample action proposals from a pretrained diffusion policy (“look back”), roll them out with a diffusion-based action-conditioned video world model (“look forward”), then rank or optimize the actions using either a learned reward model or VLM preferences. Conceptually, this is trajectory optimization / MPC with hybrid sampling + gradient optimization, interpreted through modern diffusion priors and video world models. Interestingly, we first posted the paper on arXiv in Feb 2025, when action-conditioned video world models for planning were still rare—now this direction is rapidly gaining traction. Still many open questions, e.g., • how to avoid local minima in planning • what representations work best for world models • how to balance physics priors vs. data-driven learning Paper:
1:00

Glad that our work “Inference-Time Enhancement of Generative Robot Policies via Predictive World Modeling”, led by Han Qi, has been accepted to IEEE Robotics and Automation Letters! 🎉 We propose Generative Predictive Control (GPC): sample action proposals from a pretrained diffusion policy (“look back”), roll them out with a diffusion-based action-conditioned video world model (“look forward”), then rank or optimize the actions using either a learned reward model or VLM preferences. Conceptually, this is trajectory optimization / MPC with hybrid sampling + gradient optimization, interpreted through modern diffusion priors and video world models. Interestingly, we first posted the paper on arXiv in Feb 2025, when action-conditioned video world models for planning were still rare—now this direction is rapidly gaining traction. Still many open questions, e.g., • how to avoid local minima in planning • what representations work best for world models • how to balance physics priors vs. data-driven learning Paper:

Heng Yang

18,994 просмотров • 5 месяцев назад

Unfathomable to think that Steve Jobs predicted what we see with AI today, in 1983… “The problem was you can’t just ask Aristotle questions. And I think as we look to the next 50-100yrs, if we really can come up with these machines that can capture an underlying spirit… maybe someday after a person’s dead and gone, we can ask this machine, hey what would Aristotle have said? And maybe we won’t get the right answer. But maybe we will. And that’s really exciting to me. And that’s why I’m doing what I’m doing.” He was 28 when he gave this speech at Aspen [link to whole video below], 40yrs before the ChatGPT moment. That just blows my mind. Steve Jobs has become such a clichéd tech idol, and remembered primarily for the iPhone, but if you go back and study his work, he embodies such a clarity in long term thinking paralleled by so few today. Makes you wonder how ashamed he might have been to see where Siri and Apple are in AI today.
0:48

Unfathomable to think that Steve Jobs predicted what we see with AI today, in 1983… “The problem was you can’t just ask Aristotle questions. And I think as we look to the next 50-100yrs, if we really can come up with these machines that can capture an underlying spirit… maybe someday after a person’s dead and gone, we can ask this machine, hey what would Aristotle have said? And maybe we won’t get the right answer. But maybe we will. And that’s really exciting to me. And that’s why I’m doing what I’m doing.” He was 28 when he gave this speech at Aspen [link to whole video below], 40yrs before the ChatGPT moment. That just blows my mind. Steve Jobs has become such a clichéd tech idol, and remembered primarily for the iPhone, but if you go back and study his work, he embodies such a clarity in long term thinking paralleled by so few today. Makes you wonder how ashamed he might have been to see where Siri and Apple are in AI today.

Deedy

53,240 просмотров • 4 дней назад

Super excited to share 🧠MLGym 🦾 – the first Gym environment for AI Research Agents 🤖🔬 We introduce MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing LLM agents on AI research tasks. The key contributions of our work are: 🕹️ Enables the exploration of different training algorithms for AI Research Agents such as RL 🛠️ Provides a flexible evaluation framework that can accommodate different artifacts such as models, algorithms, or predictions 🤖 Allows researchers to evaluate any model without the need to develop a custom agentic harness 🎯 Introduces 13 diverse open-ended AI Research tasks for evaluating AI Research Agents on a wide range of domains such as computer vision, natural language processing, reinforcement learning, game theory, and logical reasoning. 📈 Proposes a new evaluation metric for AI Research Agents MLGym makes it easy to: 1) Add new tasks 2) Evaluate new models 3) Integrate new agents Check out a video of the MLGym Agent to see how it performs the full pipeline of idea generation💡, implementation 👩‍💻, experimentation 👩‍🔬, and iteration 🔄 to improve on ML tasks. Huge thanks to the exceptionally talented Deepak Nathani who led this work and to all the other amazing collaborators who made this possible 🙏🫶🚀
1:32

Super excited to share 🧠MLGym 🦾 – the first Gym environment for AI Research Agents 🤖🔬 We introduce MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing LLM agents on AI research tasks. The key contributions of our work are: 🕹️ Enables the exploration of different training algorithms for AI Research Agents such as RL 🛠️ Provides a flexible evaluation framework that can accommodate different artifacts such as models, algorithms, or predictions 🤖 Allows researchers to evaluate any model without the need to develop a custom agentic harness 🎯 Introduces 13 diverse open-ended AI Research tasks for evaluating AI Research Agents on a wide range of domains such as computer vision, natural language processing, reinforcement learning, game theory, and logical reasoning. 📈 Proposes a new evaluation metric for AI Research Agents MLGym makes it easy to: 1) Add new tasks 2) Evaluate new models 3) Integrate new agents Check out a video of the MLGym Agent to see how it performs the full pipeline of idea generation💡, implementation 👩‍💻, experimentation 👩‍🔬, and iteration 🔄 to improve on ML tasks. Huge thanks to the exceptionally talented Deepak Nathani who led this work and to all the other amazing collaborators who made this possible 🙏🫶🚀

Roberta Raileanu

105,104 просмотров • 1 год назад

Today we are excited to open up Neuracore to the academic community! Neuracore is a new data foundation built to accelerate robot learning by removing one of the field’s biggest bottlenecks: capturing and working with high-fidelity multimodal robotics data. For the first time, researchers can store, view, and work with robotics data in a cloud-native system built specifically for large-scale learning, and we are making this core platform completely free for academia. The platform lets teams capture every sensor at its native rate, store and visualize data without loss, and then train and deploy models locally using our open-source code (Link in the comments). We are rolling out access to select academic institutions first. Anyone with an academic email can sign up, and if your institution is not part of the initial rollout, you will be able to join the waitlist directly. Beyond providing this infrastructure, we see an opportunity to build a global community where engineers and researchers can share, collaborate, and advance the frontier of robot learning together. Supported by our recent $3M pre-seed round led by Earlybird VC, we are excited to take this mission even further. Our long-term goal is for Neuracore to become the natural home for cutting-edge robot learning algorithms and real-world robotics experimentation, helping accelerate the next wave of Physical AI.
1:00

Today we are excited to open up Neuracore to the academic community! Neuracore is a new data foundation built to accelerate robot learning by removing one of the field’s biggest bottlenecks: capturing and working with high-fidelity multimodal robotics data. For the first time, researchers can store, view, and work with robotics data in a cloud-native system built specifically for large-scale learning, and we are making this core platform completely free for academia. The platform lets teams capture every sensor at its native rate, store and visualize data without loss, and then train and deploy models locally using our open-source code (Link in the comments). We are rolling out access to select academic institutions first. Anyone with an academic email can sign up, and if your institution is not part of the initial rollout, you will be able to join the waitlist directly. Beyond providing this infrastructure, we see an opportunity to build a global community where engineers and researchers can share, collaborate, and advance the frontier of robot learning together. Supported by our recent $3M pre-seed round led by Earlybird VC, we are excited to take this mission even further. Our long-term goal is for Neuracore to become the natural home for cutting-edge robot learning algorithms and real-world robotics experimentation, helping accelerate the next wave of Physical AI.

Neuracore

40,796 просмотров • 9 месяцев назад

This week our Genius Zone launches with tours of Genius City in Nuanu, Bali. Our Genius School launches with record enrolments. Over the last year, the growth of Nuanu has attracted over 30 projects and $150 million in investment to create a new vision for how we can live in a post-Singularity world. Genius City is the lifelong learning campus at the heart of Nuanu, based on the ABCs of the future: Abundance, Balance and Connection. We are all victims of legacy systems breeding crime, corruption and centralized power. We must fight these system with one hand, while our other hand builds the new. We go to battle with both sword and shield. What we are building at Nuanu is now attracting thousands a day. This video shows how far we've come in the last year. A big thank you to everyone who is taking part in creating such a positive future. “You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete.” - Buckminster Fuller We Will Win.
3:16

This week our Genius Zone launches with tours of Genius City in Nuanu, Bali. Our Genius School launches with record enrolments. Over the last year, the growth of Nuanu has attracted over 30 projects and $150 million in investment to create a new vision for how we can live in a post-Singularity world. Genius City is the lifelong learning campus at the heart of Nuanu, based on the ABCs of the future: Abundance, Balance and Connection. We are all victims of legacy systems breeding crime, corruption and centralized power. We must fight these system with one hand, while our other hand builds the new. We go to battle with both sword and shield. What we are building at Nuanu is now attracting thousands a day. This video shows how far we've come in the last year. A big thank you to everyone who is taking part in creating such a positive future. “You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete.” - Buckminster Fuller We Will Win.

Roger James Hamilton

26,499 просмотров • 25 дней назад

Milestone! We (robotic arms for gadgets assembly) finished the first commercial order, which brought the first revenue. Here are some learnings from this: The customer was a smart toy manufacturer. The task was to add a heatsink to Raspberry Pi. We received parts from them and returned the assembled modules back. Currently, it's done by teleoperation. Later it will be done by a remote employee via the Internet. Then it will be automated action by action, reducing the operator's time on this and making the task profitable. ps. If you have an assembly task that we can do for you asynchronically - leave a comment below. Learning 1. It's possible! This task which is usually done by the human arm with 5 fingers can be done with a two-finger gripper with the addition of a couple of simple tooling. The task was not simplified. We peeled off thin films from stickers, unpacked paper boxes, moved PCB boards full of components, etc. And no unsolvable problems have been encountered yet. Challenges: 1) The paper box shifted during the opening Solved with the plastic walls that you can lean against 2) Heat pad, stuck to the gripper instead of heat sync. Can be solved by gripper with a pump, but this time solved with the patience of the operator 3) The film on the pad is very thin. Turned out that sub-millimeter arm precision is enough to peel it off with just a regular gripper. 4) The working area has not enough space. You'll only know this by doing real tasks in bulk. This could be solved by an extra pair of long arms, but in this case, solved with the patience of the operator. I think that in the end, we will have 5-10 types of universal tooling and 5-10 types of grippers to solve almost all the problems in such assembly tasks. Learning 2. It's slow. It took 5 times more time, than doing it with human hands. But the good news is there's a lot of room for improvement. We now have specific “time for task” metrics, which we will decrease with iterations. The main reasons for slowness: 1) To rotate the gripper to a steep angle you are forced to control one robot arm with two hands instead of using both arms. We can fix this by just making more room for rotations. 2) Grabbing PCB board with two arms is hard. A slight difference in rotation can break the board, and it's hard to control these angles visually. To solve this, the best way is to use force feedback so you can feel the pressure applied to the item. 3) Accuracy and steadiness is still can be improved We will try a metal version and double the motors to do this. 4) It is physically difficult for the human hands to move with such precision To solve this, we will add a pad for the hands like in surgical robots Learning 3. It's a good business model The "Factory in the cloud" is a good business model for this stage. You send us parts and we send back assembled modules. Currently, it's more convenient than sending a robot to your place, as we can iterate/fix the robot quickly and utilize it 100% of the time. When we polish the set-up over time - we can send robots to your place. So if we can assemble something for you in the USA with Chinese prices by using modern automation - leave a comment below.
0:48

Milestone! We (robotic arms for gadgets assembly) finished the first commercial order, which brought the first revenue. Here are some learnings from this: The customer was a smart toy manufacturer. The task was to add a heatsink to Raspberry Pi. We received parts from them and returned the assembled modules back. Currently, it's done by teleoperation. Later it will be done by a remote employee via the Internet. Then it will be automated action by action, reducing the operator's time on this and making the task profitable. ps. If you have an assembly task that we can do for you asynchronically - leave a comment below. Learning 1. It's possible! This task which is usually done by the human arm with 5 fingers can be done with a two-finger gripper with the addition of a couple of simple tooling. The task was not simplified. We peeled off thin films from stickers, unpacked paper boxes, moved PCB boards full of components, etc. And no unsolvable problems have been encountered yet. Challenges: 1) The paper box shifted during the opening Solved with the plastic walls that you can lean against 2) Heat pad, stuck to the gripper instead of heat sync. Can be solved by gripper with a pump, but this time solved with the patience of the operator 3) The film on the pad is very thin. Turned out that sub-millimeter arm precision is enough to peel it off with just a regular gripper. 4) The working area has not enough space. You'll only know this by doing real tasks in bulk. This could be solved by an extra pair of long arms, but in this case, solved with the patience of the operator. I think that in the end, we will have 5-10 types of universal tooling and 5-10 types of grippers to solve almost all the problems in such assembly tasks. Learning 2. It's slow. It took 5 times more time, than doing it with human hands. But the good news is there's a lot of room for improvement. We now have specific “time for task” metrics, which we will decrease with iterations. The main reasons for slowness: 1) To rotate the gripper to a steep angle you are forced to control one robot arm with two hands instead of using both arms. We can fix this by just making more room for rotations. 2) Grabbing PCB board with two arms is hard. A slight difference in rotation can break the board, and it's hard to control these angles visually. To solve this, the best way is to use force feedback so you can feel the pressure applied to the item. 3) Accuracy and steadiness is still can be improved We will try a metal version and double the motors to do this. 4) It is physically difficult for the human hands to move with such precision To solve this, we will add a pad for the hands like in surgical robots Learning 3. It's a good business model The "Factory in the cloud" is a good business model for this stage. You send us parts and we send back assembled modules. Currently, it's more convenient than sending a robot to your place, as we can iterate/fix the robot quickly and utilize it 100% of the time. When we polish the set-up over time - we can send robots to your place. So if we can assemble something for you in the USA with Chinese prices by using modern automation - leave a comment below.

Igor Kulakov

37,266 просмотров • 1 год назад

$PLTR CTO Shyam Sankar on Deepseek and disagreeing with OpenAI CEO Sam Altman 🔮 “We are at war with China. We are in an AI arms race. I got into a disagreement with Sam Altman about this at the Senate AI summit over a year ago… There’s an opposing view that we can all get along and cooperate on these things. I’m sure the other side feels quite different realizing that their IP has been stolen. “How many times are we going to believe that anchors dragged across the sea that cut undersea cables? “China knows they’re at war. We kinda equivocate on it as a peace loving nation. “The engineering on R1 is exquisite. The optimizations they’ve done are really impressive… We have to wake up with respect for our adversary and realize that we are competing. But they absolutely did steal a lot of that through distillation of the models, and perhaps they stole even more. “We have to realize that the AI race is winner take all, and it’s going to be a whole of nation effort that extends well beyond the DoD in order for us as a nation to win.” 🇺🇸
2:09

$PLTR CTO Shyam Sankar on Deepseek and disagreeing with OpenAI CEO Sam Altman 🔮 “We are at war with China. We are in an AI arms race. I got into a disagreement with Sam Altman about this at the Senate AI summit over a year ago… There’s an opposing view that we can all get along and cooperate on these things. I’m sure the other side feels quite different realizing that their IP has been stolen. “How many times are we going to believe that anchors dragged across the sea that cut undersea cables? “China knows they’re at war. We kinda equivocate on it as a peace loving nation. “The engineering on R1 is exquisite. The optimizations they’ve done are really impressive… We have to wake up with respect for our adversary and realize that we are competing. But they absolutely did steal a lot of that through distillation of the models, and perhaps they stole even more. “We have to realize that the AI race is winner take all, and it’s going to be a whole of nation effort that extends well beyond the DoD in order for us as a nation to win.” 🇺🇸

Jawwwn

295,545 просмотров • 1 год назад

Wim Wenders on the "camera movement and blocking" in Wings of Desire (1987): Filmmaker Magazine: "What was your philosophy about camera movement and blocking in Wings of Desire?" Wenders: "As we very often had to “translate” the angels’ point of view, so to speak, we were extremely keen on moving the camera as much as possible. In the absence of Steadicam equipment we worked a lot on tracks, with dollies, cranes, jib-arms etc. But we also built ourselves devices so we could move through the air from one house to the next, for instance, and we shot the opening sequence on a helicopter, which was highly difficult in West Berlin at the time, as there were no private companies flying, just the Allies with their respective army pilots. We ended up shooting with a British pilot in an army helicopter without a proper camera mount. Today, you would do these things with gyroscopes and such. Blocking has always been my department. Henri [Alekan] kept out of it completely, and I did it with his operator, Agnès Godard. I have done shot lists for complicated sets, but usually I decide on location in the morning how we design the shots. I prefer to see the actors rehearse it, before I commit to any blocking. Camera moves weren’t the real challenge, though, for finding the angels’ points of view. It dawned on me early on that our camera had to do a more complex job. I told it to Henri. “Those angels have a very loving look at us humans. We have to find a way to teach our camera to look more lovingly.” Henri just stared at me as if I was out of my mind. “How do we do that?” Well, I didn’t know of course. But I figured we had to invest more care and love ourselves into every shot that represented what the angels saw. And that’s it, in the end. A camera can reflect on what you invest into its act of seeing. That sounds pretty lofty, I guess. But it does rub off, I tell you, if you try to imagine how angels would look at us. After all, they were some sort of metaphor for me for the better persons we carry inside ourselves, or for the children we somehow preserved in ourselves." — "“Imagine How Angels Would Look at Us”: Wim Wenders on Restoring Wings of Desire" by Jim Hemphill (Filmmaker Magazine, 2018)
0:52

Wim Wenders on the "camera movement and blocking" in Wings of Desire (1987): Filmmaker Magazine: "What was your philosophy about camera movement and blocking in Wings of Desire?" Wenders: "As we very often had to “translate” the angels’ point of view, so to speak, we were extremely keen on moving the camera as much as possible. In the absence of Steadicam equipment we worked a lot on tracks, with dollies, cranes, jib-arms etc. But we also built ourselves devices so we could move through the air from one house to the next, for instance, and we shot the opening sequence on a helicopter, which was highly difficult in West Berlin at the time, as there were no private companies flying, just the Allies with their respective army pilots. We ended up shooting with a British pilot in an army helicopter without a proper camera mount. Today, you would do these things with gyroscopes and such. Blocking has always been my department. Henri [Alekan] kept out of it completely, and I did it with his operator, Agnès Godard. I have done shot lists for complicated sets, but usually I decide on location in the morning how we design the shots. I prefer to see the actors rehearse it, before I commit to any blocking. Camera moves weren’t the real challenge, though, for finding the angels’ points of view. It dawned on me early on that our camera had to do a more complex job. I told it to Henri. “Those angels have a very loving look at us humans. We have to find a way to teach our camera to look more lovingly.” Henri just stared at me as if I was out of my mind. “How do we do that?” Well, I didn’t know of course. But I figured we had to invest more care and love ourselves into every shot that represented what the angels saw. And that’s it, in the end. A camera can reflect on what you invest into its act of seeing. That sounds pretty lofty, I guess. But it does rub off, I tell you, if you try to imagine how angels would look at us. After all, they were some sort of metaphor for me for the better persons we carry inside ourselves, or for the children we somehow preserved in ourselves." — "“Imagine How Angels Would Look at Us”: Wim Wenders on Restoring Wings of Desire" by Jim Hemphill (Filmmaker Magazine, 2018)

RadiantFilm

18,101 просмотров • 6 месяцев назад

Check out our latest work, "Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight," published in the IEEE Transactions on Robotics, where we reconcile #OptimalControl and #ReinforcementLearning, achieving the same super-human performance, but with superior generalizability, as our previous model-free deep RL! Code released! PDF: Code: Full Video: Model-free #ReinforcementLearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) provides robustness, constraint handling, and powerful online replanning capabilities. In this work, we extend our previous AC-MPC paper (Romero, ICRA'24) by taking a deeper look at how both approaches can be unified. We introduce and extend Actor-Critic Model Predictive Control (AC-MPC), a framework that embeds a differentiable MPC inside an Actor-Critic RL architecture. This integration allows the MPC-based actor to perform short-term predictive optimization, while the critic facilitates long-horizon learning and exploration. We conduct a comprehensive study that highlights AC-MPC’s key advantages: - Better out-of-distribution generalization, both against unknown disturbances and changes in the quadrotor dynamics - Improved sample efficiency - A novel empirical analysis uncovering a relationship between the critic’s value function and the MPC cost function, providing deeper insight into their interplay. We validate our method in simulation and the real world on a quadcopter flying at superhuman speeds of up to 21 m/s, matching state-of-the-art model-free RL performance, and retaining the predictive structure of MPC for more reliable out-of-distribution behavior. Reference: Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight IEEE Transactions on Robotics (T-RO), 2025 PDF: Full Video: Code: Kudos to Ángel Romero, Elie Aljalbout, Yunlong Song! University of Zurich UZH Science UZH Space Hub AUTOASSESS European Research Council (ERC) UZHai
1:46

Check out our latest work, "Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight," published in the IEEE Transactions on Robotics, where we reconcile #OptimalControl and #ReinforcementLearning, achieving the same super-human performance, but with superior generalizability, as our previous model-free deep RL! Code released! PDF: Code: Full Video: Model-free #ReinforcementLearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand, #ModelPredictiveControl (MPC) provides robustness, constraint handling, and powerful online replanning capabilities. In this work, we extend our previous AC-MPC paper (Romero, ICRA'24) by taking a deeper look at how both approaches can be unified. We introduce and extend Actor-Critic Model Predictive Control (AC-MPC), a framework that embeds a differentiable MPC inside an Actor-Critic RL architecture. This integration allows the MPC-based actor to perform short-term predictive optimization, while the critic facilitates long-horizon learning and exploration. We conduct a comprehensive study that highlights AC-MPC’s key advantages: - Better out-of-distribution generalization, both against unknown disturbances and changes in the quadrotor dynamics - Improved sample efficiency - A novel empirical analysis uncovering a relationship between the critic’s value function and the MPC cost function, providing deeper insight into their interplay. We validate our method in simulation and the real world on a quadcopter flying at superhuman speeds of up to 21 m/s, matching state-of-the-art model-free RL performance, and retaining the predictive structure of MPC for more reliable out-of-distribution behavior. Reference: Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight IEEE Transactions on Robotics (T-RO), 2025 PDF: Full Video: Code: Kudos to Ángel Romero, Elie Aljalbout, Yunlong Song! University of Zurich UZH Science UZH Space Hub AUTOASSESS European Research Council (ERC) UZHai

Davide Scaramuzza

27,279 просмотров • 7 месяцев назад

Another year of 'Machine Learning with Graphs' at the School of Computer Science and AI at Tel Aviv University has come to an end, and here’s a short summary video of me explaining Graph Neural Networks with my hands 👐 This year I had around 80 students from across Computer Science, Math, and Electrical Engineering who came to explore the magic of Graph Learning. We wrapped up with nearly 30 final research projects, and it was truly inspiring to see students come up with novel, creative ideas, combining graph methods in diverse domains and leveraging their own unique expertise. It’s always fascinating to see how graph modeling can enhance performance or solve challenges in almost every field. While some ideas didn’t fully work out (as research often goes!), many did, and I genuinely hope to see some of them published soon. Well done to all the students!!! Once again, I have to say that TAU students are simply brilliant, curious, and incredibly hard-working. Next year, I’ll continue lecturing as a Teaching Fellow at CS@TAU and I’m already excited about the many updates I plan for the course (Graph Foundation Models are coming 👀).
0:55

Another year of 'Machine Learning with Graphs' at the School of Computer Science and AI at Tel Aviv University has come to an end, and here’s a short summary video of me explaining Graph Neural Networks with my hands 👐 This year I had around 80 students from across Computer Science, Math, and Electrical Engineering who came to explore the magic of Graph Learning. We wrapped up with nearly 30 final research projects, and it was truly inspiring to see students come up with novel, creative ideas, combining graph methods in diverse domains and leveraging their own unique expertise. It’s always fascinating to see how graph modeling can enhance performance or solve challenges in almost every field. While some ideas didn’t fully work out (as research often goes!), many did, and I genuinely hope to see some of them published soon. Well done to all the students!!! Once again, I have to say that TAU students are simply brilliant, curious, and incredibly hard-working. Next year, I’ll continue lecturing as a Teaching Fellow at CS@TAU and I’m already excited about the many updates I plan for the course (Graph Foundation Models are coming 👀).

Maya Bechler-Speicher

76,069 просмотров • 10 месяцев назад

📢📢 𝐏𝐞𝐫𝐜𝐇𝐞𝐚𝐝: 𝐏𝐞𝐫𝐜𝐞𝐩𝐭𝐮𝐚𝐥 𝐇𝐞𝐚𝐝 𝐌𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐒𝐢𝐧𝐠𝐥𝐞-𝐈𝐦𝐚𝐠𝐞 𝟑𝐃 𝐇𝐞𝐚𝐝 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 & 𝐄𝐝𝐢𝐭𝐢𝐧𝐠📢📢 PercHead reconstructs realistic 3D heads from a single image and enables disentangled 3D editing via geometric controls and style inputs from images or text. At its core is a generalized 3D head decoder trained with perceptual supervision from DINOv2 and SAM 2.1. We find that our new perceptual loss formulation improves reconstruction fidelity compared to commonly-used methods such as LPIPS. Our trained reconstruction model is able to generate 3D-consistent heads from a single input image. Even with challenging side-view inputs, the model robustly infers missing regions for a coherent, high-fidelity output. In addition, our architecture seamlessly adapts to downstream tasks: by swapping the encoder, we can transform the model into a disentangled 3D editing pipeline. In this scenario, we can control geometry through - potentially hand-drawn - segmentation maps, and condition style via image or text prompt. We also provide an interactive GUI to enable the exploration of our editing pipeline. 🌍 📽️ Great work by Antonio Oroz and Tobias Kirschstein
0:30

📢📢 𝐏𝐞𝐫𝐜𝐇𝐞𝐚𝐝: 𝐏𝐞𝐫𝐜𝐞𝐩𝐭𝐮𝐚𝐥 𝐇𝐞𝐚𝐝 𝐌𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐒𝐢𝐧𝐠𝐥𝐞-𝐈𝐦𝐚𝐠𝐞 𝟑𝐃 𝐇𝐞𝐚𝐝 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 & 𝐄𝐝𝐢𝐭𝐢𝐧𝐠📢📢 PercHead reconstructs realistic 3D heads from a single image and enables disentangled 3D editing via geometric controls and style inputs from images or text. At its core is a generalized 3D head decoder trained with perceptual supervision from DINOv2 and SAM 2.1. We find that our new perceptual loss formulation improves reconstruction fidelity compared to commonly-used methods such as LPIPS. Our trained reconstruction model is able to generate 3D-consistent heads from a single input image. Even with challenging side-view inputs, the model robustly infers missing regions for a coherent, high-fidelity output. In addition, our architecture seamlessly adapts to downstream tasks: by swapping the encoder, we can transform the model into a disentangled 3D editing pipeline. In this scenario, we can control geometry through - potentially hand-drawn - segmentation maps, and condition style via image or text prompt. We also provide an interactive GUI to enable the exploration of our editing pipeline. 🌍 📽️ Great work by Antonio Oroz and Tobias Kirschstein

Matthias Niessner

18,885 просмотров • 9 месяцев назад

Today, we are launching the first publicly available AI Scientist, via the FutureHouse Platform. Our AI Scientist agents can perform a wide variety of scientific tasks better than humans. By chaining them together, we've already started to discover new biology really fast. With the platform, we are bringing these capabilities to the wider community. Watch our long-form video, in the comments below, to learn more about how the platform works and how you can use it to make new discoveries, and go to our website or see the comments below to access the platform. We are releasing three superhuman AI Scientist agents today, each with their own specialization: A general-purpose agent (Crow); An agent to automate literature reviews (Falcon); and An agent to answer the question “Has anyone done X before” (Owl). We are also releasing an experimental agent, Phoenix, that has access to a wide variety of tools for planning experiments in chemistry. More on that below. The three literature search agents (Crow, Falcon, and Owl) have benchmarked superhuman performance. They also have access to a large corpus of full scientific texts, which means that you can ask them more detailed questions about experimental protocols and study limitations that general-purpose web search agents, which usually only have access to abstracts, might miss. Our agents also use a variety of factors to distinguish source quality, so that they don’t end up relying on low-quality papers or pop-science sources. Finally, and critically, we have an API, which is intended to allow researchers to integrate our agents into their workflows. Phoenix is an experimental project we put together recently just to demonstrate what can happen if you give the agents access to lots of scientific tools. It is not better than humans at planning experiments yet, and it makes a lot more mistakes than Crow, Falcon, or Owl. We want to see all the ways you can break it! The agents we are releasing today cannot yet do all (or even most!) aspects of scientific research autonomously. However, as we show in the video, you can already use them to generate and evaluate new hypotheses and plan new experiments way faster than before. Internally, we also have dedicated agents for data analysis, hypothesis generation, protein engineering, and more, and we plan to launch these on the platform in the coming months as well. Within a year or two, it is easy to imagine that the vast majority of desk work that scientists do today will be accelerated with the help of AI agents like the ones we are releasing today. The platform is currently free-to-use. Over time, depending on how people use it, we may implement pricing plans. If you want higher rate limits, especially for research projects, get in touch. Michael Skarlinski, Andrew White 🐦‍⬛, Tyler Nadolski, Remo Storni, James Braza, Ludovico Mitchener, Michaela Hinks, as well as Jason Carman and his team for making such fantastic videos of us!
1:21

Today, we are launching the first publicly available AI Scientist, via the FutureHouse Platform. Our AI Scientist agents can perform a wide variety of scientific tasks better than humans. By chaining them together, we've already started to discover new biology really fast. With the platform, we are bringing these capabilities to the wider community. Watch our long-form video, in the comments below, to learn more about how the platform works and how you can use it to make new discoveries, and go to our website or see the comments below to access the platform. We are releasing three superhuman AI Scientist agents today, each with their own specialization: A general-purpose agent (Crow); An agent to automate literature reviews (Falcon); and An agent to answer the question “Has anyone done X before” (Owl). We are also releasing an experimental agent, Phoenix, that has access to a wide variety of tools for planning experiments in chemistry. More on that below. The three literature search agents (Crow, Falcon, and Owl) have benchmarked superhuman performance. They also have access to a large corpus of full scientific texts, which means that you can ask them more detailed questions about experimental protocols and study limitations that general-purpose web search agents, which usually only have access to abstracts, might miss. Our agents also use a variety of factors to distinguish source quality, so that they don’t end up relying on low-quality papers or pop-science sources. Finally, and critically, we have an API, which is intended to allow researchers to integrate our agents into their workflows. Phoenix is an experimental project we put together recently just to demonstrate what can happen if you give the agents access to lots of scientific tools. It is not better than humans at planning experiments yet, and it makes a lot more mistakes than Crow, Falcon, or Owl. We want to see all the ways you can break it! The agents we are releasing today cannot yet do all (or even most!) aspects of scientific research autonomously. However, as we show in the video, you can already use them to generate and evaluate new hypotheses and plan new experiments way faster than before. Internally, we also have dedicated agents for data analysis, hypothesis generation, protein engineering, and more, and we plan to launch these on the platform in the coming months as well. Within a year or two, it is easy to imagine that the vast majority of desk work that scientists do today will be accelerated with the help of AI agents like the ones we are releasing today. The platform is currently free-to-use. Over time, depending on how people use it, we may implement pricing plans. If you want higher rate limits, especially for research projects, get in touch. Michael Skarlinski, Andrew White 🐦‍⬛, Tyler Nadolski, Remo Storni, James Braza, Ludovico Mitchener, Michaela Hinks, as well as Jason Carman and his team for making such fantastic videos of us!

Sam Rodriques

725,776 просмотров • 1 год назад