Загрузка видео...

Не удалось загрузить видео

На главную

Current Vision-Language-Action (VLA) paradigms in autonomous driving primarily rely on Imitation Learning (IL), which introduces inherent challenges such as distribution shift and causal confusion. Online Reinforcement Learning offers a promising pathway to address these issues through trial-and-error learning. However, applying online reinforcement learning to VLA models in autonomous driving...

ai_bites's profile picture

AI Bites | YouTube Channel

2,270 subscribers

43,496 просмотров • 5 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

MolmoAct2 is landing in LeRobot! Ai2's open Action Reasoning Model combines a Molmo2-ER vision-language backbone with a flow-matching continuous action expert to predict robot action chunks from images, language instructions, and proprioceptive state. An open robot foundation model built for real-world control, with strong out-of-the-box performance and easy fine-tuning in LeRobot. Pick-and-place inference running on NVIDIA DGX Spark! Blog: Paper: Thanks to Ai2 Jiafei Duan Haoquan Fang
0:11

MolmoAct2 is landing in LeRobot! Ai2's open Action Reasoning Model combines a Molmo2-ER vision-language backbone with a flow-matching continuous action expert to predict robot action chunks from images, language instructions, and proprioceptive state. An open robot foundation model built for real-world control, with strong out-of-the-box performance and easy fine-tuning in LeRobot. Pick-and-place inference running on NVIDIA DGX Spark! Blog: Paper: Thanks to Ai2 Jiafei Duan Haoquan Fang

LeRobot

24,727 просмотров • 1 месяц назад

A viral paper "Language Model Represents Space and Time" recently claims that LLMs learn "world models". As much as I like Max Tegmark's works, I disagree with their definition of world model. World model is a core concept in AI agent and decision making. It is our mental simulation of how the world works given interventions (or lack thereof). A world model captures causality and intuitive physics, telling the agent what is likely and what is impossible. It can and should be used for counterfactual reasoning, i.e. "what ifs": what would happen if I knock over a cup of water? Where would I have been if I had not taken that bus? Yann LeCun Yann LeCun says it well in his position paper ( I quote: "Using such world models, animals can learn new skills with very few trials. They can predict the consequences of their actions, they can reason, plan, explore, and imagine new solutions to problems. Importantly, they can also avoid making dangerous mistakes when facing an unknown situation." The first use of the term World Model in deep policy learning is attributed to hardmaru & Jürgen Schmidhuber: In their seminal paper, an agent masters shooting skills in the popular game Doom (demo below) by learning in imagination, using an internal world model as a "physics simulator". To put in a simple Python math formula, world model learns a function F(s[0:t-1], a) -> s[t:], which takes as input the observed past and current action, and outputs plausible future states. Now the definition of World Model in Tegmark's paper seems to be about predicting GPS coordinates and time eras. I see this as just a classification task with no causal learning and simulation going on. You cannot make meaningful interventions against that model, nor can you optimize any decision making in a closed feedback loop. As for the "space & time neurons", I think they are most similar to the "sentiment neuron" that OpenAI published in 2017: Predicting GPS is conceptually no different from predicting sentiment in my opinion. I don't think their experimental results are wrong - just that their conclusion is on shaky grounds. I welcome any debate! Paper link:
0:15

A viral paper "Language Model Represents Space and Time" recently claims that LLMs learn "world models". As much as I like Max Tegmark's works, I disagree with their definition of world model. World model is a core concept in AI agent and decision making. It is our mental simulation of how the world works given interventions (or lack thereof). A world model captures causality and intuitive physics, telling the agent what is likely and what is impossible. It can and should be used for counterfactual reasoning, i.e. "what ifs": what would happen if I knock over a cup of water? Where would I have been if I had not taken that bus? Yann LeCun Yann LeCun says it well in his position paper ( I quote: "Using such world models, animals can learn new skills with very few trials. They can predict the consequences of their actions, they can reason, plan, explore, and imagine new solutions to problems. Importantly, they can also avoid making dangerous mistakes when facing an unknown situation." The first use of the term World Model in deep policy learning is attributed to hardmaru & Jürgen Schmidhuber: In their seminal paper, an agent masters shooting skills in the popular game Doom (demo below) by learning in imagination, using an internal world model as a "physics simulator". To put in a simple Python math formula, world model learns a function F(s[0:t-1], a) -> s[t:], which takes as input the observed past and current action, and outputs plausible future states. Now the definition of World Model in Tegmark's paper seems to be about predicting GPS coordinates and time eras. I see this as just a classification task with no causal learning and simulation going on. You cannot make meaningful interventions against that model, nor can you optimize any decision making in a closed feedback loop. As for the "space & time neurons", I think they are most similar to the "sentiment neuron" that OpenAI published in 2017: Predicting GPS is conceptually no different from predicting sentiment in my opinion. I don't think their experimental results are wrong - just that their conclusion is on shaky grounds. I welcome any debate! Paper link:

Jim Fan

593,943 просмотров • 2 лет назад

🤔 How to fine-tune an Imitation Learning policy (e.g., Diffusion Policy, ACT) with RL? As an RL practitioner, I’ve been struggling with this problem for a while. Here’s why it’s tough: 1️⃣ Special designs (usually for multimodal action distributions) in modern IL models make them non-trivial to fine-tune by RL. 2️⃣ Large policy models + RL's poor sample efficiency = a nightmare But finally, we figured out a simple solution that works for any model architecture! 🌟 Check out our #ICLR2025 paper: “Policy Decorator: Model-Agnostic Online Refinement for Large Policy Models”, led by my amazing mentee Xiu Yuan. 🔗 🧵 Read more below!
0:16

🤔 How to fine-tune an Imitation Learning policy (e.g., Diffusion Policy, ACT) with RL? As an RL practitioner, I’ve been struggling with this problem for a while. Here’s why it’s tough: 1️⃣ Special designs (usually for multimodal action distributions) in modern IL models make them non-trivial to fine-tune by RL. 2️⃣ Large policy models + RL's poor sample efficiency = a nightmare But finally, we figured out a simple solution that works for any model architecture! 🌟 Check out our #ICLR2025 paper: “Policy Decorator: Model-Agnostic Online Refinement for Large Policy Models”, led by my amazing mentee Xiu Yuan. 🔗 🧵 Read more below!

Tongzhou Mu 🤖🦾🦿

16,959 просмотров • 1 год назад

Model-Free Reinforcement Learning (MFRL) has been alluring, especially with supercharged compute with physics on GPU. However, the methods use 0-th order gradients, and are often not the best optimizers. Can we do better than PPO in continuous control for robotics? Turns out yes! 🥳 tl;dr: Faster, better RL than PPO in continuous control 💪 The answer lies in using more information from the simulation. We are juicing the simulation on GPU as it is, why not use it for gradients as well? This has been a driving question in a series of our works. We first studied this problem in ICLR 2022 paper on Short Horizon Actor Critic Naive gradient based methods are stuck in local minima and have exploding/vanishing gradients. SHAC solved this problem truncated rollouts and model based value estimation, where the model is Differentiable Sim. This boosted sample efficiency and wall-clock time immensely especially in high dimensional systems such as humanoids Yet, given enough compute PPO often caught up. Our follow up paper on on Adaptive Horizon Actor Critic at ICML 2024 discovers the cause and provides a fix. However, we find that even when given ground-truth dynamics, not all gradients are useful due to sample error. 1st-Order Model-Based Reinforcement Learning methods employing differentiable simulation provide gradients with reduced variance but are susceptible to bias in scenarios involving stiff dynamics, such as physical contact. We find that back-propagating through contact and long trajectories drastically reduces gradient accuracy. Using this insight, we propose AHAC to dynamically adapt its roll-out horizon to avoid differentiating through stiff contact. AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes (wall clock) and outperforms PPO by 40%, even in the limit of data provided to PPO. This work is led by Ignat Georgiev alongside Krishnan Srinivasan, Jie Xu, Eric Heiden and ample assistance from warp team at NVIDIA Robotics (Miles Macklin)
0:10

Model-Free Reinforcement Learning (MFRL) has been alluring, especially with supercharged compute with physics on GPU. However, the methods use 0-th order gradients, and are often not the best optimizers. Can we do better than PPO in continuous control for robotics? Turns out yes! 🥳 tl;dr: Faster, better RL than PPO in continuous control 💪 The answer lies in using more information from the simulation. We are juicing the simulation on GPU as it is, why not use it for gradients as well? This has been a driving question in a series of our works. We first studied this problem in ICLR 2022 paper on Short Horizon Actor Critic Naive gradient based methods are stuck in local minima and have exploding/vanishing gradients. SHAC solved this problem truncated rollouts and model based value estimation, where the model is Differentiable Sim. This boosted sample efficiency and wall-clock time immensely especially in high dimensional systems such as humanoids Yet, given enough compute PPO often caught up. Our follow up paper on on Adaptive Horizon Actor Critic at ICML 2024 discovers the cause and provides a fix. However, we find that even when given ground-truth dynamics, not all gradients are useful due to sample error. 1st-Order Model-Based Reinforcement Learning methods employing differentiable simulation provide gradients with reduced variance but are susceptible to bias in scenarios involving stiff dynamics, such as physical contact. We find that back-propagating through contact and long trajectories drastically reduces gradient accuracy. Using this insight, we propose AHAC to dynamically adapt its roll-out horizon to avoid differentiating through stiff contact. AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes (wall clock) and outperforms PPO by 40%, even in the limit of data provided to PPO. This work is led by Ignat Georgiev alongside Krishnan Srinivasan, Jie Xu, Eric Heiden and ample assistance from warp team at NVIDIA Robotics (Miles Macklin)

Animesh Garg

52,300 просмотров • 2 лет назад

The Hidden Language of Diffusion Models paper page: tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation
0:19

The Hidden Language of Diffusion Models paper page: tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation

AK

41,746 просмотров • 3 лет назад

Did you know that babies cry with an accent? This week I’ll be focusing on language acquisition during infancy. Yesterday we talked about how newborns enter the world already recognizing the distinct tones of their mothers’ voices, as well as the rhythms and patterns of what will become their native language. And that recognition is not only receptive (the language we hear and understand), but expressive (the language we speak). Which leads us to crying with accent. A study that compared the cries of French and German infants identified distinct patterns their cries - measured as early as the day after their births - that mirror the unique melodies (or prosody) of their native languages. The French infants ended their cries with a lilt that mirrored French, while German infants began their cries with intensity then dropped off at the end, mirroring the vocalizations of their German parents. It’s still more evidence that some basic forms of language learning are influenced by children’s experience (learning) in utero. When it comes to our earliest vocalizations, newborns rely heavily on cries - which often tend to be a bit screechier than they will become in time - alongside gurgles, grunts and squeaks (like those shown in the video here.) But in just a matter of weeks these basic sounds make way for a much broader array of vocalizations. We’ll look at a few of those tomorrow. This new arrival was shared to TT by
0:14

Did you know that babies cry with an accent? This week I’ll be focusing on language acquisition during infancy. Yesterday we talked about how newborns enter the world already recognizing the distinct tones of their mothers’ voices, as well as the rhythms and patterns of what will become their native language. And that recognition is not only receptive (the language we hear and understand), but expressive (the language we speak). Which leads us to crying with accent. A study that compared the cries of French and German infants identified distinct patterns their cries - measured as early as the day after their births - that mirror the unique melodies (or prosody) of their native languages. The French infants ended their cries with a lilt that mirrored French, while German infants began their cries with intensity then dropped off at the end, mirroring the vocalizations of their German parents. It’s still more evidence that some basic forms of language learning are influenced by children’s experience (learning) in utero. When it comes to our earliest vocalizations, newborns rely heavily on cries - which often tend to be a bit screechier than they will become in time - alongside gurgles, grunts and squeaks (like those shown in the video here.) But in just a matter of weeks these basic sounds make way for a much broader array of vocalizations. We’ll look at a few of those tomorrow. This new arrival was shared to TT by

Dan Wuori

115,574 просмотров • 1 год назад

𝗜'𝘃𝗲 𝗵𝗲𝗮𝗿𝗱 𝘁𝗵𝗶𝘀 𝗮 𝗹𝗼𝘁 𝗿𝗲𝗰𝗲𝗻𝘁𝗹𝘆: "𝗪𝗲 𝘁𝗿𝗮𝗶𝗻𝗲𝗱 𝗼𝘂𝗿 𝗿𝗼𝗯𝗼𝘁 𝗼𝗻 𝗼𝗻𝗲 𝗼𝗯𝗷𝗲𝗰𝘁 𝗮𝗻𝗱 𝗶𝘁 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘀𝗲𝗱 𝘁𝗼 𝗮 𝗻𝗼𝘃𝗲𝗹 𝗼𝗯𝗷𝗲𝗰𝘁 - 𝘁𝗵𝗲𝘀𝗲 𝗻𝗲𝘄 𝗩𝗟𝗔 𝗺𝗼𝗱𝗲𝗹𝘀 𝗮𝗿𝗲 𝗰𝗿𝗮𝘇𝘆!" Let's talk about what's actually happening in that "A" (Action) part of your VLA model. The Vision and Language components? They're incredible. Pre-trained on internet-scale data, they understand objects, spatial relationships, and task instructions better than ever. But the Action component? That's still learned from scratch on your specific robot demonstrations. 𝗛𝗲𝗿𝗲'𝘀 𝘁𝗵𝗲 𝗿𝗲𝗮𝗹𝗶𝘁𝘆: Your VLA model has internet-scale understanding of what a screwdriver looks like and what "tighten the screw" means. But the actual motor pattern for "rotating wrist while applying downward pressure"? That comes from your 500 robot demos. 𝗪𝗵𝗮𝘁 𝘁𝗵𝗶𝘀 𝗺𝗲𝗮𝗻𝘀 𝗳𝗼𝗿 "𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘀𝗮𝘁𝗶𝗼𝗻": • 𝗩𝗶𝘀𝗶𝗼𝗻 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘀𝗮𝘁𝗶𝗼𝗻: Recognises novel objects instantly (thanks to pre-training) • 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘀𝗮𝘁𝗶𝗼𝗻: Understands new task instructions (thanks to pre-training) • 𝗔𝗰𝘁𝗶𝗼𝗻 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘀𝗮𝘁𝗶𝗼𝗻: Still limited to motor patterns seen during robot training Ask that same robot to "unscrew the bottle cap" and it fails because: • Vision: Recognises bottle and cap • Language: Understands "unscrew" • Action: Never learned the "twist while pulling" motor pattern 𝗧𝗵𝗲 𝗵𝗮𝗿𝗱 𝘁𝗿𝘂𝘁𝗵 𝗮𝗯𝗼𝘂𝘁 𝗩𝗟𝗔 𝗺𝗼𝗱𝗲𝗹𝘀: The "VL" gives you incredible zero-shot understanding. The "A" still requires task-specific demonstrations. We've cracked the perception and reasoning problem. We haven't cracked the motor generalisation problem.
0:14

𝗜'𝘃𝗲 𝗵𝗲𝗮𝗿𝗱 𝘁𝗵𝗶𝘀 𝗮 𝗹𝗼𝘁 𝗿𝗲𝗰𝗲𝗻𝘁𝗹𝘆: "𝗪𝗲 𝘁𝗿𝗮𝗶𝗻𝗲𝗱 𝗼𝘂𝗿 𝗿𝗼𝗯𝗼𝘁 𝗼𝗻 𝗼𝗻𝗲 𝗼𝗯𝗷𝗲𝗰𝘁 𝗮𝗻𝗱 𝗶𝘁 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘀𝗲𝗱 𝘁𝗼 𝗮 𝗻𝗼𝘃𝗲𝗹 𝗼𝗯𝗷𝗲𝗰𝘁 - 𝘁𝗵𝗲𝘀𝗲 𝗻𝗲𝘄 𝗩𝗟𝗔 𝗺𝗼𝗱𝗲𝗹𝘀 𝗮𝗿𝗲 𝗰𝗿𝗮𝘇𝘆!" Let's talk about what's actually happening in that "A" (Action) part of your VLA model. The Vision and Language components? They're incredible. Pre-trained on internet-scale data, they understand objects, spatial relationships, and task instructions better than ever. But the Action component? That's still learned from scratch on your specific robot demonstrations. 𝗛𝗲𝗿𝗲'𝘀 𝘁𝗵𝗲 𝗿𝗲𝗮𝗹𝗶𝘁𝘆: Your VLA model has internet-scale understanding of what a screwdriver looks like and what "tighten the screw" means. But the actual motor pattern for "rotating wrist while applying downward pressure"? That comes from your 500 robot demos. 𝗪𝗵𝗮𝘁 𝘁𝗵𝗶𝘀 𝗺𝗲𝗮𝗻𝘀 𝗳𝗼𝗿 "𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘀𝗮𝘁𝗶𝗼𝗻": • 𝗩𝗶𝘀𝗶𝗼𝗻 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘀𝗮𝘁𝗶𝗼𝗻: Recognises novel objects instantly (thanks to pre-training) • 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘀𝗮𝘁𝗶𝗼𝗻: Understands new task instructions (thanks to pre-training) • 𝗔𝗰𝘁𝗶𝗼𝗻 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘀𝗮𝘁𝗶𝗼𝗻: Still limited to motor patterns seen during robot training Ask that same robot to "unscrew the bottle cap" and it fails because: • Vision: Recognises bottle and cap • Language: Understands "unscrew" • Action: Never learned the "twist while pulling" motor pattern 𝗧𝗵𝗲 𝗵𝗮𝗿𝗱 𝘁𝗿𝘂𝘁𝗵 𝗮𝗯𝗼𝘂𝘁 𝗩𝗟𝗔 𝗺𝗼𝗱𝗲𝗹𝘀: The "VL" gives you incredible zero-shot understanding. The "A" still requires task-specific demonstrations. We've cracked the perception and reasoning problem. We haven't cracked the motor generalisation problem.

Stephen James

51,332 просмотров • 11 месяцев назад

The term "continual learning" has become overloaded if you see it as an ML problem. One classic thread is about memorization: regularization-based continual learning methods, such as EWC, MAS, and SI, estimate which parameters mattered for previous tasks and resist changing them too much. One modern thread is about adaptation: test-time training and inference-time learning methods, such as TTT, adapt part of the model on the incoming test stream before making predictions. These are sometimes discussed as separate threads. But in modern scalable architectures, I think they are better seen as complementary constraints: a model that learns quickly at test time also benefits from a mechanism for deciding what not to forget. In our #ECCV2026 paper, we study this in large-scale 4D reconstruction: how to build fast spatial memory that can adapt over long observation streams while reducing collapse and forgetting. Instead of using fully plastic test-time updates, we stabilize fast-weight adaptation with an elastic prior that balances adaptation and memory. Key ideas: - Elastic Test-Time Training: Fisher-weighted consolidation for fast-weight updates - EMA anchor weights that provide a moving reference for stability - Chunk-by-chunk inference for long 3D/4D observation streams We show that this scales across large 3D/4D pretraining settings, including both LRM-style and LVSM-style models, and improves reconstruction across benchmarks including Stereo4D, NVIDIA, and DL3DV-140. We release model checkpoints across different design choices: resolution, post-training curriculum, and whether the model uses an explicit 4DGS intermediate representation. - Homepage: - Paper: - Code: - Models: This work is co-led with Xueyang Yu, contributed by Haoyu Zhen Yuncong Yang, and advised by Michigan SLED Lab Chuang Gan.
0:14

The term "continual learning" has become overloaded if you see it as an ML problem. One classic thread is about memorization: regularization-based continual learning methods, such as EWC, MAS, and SI, estimate which parameters mattered for previous tasks and resist changing them too much. One modern thread is about adaptation: test-time training and inference-time learning methods, such as TTT, adapt part of the model on the incoming test stream before making predictions. These are sometimes discussed as separate threads. But in modern scalable architectures, I think they are better seen as complementary constraints: a model that learns quickly at test time also benefits from a mechanism for deciding what not to forget. In our #ECCV2026 paper, we study this in large-scale 4D reconstruction: how to build fast spatial memory that can adapt over long observation streams while reducing collapse and forgetting. Instead of using fully plastic test-time updates, we stabilize fast-weight adaptation with an elastic prior that balances adaptation and memory. Key ideas: - Elastic Test-Time Training: Fisher-weighted consolidation for fast-weight updates - EMA anchor weights that provide a moving reference for stability - Chunk-by-chunk inference for long 3D/4D observation streams We show that this scales across large 3D/4D pretraining settings, including both LRM-style and LVSM-style models, and improves reconstruction across benchmarks including Stereo4D, NVIDIA, and DL3DV-140. We release model checkpoints across different design choices: resolution, post-training curriculum, and whether the model uses an explicit 4DGS intermediate representation. - Homepage: - Paper: - Code: - Models: This work is co-led with Xueyang Yu, contributed by Haoyu Zhen Yuncong Yang, and advised by Michigan SLED Lab Chuang Gan.

Martin Ziqiao Ma

32,705 просмотров • 24 дней назад

Introducing VL-JEPA: Vision-Language Joint Embedding Predictive Architecture for streaming, live action recognition, retrieval, VQA, and classification tasks with better performance and higher efficiency than large VLMs. • VL-JEPA is the first non-generative model that can perform general-domain vision-language tasks in real-time, built on a joint embedding predictive architecture. • We demonstrate in controlled experiments that VL-JEPA, trained with latent space embedding prediction, outperforms VLMs that rely on data space token prediction. • We show that VL-JEPA delivers significant efficiency gains over VLMs for online video streaming applications, thanks to its non-autoregressive design and native support for selective decoding. • We highlight that our VL-JEPA model, with an unified model architecture, can effectively handle a wide range of classification, retrieval, and VQA tasks at the same time. by Delong Chen (陈德龙) Mustafa Shukor Théo Moutakanni Willy Jade Lei Yu Tejaswi Kasarla Allen Bolourchi Yann LeCun Pascale Fung
1:51

Introducing VL-JEPA: Vision-Language Joint Embedding Predictive Architecture for streaming, live action recognition, retrieval, VQA, and classification tasks with better performance and higher efficiency than large VLMs. • VL-JEPA is the first non-generative model that can perform general-domain vision-language tasks in real-time, built on a joint embedding predictive architecture. • We demonstrate in controlled experiments that VL-JEPA, trained with latent space embedding prediction, outperforms VLMs that rely on data space token prediction. • We show that VL-JEPA delivers significant efficiency gains over VLMs for online video streaming applications, thanks to its non-autoregressive design and native support for selective decoding. • We highlight that our VL-JEPA model, with an unified model architecture, can effectively handle a wide range of classification, retrieval, and VQA tasks at the same time. by Delong Chen (陈德龙) Mustafa Shukor Théo Moutakanni Willy Jade Lei Yu Tejaswi Kasarla Allen Bolourchi Yann LeCun Pascale Fung

Pascale Fung

90,144 просмотров • 7 месяцев назад

As a newly appointed 𝗔𝘀𝘀𝗶𝘀𝘁𝗮𝗻𝘁 𝗣𝗿𝗼𝗳𝗲𝘀𝘀𝗼𝗿 at Imperial College London, I'm thrilled to announce the 𝗦𝗮𝗳𝗲 𝗪𝗵𝗼𝗹𝗲-𝗯𝗼𝗱𝘆 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝘁 𝗥𝗼𝗯𝗼𝘁𝗶𝗰𝘀 𝗟𝗮𝗯 (𝗦𝗪𝗜𝗥𝗟) at 𝗜𝗺𝗽𝗲𝗿𝗶𝗮𝗹 𝗖𝗼𝗹𝗹𝗲𝗴𝗲 𝗟𝗼𝗻𝗱𝗼𝗻. 𝗦𝗮𝗳𝗲 𝗪𝗵𝗼𝗹𝗲-𝗯𝗼𝗱𝘆 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝘁 𝗥𝗼𝗯𝗼𝘁𝗶𝗰𝘀 𝗟𝗮𝗯 (𝗦𝗪𝗜𝗥𝗟) ( is a new research lab focused on the intersection of safety and intelligence in next-generation robotics. We're hiring exceptional PhD students who are passionate about pushing the boundaries of robot learning. 𝗪𝗵𝗮𝘁 𝗺𝗮𝗸𝗲𝘀 𝗦𝗪𝗜𝗥𝗟 𝘂𝗻𝗶𝗾𝘂𝗲? We operate at the exciting convergence of: • Online & offline reinforcement learning • Imitation learning & human demonstrations • Sample-efficient learning methods • Whole-body and soft robotics systems We're 𝗹𝗼𝗼𝗸𝗶𝗻𝗴 𝗳𝗼𝗿 𝗽𝗿𝗼𝘀𝗽𝗲𝗰𝘁𝗶𝘃𝗲 𝗣𝗵𝗗 𝘀𝘁𝘂𝗱𝗲𝗻𝘁𝘀 interested in: • Developing safe exploration algorithms for robotic systems • Creating sample-efficient learning methods that minimize real-world trials • Building foundation models for robotics with safety guarantees • Advancing soft robotics and compliant human-robot interaction • Bridging theory and practice in embodied AI Why now? As robots become more capable and work closer with humans, we need systems that are both intelligent enough to handle complex tasks 𝗔𝗡𝗗 safe enough for real-world deployment. Traditional approaches treat safety and intelligence as competing priorities, we believe they're synergistic. If you're a motivated researcher who wants to develop the theoretical foundations and practical algorithms for tomorrow's safe, intelligent robots, I'd love to hear from you. Want to join? Apply via
0:15

As a newly appointed 𝗔𝘀𝘀𝗶𝘀𝘁𝗮𝗻𝘁 𝗣𝗿𝗼𝗳𝗲𝘀𝘀𝗼𝗿 at Imperial College London, I'm thrilled to announce the 𝗦𝗮𝗳𝗲 𝗪𝗵𝗼𝗹𝗲-𝗯𝗼𝗱𝘆 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝘁 𝗥𝗼𝗯𝗼𝘁𝗶𝗰𝘀 𝗟𝗮𝗯 (𝗦𝗪𝗜𝗥𝗟) at 𝗜𝗺𝗽𝗲𝗿𝗶𝗮𝗹 𝗖𝗼𝗹𝗹𝗲𝗴𝗲 𝗟𝗼𝗻𝗱𝗼𝗻. 𝗦𝗮𝗳𝗲 𝗪𝗵𝗼𝗹𝗲-𝗯𝗼𝗱𝘆 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝘁 𝗥𝗼𝗯𝗼𝘁𝗶𝗰𝘀 𝗟𝗮𝗯 (𝗦𝗪𝗜𝗥𝗟) ( is a new research lab focused on the intersection of safety and intelligence in next-generation robotics. We're hiring exceptional PhD students who are passionate about pushing the boundaries of robot learning. 𝗪𝗵𝗮𝘁 𝗺𝗮𝗸𝗲𝘀 𝗦𝗪𝗜𝗥𝗟 𝘂𝗻𝗶𝗾𝘂𝗲? We operate at the exciting convergence of: • Online & offline reinforcement learning • Imitation learning & human demonstrations • Sample-efficient learning methods • Whole-body and soft robotics systems We're 𝗹𝗼𝗼𝗸𝗶𝗻𝗴 𝗳𝗼𝗿 𝗽𝗿𝗼𝘀𝗽𝗲𝗰𝘁𝗶𝘃𝗲 𝗣𝗵𝗗 𝘀𝘁𝘂𝗱𝗲𝗻𝘁𝘀 interested in: • Developing safe exploration algorithms for robotic systems • Creating sample-efficient learning methods that minimize real-world trials • Building foundation models for robotics with safety guarantees • Advancing soft robotics and compliant human-robot interaction • Bridging theory and practice in embodied AI Why now? As robots become more capable and work closer with humans, we need systems that are both intelligent enough to handle complex tasks 𝗔𝗡𝗗 safe enough for real-world deployment. Traditional approaches treat safety and intelligence as competing priorities, we believe they're synergistic. If you're a motivated researcher who wants to develop the theoretical foundations and practical algorithms for tomorrow's safe, intelligent robots, I'd love to hear from you. Want to join? Apply via

Stephen James

16,552 просмотров • 9 месяцев назад

I believe solving robotics = 90% engineering + 10% research vision. Project GR00T is NVIDIA's moonshot initiative to build physical AGI for humanoid robots. The GEAR Lab is assembling a crack team right now. Join us! Openings: - Sr. Research Engineer, Robotics Systems - Sr. RE, Reinforcement Learning - Sr. RE, Foundation Model Training Infrastructure - Sr. RE, Simulation - Sr. RE, ML Data Pipelines - Research Scientist - Research Intern (both part-time and summer full-time in 2025) For the Sr. positions, we strongly prefer candidates with many years of engineering experience at robotics/autonomous driving companies, or MLOps/large-scale AI teams at big techs. For interns, we welcome ace robotics hackers anywhere! Show me your past works. Job links in the thread. Apply today! Your resumes will be my best Christmas gifts:
0:16

I believe solving robotics = 90% engineering + 10% research vision. Project GR00T is NVIDIA's moonshot initiative to build physical AGI for humanoid robots. The GEAR Lab is assembling a crack team right now. Join us! Openings: - Sr. Research Engineer, Robotics Systems - Sr. RE, Reinforcement Learning - Sr. RE, Foundation Model Training Infrastructure - Sr. RE, Simulation - Sr. RE, ML Data Pipelines - Research Scientist - Research Intern (both part-time and summer full-time in 2025) For the Sr. positions, we strongly prefer candidates with many years of engineering experience at robotics/autonomous driving companies, or MLOps/large-scale AI teams at big techs. For interns, we welcome ace robotics hackers anywhere! Show me your past works. Job links in the thread. Apply today! Your resumes will be my best Christmas gifts:

Jim Fan

103,177 просмотров • 1 год назад

Fine-tune DeepSeek-OCR on your own language! (100% local) DeepSeek-OCR is a 3B-parameter vision model that achieves 97% precision while using 10× fewer vision tokens than text-based LLMs. It handles tables, papers, and handwriting without killing your GPU or budget. Why it matters: Most vision models treat documents as massive sequences of tokens, making long-context processing expensive and slow. DeepSeek-OCR uses context optical compression to convert 2D layouts into vision tokens, enabling efficient processing of complex documents. The best part? You can easily fine-tune it for your specific use case on a single GPU. I used Unsloth to run this experiment on Persian text and saw an 88.26% improvement in character error rate. ↳ Base model: 149% character error rate (CER) ↳ Fine-tuned model: 60% CER (57% more accurate) ↳ Training time: 60 steps on a single GPU Persian was just the test case. You can swap in your own dataset for any language, document type, or specific domain you're working with. I've shared the complete guide in the next tweet - all the code, notebooks, and environment setup ready to run with a single click. Everything is 100% open-source!
0:14

Fine-tune DeepSeek-OCR on your own language! (100% local) DeepSeek-OCR is a 3B-parameter vision model that achieves 97% precision while using 10× fewer vision tokens than text-based LLMs. It handles tables, papers, and handwriting without killing your GPU or budget. Why it matters: Most vision models treat documents as massive sequences of tokens, making long-context processing expensive and slow. DeepSeek-OCR uses context optical compression to convert 2D layouts into vision tokens, enabling efficient processing of complex documents. The best part? You can easily fine-tune it for your specific use case on a single GPU. I used Unsloth to run this experiment on Persian text and saw an 88.26% improvement in character error rate. ↳ Base model: 149% character error rate (CER) ↳ Fine-tuned model: 60% CER (57% more accurate) ↳ Training time: 60 steps on a single GPU Persian was just the test case. You can swap in your own dataset for any language, document type, or specific domain you're working with. I've shared the complete guide in the next tweet - all the code, notebooks, and environment setup ready to run with a single click. Everything is 100% open-source!

Akshay 🚀

126,091 просмотров • 8 месяцев назад

Uber is Dead, my reflections on Waymo I’ve been in San Francisco for just over a week, during which I’ve taken 7 rides with Waymo, a similar number with Uber, and a few with FSD Teslas. My journey to SFO via Uber was alarming—the driver veered out of the lane multiple times and nearly crashed on a ramp, seemingly vying for a one-star rating or to genuinely scare me. Conversely, my experiences with Waymo were virtually flawless, if you don’t consider overly cautious driving a fault. I experienced a minor hiccup when we got stuck behind parked cars because the vehicle thought they were queuing at a red light. It quickly resolved the confusion and moved on, which was rather amusing. Waymo, and other Level 5 autonomous vehicles, are poised to revolutionize the movement of people and goods. The most apt analogy I can think of is that Waymo is transforming the real world into an automated Amazon warehouse, with people as the goods and Waymo vehicles as the robots shuttling them around. With the advent of personal transportation becoming incredibly affordable, sending anything from point A to point B using a self-driving electric vehicle will soon be within easy reach. One of Waymo’s standout features is privacy. Riding in an Uber often means being subjected to the driver’s loud group chats on some app, making the journey neither quiet nor private. In contrast, Waymo offers a fully private experience, allowing you to have confidential phone conversations or chat freely with fellow passengers without distraction. Waymo also reimagines the concept of a car. Without the need for a driver, we can eliminate the front console, reduce weight, and remove the steering wheel. This opens up possibilities for passenger seats to be reoriented, perhaps facing backwards, or for the vehicle to become a mobile living room. Tomorrow’s vehicle designs will differ drastically from today’s. Destinations that are currently expensive and logistically complicated to reach via Taxi/Uber, often lying outside public transport routes, can be simplified to a single “Waymo” journey. This could shift the current model of “Uber + public transport + Uber” to a more streamlined experience. As more cars become self-driving, we could see a reduction in the amount of time cars are parked—from 99% of their lifetime to perhaps just 25%. This not only improves unit economics but could also decrease the number of cars on the road. This transition represents one of the most significant shifts for Generation X. In conclusion, the future is autonomous, electric, and efficient. Uber, as we know it, is dead.
0:14

Uber is Dead, my reflections on Waymo I’ve been in San Francisco for just over a week, during which I’ve taken 7 rides with Waymo, a similar number with Uber, and a few with FSD Teslas. My journey to SFO via Uber was alarming—the driver veered out of the lane multiple times and nearly crashed on a ramp, seemingly vying for a one-star rating or to genuinely scare me. Conversely, my experiences with Waymo were virtually flawless, if you don’t consider overly cautious driving a fault. I experienced a minor hiccup when we got stuck behind parked cars because the vehicle thought they were queuing at a red light. It quickly resolved the confusion and moved on, which was rather amusing. Waymo, and other Level 5 autonomous vehicles, are poised to revolutionize the movement of people and goods. The most apt analogy I can think of is that Waymo is transforming the real world into an automated Amazon warehouse, with people as the goods and Waymo vehicles as the robots shuttling them around. With the advent of personal transportation becoming incredibly affordable, sending anything from point A to point B using a self-driving electric vehicle will soon be within easy reach. One of Waymo’s standout features is privacy. Riding in an Uber often means being subjected to the driver’s loud group chats on some app, making the journey neither quiet nor private. In contrast, Waymo offers a fully private experience, allowing you to have confidential phone conversations or chat freely with fellow passengers without distraction. Waymo also reimagines the concept of a car. Without the need for a driver, we can eliminate the front console, reduce weight, and remove the steering wheel. This opens up possibilities for passenger seats to be reoriented, perhaps facing backwards, or for the vehicle to become a mobile living room. Tomorrow’s vehicle designs will differ drastically from today’s. Destinations that are currently expensive and logistically complicated to reach via Taxi/Uber, often lying outside public transport routes, can be simplified to a single “Waymo” journey. This could shift the current model of “Uber + public transport + Uber” to a more streamlined experience. As more cars become self-driving, we could see a reduction in the amount of time cars are parked—from 99% of their lifetime to perhaps just 25%. This not only improves unit economics but could also decrease the number of cars on the road. This transition represents one of the most significant shifts for Generation X. In conclusion, the future is autonomous, electric, and efficient. Uber, as we know it, is dead.

Linus ✦ Ekenstam

6,099,881 просмотров • 2 лет назад

Learning is something you and your baby do together. You can think of the process as happening in three distinct stages, during which skills are transferred gradually from you to your little one: During the first stage, your baby is observing the behavior and skills of others. During the second, they begin to emulate these behaviors - and can find success with the support of a helpful adult (you) or more expert peer (often a sibling). And gradually they internalize these skills and perform them all by themselves. This video is a great example of the shared second phase. Infants explore the world with their mouths. But an important lesson of toddlerhood is that some things are for putting in our mouths, while others are not. This little one knows that we don’t eat the Play-Doh. But it sure is tempting! Watch as he breaks off a piece and brings it to his mouth. As he does his eyes lift and he realizes that Mom is watching - which alone prompts some introspection. He grins broadly, shakes his head and exclaims “No, no, no” - using Mom’s past words to affirm his decision to place the Play-Doh back on the table. Left to his own devices, who knows? But together, without exchanging a word, he managed to make the right choice. As a parent it’s important to remember the key role you play in the learning process. And that extends to your child’s behavior. Self-regulation begins as co-regulation. So be there. This sweet little guy was shared to IG by parentosa.
0:10

Learning is something you and your baby do together. You can think of the process as happening in three distinct stages, during which skills are transferred gradually from you to your little one: During the first stage, your baby is observing the behavior and skills of others. During the second, they begin to emulate these behaviors - and can find success with the support of a helpful adult (you) or more expert peer (often a sibling). And gradually they internalize these skills and perform them all by themselves. This video is a great example of the shared second phase. Infants explore the world with their mouths. But an important lesson of toddlerhood is that some things are for putting in our mouths, while others are not. This little one knows that we don’t eat the Play-Doh. But it sure is tempting! Watch as he breaks off a piece and brings it to his mouth. As he does his eyes lift and he realizes that Mom is watching - which alone prompts some introspection. He grins broadly, shakes his head and exclaims “No, no, no” - using Mom’s past words to affirm his decision to place the Play-Doh back on the table. Left to his own devices, who knows? But together, without exchanging a word, he managed to make the right choice. As a parent it’s important to remember the key role you play in the learning process. And that extends to your child’s behavior. Self-regulation begins as co-regulation. So be there. This sweet little guy was shared to IG by parentosa.

Dan Wuori

75,169 просмотров • 2 лет назад

THE KILLER of an MBBS Student & 2 other citizens The accused is Golden Sahani, reportedly a close aide of Minister Sanjay Nishad, and is said to be involved in the property business & local politics in Gorakhpur. On Holi, allegedly under the influence of alcohol, he rammed his Fortuner into three MBBS students, crushing them. One of them, Aakash Pandey, a 3rd-year MBBS student, died on the spot. When the police arrested him, he was seen walking arrogantly with his hands in his pockets, perhaps confident that nothing serious will happen to him. People believe that because of his political connections, he may walk out within a month or two and return to public life as if nothing happened. But what about the parents who lost their only son? A young, promising medical student lost his life in a moment of reckless drunken driving. Shockingly, a video from the same day also surfaced in which he was seen abusing and threatening a woman but the police allegedly failed to take timely action. If timely action had been taken by the police earlier, perhaps a bright young life would not have been lost today. Strict action must be taken so that such tragedies are never repeated #MedTwitter PMO India Yogi Adityanath UP POLICE #JusticeForAakashPandey
0:19

THE KILLER of an MBBS Student & 2 other citizens The accused is Golden Sahani, reportedly a close aide of Minister Sanjay Nishad, and is said to be involved in the property business & local politics in Gorakhpur. On Holi, allegedly under the influence of alcohol, he rammed his Fortuner into three MBBS students, crushing them. One of them, Aakash Pandey, a 3rd-year MBBS student, died on the spot. When the police arrested him, he was seen walking arrogantly with his hands in his pockets, perhaps confident that nothing serious will happen to him. People believe that because of his political connections, he may walk out within a month or two and return to public life as if nothing happened. But what about the parents who lost their only son? A young, promising medical student lost his life in a moment of reckless drunken driving. Shockingly, a video from the same day also surfaced in which he was seen abusing and threatening a woman but the police allegedly failed to take timely action. If timely action had been taken by the police earlier, perhaps a bright young life would not have been lost today. Strict action must be taken so that such tragedies are never repeated #MedTwitter PMO India Yogi Adityanath UP POLICE #JusticeForAakashPandey

Indian Doctor🇮🇳

37,586 просмотров • 4 месяцев назад

🚨 SCIENTISTS JUST BUILT A CHIP THAT CAN SEE, THINK, AND REMEMBER ALL AT THE SAME TIME. And it works more like a biological brain than a traditional computer. Researchers at RMIT University have created a neuromorphic vision chip that mimics the human eye and brain. Unlike conventional systems that capture images and send data to external processors, this chip performs sensing, processing, and memory storage directly where the light hits. The active layer is thousands of times thinner than a human hair. It uses doped indium oxide to detect light, process the information on-chip, and retain what it sees over time without constant electrical refreshing. Why this matters: • It dramatically cuts energy use and latency by eliminating data transfer to separate processors • Enables much faster real-time decision making for autonomous systems • Works more like biological vision than traditional machine vision • Could power the next generation of efficient edge AI in vehicles, robots, and remote sensors The deeper implication: For decades, we’ve built vision systems by bolting cameras, processors, and memory together like separate organs. This chip collapses those functions into one biological-style unit. It’s a step toward machines that don’t just “see” but actually perceive and remember in a more efficient, brain-like way. If scaled successfully, it could become a foundational component for autonomous systems that need to operate intelligently with minimal power and minimal delay. We’re moving from cameras that take pictures to chips that truly see. How do you think neuromorphic vision chips like this will change what’s possible for self-driving cars and autonomous robots? Follow for more frontier neuromorphic computing, AI hardware, and brain-inspired technology.
0:10

🚨 SCIENTISTS JUST BUILT A CHIP THAT CAN SEE, THINK, AND REMEMBER ALL AT THE SAME TIME. And it works more like a biological brain than a traditional computer. Researchers at RMIT University have created a neuromorphic vision chip that mimics the human eye and brain. Unlike conventional systems that capture images and send data to external processors, this chip performs sensing, processing, and memory storage directly where the light hits. The active layer is thousands of times thinner than a human hair. It uses doped indium oxide to detect light, process the information on-chip, and retain what it sees over time without constant electrical refreshing. Why this matters: • It dramatically cuts energy use and latency by eliminating data transfer to separate processors • Enables much faster real-time decision making for autonomous systems • Works more like biological vision than traditional machine vision • Could power the next generation of efficient edge AI in vehicles, robots, and remote sensors The deeper implication: For decades, we’ve built vision systems by bolting cameras, processors, and memory together like separate organs. This chip collapses those functions into one biological-style unit. It’s a step toward machines that don’t just “see” but actually perceive and remember in a more efficient, brain-like way. If scaled successfully, it could become a foundational component for autonomous systems that need to operate intelligently with minimal power and minimal delay. We’re moving from cameras that take pictures to chips that truly see. How do you think neuromorphic vision chips like this will change what’s possible for self-driving cars and autonomous robots? Follow for more frontier neuromorphic computing, AI hardware, and brain-inspired technology.

TheNewPhysics

23,196 просмотров • 29 дней назад

Brilliant open-source tool from Anthropic Visualize internal LLM pathways through open attribution graphs. → Attribution graphs reveal which neurons and features drive each output token, turning the LLM’s black box into a clear map. They open-sourced a library that generates attribution graphs to trace how LLMs arrive at outputs. These graphs reveal internal activation flows and feature influences in popular open-weights models. 🧠 Attribution Graphs Graphs link model activations to output tokens. Nodes represent neurons or features. Edges show influence strength. Researchers inspect paths to see which components drive each decision. 🛠️ Toolchain and Frontend Circuit-tracer library works with models like Gemma-2-2b and Llama-3.2-1b. Demo notebooks illustrate multi-step reasoning examples. Neuronpedia hosts an interactive UI for graph exploration, annotation, and sharing. 🔍 Hypothesis Testing Users tweak feature values in graphs, regenerate outputs, and observe changes. This lets researchers validate whether specific circuits control behaviors like language translation or reasoning chains.
0:17

Brilliant open-source tool from Anthropic Visualize internal LLM pathways through open attribution graphs. → Attribution graphs reveal which neurons and features drive each output token, turning the LLM’s black box into a clear map. They open-sourced a library that generates attribution graphs to trace how LLMs arrive at outputs. These graphs reveal internal activation flows and feature influences in popular open-weights models. 🧠 Attribution Graphs Graphs link model activations to output tokens. Nodes represent neurons or features. Edges show influence strength. Researchers inspect paths to see which components drive each decision. 🛠️ Toolchain and Frontend Circuit-tracer library works with models like Gemma-2-2b and Llama-3.2-1b. Demo notebooks illustrate multi-step reasoning examples. Neuronpedia hosts an interactive UI for graph exploration, annotation, and sharing. 🔍 Hypothesis Testing Users tweak feature values in graphs, regenerate outputs, and observe changes. This lets researchers validate whether specific circuits control behaviors like language translation or reasoning chains.

Rohan Paul

27,197 просмотров • 1 год назад

🧬 We have many foundation models or language models for DNAs, but can we control them? We introduce Ctrl-DNA: Controllable Cell-Type-Specific Regulatory DNA Design via Constrained RL — a reinforcement learning framework for controllable cis-regulatory sequence generation. Paper: Code: 🔬What’s the challenge? Designing regulatory DNA that is both highly expressive in target cell types and inactive in others is essential for synthetic biology, gene therapy, and precision medicine. Yet, controlling these trade-offs is challenging due to sparse, sequence-level rewards and biological constraints. 🔥Why Ctrl-DNA? Ctrl-DNA fine-tunes pre-trained DNA language models using a value model free, Lagrangian-guided RL framework, enabling flexible and customizable constraint optimization. Users can define application-specific thresholds across cell types, balancing expression strength with specificity. ✅ Maximize target-cell expression ✅ Constrain off-target activity under user-defined thresholds ✅ Preserve cell-type-specific TF motif structure Benchmarked on human enhancer and promoter datasets, Ctrl-DNA consistently outperforms prior methods, achieving stronger specificity, higher fitness, and more biologically grounded sequence generation — all with direct control over regulatory trade-offs. Shoutout to the PhD students Xingyu Chen (Xingyu Chen ) and Rex Ma (Rex Ma) for their amazing work leading this project!
0:12

🧬 We have many foundation models or language models for DNAs, but can we control them? We introduce Ctrl-DNA: Controllable Cell-Type-Specific Regulatory DNA Design via Constrained RL — a reinforcement learning framework for controllable cis-regulatory sequence generation. Paper: Code: 🔬What’s the challenge? Designing regulatory DNA that is both highly expressive in target cell types and inactive in others is essential for synthetic biology, gene therapy, and precision medicine. Yet, controlling these trade-offs is challenging due to sparse, sequence-level rewards and biological constraints. 🔥Why Ctrl-DNA? Ctrl-DNA fine-tunes pre-trained DNA language models using a value model free, Lagrangian-guided RL framework, enabling flexible and customizable constraint optimization. Users can define application-specific thresholds across cell types, balancing expression strength with specificity. ✅ Maximize target-cell expression ✅ Constrain off-target activity under user-defined thresholds ✅ Preserve cell-type-specific TF motif structure Benchmarked on human enhancer and promoter datasets, Ctrl-DNA consistently outperforms prior methods, achieving stronger specificity, higher fitness, and more biologically grounded sequence generation — all with direct control over regulatory trade-offs. Shoutout to the PhD students Xingyu Chen (Xingyu Chen ) and Rex Ma (Rex Ma) for their amazing work leading this project!

Bo Wang

30,719 просмотров • 1 год назад

Attempt to Kill? XL6 Driver Drags Man on Bonnet in Broad Daylight In Bengaluru A shocking case of road rage was caught on camera on 11th February 2026 at around 12:45 PM on Old Airport Road in Bengaluru. In the disturbing visuals, a Maruti Suzuki XL6 bearing registration number KA03NN4501 can be seen speeding recklessly while a man is lying on the bonnet of the moving vehicle. Eyewitnesses reported that the driver was driving at a very high speed, endangering not only the man on the bonnet but also other commuters on the busy stretch. The man on the bonnet was seen shouting and pleading with the driver to stop the car, but the driver allegedly continued to accelerate without concern for human life. The act has raised serious concerns about public safety and reckless behavior on city roads. Such behavior is not just irresponsible it is criminal. Dragging a person on the bonnet of a speeding car shows a complete disregard for human life and traffic laws. Roads are meant for safe travel, not for settling personal disputes in a life-threatening manner. Incidents like these could have easily resulted in a fatality.Upon further verification, it has come to light that the driver is reportedly a previous offender in cases related to rash driving. Additionally, checks revealed that the vehicle’s insurance policy had expired, which is another violation of the Motor Vehicles Act. Operating a vehicle without valid insurance and engaging in repeated reckless driving reflects blatant negligence and a lack of accountability. According to the DCP EAST, a case has been registered against the driver, and further investigation is underway. Authorities have assured that strict action will be taken as per the law. This incident once again highlights the growing menace of road rage and reckless driving in Bengaluru. Strict enforcement, heavy penalties, and cancellation of licenses for repeat offenders are necessary to ensure that such dangerous individuals are kept off the roads before innocent lives are lost. #bangalore #bengaluru #roadrage ಬೆಂಗಳೂರು ನಗರ ಪೊಲೀಸ್‌ BengaluruCityPolice ಬೆಂಗಳೂರು ಸಂಚಾರ ಪೊಲೀಸ್ BengaluruTrafficPolice CP Bengaluru ಪೊಲೀಸ್ ಆಯುಕ್ತ ಬೆಂಗಳೂರು Joint CP, Traffic, Bengaluru alok kumar DGP KARNATAKA ಕರ್ನಾಟಕ ರಾಜ್ಯ ಪೊಲೀಸ್ Karnataka State Police S. Lalitha ChristinMathewPhilip
0:18

Attempt to Kill? XL6 Driver Drags Man on Bonnet in Broad Daylight In Bengaluru A shocking case of road rage was caught on camera on 11th February 2026 at around 12:45 PM on Old Airport Road in Bengaluru. In the disturbing visuals, a Maruti Suzuki XL6 bearing registration number KA03NN4501 can be seen speeding recklessly while a man is lying on the bonnet of the moving vehicle. Eyewitnesses reported that the driver was driving at a very high speed, endangering not only the man on the bonnet but also other commuters on the busy stretch. The man on the bonnet was seen shouting and pleading with the driver to stop the car, but the driver allegedly continued to accelerate without concern for human life. The act has raised serious concerns about public safety and reckless behavior on city roads. Such behavior is not just irresponsible it is criminal. Dragging a person on the bonnet of a speeding car shows a complete disregard for human life and traffic laws. Roads are meant for safe travel, not for settling personal disputes in a life-threatening manner. Incidents like these could have easily resulted in a fatality.Upon further verification, it has come to light that the driver is reportedly a previous offender in cases related to rash driving. Additionally, checks revealed that the vehicle’s insurance policy had expired, which is another violation of the Motor Vehicles Act. Operating a vehicle without valid insurance and engaging in repeated reckless driving reflects blatant negligence and a lack of accountability. According to the DCP EAST, a case has been registered against the driver, and further investigation is underway. Authorities have assured that strict action will be taken as per the law. This incident once again highlights the growing menace of road rage and reckless driving in Bengaluru. Strict enforcement, heavy penalties, and cancellation of licenses for repeat offenders are necessary to ensure that such dangerous individuals are kept off the roads before innocent lives are lost. #bangalore #bengaluru #roadrage ಬೆಂಗಳೂರು ನಗರ ಪೊಲೀಸ್‌ BengaluruCityPolice ಬೆಂಗಳೂರು ಸಂಚಾರ ಪೊಲೀಸ್ BengaluruTrafficPolice CP Bengaluru ಪೊಲೀಸ್ ಆಯುಕ್ತ ಬೆಂಗಳೂರು Joint CP, Traffic, Bengaluru alok kumar DGP KARNATAKA ಕರ್ನಾಟಕ ರಾಜ್ಯ ಪೊಲೀಸ್ Karnataka State Police S. Lalitha ChristinMathewPhilip

Karnataka Portfolio

29,082 просмотров • 5 месяцев назад

🚀 Introducing EgoExo Forge - built on top of Rerun, Gradio, and Hugging Face hub (I’ll be in San Francisco July 21–29 — if you’re into robotics, egocentric AI, large-scale data collection, or just want to chat, DM me!) In my opinion, large-scale, diverse, and high-quality data is still the largest bottleneck for generalized robotics deployment. I believe that some version of imitation learning from human examples will be the most scalable + clean way to train humanoid robots 🤖 (similar to what Tesla did for Full Self Driving). Teleop is too expensive to collect a large enough dataset in a reasonable manner, so passive collection via egocentric (and in certain cases, exocentric) views feels like the right bet. Over the past few months, I've been trying to build out the scaffolding for this and using Rerun as my underlying infrastructure. Data being collected needs to be easily inspectable + time series and rerun provides the right tooling for this. My goal is to first build out a ground truth representative dataset from already existing open source data, generate some reasonable baselines, and then go out and collect my own data that adheres to the defined schema. 🔍 Starting with open-source datasets 1. EgoDex from Apple 2. HOCap from Nvidia and the University of Texas at Dallas 3. Assembly101 from Meta All these different datasets have different sensor configurations + annotations, so my goal with egoexo-forge is to have one consistent labeling scheme + data layout. I built a data pipeline that aligns all of the different datasets in one general schema assuming the COCO133 keypoint layout that allows for exo+ego, ego only, or exo only Since the scaffolding is already there, it becomes MUCH easier to add other datasets. So the next ones that I'll be including are HD-EPIC kitchens dataset, HOT3D, and finally my own personal iPhone + insta360 go collection method. Once I have a diverse variety of datasets, I'll double down on what I believe to be the key algorithms required to make useful data for imitation learning 📊 1. Camera Pose estimation via SLAM/SFM for ego perspective (and automatic calibration for exo) 2. Human pose estimation for both egocentric + exocentric views 3. Metric 3D reconstruction + object tracking I'll be setting up reasonable open-source baselines for each of these to validate that these datasets work, and then finally try to use the generated datasets for some imitation learning via the pi0-lerobot repo I've been working on. I plan on making a blog post + providing more info on all of this in the near future so stay tuned
0:18

🚀 Introducing EgoExo Forge - built on top of Rerun, Gradio, and Hugging Face hub (I’ll be in San Francisco July 21–29 — if you’re into robotics, egocentric AI, large-scale data collection, or just want to chat, DM me!) In my opinion, large-scale, diverse, and high-quality data is still the largest bottleneck for generalized robotics deployment. I believe that some version of imitation learning from human examples will be the most scalable + clean way to train humanoid robots 🤖 (similar to what Tesla did for Full Self Driving). Teleop is too expensive to collect a large enough dataset in a reasonable manner, so passive collection via egocentric (and in certain cases, exocentric) views feels like the right bet. Over the past few months, I've been trying to build out the scaffolding for this and using Rerun as my underlying infrastructure. Data being collected needs to be easily inspectable + time series and rerun provides the right tooling for this. My goal is to first build out a ground truth representative dataset from already existing open source data, generate some reasonable baselines, and then go out and collect my own data that adheres to the defined schema. 🔍 Starting with open-source datasets 1. EgoDex from Apple 2. HOCap from Nvidia and the University of Texas at Dallas 3. Assembly101 from Meta All these different datasets have different sensor configurations + annotations, so my goal with egoexo-forge is to have one consistent labeling scheme + data layout. I built a data pipeline that aligns all of the different datasets in one general schema assuming the COCO133 keypoint layout that allows for exo+ego, ego only, or exo only Since the scaffolding is already there, it becomes MUCH easier to add other datasets. So the next ones that I'll be including are HD-EPIC kitchens dataset, HOT3D, and finally my own personal iPhone + insta360 go collection method. Once I have a diverse variety of datasets, I'll double down on what I believe to be the key algorithms required to make useful data for imitation learning 📊 1. Camera Pose estimation via SLAM/SFM for ego perspective (and automatic calibration for exo) 2. Human pose estimation for both egocentric + exocentric views 3. Metric 3D reconstruction + object tracking I'll be setting up reasonable open-source baselines for each of these to validate that these datasets work, and then finally try to use the generated datasets for some imitation learning via the pi0-lerobot repo I've been working on. I plan on making a blog post + providing more info on all of this in the near future so stay tuned

Pablo Vela

32,085 просмотров • 1 год назад