正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Introducing Masked Trajectory Modeling (MTM), a new general-purpose framework for sequential decision making. A single transformer trained with MTM can exhibit multiple capabilities by simply choosing different masking patterns at inference time. Accepted at ICML 2023. 🧵👇

Philipp Wu

3,113 subscribers

93,121 次观看 • 3 年前 •via X (Twitter)

教育科学技术新闻政治

Anya Rossi• Live Now

Private livecam show

11 条评论

Philipp Wu 的头像

Philipp Wu3 年前

MTM trains by reconstructing a masked input sequence using random autoregressive masking (think BERT with a prediction prior). We prompt the model with different masking patterns to achieve specific capabilities such as behavior cloning and inverse dynamics.

Philipp Wu 的头像

Philipp Wu3 年前

We also evaluate MTM in offline RL settings, such as D4RL. At test time we prompt the model with a desired return. We find that MTM is comparable to prior work while also also having the capability to be used for other tasks.

Philipp Wu 的头像

Philipp Wu3 年前

MTM is able to train on datasets with missing data modes. Ex: we might have a large dataset of state trajectories, but only a small amount with action labels. MTM is able to train on both without any change to the algorithm, simply by treating missing data modes as a forced mask.

Philipp Wu 的头像

Philipp Wu3 年前

We test if MTM is able to leverage the trajectories without actions. Here we compare various methods that only train on the labeled subset and our Heteromodal MTM which also trains on trajectories without action labels. Heteromodal MTM is able to improve task performance.

Philipp Wu 的头像

Philipp Wu3 年前

MTM can also be used to learn representations. After pretraining on generic trajectories, we use TD3 on top of the learned MTM representations. We find that these representations help with faster learning and in some cases, improve on asymptotic performance.

Philipp Wu 的头像

Philipp Wu3 年前

If you are interested in this line of work, also checkout MaskDP ( from my lab mate @fangchenliu. Our work extends on some of the ideas presented here and explores a different set of capabilities and features of this paradigm.

Philipp Wu 的头像

Philipp Wu3 年前

If you want to get a quick start with MTM, checkout this notebook which illustrates how to train and use MTM on a simple sinusoidal dataset.

Philipp Wu 的头像

Philipp Wu3 年前

Thanks for the help and assistance of amazing collaborators! @arjunmajum, @kevinleestone, @yixin_lin_, @IMordatch, @pabbeel, @aravindr93 If you are interested in more details checkout the links below! Website: Code:

Philipp Wu 的头像

Philipp Wu3 年前

Paper now out on arxiv!

kourosh hakhamaneshi 的头像

kourosh hakhamaneshi3 年前

Nice. It finally worked. Congrats. 🎉

Philipp Wu 的头像

Philipp Wu3 年前

😂 thanks kourosh!!

相关视频

Introducing Lemon Slice Live: the first real-time audio-video AI model Anyone can video chat with any character by just uploading a single photo. Every pixel on the screen is rendered at 25fps in real-time by a transformer model. Here’s how we built it + a live demo! 👇

Introducing Lemon Slice Live: the first real-time audio-video AI model Anyone can video chat with any character by just uploading a single photo. Every pixel on the screen is rendered at 25fps in real-time by a transformer model. Here’s how we built it + a live demo! 👇

LemonSlice

171,007 次观看 • 1 年前

Introducing MIRA. A playable, multiplayer world model. A dream of Rocket League. Trained on 10k hours of data collected with publicly available bots, MIRA learns the dynamics of a four-player game. The model runs in real time at 20 fps, based on the keys you and the other players press. Built by General Intuition and kyutai, in collaboration with Epic Games. Not used to develop Rocket League. ▶️ Play the demo, read the technical report, and explore the open-source code at At ICML? Find us at Booth 111 to try it yourself and dig into the results with the team.

Introducing MIRA. A playable, multiplayer world model. A dream of Rocket League. Trained on 10k hours of data collected with publicly available bots, MIRA learns the dynamics of a four-player game. The model runs in real time at 20 fps, based on the keys you and the other players press. Built by General Intuition and kyutai, in collaboration with Epic Games. Not used to develop Rocket League. ▶️ Play the demo, read the technical report, and explore the open-source code at At ICML? Find us at Booth 111 to try it yourself and dig into the results with the team.

General Intuition

455,097 次观看 • 18 天前

New to Featured Workflows: Seamless Transitions. Now you can create multiple generations with a single consistent subject interacting with different objects across various scenes and environments. All in a single, easy to use Workflow without the need for complex prompting. Explore the full collection of Featured Workflows or create your own at

New to Featured Workflows: Seamless Transitions. Now you can create multiple generations with a single consistent subject interacting with different objects across various scenes and environments. All in a single, easy to use Workflow without the need for complex prompting. Explore the full collection of Featured Workflows or create your own at

Runway

52,203 次观看 • 6 个月前

Punchline: World models == VQA (about the future)! Planning with world models can be powerful for robotics/control. But most world models are video generators trained to predict everything, including irrelevant pixels and distractions. We ask - what if a world model only predicted the semantic information necessary for decision-making? Introducing Semantic World Models (SWM). Given an observation and an action sequence, SWMs cast modeling as answering textual questions about the future outcome resulting from the actions. Recasting world modeling as a VQA problem lets us directly leverage the pretrained knowledge and machinery of VLMs for generalizable modeling. We had a lot of fun thinking about how this work helps connect these two seemingly very different fields of study - VLMs and world models! 🧵(1/6) Paper: Fun demo:

Punchline: World models == VQA (about the future)! Planning with world models can be powerful for robotics/control. But most world models are video generators trained to predict everything, including irrelevant pixels and distractions. We ask - what if a world model only predicted the semantic information necessary for decision-making? Introducing Semantic World Models (SWM). Given an observation and an action sequence, SWMs cast modeling as answering textual questions about the future outcome resulting from the actions. Recasting world modeling as a VQA problem lets us directly leverage the pretrained knowledge and machinery of VLMs for generalizable modeling. We had a lot of fun thinking about how this work helps connect these two seemingly very different fields of study - VLMs and world models! 🧵(1/6) Paper: Fun demo:

Abhishek Gupta

61,208 次观看 • 9 个月前

Germany's Chancellor Merz: For the first time, I have accepted the offer made by the French President to jointly think about nuclear deterrence. Historically speaking, General Charles de Gaulle had already offered the Federal Republic of Germany a French nuclear umbrella many decades ago. Germany never accepted that offer. At the time, we didn't need to, because we lived in a different world. But the world we live in today requires new answers. That is why we will, for the first time, take part in a French nuclear exercise with conventional forces in order to determine where we can achieve our shared objective of stronger deterrence in Europe.

Germany's Chancellor Merz: For the first time, I have accepted the offer made by the French President to jointly think about nuclear deterrence. Historically speaking, General Charles de Gaulle had already offered the Federal Republic of Germany a French nuclear umbrella many decades ago. Germany never accepted that offer. At the time, we didn't need to, because we lived in a different world. But the world we live in today requires new answers. That is why we will, for the first time, take part in a French nuclear exercise with conventional forces in order to determine where we can achieve our shared objective of stronger deterrence in Europe.

Clash Report

282,284 次观看 • 7 天前

Meta just released MusicGen, a simple and controllable model for music generation MusicGen is a single stage auto-regressive Transformer model trained over a 32kHz EnCodec tokenizer with 4 codebooks sampled at 50 Hz. Unlike existing methods like MusicLM, MusicGen doesn't not require a self-supervised semantic representation, and it generates all 4 codebooks in one pass. By introducing a small delay between the codebooks, can predict them in parallel, thus having only 50 auto-regressive steps per second of audio try out the Gradio demo: Models on Hugging Face: github:

Sensitive content

Meta just released MusicGen, a simple and controllable model for music generation MusicGen is a single stage auto-regressive Transformer model trained over a 32kHz EnCodec tokenizer with 4 codebooks sampled at 50 Hz. Unlike existing methods like MusicLM, MusicGen doesn't not require a self-supervised semantic representation, and it generates all 4 codebooks in one pass. By introducing a small delay between the codebooks, can predict them in parallel, thus having only 50 auto-regressive steps per second of audio try out the Gradio demo: Models on Hugging Face: github:

AK

627,450 次观看 • 3 年前

Can LLMs Self-Verify? Much better than you'd expect. LLMs are increasingly used as parallel reasoners, sampling many solutions at once. Choosing the right answer is the real bottleneck. We show that pairwise self-verification is a powerful primitive. Introducing V1, a framework that unifies generation and self-verification: 💡 Pairwise self-verification beats pointwise scoring, improving test-time scaling 💡 V1-Infer: Efficient tournament-style ranking that improves self-verification 💡 V1-PairRL: RL training where generation and verification co-evolve for developing better self-verifiers 🧵👇

Can LLMs Self-Verify? Much better than you'd expect. LLMs are increasingly used as parallel reasoners, sampling many solutions at once. Choosing the right answer is the real bottleneck. We show that pairwise self-verification is a powerful primitive. Introducing V1, a framework that unifies generation and self-verification: 💡 Pairwise self-verification beats pointwise scoring, improving test-time scaling 💡 V1-Infer: Efficient tournament-style ranking that improves self-verification 💡 V1-PairRL: RL training where generation and verification co-evolve for developing better self-verifiers 🧵👇

Harman Singh @ ICML 🇰🇷🇰🇷

107,027 次观看 • 4 个月前

A breakthrough in real-time video generation. As a research preview developed with NVIDIA and shared at NVIDIAGTC this week, we trained a new real-time video model running on Vera Rubin. HD videos generate instantly, with time-to-first-frame under 100ms. Unlocking an entirely new creative paradigm and bolstering the foundations of our General World Model, GWM-1. Real-time generation opens a fundamentally different design space for video models and world simulation. We're investing in co-designing our models alongside advances in hardware to keep pushing this frontier.

A breakthrough in real-time video generation. As a research preview developed with NVIDIA and shared at NVIDIAGTC this week, we trained a new real-time video model running on Vera Rubin. HD videos generate instantly, with time-to-first-frame under 100ms. Unlocking an entirely new creative paradigm and bolstering the foundations of our General World Model, GWM-1. Real-time generation opens a fundamentally different design space for video models and world simulation. We're investing in co-designing our models alongside advances in hardware to keep pushing this frontier.

Runway

1,162,775 次观看 • 4 个月前

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.

MrNeRF

34,782 次观看 • 1 年前

No way this isn’t the future of browsing. Safari in visionOS 27 lets you view up to five tabs side by side in a carousel of webpages, each as large as a poster, making it much easier to compare products, services, or travel accommodations. It’s also a major productivity boost for anyone who needs to monitor multiple sources of information at once or frequently jump between different websites. Instead of managing tabs, you can simply see everything at a glance. Super cool.

No way this isn’t the future of browsing. Safari in visionOS 27 lets you view up to five tabs side by side in a carousel of webpages, each as large as a poster, making it much easier to compare products, services, or travel accommodations. It’s also a major productivity boost for anyone who needs to monitor multiple sources of information at once or frequently jump between different websites. Instead of managing tabs, you can simply see everything at a glance. Super cool.

Phil Traut ᯅ

32,130 次观看 • 1 个月前

Sitting only a few feet away from Elder Gérald Caussé in the Joseph Smith Memorial Building on Friday, Nov. 7, was a grand piano. Following his remarks that morning, he mentioned his love again of his parents for their decision to become pioneers and for their perseverance that shined as an example to him. He paired their example with that of the pioneers of the Church and expressed his admiration for them. Elder Caussé sat down at the piano and began to play a hymn, first with a single finger, then both hands, then with his whole heart with the high standard that only a classically trained pianist can exhibit.

Sitting only a few feet away from Elder Gérald Caussé in the Joseph Smith Memorial Building on Friday, Nov. 7, was a grand piano. Following his remarks that morning, he mentioned his love again of his parents for their decision to become pioneers and for their perseverance that shined as an example to him. He paired their example with that of the pioneers of the Church and expressed his admiration for them. Elder Caussé sat down at the piano and began to play a hymn, first with a single finger, then both hands, then with his whole heart with the high standard that only a classically trained pianist can exhibit.

Church News

39,972 次观看 • 8 个月前

timelapse #74 (11.5 hrs): - 95% done the most insane transformer training and inference chapter ever (competing w/ llm.c at this point) - talking with Luminal team - contract work - watching Minecraft videos while waiting for claude code and build scripts - starting learning multiple things at same time so I can parallelize chapter creation in my book based on what im feeling at a given moment - went a layer deeper into quantization: training challenges, group-wise vs block-wise vs tensor-wise vs channel-wise vs all the wises, input type vs compute type vs accumulate type vs epilogue, dealing w/ outliers

timelapse #74 (11.5 hrs): - 95% done the most insane transformer training and inference chapter ever (competing w/ llm.c at this point) - talking with Luminal team - contract work - watching Minecraft videos while waiting for claude code and build scripts - starting learning multiple things at same time so I can parallelize chapter creation in my book based on what im feeling at a given moment - went a layer deeper into quantization: training challenges, group-wise vs block-wise vs tensor-wise vs channel-wise vs all the wises, input type vs compute type vs accumulate type vs epilogue, dealing w/ outliers

Elliot Arledge

121,805 次观看 • 10 个月前

In this demo we extend our prior work on obstacles avoidance at aggressive speeds, showcasing our Thar based autonomous vehicle navigating at near drift speeds, progressing towards our endeavour of Level-5 autonomy. Our autonomous vehicle at Swaayatt Robots was tasked with avoidance of traffic cones on the road, placed in a zig-zag fashion, at aggressive speeds. The location of the marked cones was not known to the planner beforehand. The #autonomousdriving task, i.e., motion planning (time parametrized trajectory computation) and decision making, was made even more challenging by restricting the AI agents to not act on obstacles unless they are within 24m radius. Level-5 #autonomousvehicles should be able to react quickly to overtake, or to avoid, any sudden unforeseeable obstacle or pedestrian on the road to avoid fatalities -- a capability demonstrated by our novel motion planning and decision making algorithmic framework over here. Our previous demo showcased our Bolero based platform consistently keeping speeds beyond 45 KMPH for most part, slowing down to only 39 KMPH at one point. Given Thar has lesser body roll, our framework successfully kept speeds well above 47 KMPH (even at the points of avoidance of obstacles), with speeds reaching as high as 55 KMPH. A typical human driver would feel uncomfortable at speeds beyond 40 KMPH in such as scenario. The entire algorithmic framework with 5 classical (one #reinforcementlearning-) agents , runs at 800+ Hz on a regular i7 processor, single thread. This algorithmic framework is being further scaled up with end-to-end deep reinforcement learning, and will be showcased in the month of March. #deeplearning #machinelearning #motionplanning

In this demo we extend our prior work on obstacles avoidance at aggressive speeds, showcasing our Thar based autonomous vehicle navigating at near drift speeds, progressing towards our endeavour of Level-5 autonomy. Our autonomous vehicle at Swaayatt Robots was tasked with avoidance of traffic cones on the road, placed in a zig-zag fashion, at aggressive speeds. The location of the marked cones was not known to the planner beforehand. The #autonomousdriving task, i.e., motion planning (time parametrized trajectory computation) and decision making, was made even more challenging by restricting the AI agents to not act on obstacles unless they are within 24m radius. Level-5 #autonomousvehicles should be able to react quickly to overtake, or to avoid, any sudden unforeseeable obstacle or pedestrian on the road to avoid fatalities -- a capability demonstrated by our novel motion planning and decision making algorithmic framework over here. Our previous demo showcased our Bolero based platform consistently keeping speeds beyond 45 KMPH for most part, slowing down to only 39 KMPH at one point. Given Thar has lesser body roll, our framework successfully kept speeds well above 47 KMPH (even at the points of avoidance of obstacles), with speeds reaching as high as 55 KMPH. A typical human driver would feel uncomfortable at speeds beyond 40 KMPH in such as scenario. The entire algorithmic framework with 5 classical (one #reinforcementlearning-) agents , runs at 800+ Hz on a regular i7 processor, single thread. This algorithmic framework is being further scaled up with end-to-end deep reinforcement learning, and will be showcased in the month of March. #deeplearning #machinelearning #motionplanning

Sanjeev Sharma

12,839 次观看 • 1 年前

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

AK

126,585 次观看 • 2 年前

Presenting our 6th Environmental, Social and Governance Report. The Konark Wheel at the Sun Temple in Odisha, associated with the cosmic cycle of ‘dharma’ (righteousness) and ‘karma’ (action), its eight spokes represent eight ‘prahar’ (time division) of a day, symbolising discipline and balance at the heart of governance. It reminds us that governance is not static but guided by purpose and measured decision-making. These values are at the foundation of our corporate governance framework. Just like the sun’s perpetual motion, the wheel’s circular form echoes the Bank’s continuous efforts in environmental conservation, societal development and sustainable business practices.

Presenting our 6th Environmental, Social and Governance Report. The Konark Wheel at the Sun Temple in Odisha, associated with the cosmic cycle of ‘dharma’ (righteousness) and ‘karma’ (action), its eight spokes represent eight ‘prahar’ (time division) of a day, symbolising discipline and balance at the heart of governance. It reminds us that governance is not static but guided by purpose and measured decision-making. These values are at the foundation of our corporate governance framework. Just like the sun’s perpetual motion, the wheel’s circular form echoes the Bank’s continuous efforts in environmental conservation, societal development and sustainable business practices.

ICICI Bank

2,244,501 次观看 • 11 个月前

Thomas Massie just set a trap for the next Attorney General, Todd Blanche, by introducing his new Epstein Files Transparency Act 2. The bill would require the U.S. Attorney General to cooperate with state Attorneys to help prosecute Jeffrey Epstein's top clients under state law. The bill aims to provide maximum disclosure of all the crimes against children that took place at Epstein's Zorro Ranch property. "Crimes were committed there." "The Attorney General of New Mexico would like to prosecute those crimes, so we're making it easier with this bill."

Thomas Massie just set a trap for the next Attorney General, Todd Blanche, by introducing his new Epstein Files Transparency Act 2. The bill would require the U.S. Attorney General to cooperate with state Attorneys to help prosecute Jeffrey Epstein's top clients under state law. The bill aims to provide maximum disclosure of all the crimes against children that took place at Epstein's Zorro Ranch property. "Crimes were committed there." "The Attorney General of New Mexico would like to prosecute those crimes, so we're making it easier with this bill."

Shadow of Ezra

377,184 次观看 • 8 天前

Type a sentence, get any sound - from talking cats to singing saxophones. Brilliant release by NVIDIA ✨ NVIDIA just unveiled Fugatto, a groundbreaking 2.5B parameter audio AI model that can generate and transform any combination of music, voices, and sounds using text prompts and audio inputs Fugatto could ultimately allow developers and creators to bring sounds to life simply by inputting text prompts, → The model demonstrates unique capabilities like creating hybrid sounds (trumpet barking), changing accents/emotions in voices, and allowing fine-grained control over sound transitions - trained on millions of audio samples using 32 NVIDIA H100 GPUs 👨‍🔧 Architecture Built as a foundational generative transformer model leveraging NVIDIA's previous work in speech modeling and audio understanding. The training process involved creating a specialized blended dataset containing millions of audio samples → ComposableART's Innovation in Audio Control Introduces a novel technique allowing combination of instructions that were only seen separately during training. Users can blend different audio attributes and control their intensity → Temporal Interpolation Capabilities Enables generation of evolving soundscapes with precise control over transitions. Can create dynamic audio sequences like rainstorms fading into birdsong at dawn → Processes both text and audio inputs flexibly, enabling tasks like removing instruments from songs or modifying specific audio characteristics while preserving others → Shows capabilities beyond its training data, creating entirely new sound combinations through interaction between different trained abilities 🔍 Real-world Applications → Allows rapid prototyping of musical ideas, style experimentation, and real-time sound creation during studio sessions → Enables dynamic audio asset generation matching gameplay situations, reducing pre-recorded audio requirements → Can modify voice characteristics for language learning applications, allowing content delivery in familiar voices NVIDIA AI Developer

Type a sentence, get any sound - from talking cats to singing saxophones. Brilliant release by NVIDIA ✨ NVIDIA just unveiled Fugatto, a groundbreaking 2.5B parameter audio AI model that can generate and transform any combination of music, voices, and sounds using text prompts and audio inputs Fugatto could ultimately allow developers and creators to bring sounds to life simply by inputting text prompts, → The model demonstrates unique capabilities like creating hybrid sounds (trumpet barking), changing accents/emotions in voices, and allowing fine-grained control over sound transitions - trained on millions of audio samples using 32 NVIDIA H100 GPUs 👨‍🔧 Architecture Built as a foundational generative transformer model leveraging NVIDIA's previous work in speech modeling and audio understanding. The training process involved creating a specialized blended dataset containing millions of audio samples → ComposableART's Innovation in Audio Control Introduces a novel technique allowing combination of instructions that were only seen separately during training. Users can blend different audio attributes and control their intensity → Temporal Interpolation Capabilities Enables generation of evolving soundscapes with precise control over transitions. Can create dynamic audio sequences like rainstorms fading into birdsong at dawn → Processes both text and audio inputs flexibly, enabling tasks like removing instruments from songs or modifying specific audio characteristics while preserving others → Shows capabilities beyond its training data, creating entirely new sound combinations through interaction between different trained abilities 🔍 Real-world Applications → Allows rapid prototyping of musical ideas, style experimentation, and real-time sound creation during studio sessions → Enables dynamic audio asset generation matching gameplay situations, reducing pre-recorded audio requirements → Can modify voice characteristics for language learning applications, allowing content delivery in familiar voices NVIDIA AI Developer

Rohan Paul

96,354 次观看 • 1 年前

Today at Meta FAIR we’re announcing three new cutting-edge developments in robotics and touch perception — and releasing a collection of artifacts to empower the community to build on this work. Details on all of this new work ➡️ 1️⃣ Meta Sparsh is the first general-purpose encoder for vision-based tactile sensing that works across many tactile sensors and many tasks. Trained on 460K+ tactile images using self-supervised learning. 2️⃣ Meta Digit 360 is a breakthrough artificial fingertip-based tactile sensor, equipped with 18+ sensing features to deliver detailed touch data with human-level precision and touch-sensing capabilities. 3️⃣ Meta Digit Plexus is a standardized platform for robotic sensor connections and interactions. It provides a hardware-software solution to integrate tactile sensors on a single robot hand and enables seamless data collection, control and analysis over a single cable. The potential impact of expanding capabilities and components like these for the open source community ranges from medical research to supply chain, manufacturing and much more. We’re excited to continue this work with the broader community.

Today at Meta FAIR we’re announcing three new cutting-edge developments in robotics and touch perception — and releasing a collection of artifacts to empower the community to build on this work. Details on all of this new work ➡️ 1️⃣ Meta Sparsh is the first general-purpose encoder for vision-based tactile sensing that works across many tactile sensors and many tasks. Trained on 460K+ tactile images using self-supervised learning. 2️⃣ Meta Digit 360 is a breakthrough artificial fingertip-based tactile sensor, equipped with 18+ sensing features to deliver detailed touch data with human-level precision and touch-sensing capabilities. 3️⃣ Meta Digit Plexus is a standardized platform for robotic sensor connections and interactions. It provides a hardware-software solution to integrate tactile sensors on a single robot hand and enables seamless data collection, control and analysis over a single cable. The potential impact of expanding capabilities and components like these for the open source community ranges from medical research to supply chain, manufacturing and much more. We’re excited to continue this work with the broader community.

AI at Meta

453,260 次观看 • 1 年前

New short course with Mistral AI ! Mistral's open-source Mixtral 8x7B model uses a "mixture of experts" (MoE) architecture. Unlike a standard transformer, an MoE model has multiple expert feed-forward networks (8 in this case), with a gating network selecting two experts at inference time. This enables MoE to match the performance of a large model but faster inference. Mixtral 8x7B has 46.7B parameters but activates only 12.9B at inference to predict the next token. In Getting Started with Mistral, you’ll learn from Mistral’s Sophia Yang, Ph.D. to: - Explore Mistral's open-source models (Mistral 7B, Mixtral 8x7B) and commercial models via API calls and Mistral AI's Le Chat website - Implement JSON mode to generate structured outputs to integrate directly into larger software systems. - Use function calling for Tool Use, such as calling custom Python code that queries tabular data - Ground your LLM's response with external knowledge sources using RAG - Build a Mistral-powered chat interface that can reference external documents This course will help deepen your prompt engineering skills. Please sign up here:

New short course with Mistral AI ! Mistral's open-source Mixtral 8x7B model uses a "mixture of experts" (MoE) architecture. Unlike a standard transformer, an MoE model has multiple expert feed-forward networks (8 in this case), with a gating network selecting two experts at inference time. This enables MoE to match the performance of a large model but faster inference. Mixtral 8x7B has 46.7B parameters but activates only 12.9B at inference to predict the next token. In Getting Started with Mistral, you’ll learn from Mistral’s Sophia Yang, Ph.D. to: - Explore Mistral's open-source models (Mistral 7B, Mixtral 8x7B) and commercial models via API calls and Mistral AI's Le Chat website - Implement JSON mode to generate structured outputs to integrate directly into larger software systems. - Use function calling for Tool Use, such as calling custom Python code that queries tabular data - Ground your LLM's response with external knowledge sources using RAG - Build a Mistral-powered chat interface that can reference external documents This course will help deepen your prompt engineering skills. Please sign up here:

Andrew Ng

387,018 次观看 • 2 年前

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏

Ilir Aliu

46,285 次观看 • 1 年前