正在加载视频...

视频加载失败

Deep SSMs are great in modeling audio / language, but they really excel in modeling continuous, raw sensor data. Introducing Hierarchical SSMs (HiSS), a new framework that improves upon Transformers, RNNs, S4 & Mamba for a range of sensors (touch, accelerometers & IMUs). 🧵👇

19,166 次观看 • 2 年前 •via X (Twitter)

5 条评论

Lerrel Pinto 的头像
Lerrel Pinto2 年前

HiSS is conceptually simple: Just divide a long raw sequence into chunks which are processed into chunked features by a low-level SSM. A high-level SSM then maps the features to an output sequence. This temporal hierarchy is critical to learning from small sensor datasets!

Lerrel Pinto 的头像
Lerrel Pinto2 年前

To study continuous sequence prediction (CSP), we put together a benchmark called CSP-Bench that has 6 datasets, 3 created in-house and 3 curated from prior work. CSP-Bench has a total of 40hrs of data across touch sensors (ReSkin, XELA), accelerometers, and IMU data.

Lerrel Pinto 的头像
Lerrel Pinto2 年前

Interesting insights: 💡Flat SSM models (S4 & Mamba) are much stronger than causal transformers or RNNs by ~10% MSE on avg. 💡HiSS further improves upon flat SSMs by ~23% MSE on avg. 💡HiSS is also generally better on smaller datasets

Lerrel Pinto 的头像
Lerrel Pinto2 年前

All the benchmark data (CSP-Bench) and code is public on our project page: HiSS was created by @Raunaqmb w/ Chenyu, Venkatesh, Carmel, Abhinav & Tess.

Masato Kobayashi 的头像
Masato Kobayashi2 年前

This is so great! Thank you very much!

相关视频

🚨 BOMBSHELL: The CHILLING Epstein Files Clip That Shows How the GROOMING Operation Worked —He Used The Lure Of "Modeling" to TRAP His Victims!🤯 This isn't just about abuse; this is about the method. This clip is the smoking gun that proves this was a sophisticated, manipulative grooming operation disguised as a "modeling" shoot. The clip starts with Jeffrey Epstein coaching a young woman with a European accent to pose for a photo. She's trying, but it's not enough for him. He says, "Is that the best you can do?" She replies, "Yes." Epstein pushes back: "No, come on, do like a modeling photo." The girl is perplexed. She answers back: "A modeling photo?" And then... REDACTED. The audio goes silent. The government cuts the tape right at the most critical moment. When the audio returns, we hear the aftermath of whatever a "modeling photo" entailed. A male voice says, "That's not... Oh no. Your boob is sticking out." The woman denies it. This is the playbook. Epstein is using the alluring promise of "modeling" to lure this young woman into a specific, exploitative pose. The fact that she seems confused or unsure proves she's not a willing participant in the way they'd have you believe. She's being manipulated. And the government's redaction is the second crime here. What happened in those silent seconds? What instructions were given? What was said that they don't want the public to hear? They are actively protecting the methodology of the grooming operation. This is a pattern i keep seeing in all of these videos in the files. This clip is the key. It shows the premeditation, the coaching, the bait-and-switch, and the system of exploitation. It proves this wasn't random; it was a well-oiled machine, using the dream of modeling as a trap. They can redact the audio, but they can't redact the truth. We know what this "modeling" led to. We know what this operation was all about. The redaction is the confession. They are hiding how the operation worked. Share this everywhere. The world needs to see the manipulation for themselves.
0:30

Sensitive content

🚨 BOMBSHELL: The CHILLING Epstein Files Clip That Shows How the GROOMING Operation Worked —He Used The Lure Of "Modeling" to TRAP His Victims!🤯 This isn't just about abuse; this is about the method. This clip is the smoking gun that proves this was a sophisticated, manipulative grooming operation disguised as a "modeling" shoot. The clip starts with Jeffrey Epstein coaching a young woman with a European accent to pose for a photo. She's trying, but it's not enough for him. He says, "Is that the best you can do?" She replies, "Yes." Epstein pushes back: "No, come on, do like a modeling photo." The girl is perplexed. She answers back: "A modeling photo?" And then... REDACTED. The audio goes silent. The government cuts the tape right at the most critical moment. When the audio returns, we hear the aftermath of whatever a "modeling photo" entailed. A male voice says, "That's not... Oh no. Your boob is sticking out." The woman denies it. This is the playbook. Epstein is using the alluring promise of "modeling" to lure this young woman into a specific, exploitative pose. The fact that she seems confused or unsure proves she's not a willing participant in the way they'd have you believe. She's being manipulated. And the government's redaction is the second crime here. What happened in those silent seconds? What instructions were given? What was said that they don't want the public to hear? They are actively protecting the methodology of the grooming operation. This is a pattern i keep seeing in all of these videos in the files. This clip is the key. It shows the premeditation, the coaching, the bait-and-switch, and the system of exploitation. It proves this wasn't random; it was a well-oiled machine, using the dream of modeling as a trap. They can redact the audio, but they can't redact the truth. We know what this "modeling" led to. We know what this operation was all about. The redaction is the confession. They are hiding how the operation worked. Share this everywhere. The world needs to see the manipulation for themselves.

Project Constitution

54,678 次观看 • 3 个月前

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

AK

148,998 次观看 • 1 年前

New short course: Build Long-Context AI Apps with Jamba. Learn about state space models (SSMs), which have emerged as an alternative to transformers! Specifically, Jamba is a hybrid transformer-Mamba architecture that combines strengths of the transformer with ideas from SSMs. This course is built with AI21 Labs and taught by Chen Wang and Chen Almagor. The transformer architecture is computationally expensive when handling very long input contexts. But there's an alternative called Mamba, a selective state space model that can process very long contexts with a much lower computational cost. However, researchers found that the pure Mamba architecture underperforms in understanding the context, and gives lower-quality responses. To overcome this, AI21 developed the Jamba model, which combines Mamba's computational efficiency with the transformer's attention mechanism to help with the output quality. In this course, you’ll learn about how state space models, and Jamba, work. You’ll also learn how to prompt Jamba, use it to process long documents, and build long-context RAG apps. - Learn how Jamba combines transformer and state space model architectures to achieve high performance and quality - Use the AI21 SDK, with an example of prompting over a large 200k-token annual financial report of Nvidia - Use Jamba for tool-calling, with hands-on examples from calling simple arithmetic calculations to a function that returns quarterly company financial reports. - Learn how training for long context is done, and the metrics used for its evaluation - Create a RAG app using the AI21 Conversational RAG tool and build your own RAG pipeline that uses Jamba and LangChain. By the end of this course, you'll learn how to build applications that can handle context as long as an entire book. Please sign up here:

Andrew Ng

77,792 次观看 • 1 年前