Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

🎮 We release VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents (w/ Junyi Zhang Jiaxin Ge) 🌐 With 17 environments across multiple domains, we show systematically the brittleness of VLMs in visual interaction, and what training leads to. 🧵[1/8]

zwcolin's profile picture

Zirui "Colin" Wang

1,401 subscribers

40,334 Aufrufe • vor 4 Monaten •via X (Twitter)

BildungWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Happy to share our new work on Navigation World Models! 🔥🔥 Navigation is a fundamental skill of agents with visual-motor capabilities. We train a single World Model across multiple environments and diverse agent data. w/ Gaoyue Zhou, Danny Tran, trevordarrell and Yann LeCun.
1:01

Happy to share our new work on Navigation World Models! 🔥🔥 Navigation is a fundamental skill of agents with visual-motor capabilities. We train a single World Model across multiple environments and diverse agent data. w/ Gaoyue Zhou, Danny Tran, trevordarrell and Yann LeCun.

Amir Bar

83,539 Aufrufe • vor 1 Jahr

🚨Thrilled to present VisualAgentBench (VAB) with Yu Gu and Tianjie, where we enable both TRAINING & TESTING of visual foundation agents across 5 different environments! In all 17 large multimodal models (LMMs) are tested. Find our paper, data, and more insights below 👇 Paper: Code & Data: Thanks AK for sharing on today’s arxiv on HF!
2:41

🚨Thrilled to present VisualAgentBench (VAB) with Yu Gu and Tianjie, where we enable both TRAINING & TESTING of visual foundation agents across 5 different environments! In all 17 large multimodal models (LMMs) are tested. Find our paper, data, and more insights below 👇 Paper: Code & Data: Thanks AK for sharing on today’s arxiv on HF!

Xiao Liu (Shaw)

22,594 Aufrufe • vor 1 Jahr

Are black-box VLAs really the endgame for generalist robots? With SAGA, we explore an alternative path to efficiently generalize across environments, behaviors, and user specifications by explicitly grounding visual affordances in 3D space. 🧵
0:36

Are black-box VLAs really the endgame for generalist robots? With SAGA, we explore an alternative path to efficiently generalize across environments, behaviors, and user specifications by explicitly grounding visual affordances in 3D space. 🧵

Kuan Fang

197,834 Aufrufe • vor 6 Monaten

PufferLib 3.0: We trained reinforcement learning agents on 1 Petabyte / 12,000 years of data with 1 server. Now you can, too! Our latest release includes algorithmic breakthroughs, massively faster training, and 10 new environments. Live demos on our site. Volume on for trailer!
0:50

PufferLib 3.0: We trained reinforcement learning agents on 1 Petabyte / 12,000 years of data with 1 server. Now you can, too! Our latest release includes algorithmic breakthroughs, massively faster training, and 10 new environments. Live demos on our site. Volume on for trailer!

Joseph Suarez 🐡

194,632 Aufrufe • vor 11 Monaten

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵
0:30

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Robotic Systems Lab

39,143 Aufrufe • vor 29 Tagen

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically *retrieved* “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are *not* collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant *sub-trajectories* from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)
0:43

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically *retrieved* “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are *not* collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant *sub-trajectories* from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

Abhishek Gupta

12,045 Aufrufe • vor 1 Jahr

Today we're announcing cua-bench: a framework for benchmarking, training data, and RL environments for computer-use AI agents. Why? Current agents show 10x variance across minor UI changes. Here's how we're fixing it.
2:02

Today we're announcing cua-bench: a framework for benchmarking, training data, and RL environments for computer-use AI agents. Why? Current agents show 10x variance across minor UI changes. Here's how we're fixing it.

Cua

189,503 Aufrufe • vor 6 Monaten

We asked Mika Senghaas how RL environments need to change for better agents. His take: stop treating environments like static snapshots. Today, researchers manually define tasks and rewards. Prime Intellect is working toward a loop where agents solve tasks and help create harder ones. “Up to now, environments are still these static snapshots. The task in and of itself is sort of hard-coded.” “What gets really interesting is once we can fully close the loop.” “The policy you’re training is not only a solver... it also becomes an instance creating new tasks on the fly.”
1:32

We asked Mika Senghaas how RL environments need to change for better agents. His take: stop treating environments like static snapshots. Today, researchers manually define tasks and rewards. Prime Intellect is working toward a loop where agents solve tasks and help create harder ones. “Up to now, environments are still these static snapshots. The task in and of itself is sort of hard-coded.” “What gets really interesting is once we can fully close the loop.” “The policy you’re training is not only a solver... it also becomes an instance creating new tasks on the fly.”

MTS

15,324 Aufrufe • vor 27 Tagen

😈 Today, we introduce WebGym, the largest-to-date open-source RL environment for web agent training that contains 300k tasks and a rollout framework optimized specifically for web environments' rollout speed. We reveal the effects of essential scaling directions we observe with WebGym. 1/n
0:26

😈 Today, we introduce WebGym, the largest-to-date open-source RL environment for web agent training that contains 300k tasks and a rollout framework optimized specifically for web environments' rollout speed. We reveal the effects of essential scaling directions we observe with WebGym. 1/n

Jack Bai

45,075 Aufrufe • vor 5 Monaten

If we train VLAs to respond to diverse multimodal prompts, then we can steer them better: [grasp the carrot]/[move to x,y,z]/[put the carrot on the plate]. With many levels of detail, powerful VLMs can step in and steer the model to success much more often! More below 👇
2:11

If we train VLAs to respond to diverse multimodal prompts, then we can steer them better: [grasp the carrot]/[move to x,y,z]/[put the carrot on the plate]. With many levels of detail, powerful VLMs can step in and steer the model to success much more often! More below 👇

Sergey Levine

20,933 Aufrufe • vor 4 Monaten

I am thrilled to introduce OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code. Led by Maxence Faldor and Jenny Zhang, with Antoine Cully and myself. 🧵👇
1:03

I am thrilled to introduce OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code. Led by Maxence Faldor and Jenny Zhang, with Antoine Cully and myself. 🧵👇

Jeff Clune

170,209 Aufrufe • vor 2 Jahren

TRADING HIGH-VOLATILITY ENVIRONMENTS High-volatility environments are the ones we live for. That is when we size up aggressively & make all of our big money, right? That is the mindset of many traders & I think they're missing this key concept which leads to so many blowups.
3:46

TRADING HIGH-VOLATILITY ENVIRONMENTS High-volatility environments are the ones we live for. That is when we size up aggressively & make all of our big money, right? That is the mindset of many traders & I think they're missing this key concept which leads to so many blowups.

Lance Breitstein 🇺🇸🌎

47,325 Aufrufe • vor 1 Jahr

You told us you’re running multiple AI agents and wanted a better UX. We listened and shipped it! Here’s what’s new in the latest Visual Studio Code release: 🗂️ Unified agent sessions workspace for local, background, and cloud agents 💻 Claude and Codex support for local and cloud agents 🔀 Parallel subagents 🌐 Integrated browser And more...
1:09

You told us you’re running multiple AI agents and wanted a better UX. We listened and shipped it! Here’s what’s new in the latest Visual Studio Code release: 🗂️ Unified agent sessions workspace for local, background, and cloud agents 💻 Claude and Codex support for local and cloud agents 🔀 Parallel subagents 🌐 Integrated browser And more...

Visual Studio Code

586,802 Aufrufe • vor 4 Monaten

"So, in the body, we can create an environment that's very hospitable for cancer. We can create an environment that's very inhospitable to cancer. Number one, it has few areas of fungal hypoxia—it has few areas of the body that are deprived of oxygen. Healthy circulation and movement, sunlight, things like red light therapy. The second thing is we don’t have an environment that is very acidic. We know that alkaline environments tend to be disease-free environments. Acidic environments tend to be disease-prone environments." Gary Brecka
8:20

"So, in the body, we can create an environment that's very hospitable for cancer. We can create an environment that's very inhospitable to cancer. Number one, it has few areas of fungal hypoxia—it has few areas of the body that are deprived of oxygen. Healthy circulation and movement, sunlight, things like red light therapy. The second thing is we don’t have an environment that is very acidic. We know that alkaline environments tend to be disease-free environments. Acidic environments tend to be disease-prone environments." Gary Brecka

Shawn Ryan

272,926 Aufrufe • vor 1 Jahr

Attritable Swarm 🇺🇸 Swarm Defense (Swarm Defense), Michigan based firm is a manufacturer of low-cost, scalable drone swarms systems built for military training and CUAS operations. "Our swarm technology enables fast, scalable deployment of low-cost, attritable drones. Built for coordination, redundancy, and adaptability, the system allows for seamless operation across complex environments with precision and control." Swarm Defense has been invited to compete in The Phase I evaluation- the Gauntlet. Learn more >>
0:42

Attritable Swarm 🇺🇸 Swarm Defense (Swarm Defense), Michigan based firm is a manufacturer of low-cost, scalable drone swarms systems built for military training and CUAS operations. "Our swarm technology enables fast, scalable deployment of low-cost, attritable drones. Built for coordination, redundancy, and adaptability, the system allows for seamless operation across complex environments with precision and control." Swarm Defense has been invited to compete in The Phase I evaluation- the Gauntlet. Learn more >>

Counter Unmanned Systems

116,161 Aufrufe • vor 3 Monaten

Yesterday, we signed a seven figure contract, growing our enterprise revenue 10x since last year. Today, we launch our Act 2: Gitpod Flex. We want to give you all the building blocks to automate your software development lifecycle, starting with development environments. Here is what you get. -Automations: programmable tasks and services triggered on events unique to development environments - Gitpod Desktop: bringing cloud development environments to your laptop, starting with Mac - Runners: self-host in your cloud in three minutes - Dev Container: for base images, tools, and extensions - Zero-trust environments: ‘never trust, always verify’ every actor and service Try it for free in early access, no waitlist.
1:31

Yesterday, we signed a seven figure contract, growing our enterprise revenue 10x since last year. Today, we launch our Act 2: Gitpod Flex. We want to give you all the building blocks to automate your software development lifecycle, starting with development environments. Here is what you get. -Automations: programmable tasks and services triggered on events unique to development environments - Gitpod Desktop: bringing cloud development environments to your laptop, starting with Mac - Runners: self-host in your cloud in three minutes - Dev Container: for base images, tools, and extensions - Zero-trust environments: ‘never trust, always verify’ every actor and service Try it for free in early access, no waitlist.

Johannes Landgraf

26,073 Aufrufe • vor 1 Jahr

VITA Towards Open-Source Interactive Omni Multimodal LLM discuss: The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. To the best of our knowledge, we are the first to exploit non-awakening interaction and audio interrupt in MLLM. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research.
2:21

VITA Towards Open-Source Interactive Omni Multimodal LLM discuss: The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. To the best of our knowledge, we are the first to exploit non-awakening interaction and audio interrupt in MLLM. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research.

AK

23,958 Aufrufe • vor 1 Jahr

MURDER DRONES EPISODE 7 & 8 coming Spring 2024 We’re hard at work on the final episodes of Murder Drones Season. With bigger fights, environments, scenes and overall animation we can't wait to show you whats in store 😈
1:30

MURDER DRONES EPISODE 7 & 8 coming Spring 2024 We’re hard at work on the final episodes of Murder Drones Season. With bigger fights, environments, scenes and overall animation we can't wait to show you whats in store 😈

GLITCH

682,689 Aufrufe • vor 2 Jahren

🚀 Training AI agents isn’t about teaching them which buttons to click. It’s about judgment. Labelbox builds RL environments with domain experts, creating thousands of real-world scenarios where AI agents learn how to handle complex decisions across industries.
1:30

🚀 Training AI agents isn’t about teaching them which buttons to click. It’s about judgment. Labelbox builds RL environments with domain experts, creating thousands of real-world scenarios where AI agents learn how to handle complex decisions across industries.

Labelbox

22,158,127 Aufrufe • vor 2 Monaten

🧵Data sharing between the police and Immigration Enforcement traps migrant victims of serious crime in harmful environments. Urgent change is needed to protect them. We support the Dame Nicole Jacobs’s firewall amendment to the Victims and Prisoners Bill. 1/2
1:43

🧵Data sharing between the police and Immigration Enforcement traps migrant victims of serious crime in harmful environments. Urgent change is needed to protect them. We support the Dame Nicole Jacobs’s firewall amendment to the Victims and Prisoners Bill. 1/2

Latin American Women's Rights Service - LAWRS

16,934 Aufrufe • vor 2 Jahren