Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

LERF: Language Embedded Radiance Fields TL;DR: Grounding CLIP vectors volumetrically inside a NeRF allows flexible natural language queries in 3D abs: project page:

AK

525,397 subscribers

251,289 просмотров • 3 лет назад •via X (Twitter)

Наука и технологии Образование

Anya Rossi• Live Now

Private livecam show

Комментарии: 9

Фото профиля Fabien Benetou

Fabien Benetou3 лет назад

Damn...

Фото профиля Allknight

Allknight3 лет назад

@zenstyle Now do Sarah Connor >:D

Фото профиля potatoruning

potatoruning3 лет назад

looks good

Фото профиля David Asem

David Asem3 лет назад

For a sex maybe 🤔 fix the @AR15COM has a fix

Фото профиля Viho

Viho3 лет назад

🤩

Фото профиля MoonGotArt

MoonGotArt3 лет назад

Will this be open source?

Фото профиля Bill Xue

Bill Xue3 лет назад

@SaveToNotion #tweet

Фото профиля Guruprasad Venkataraghavan

Guruprasad Venkataraghavan3 лет назад

@memdotai mem it

Фото профиля $maczter

$maczter3 лет назад

@icreatelife 🤯🤯🤯

Похожие видео

LangSplat: 3D Language Gaussian Splatting paper page: ground CLIP features into a set of 3D language Gaussians, which attains precise 3D language fields while being 199 × faster than LERF

LangSplat: 3D Language Gaussian Splatting paper page: ground CLIP features into a set of 3D language Gaussians, which attains precise 3D language fields while being 199 × faster than LERF

AK

114,256 просмотров • 2 лет назад

How can we learn 3D visual grounding with natural supervision—only looking at QA pairs, without ground truth bounding boxes or object classification labels? We inject explicit language priors, e.g., the symmetric property that A near B ⇒ B near A, in structured vision models.

How can we learn 3D visual grounding with natural supervision—only looking at QA pairs, without ground truth bounding boxes or object classification labels? We inject explicit language priors, e.g., the symmetric property that A near B ⇒ B near A, in structured vision models.

Joy Hsu

14,843 просмотров • 2 лет назад

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

AK

62,768 просмотров • 3 лет назад

TidyBot: Personalized Robot Assistance with Large Language Models approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios abs: project page: paper page:

TidyBot: Personalized Robot Assistance with Large Language Models approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios abs: project page: paper page:

AK

326,009 просмотров • 3 лет назад

Introducing AskAnon The plug-in that will turn every project and protocol from DeFi to DeFAI Ask Anon will be a user-facing widget embedded on DeFi project websites, enabling users to have seamless interactions with protocols through natural language prompts. Gemma will also be built into Ask Anon making it not just a tool for actions but a gateway to informed decisions.

Introducing AskAnon The plug-in that will turn every project and protocol from DeFi to DeFAI Ask Anon will be a user-facing widget embedded on DeFi project websites, enabling users to have seamless interactions with protocols through natural language prompts. Gemma will also be built into Ask Anon making it not just a tool for actions but a gateway to informed decisions.

Hey Anon

104,961 просмотров • 1 год назад

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,798 просмотров • 3 лет назад

a robot can recognize every object in a room and still not know how to open a single drawer knowing there's a cabinet is scene understanding. knowing which handle to grab, which direction to pull, and that it's a drawer not a door is affordance understanding almost no dataset has it in 3D SceneFun3D: 14,800+ functional elements across 710 laser-scanned indoor scenes. every handle, knob, button, and switch labeled with its affordance, motion axis, motion type, and a natural language task description each scene's 3D point cloud with its iPad video recordings in fiftyone. the affordance annotations live in 3D and project into the video frames check it out here

a robot can recognize every object in a room and still not know how to open a single drawer knowing there's a cabinet is scene understanding. knowing which handle to grab, which direction to pull, and that it's a drawer not a door is affordance understanding almost no dataset has it in 3D SceneFun3D: 14,800+ functional elements across 710 laser-scanned indoor scenes. every handle, knob, button, and switch labeled with its affordance, motion axis, motion type, and a natural language task description each scene's 3D point cloud with its iPad video recordings in fiftyone. the affordance annotations live in 3D and project into the video frames check it out here

harpreet

18,800 просмотров • 3 месяцев назад

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

Tom Yeh

67,924 просмотров • 2 лет назад

🚀 Exciting news! The GeoAI Python package just got a major upgrade. It now supports interactive geospatial data visualization powered by AI Agents. 🌍✨ GitHub: Full tutorial: 🌍 In this tutorial, you’ll learn how to build interactive maps powered by AI agents that respond to natural language queries. Watch how you can fly to any city, add raster or vector layers, visualize 3D buildings, and explore geospatial data—all without writing code. Perfect for developers, researchers, and anyone curious about AI, GIS, and interactive mapping. #geospatial #mapping #python #GeoAI #AI

🚀 Exciting news! The GeoAI Python package just got a major upgrade. It now supports interactive geospatial data visualization powered by AI Agents. 🌍✨ GitHub: Full tutorial: 🌍 In this tutorial, you’ll learn how to build interactive maps powered by AI agents that respond to natural language queries. Watch how you can fly to any city, add raster or vector layers, visualize 3D buildings, and explore geospatial data—all without writing code. Perfect for developers, researchers, and anyone curious about AI, GIS, and interactive mapping. #geospatial #mapping #python #GeoAI #AI

Qiusheng Wu

12,908 просмотров • 1 год назад

GraphAI × Fetch.ai - ASI-1 Integration Demo Akhil Verghese, our Head AI Scientist, walks through how Fetch.ai 's ASI-1 LLM now runs natively inside GraphEngine, answering real on-chain analysis queries in natural language; from wallet tracking to token flow interpretation. This demo is more than a feature showcase, it’s the first visible milestone in the GraphAI × Fetch.ai collaboration, illustrating how ASI-1 and GraphEngine together can power agentic systems that understand, reason over, and act on decentralised finance in real time. A first glimpse at an AI-driven, on-chain future where intelligence and decentralised execution converge. Watch the demo.

GraphAI × Fetch.ai - ASI-1 Integration Demo Akhil Verghese, our Head AI Scientist, walks through how Fetch.ai 's ASI-1 LLM now runs natively inside GraphEngine, answering real on-chain analysis queries in natural language; from wallet tracking to token flow interpretation. This demo is more than a feature showcase, it’s the first visible milestone in the GraphAI × Fetch.ai collaboration, illustrating how ASI-1 and GraphEngine together can power agentic systems that understand, reason over, and act on decentralised finance in real time. A first glimpse at an AI-driven, on-chain future where intelligence and decentralised execution converge. Watch the demo.

GraphAI

29,810 просмотров • 10 месяцев назад

At Avalon we are building "Real-time creating" - the ability to generate gameplay ready persistent worlds prompted from text. While others are building real-time video world models, Avalon is building real-time world generation inside a fully playable, persistent multiplayer engine. Internally running at 3840×2180 at 60 FPS. Built on Unreal Engine. Multiplayer by default. Persistent by default. Gameplay-ready by default. This is not a video latent replay. Not a simulation of interaction. It is a real 3D world with physics, logic, and authoritative multiplayer state. Avalon is trained on proprietary Avalon interaction data and powered by a hybrid system that combines language understanding, 3D model generation, procedural systems, and structured gameplay logic synthesis. Players can walk through a live world and generate environments, assets, mechanics, and entirely new gameplay modes using natural language. We accomplish this through a combination of 3D model generation, game logic generation based on our proprietary systems, and AI driven world creation. While other players are inside it. Changes persist instantly. State is synchronized in real time. Creation happens inside the world, not outside of it. Describe a biome. Spawn a civilization. Create a survival mode. Build a dungeon crawler. Launch a new game inside the world. Avalon interprets intent and integrates it directly into the live multiplayer environment. This is not a world model predicting video. This is a gameplay engine that understands language. If you can describe it, you can build it. And others can walk into it instantly.

At Avalon we are building "Real-time creating" - the ability to generate gameplay ready persistent worlds prompted from text. While others are building real-time video world models, Avalon is building real-time world generation inside a fully playable, persistent multiplayer engine. Internally running at 3840×2180 at 60 FPS. Built on Unreal Engine. Multiplayer by default. Persistent by default. Gameplay-ready by default. This is not a video latent replay. Not a simulation of interaction. It is a real 3D world with physics, logic, and authoritative multiplayer state. Avalon is trained on proprietary Avalon interaction data and powered by a hybrid system that combines language understanding, 3D model generation, procedural systems, and structured gameplay logic synthesis. Players can walk through a live world and generate environments, assets, mechanics, and entirely new gameplay modes using natural language. We accomplish this through a combination of 3D model generation, game logic generation based on our proprietary systems, and AI driven world creation. While other players are inside it. Changes persist instantly. State is synchronized in real time. Creation happens inside the world, not outside of it. Describe a biome. Spawn a civilization. Create a survival mode. Build a dungeon crawler. Launch a new game inside the world. Avalon interprets intent and integrates it directly into the live multiplayer environment. This is not a world model predicting video. This is a gameplay engine that understands language. If you can describe it, you can build it. And others can walk into it instantly.

AVALON

66,145 просмотров • 8 месяцев назад

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: Code: Paper: Models: Work done with Jianyuan Minghao Chen Christian Rupprecht and Andrea Vedaldi

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: Code: Paper: Models: Work done with Jianyuan Minghao Chen Christian Rupprecht and Andrea Vedaldi

Stan Szymanowicz

33,062 просмотров • 6 месяцев назад

✨We are excited to open-source Tencent HY-Motion 1.0, a billion-parameter text-to-motion model built on the Diffusion Transformer (DiT) architecture and flow matching. Tencent HY-Motion 1.0 empowers developers and individual creators alike by transforming natural language into high-fidelity, fluid, and diverse 3D character animations, delivering exceptional instruction-following capabilities across a broad range of categories. The generated 3D animation assets can be seamlessly integrated into typical 3D animation pipelines.🎮🎥 Highlights: 🔹Billion-Scale DiT: Successfully scaled flow-matching DiT to 1B+ parameters, setting a new ceiling for instruction-following capability and generated motion quality. 🔹Full-Stage Training Strategy: The industry’s first motion generation model featuring a complete Pre-training → SFT → RL loop to optimize physical plausibility and semantic accuracy. 🔹Comprehensive Category Coverage: Features 200+ motion categories across 6 major classes—the most comprehensive in the industry, curated via a meticulous data pipeline. 🌐Project Page: 🔗Github: 🤗Hugging Face: 📄Technical report:

✨We are excited to open-source Tencent HY-Motion 1.0, a billion-parameter text-to-motion model built on the Diffusion Transformer (DiT) architecture and flow matching. Tencent HY-Motion 1.0 empowers developers and individual creators alike by transforming natural language into high-fidelity, fluid, and diverse 3D character animations, delivering exceptional instruction-following capabilities across a broad range of categories. The generated 3D animation assets can be seamlessly integrated into typical 3D animation pipelines.🎮🎥 Highlights: 🔹Billion-Scale DiT: Successfully scaled flow-matching DiT to 1B+ parameters, setting a new ceiling for instruction-following capability and generated motion quality. 🔹Full-Stage Training Strategy: The industry’s first motion generation model featuring a complete Pre-training → SFT → RL loop to optimize physical plausibility and semantic accuracy. 🔹Comprehensive Category Coverage: Features 200+ motion categories across 6 major classes—the most comprehensive in the industry, curated via a meticulous data pipeline. 🌐Project Page: 🔗Github: 🤗Hugging Face: 📄Technical report:

Tencent Hy

329,750 просмотров • 9 месяцев назад

Robots might learn better from video than from language! 📼 Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle “what will happen next,” and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10× less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can “imagine” motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Robots might learn better from video than from language! 📼 Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle “what will happen next,” and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10× less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can “imagine” motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

50,130 просмотров • 9 месяцев назад

Nvidia announces GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning paper page: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.

AK

141,058 просмотров • 2 лет назад

🚨🚨Can agents earn money, run a business, or even self-organize a society in the physical social world? 🤖🤖 Can agents learn continually to survive and thrive in embodied environments, like how human babies grow? 👶 Super excited to introduce SimWorld, an open-ended simulator of LLM agents in infinite, realistic embodied worlds. SimWorld features 3 key designs: 1⃣Open-ended realistic world simulation - built on Unreal Engine 5, with accurate physical social dynamics - 100+ built-in environments (city, island, wilderness ...) - language-controllable procedural generation - text-to-3D asset generation 2⃣Native interface for LLM/VLM agents - Gym-like agent-environment interaction APIs - plug in any LLMs/VLMs (GPTs, Gemini, Qwen ...) - rich multi-modal perception - open-vocabulary natural-language action outputs 3⃣Diverse physical and social reasoning scenarios - long-horizon embodied reasoning - multi-agent collaboration / competition - easily customizable for any reasoning tasks SimWorld is fully open-sourced, with a hope to become a foundational infrastructure for real-world agent research across disciplines: robotics, economy, public health, education, etc. Project website + more details in the thread👇 ...1/

🚨🚨Can agents earn money, run a business, or even self-organize a society in the physical social world? 🤖🤖 Can agents learn continually to survive and thrive in embodied environments, like how human babies grow? 👶 Super excited to introduce SimWorld, an open-ended simulator of LLM agents in infinite, realistic embodied worlds. SimWorld features 3 key designs: 1⃣Open-ended realistic world simulation - built on Unreal Engine 5, with accurate physical social dynamics - 100+ built-in environments (city, island, wilderness ...) - language-controllable procedural generation - text-to-3D asset generation 2⃣Native interface for LLM/VLM agents - Gym-like agent-environment interaction APIs - plug in any LLMs/VLMs (GPTs, Gemini, Qwen ...) - rich multi-modal perception - open-vocabulary natural-language action outputs 3⃣Diverse physical and social reasoning scenarios - long-horizon embodied reasoning - multi-agent collaboration / competition - easily customizable for any reasoning tasks SimWorld is fully open-sourced, with a hope to become a foundational infrastructure for real-world agent research across disciplines: robotics, economy, public health, education, etc. Project website + more details in the thread👇 ...1/

Lianhui Qin@COLM2026

65,331 просмотров • 10 месяцев назад

I used to think 3D was way too much work for a quick creative idea Then I tried this workflow in MiniMax Design GPT-6 Astra can work with Blender to build the 3D scene from a natural-language prompt, which honestly changes how I think about 3D creation I don’t need to start by figuring out every Blender operation myself I can just describe the scene I have in mind and let Astra handle the technical side Then I can take that 3D result straight into H3 Max Turbo and turn it into a video And the speed here is no joke 15s video → 20s generation Starting at $0.018/s That makes experimenting with different ideas feel so much less expensive in terms of time But the more I use MiniMax Design, the more I realize 3D is only one part of it GPT Image 2.5 opens up another route — generate the visuals and storyboards first, then use H3 Max Turbo to bring those shots together into a video So now I’m basically playing with two different creative paths: Astra + Blender → 3D → H3 Max Turbo GPT Image 2.5 → Storyboard → H3 Max Turbo And having all of these connected inside MiniMax Design makes the whole process feel much more flexible Honestly, I started this just wanting to see what Astra could do with 3D Now I’m way more curious about what happens when I start mixing all these workflows together

I used to think 3D was way too much work for a quick creative idea Then I tried this workflow in MiniMax Design GPT-6 Astra can work with Blender to build the 3D scene from a natural-language prompt, which honestly changes how I think about 3D creation I don’t need to start by figuring out every Blender operation myself I can just describe the scene I have in mind and let Astra handle the technical side Then I can take that 3D result straight into H3 Max Turbo and turn it into a video And the speed here is no joke 15s video → 20s generation Starting at $0.018/s That makes experimenting with different ideas feel so much less expensive in terms of time But the more I use MiniMax Design, the more I realize 3D is only one part of it GPT Image 2.5 opens up another route — generate the visuals and storyboards first, then use H3 Max Turbo to bring those shots together into a video So now I’m basically playing with two different creative paths: Astra + Blender → 3D → H3 Max Turbo GPT Image 2.5 → Storyboard → H3 Max Turbo And having all of these connected inside MiniMax Design makes the whole process feel much more flexible Honestly, I started this just wanting to see what Astra could do with 3D Now I’m way more curious about what happens when I start mixing all these workflows together

haytam

83,103 просмотров • 1 месяц назад

We’re excited to officially launch LlamaParse, the first genAI-native document parsing solution. Not only is it better at parsing out images/tables/charts 📊📈 than virtually every other parser, it is now steerable through natural language instructions - output the document in whatever format you desire! It is also the only parsing solution that seamlessly allows you to build accurate RAG over complex documents, free of hallucinations 🔥 We launched it in private preview a few weeks ago and hit 2k users, 1M total PDF pages parsed. And now it’s better than ever. LlamaParse contains the following killer features: ✅ SOTA table/chart extraction ✅ Seamless integration with LlamaIndex 🦙 advanced RAG/agents ✅✨ Natural language Parsing Instructions ✅✨JSON mode and image extraction ✅✨Support for ~10 document types (.pdf, .pptx, .docx, .xml) and more Our pricing is simple: 1k free per day, and additional pages at 0.3c a page, or $3 for 1k pages. If you want advanced document RAG and/or private deployments, come get in touch with us to chat about LlamaCloud. Check out our full blog post here: LlamaParse client repo: Signup at 🦙☁️: Come talk to us:

We’re excited to officially launch LlamaParse, the first genAI-native document parsing solution. Not only is it better at parsing out images/tables/charts 📊📈 than virtually every other parser, it is now steerable through natural language instructions - output the document in whatever format you desire! It is also the only parsing solution that seamlessly allows you to build accurate RAG over complex documents, free of hallucinations 🔥 We launched it in private preview a few weeks ago and hit 2k users, 1M total PDF pages parsed. And now it’s better than ever. LlamaParse contains the following killer features: ✅ SOTA table/chart extraction ✅ Seamless integration with LlamaIndex 🦙 advanced RAG/agents ✅✨ Natural language Parsing Instructions ✅✨JSON mode and image extraction ✅✨Support for ~10 document types (.pdf, .pptx, .docx, .xml) and more Our pricing is simple: 1k free per day, and additional pages at 0.3c a page, or $3 for 1k pages. If you want advanced document RAG and/or private deployments, come get in touch with us to chat about LlamaCloud. Check out our full blog post here: LlamaParse client repo: Signup at 🦙☁️: Come talk to us:

LlamaIndex 🦙

143,198 просмотров • 2 лет назад

Today, I am launching Paper Breakdown. - PBD gets you academic paper recommendations and lets you study CS/ML/AI research with LLM agents. - It highlights relevant sections directly in the actual PDF - generates flowcharts/illustrations too - we provide an in-build screenshot tool to send images to the agent directly from the paper. - we also got agentic paper search that allows you to search our database of 70,000+ CS and ML Arxiv papers in seconds using natural language. I have been building PBD for almost half a year - it all started as a means for me to keep up with research and use AI to produce visuals and scripts for my own YouTube videos. I have developed it enough to confidently recommend it to you. Visit our landing page to learn more.

Today, I am launching Paper Breakdown. - PBD gets you academic paper recommendations and lets you study CS/ML/AI research with LLM agents. - It highlights relevant sections directly in the actual PDF - generates flowcharts/illustrations too - we provide an in-build screenshot tool to send images to the agent directly from the paper. - we also got agentic paper search that allows you to search our database of 70,000+ CS and ML Arxiv papers in seconds using natural language. I have been building PBD for almost half a year - it all started as a means for me to keep up with research and use AI to produce visuals and scripts for my own YouTube videos. I have developed it enough to confidently recommend it to you. Visit our landing page to learn more.

AVB

14,579 просмотров • 9 месяцев назад