How do we create realistic models of dressed humans... directly from visual data? We introduce PhysAvatar, a framework that estimates the shape, appearance, and physical parameters of dressed human avatars from multi-view videos. Page: (1/6)show more

Diffuman4D: 4D Consistent Human View Synthesis from Sparse-View Videos... with Spatio-Temporal Diffusion Models Contributions: • We introduce Diffuman4D, a novel diffusion model that generates spatio-temporally consistent and high-resolution (1024p) human videos from sparse-view video inputs. • We propose a sliding iterative denoising mechanism that enhances both the spatial and temporal consistency of generated long-term videos while maintaining efficient inference. • We design a human pose conditioning scheme to enhance the appearance quality and motion accuracy of generated human videos. • We plan to release our processed version of the DNA-Rendering dataset, which we believe will benefit future research in this area.show more

(1/6) X-Humanoid 🤖: Scaling up data for Humanoid Robots.... We convert human daily activity videos (from Ego-Exo4D) into humanoid videos (i.e., Tesla Optimus) performing tasks like cooking or fixing a bike. This data can be potentially used to train robot policies and world models. 🔥 Project page: Paper link:show more

(1/3) Can we turn text-to-image models into photorealistic 3D... generators? ViewDiff (#CVPR2024) produces realistic, multi-view consistent images of real-world 3D objects in authentic surroundings. Website Video How does it work?show more

WeatherEdit: Controllable Weather Editing with 4D Gaussian Field Contributions:... 1. Based on our analysis of weather editing characteristics, we introduce WeatherEdit, a comprehensive and efficient framework for realistic and controllable weather generation. Compared with existing methods that focus on either background editing or static weather effects, a progressive 2D-to-4D transformation process in WeatherEdit enhances adaptability across a wider range of scenarios. 2. We introduce an all-in-one adapter to enable a diffusion model for multi-weather (snowy, rainy, and fog) synthesis, along with a Temporal-View attention to ensure consistent editing across multi-frame and multi-view. 3. We design a 4D Gaussian field for weather particle modeling, enabling plausible simulation of raindrops, snowflakes, and fog with controllable severity. 4. We demonstrate WeatherEdit’s effectiveness in generating realistic, consistent, and controllable weather effects in 3D driving scenes, showcasing its applicability to real-world scenarios.show more

Wonderland: Navigating 3D Scenes from a Single Image Contributions:... • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.show more

OmniHuman 1 - creating lifelike human videos based on... a single human image OmniHuman 1 from Bytedance is an end-to-end multimodality-conditioned human video generation framework, which can generate human videos based on a single human image and motion signals It can support input of any aspect ratio in terms of speech. It significantly improves the handling of gestures, which is a challenge for existing methods, and produces highly realistic results Below some examplesshow more

New research from BlackForestLabsAI - Unofficial 🥳 Meet Self-Flow:... our self-supervised framework for image, audio, video & world models 🤖 Do generative models really need DINO to learn strong representations? We propose teaching them directly via a joint framework instead 🧵show more

Checkout ObjectForesight! We tackle the problem of predicting future... 3D object trajectories given a past context, purely from human videos of everyday interactions! [1/n]show more

I am posting after a Long Time on Twitter,... but its to announce a big change. I have started and we are Collecting Egocentric Data at Scale from India, Covering 300 + Commercial Locations 1500 + Households This is the network and base we have built in just past 2 months, as the Robotics companies, VLMs and World Models increase their requirements on Real World Data collection, Human Loops will be keep scaling out capacity. We are on track to collect 1M Hours of Egocentric in the coming 6 months for our clients exclusively. We are maintaining 95% Quality standards across industrial data and running a End - End Operational Management complying all Indian Laws and compensating our partners/operators. (From Environment sourcing, to hardware, Legal contracts, deployment, training, collection, processing) Check our Samples: - Commercial Videos -- Household Videos --- Multimodel ---- Egocentric + Live Audio Narration Links : On the Journey to become #1 India Physical AI Data Partner. We are also building our capacity as annotation and labelling partner for data companies to become end-end partner for companies. A big change from the world of crypto and web3 but physical AI and data space is where i want to build my next venture #Egocentric #EgocentricIndia #PhysicalAI #Robotics #India #AIdata #data #Multimodeldata #worldmodels #VLMs #Humanloops #Egocentridata #Multimodeldatashow more

How can we address the scarcity of data required... for specialized AI? Learn about Simula, a framework that reframes synthetic data generation as dataset-level mechanism design. By using reasoning to architect datasets from first principles, Simula enables fine-grained control over coverage, complexity, and quality. More →show more

🚨How do LLMs acquire human values?🤔 We often point... to preference optimization. However, in our new work, we trace how and when model values shift during post-training and uncover surprising dynamics. We ask: How do data, algorithms, and their interaction shape model values?🧵show more

System identification (sysid) is the process of finding the... physical parameters that make a simulation match reality. If you're training an RL locomotion policy in simulation, the accuracy of your motor model directly affects how well the policy transfers to the real robot. A recent git commit by Kevin Zakka added a sysid toolbox to MuJoCo which automates this process: you provide recorded motor data and a MuJoCo model, and it optimizes the model parameters to minimize the difference between simulated and real trajectories. For my RobStride Dynamics RS02 QDD motors (17 Nm peak, 7.75:1 gear), I built a Rust tool that sends multi-sine torque excitation at 1 kHz and records position/velocity feedback. I then feed this data into MuJoCo's sysid optimizer.show more

Artificial Intelligence (#AI) models that process and interpret visual... data are called large vision models. These models, like large language models, are trained on large amounts of data. #midjourney #dalle #stablediffusion (1/8) 🧵 ⤵️show more

"Mothers Are Separated From Children To Create Milk. Animals... Are Sick And In Crowded, Prison - Like Conditions To Make Dairy And Eggs. It Doesn't Take A Lot To Draw The Line From How We Treat Animals To How We Treat Humans." -Natalie Portmanshow more

How can we leverage diverse human videos to improve... robot manipulation? Excited to introduce EgoVLA — a Vision-Language-Action model trained on egocentric human videos by explicitly modeling wrist & hand motion. We build a shared action space between humans and robots, enabling seamless transfer. With some robot demos, EgoVLA becomes a powerful, generalizable robot policy.show more

(1/2) MonoNPHM will be presented as a #CVPR2024 Highlight!... Our Neural Parametric Head Model parametrizes both geometry and appearance. With the learned model, we can then 3D reconstruct and track human heads from images or videos.show more

We evolved from walking into girls undressing/getting dressed to... walking in on girls using the restroom.show more

UK Conservative Party Leader Kemi Badenoch: “The UK is... a multi-faith, multi-ethnic, multicultural success story, and we believe that many of our greatest strengths derive from the diversity of our population.” Thoughts?show more

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents... at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.show more