Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

🚀Announcing NeRSemble 3D Head Avatar Benchmark v2 Version 2 of the NeRSemble 3D Head Avatar Benchmark systematically evaluates several aspects of 3D head avatar creation. Our goal is to drive progress toward more realistic, robust, and generalizable avatar methods. 🔬Benchmark Tasks The NeRSemble Benchmark v2 features three core challenges:...

MattNiessner's profile picture

Matthias Niessner

48,393 subscribers

29,954 Aufrufe • vor 4 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!
0:30

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!

Matthias Niessner

96,186 Aufrufe • vor 8 Monaten

📢Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction📢 -> highly accurate face reconstruction by training powerful VITs via surface normals and UV-coordinates estimation. The geometric cues from our 2D foundation model backbone constrain the 3DMM parameters, which allows us to achieve remarkable reconstruction accuracy - works for both single image and videos! In addition, we introduce a new 3D face reconstruction benchmark that evaluates both neutral and posed face geometry. 🌍 📷 Great work by Simon Giebenhain Tobias Kirschstein Martin Rünz Lourdes Agapito
0:32

📢Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction📢 -> highly accurate face reconstruction by training powerful VITs via surface normals and UV-coordinates estimation. The geometric cues from our 2D foundation model backbone constrain the 3DMM parameters, which allows us to achieve remarkable reconstruction accuracy - works for both single image and videos! In addition, we introduce a new 3D face reconstruction benchmark that evaluates both neutral and posed face geometry. 🌍 📷 Great work by Simon Giebenhain Tobias Kirschstein Martin Rünz Lourdes Agapito

Matthias Niessner

62,269 Aufrufe • vor 1 Jahr

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito
0:38

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito

Matthias Niessner

74,763 Aufrufe • vor 1 Jahr

📢📢 𝐏𝐞𝐫𝐜𝐇𝐞𝐚𝐝: 𝐏𝐞𝐫𝐜𝐞𝐩𝐭𝐮𝐚𝐥 𝐇𝐞𝐚𝐝 𝐌𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐒𝐢𝐧𝐠𝐥𝐞-𝐈𝐦𝐚𝐠𝐞 𝟑𝐃 𝐇𝐞𝐚𝐝 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 & 𝐄𝐝𝐢𝐭𝐢𝐧𝐠📢📢 PercHead reconstructs realistic 3D heads from a single image and enables disentangled 3D editing via geometric controls and style inputs from images or text. At its core is a generalized 3D head decoder trained with perceptual supervision from DINOv2 and SAM 2.1. We find that our new perceptual loss formulation improves reconstruction fidelity compared to commonly-used methods such as LPIPS. Our trained reconstruction model is able to generate 3D-consistent heads from a single input image. Even with challenging side-view inputs, the model robustly infers missing regions for a coherent, high-fidelity output. In addition, our architecture seamlessly adapts to downstream tasks: by swapping the encoder, we can transform the model into a disentangled 3D editing pipeline. In this scenario, we can control geometry through - potentially hand-drawn - segmentation maps, and condition style via image or text prompt. We also provide an interactive GUI to enable the exploration of our editing pipeline. 🌍 📽️ Great work by Antonio Oroz and Tobias Kirschstein
0:30

📢📢 𝐏𝐞𝐫𝐜𝐇𝐞𝐚𝐝: 𝐏𝐞𝐫𝐜𝐞𝐩𝐭𝐮𝐚𝐥 𝐇𝐞𝐚𝐝 𝐌𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐒𝐢𝐧𝐠𝐥𝐞-𝐈𝐦𝐚𝐠𝐞 𝟑𝐃 𝐇𝐞𝐚𝐝 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 & 𝐄𝐝𝐢𝐭𝐢𝐧𝐠📢📢 PercHead reconstructs realistic 3D heads from a single image and enables disentangled 3D editing via geometric controls and style inputs from images or text. At its core is a generalized 3D head decoder trained with perceptual supervision from DINOv2 and SAM 2.1. We find that our new perceptual loss formulation improves reconstruction fidelity compared to commonly-used methods such as LPIPS. Our trained reconstruction model is able to generate 3D-consistent heads from a single input image. Even with challenging side-view inputs, the model robustly infers missing regions for a coherent, high-fidelity output. In addition, our architecture seamlessly adapts to downstream tasks: by swapping the encoder, we can transform the model into a disentangled 3D editing pipeline. In this scenario, we can control geometry through - potentially hand-drawn - segmentation maps, and condition style via image or text prompt. We also provide an interactive GUI to enable the exploration of our editing pipeline. 🌍 📽️ Great work by Antonio Oroz and Tobias Kirschstein

Matthias Niessner

18,855 Aufrufe • vor 9 Monaten

I made a digital twin of myself from 10 seconds of video. In the clip: left is the real me, middle is a leading avatar model, right is Mirage Avatar X. Watch the eyes. The difference is not subtle. I have been testing AI avatar models since my first clone in 2023. Every one of them was impressive for about 30 seconds, then your brain caught up. Still eyes. One polite expression. A mouth doing all the work. Avatar X is the first model where that moment never came. Here is what makes it different: It is trained on you. Avatar X preserves your identity. Most avatar models can copy your appearance. Avatar X captures the subtle details that make you you. The way you move, the way you express yourself, and the way you naturally deliver speech. It looks like you. It moves like you. It sounds like you. It understands non-verbal performance Laughing, crying, yawning, sighing. These are the moments where most avatar models fall apart, trying to lip-sync through sounds that aren't words. Avatar X responds naturally, generating realistic facial expressions and micro-expressions instead of forcing every sound into speech. The expression goes beyond the lips Expressions are driven by the audio, through the whole face and body. Ask a question and it furrows its brows and shrugs on the tone. No other model does this to this degree. No quality degradation The first second and the last second look the same. Other models lose quality the longer the video runs. 10 seconds of input That is the entire requirement. Other models need 15 seconds, some even 1 to five minutes. Three years ago my AI clone was a party trick. This one can carry my face, my expressions and my delivery without me in the room. The bar for AI avatars just moved. Avatar X is live today. → Try it here:
0:30

I made a digital twin of myself from 10 seconds of video. In the clip: left is the real me, middle is a leading avatar model, right is Mirage Avatar X. Watch the eyes. The difference is not subtle. I have been testing AI avatar models since my first clone in 2023. Every one of them was impressive for about 30 seconds, then your brain caught up. Still eyes. One polite expression. A mouth doing all the work. Avatar X is the first model where that moment never came. Here is what makes it different: It is trained on you. Avatar X preserves your identity. Most avatar models can copy your appearance. Avatar X captures the subtle details that make you you. The way you move, the way you express yourself, and the way you naturally deliver speech. It looks like you. It moves like you. It sounds like you. It understands non-verbal performance Laughing, crying, yawning, sighing. These are the moments where most avatar models fall apart, trying to lip-sync through sounds that aren't words. Avatar X responds naturally, generating realistic facial expressions and micro-expressions instead of forcing every sound into speech. The expression goes beyond the lips Expressions are driven by the audio, through the whole face and body. Ask a question and it furrows its brows and shrugs on the tone. No other model does this to this degree. No quality degradation The first second and the last second look the same. Other models lose quality the longer the video runs. 10 seconds of input That is the entire requirement. Other models need 15 seconds, some even 1 to five minutes. Three years ago my AI clone was a party trick. This one can carry my face, my expressions and my delivery without me in the room. The bar for AI avatars just moved. Avatar X is live today. → Try it here:

Linus ✦ Ekenstam

20,752 Aufrufe • vor 18 Tagen

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.
0:50

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,798 Aufrufe • vor 3 Jahren

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at
2:33

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

106,425 Aufrufe • vor 2 Tagen

The day of Parallel Avatars Manifest has come! Sharing some thoughts on them and their place within the Parallel ecosystem… 1. Avatars have immediate utility within the BETA release of the TCG. Matching the Parallel of the Avatar you use to the Parallel of the deck your playing [in ranked matches] will boost $PRIME emissions. Given this, consider if you want one for your favourite Parallel or one for each Parallel to have flexibility in what deck you play. Initial emissions will be set to 11% to ensure all systems are functioning as intended in BETA. 2. Beyond their function in the TCG, Avatars represent a new cornerstone for Parallel as we aspire to more immersive games and story telling. Although we plan to release expansions to the TCG [Planetfall is in development] the Avatars represent our first step towards ‘more ways to play’ and a persistent identity for gamers within the Parallel ecosystem. 3. We are working on finishing full bodies and a migration of Avatars into UE5. This is a process that will take time but ultimately will unlock their potential for use in content and games like Colony. 4. If you obtained an Avatar, you can pay a fee in $PRIME to peak and reveal the door of the Parallel you got but can’t peak to reveal their identity. If you do choose to peak at your avatar, Opensea will reflect this so everyone is aware which Parallel the Avatar belongs to. 5. Although there is varying rarity to the Avatars, there are six hand drawn 1/1 Avatars in the collection. One for each Parallel and one mystery. Special recognition is due to the the team that developed these 3D generated Avatars. It was no small task to develop this program and the detail that has gone into it is unmatched. I am certain it will speak for itself upon reveal. In true Parallel fashion, we stand in a category of our own and bring you something unlike anything the space has seen before. Thank you for your support and may your pulls be Prime. //
0:24

The day of Parallel Avatars Manifest has come! Sharing some thoughts on them and their place within the Parallel ecosystem… 1. Avatars have immediate utility within the BETA release of the TCG. Matching the Parallel of the Avatar you use to the Parallel of the deck your playing [in ranked matches] will boost $PRIME emissions. Given this, consider if you want one for your favourite Parallel or one for each Parallel to have flexibility in what deck you play. Initial emissions will be set to 11% to ensure all systems are functioning as intended in BETA. 2. Beyond their function in the TCG, Avatars represent a new cornerstone for Parallel as we aspire to more immersive games and story telling. Although we plan to release expansions to the TCG [Planetfall is in development] the Avatars represent our first step towards ‘more ways to play’ and a persistent identity for gamers within the Parallel ecosystem. 3. We are working on finishing full bodies and a migration of Avatars into UE5. This is a process that will take time but ultimately will unlock their potential for use in content and games like Colony. 4. If you obtained an Avatar, you can pay a fee in $PRIME to peak and reveal the door of the Parallel you got but can’t peak to reveal their identity. If you do choose to peak at your avatar, Opensea will reflect this so everyone is aware which Parallel the Avatar belongs to. 5. Although there is varying rarity to the Avatars, there are six hand drawn 1/1 Avatars in the collection. One for each Parallel and one mystery. Special recognition is due to the the team that developed these 3D generated Avatars. It was no small task to develop this program and the detail that has gone into it is unmatched. I am certain it will speak for itself upon reveal. In true Parallel fashion, we stand in a category of our own and bring you something unlike anything the space has seen before. Thank you for your support and may your pulls be Prime. //

//Kalos

25,585 Aufrufe • vor 3 Jahren

🚀 Announcing Echo — our new frontier model for 3D world generation. Echo turns a simple text prompt or image into a fully explorable, 3D-consistent world. Instead of disconnected views, the result is a single, coherent spatial representation you can move through freely. This is part of a bigger shift in AI: from generating pixels and tokens to generating spaces. Echo predicts a geometry-grounded 3D scene at metric scale, meaning every novel view, depth map, and interaction comes from the same underlying world — not independent hallucinations. Once generated, the world is interactive in real time. You control the camera, explore from any angle, and render instantly — even on low-end hardware, directly in the browser. High-quality 3D world exploration is no longer gated by expensive equipment. Under the hood, Echo infers a physically grounded 3D representation and converts it into a renderable format. For our web demo, we use 3D Gaussian Splatting (3DGS) for fast, GPU-friendly rendering — but the representation itself is flexible and can be easily adapted. Why this matters: consistent 3D worlds unlock real workflows — digital twins, 3D design, game environments, robotics simulation, and more. From a single photo or a line of text, Echo builds worlds that are reliable, editable, and spatially faithful. Echo also enables scene editing and restyling. Change materials, remove or add objects, explore design variations — all while preserving global 3D consistency. Editing no longer breaks the world. This is only the beginning. Echo is the foundation for future world models with dynamics, physical reasoning, and richer interaction — environments that don’t just look right, but behave right. Explore the generated worlds on our website and sign up for the closed beta. The era of spatial intelligence starts here. 🌍 #Echo #WorldModels #SpatialAI #3DFoundationModels Check it out:
1:07

🚀 Announcing Echo — our new frontier model for 3D world generation. Echo turns a simple text prompt or image into a fully explorable, 3D-consistent world. Instead of disconnected views, the result is a single, coherent spatial representation you can move through freely. This is part of a bigger shift in AI: from generating pixels and tokens to generating spaces. Echo predicts a geometry-grounded 3D scene at metric scale, meaning every novel view, depth map, and interaction comes from the same underlying world — not independent hallucinations. Once generated, the world is interactive in real time. You control the camera, explore from any angle, and render instantly — even on low-end hardware, directly in the browser. High-quality 3D world exploration is no longer gated by expensive equipment. Under the hood, Echo infers a physically grounded 3D representation and converts it into a renderable format. For our web demo, we use 3D Gaussian Splatting (3DGS) for fast, GPU-friendly rendering — but the representation itself is flexible and can be easily adapted. Why this matters: consistent 3D worlds unlock real workflows — digital twins, 3D design, game environments, robotics simulation, and more. From a single photo or a line of text, Echo builds worlds that are reliable, editable, and spatially faithful. Echo also enables scene editing and restyling. Change materials, remove or add objects, explore design variations — all while preserving global 3D consistency. Editing no longer breaks the world. This is only the beginning. Echo is the foundation for future world models with dynamics, physical reasoning, and richer interaction — environments that don’t just look right, but behave right. Explore the generated worlds on our website and sign up for the closed beta. The era of spatial intelligence starts here. 🌍 #Echo #WorldModels #SpatialAI #3DFoundationModels Check it out:

SpAItial AI

176,105 Aufrufe • vor 8 Monaten

🚀AI NEXUS 2.0 - A NEW ERA BEGINS Happy One Year Anniversary! During the bear… we proved the model: • Over 350K installs of the AI Nexus Mobile App • 150K wallets onboarded to Multiversᕽ • Turning people’s socials → playable 3D avatars in minutes • 60+ partner projects onboarded to our ecosystem But it's not enough. Now, we upgrade... Welcome to AI Nexus 2.0 AI Nexus 2.0 = AI + Gaming Infrastructure. Powered by $A1X. Cross-chain. Creator-driven. Identity-first. Built for everyone: from everyday mobile users to full game studios. AI Nexus becomes a superlayer for digital identity, gaming and AI-powered creation. And over the coming weeks (before end of the year), we’ll reveal every part of it in clear, bite-sized drops. What’s Coming in AI Nexus 2.0? 1. Two brand-new apps (launching sooner than you think!). 2. Deeper strategic, revenue-driven integrations with Ice Open Network, Multiversᕽ, , Linera ⛓️ & more. 3. Marketplace for Avatars / Assets; powered by NFTs. 4. SDK for effortless 3D avatar creation. 5. $A1X total supply reduction. 6. More AI tools coming to AI Nexus Mobile App & games to the Clone Machine. 7. New staking + developer incentive programs. AI Nexus 2.0 is bigger, broader, and built to scale. And we’re just getting started.
0:38

🚀AI NEXUS 2.0 - A NEW ERA BEGINS Happy One Year Anniversary! During the bear… we proved the model: • Over 350K installs of the AI Nexus Mobile App • 150K wallets onboarded to Multiversᕽ • Turning people’s socials → playable 3D avatars in minutes • 60+ partner projects onboarded to our ecosystem But it's not enough. Now, we upgrade... Welcome to AI Nexus 2.0 AI Nexus 2.0 = AI + Gaming Infrastructure. Powered by $A1X. Cross-chain. Creator-driven. Identity-first. Built for everyone: from everyday mobile users to full game studios. AI Nexus becomes a superlayer for digital identity, gaming and AI-powered creation. And over the coming weeks (before end of the year), we’ll reveal every part of it in clear, bite-sized drops. What’s Coming in AI Nexus 2.0? 1. Two brand-new apps (launching sooner than you think!). 2. Deeper strategic, revenue-driven integrations with Ice Open Network, Multiversᕽ, , Linera ⛓️ & more. 3. Marketplace for Avatars / Assets; powered by NFTs. 4. SDK for effortless 3D avatar creation. 5. $A1X total supply reduction. 6. More AI tools coming to AI Nexus Mobile App & games to the Clone Machine. 7. New staking + developer incentive programs. AI Nexus 2.0 is bigger, broader, and built to scale. And we’re just getting started.

AI Nexus

172,589 Aufrufe • vor 8 Monaten

great to see more people generating 3d avatars with our new text-to-3d feature in forge. this marks a step in the right direction in putting powerful creation tools directly in the hands of everyone. we built forge entirely from the ground up over the past months as one of the key releases on our roadmap. having full proprietary ownership of the technology gives us complete control to shape its direction without depending on external platforms or third-party licenses. building on this foundation, our upcoming studio feature will let users generate high-quality accessories, clothing, and environments simply by typing natural language prompts. a single description can produce fully textured, production-ready 3d assets in seconds, with options to create multiple variations and refine them through follow-up instructions.every asset created in studio integrates seamlessly with the 3d ai agents made in forge on users can instantly apply clothing and accessories with automatic fitting, layer multiple items, and place their agents inside custom-generated environments. all clothing and accessories come pre-rigged and optimized, while environments include proper lighting and geometry for immediate use in games, animation, virtual worlds, and more. we have spent months thoughtfully designing how can deliver real, sustainable value back to the community. we will continue to share more details on token utility use cases and the economic flywheel we have built. the goal is to create a self-reinforcing system where creators earn through royalties, autonomous agents drive on-chain activity, and platform growth directly benefits active community members and token holders. together, these features enable a complete creative flow. from a simple idea, anyone can quickly build fully realized 3d ai agents standing in rich, custom scenes. we are excited to see what the community builds next.
0:24

great to see more people generating 3d avatars with our new text-to-3d feature in forge. this marks a step in the right direction in putting powerful creation tools directly in the hands of everyone. we built forge entirely from the ground up over the past months as one of the key releases on our roadmap. having full proprietary ownership of the technology gives us complete control to shape its direction without depending on external platforms or third-party licenses. building on this foundation, our upcoming studio feature will let users generate high-quality accessories, clothing, and environments simply by typing natural language prompts. a single description can produce fully textured, production-ready 3d assets in seconds, with options to create multiple variations and refine them through follow-up instructions.every asset created in studio integrates seamlessly with the 3d ai agents made in forge on users can instantly apply clothing and accessories with automatic fitting, layer multiple items, and place their agents inside custom-generated environments. all clothing and accessories come pre-rigged and optimized, while environments include proper lighting and geometry for immediate use in games, animation, virtual worlds, and more. we have spent months thoughtfully designing how can deliver real, sustainable value back to the community. we will continue to share more details on token utility use cases and the economic flywheel we have built. the goal is to create a self-reinforcing system where creators earn through royalties, autonomous agents drive on-chain activity, and platform growth directly benefits active community members and token holders. together, these features enable a complete creative flow. from a simple idea, anyone can quickly build fully realized 3d ai agents standing in rich, custom scenes. we are excited to see what the community builds next.

nich

19,611 Aufrufe • vor 2 Monaten

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.
0:21

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

AK

126,595 Aufrufe • vor 2 Jahren

Wow. Recreating the Shawshank Redemption prison in 3D from a single video, in real time (!) Just read the MASt3R-SLAM paper and it's pretty neat. These folks basically built a real-time dense SLAM system on top of MASt3R, which is a transformer-based neural network that can do 3d reconstruction and localization from uncalibrated image pairs. The cool part is they don't need a fixed camera model -- it just works with arbitrary cameras -- think different focal lengths, sensor sizes, even handling zooming in video (FMV drone video anyone?!). If you've done photogrammetry or played with NeRFs you know that is a HUGE deal. They've solved some tricky problems like efficient point matching and tracking, plus they've figured out how to fuse point clouds and handle loop closures in real-time. Their system runs at about 15 FPS on a 4090 and produces both camera poses and dense geometry. When they know the camera calibration, they get SOTA results across several benchmarks, but even without calibration, they still perform well. What's interesting is the approach -- most recent SLAM work has built on DROID-SLAM's architecture, but these folks went a different direction by leveraging a strong 3D reconstruction prior. Seems to give them more coherent geometry, which makes sense since that's what MASt3R was designed for. For anyone who cares about monocular SLAM and 3D reconstruction, this feels like a significant step toward plug-and-play dense SLAM without calibration headaches -- perfect for drones, robots, AR/VR -- the works!
0:29

Wow. Recreating the Shawshank Redemption prison in 3D from a single video, in real time (!) Just read the MASt3R-SLAM paper and it's pretty neat. These folks basically built a real-time dense SLAM system on top of MASt3R, which is a transformer-based neural network that can do 3d reconstruction and localization from uncalibrated image pairs. The cool part is they don't need a fixed camera model -- it just works with arbitrary cameras -- think different focal lengths, sensor sizes, even handling zooming in video (FMV drone video anyone?!). If you've done photogrammetry or played with NeRFs you know that is a HUGE deal. They've solved some tricky problems like efficient point matching and tracking, plus they've figured out how to fuse point clouds and handle loop closures in real-time. Their system runs at about 15 FPS on a 4090 and produces both camera poses and dense geometry. When they know the camera calibration, they get SOTA results across several benchmarks, but even without calibration, they still perform well. What's interesting is the approach -- most recent SLAM work has built on DROID-SLAM's architecture, but these folks went a different direction by leveraging a strong 3D reconstruction prior. Seems to give them more coherent geometry, which makes sense since that's what MASt3R was designed for. For anyone who cares about monocular SLAM and 3D reconstruction, this feels like a significant step toward plug-and-play dense SLAM without calibration headaches -- perfect for drones, robots, AR/VR -- the works!

Bilawal Sidhu

704,021 Aufrufe • vor 1 Jahr

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: **Highlights:** 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch
0:22

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: **Highlights:** 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch

Bo Wang

178,572 Aufrufe • vor 2 Jahren

BMW's upcoming all-electric iX3 SUV: • Expected EPA range: Over 400 miles (according to BMW) • Up to 400kW charging speeds. Can add ~200 miles of range in 10 mins • New “superbrain” will provide 20X the computing power of BMW’s previous system • Redesigned cockpit: Panoramic display shows information from A-pillar to A-pillar, 3D Head-Up, central display with matrix LEDs • Uses 2,000 feet less wiring than before • 800-volt • Bidirectional charging functions: Vehicle-to-Home (V2H) and Vehicle-to-Grid (V2G) • 20% lower drag coefficient • BMW: "BMW Group’s latest driver assistance systems are among the world’s best. the iX3 merges the proven, rule-based assistance system with a new system based on AI. In an optimum symbiosis, this combination supports the driver even more precisely and reliably in challenging situations on the road. The Superbrain of Automated Driving pools all customer functions for automated driving and has a 20x faster processing speed than the previous system." • Production version of the iX3 will be unveiled in Munich next month (Sept). Will go on sale in the US in Q2 2026. • Expected starting price: Between $55k-$60k. BMW CEO on the new iX3: “This will be the benchmark of the industry. BMW can build superior electric cars, and the rest of the market will have to answer.”
0:39

BMW's upcoming all-electric iX3 SUV: • Expected EPA range: Over 400 miles (according to BMW) • Up to 400kW charging speeds. Can add ~200 miles of range in 10 mins • New “superbrain” will provide 20X the computing power of BMW’s previous system • Redesigned cockpit: Panoramic display shows information from A-pillar to A-pillar, 3D Head-Up, central display with matrix LEDs • Uses 2,000 feet less wiring than before • 800-volt • Bidirectional charging functions: Vehicle-to-Home (V2H) and Vehicle-to-Grid (V2G) • 20% lower drag coefficient • BMW: "BMW Group’s latest driver assistance systems are among the world’s best. the iX3 merges the proven, rule-based assistance system with a new system based on AI. In an optimum symbiosis, this combination supports the driver even more precisely and reliably in challenging situations on the road. The Superbrain of Automated Driving pools all customer functions for automated driving and has a 20x faster processing speed than the previous system." • Production version of the iX3 will be unveiled in Munich next month (Sept). Will go on sale in the US in Q2 2026. • Expected starting price: Between $55k-$60k. BMW CEO on the new iX3: “This will be the benchmark of the industry. BMW can build superior electric cars, and the rest of the market will have to answer.”

Sawyer Merritt

43,218 Aufrufe • vor 1 Jahr

Today, we’re pushing a major update to Edison Analysis, our data analysis agent, which is tuned for scientific research and SOTA across data analysis benchmarks. In contrast to Kosmos, which runs for 6-12 hours and produces tens of thousands of lines of code, Edison Analysis runs for seconds to minutes and is best for specific, well-defined computational tasks. It is available both on our platform under the Analysis tab, and via API, and costs only one credit per run, so it is available to users on both free and paid tiers. Edison Analysis is a modified version of the data analysis agent Kosmos uses in its trajectories. Try it out! One of the most important improvements over our previous data analysis agents has been the addition of a specialized data retrieval tool. Edison Analysis can either use this tool to access data, or can pull data down directly via API. To evaluate this tool, we ranked the most commonly used public data repositories across recent papers from BioRxiv, and created a new benchmark that measures the ability of a language agent system to retrieve raw data from those sources. Edison Analysis gets 71% on this benchmark, and we’ll be working to increase this over time. You can read more about our benchmarks in the our blog post, link below. Some features worth highlighting: 1. Edison Analysis produces a report on the analysis it runs, along with a Jupyter notebook that you can download to reproduce the analysis yourself. Every figure it produces is linked back to the specific lines of code used to produce the figure, to make it easy to reproduce. 2. It works well with both Python and R. 3. One of the best uses for Edison Analysis is to use it to retrieve datasets that you can then analyze with Kosmos. We have a bunch of major improvements to Edison Analysis coming in the next few months that we’re excited to share. In the meantime, congratulations to the team, especially Ludovico Mitchener, Jon Laurent, Conor Igoe , Alex Andonian, and many more.
1:12

Today, we’re pushing a major update to Edison Analysis, our data analysis agent, which is tuned for scientific research and SOTA across data analysis benchmarks. In contrast to Kosmos, which runs for 6-12 hours and produces tens of thousands of lines of code, Edison Analysis runs for seconds to minutes and is best for specific, well-defined computational tasks. It is available both on our platform under the Analysis tab, and via API, and costs only one credit per run, so it is available to users on both free and paid tiers. Edison Analysis is a modified version of the data analysis agent Kosmos uses in its trajectories. Try it out! One of the most important improvements over our previous data analysis agents has been the addition of a specialized data retrieval tool. Edison Analysis can either use this tool to access data, or can pull data down directly via API. To evaluate this tool, we ranked the most commonly used public data repositories across recent papers from BioRxiv, and created a new benchmark that measures the ability of a language agent system to retrieve raw data from those sources. Edison Analysis gets 71% on this benchmark, and we’ll be working to increase this over time. You can read more about our benchmarks in the our blog post, link below. Some features worth highlighting: 1. Edison Analysis produces a report on the analysis it runs, along with a Jupyter notebook that you can download to reproduce the analysis yourself. Every figure it produces is linked back to the specific lines of code used to produce the figure, to make it easy to reproduce. 2. It works well with both Python and R. 3. One of the best uses for Edison Analysis is to use it to retrieve datasets that you can then analyze with Kosmos. We have a bunch of major improvements to Edison Analysis coming in the next few months that we’re excited to share. In the meantime, congratulations to the team, especially Ludovico Mitchener, Jon Laurent, Conor Igoe , Alex Andonian, and many more.

Sam Rodriques

61,934 Aufrufe • vor 8 Monaten

We’re thrilled to share that our MERFISH+ preprint is now live on bioRxiv!👉 In this work, the Bintu and Zhu labs (UCSD) developed MERFISH+, a next-generation spatial genomics platform that combines genome-wide RNA and epigenetic imaging over a large field of view. By introducing acrydite-modified probes covalently anchored to hydrogels, MERFISH+ achieves remarkable imaging stability and enables >1,800-gene, multi-modal, and multi-month experiments. With this platform, they, together with the Chi lab at UCSD, profiled a whole developing human heart at 12 post-conception week with merely two slides, resulting in a total of 53 slides, 3.1 million single cells and more than 30 cell types. Building upon our previous 3D reconstruction and modeling framework, Spateo ( we reconstruct the 3D human heart that nicely captures the anatomical structure of the heart, including the intricate vasculature network. Sophisticated analyses provide a holistic view of an entire organ and enable systematic characterization of 3D cellular neighborhoods and transcriptional gradients of substructures such as the descending arteries. Furthermore, using a generative integration framework for spatial multimodal data (Spateo-VI), we harmonized these MERFISH+ transcriptomic and chromatin data to reconstruct a 3D spatially-resolved multi-omics atlas of the developing human heart, shared at and MERFISH+ thus sets a new standard for large-format, multi-omic spatial profiling, enabling holistic, 3D characterization of organs at subcellular resolution. Huge congratulations to first authors Colin Kern, qingquan Zhang, @YifanLu2024 , and Jacqueline Eschbach, and to all collaborators from the Bintu, Zhu, Chi, and Qiu labs for this amazing team effort. Thanks for your diligence, creativity, and hard work on this project. We’re grateful for support from Arc Institute and our generous donors. Our lab is expanding—if you’re excited about building the next generation of single-cell and spatial genomics techniques and predictive single cell and spatial foundation models, we’re hiring! If you are interested, please reach out to me via direct message or email at xiaojie@stanford.edu. We are excited for any potential collaborations along this line of research in Stanford, UCSF and Berkeley and other labs as well.
0:42

We’re thrilled to share that our MERFISH+ preprint is now live on bioRxiv!👉 In this work, the Bintu and Zhu labs (UCSD) developed MERFISH+, a next-generation spatial genomics platform that combines genome-wide RNA and epigenetic imaging over a large field of view. By introducing acrydite-modified probes covalently anchored to hydrogels, MERFISH+ achieves remarkable imaging stability and enables >1,800-gene, multi-modal, and multi-month experiments. With this platform, they, together with the Chi lab at UCSD, profiled a whole developing human heart at 12 post-conception week with merely two slides, resulting in a total of 53 slides, 3.1 million single cells and more than 30 cell types. Building upon our previous 3D reconstruction and modeling framework, Spateo ( we reconstruct the 3D human heart that nicely captures the anatomical structure of the heart, including the intricate vasculature network. Sophisticated analyses provide a holistic view of an entire organ and enable systematic characterization of 3D cellular neighborhoods and transcriptional gradients of substructures such as the descending arteries. Furthermore, using a generative integration framework for spatial multimodal data (Spateo-VI), we harmonized these MERFISH+ transcriptomic and chromatin data to reconstruct a 3D spatially-resolved multi-omics atlas of the developing human heart, shared at and MERFISH+ thus sets a new standard for large-format, multi-omic spatial profiling, enabling holistic, 3D characterization of organs at subcellular resolution. Huge congratulations to first authors Colin Kern, qingquan Zhang, @YifanLu2024 , and Jacqueline Eschbach, and to all collaborators from the Bintu, Zhu, Chi, and Qiu labs for this amazing team effort. Thanks for your diligence, creativity, and hard work on this project. We’re grateful for support from Arc Institute and our generous donors. Our lab is expanding—if you’re excited about building the next generation of single-cell and spatial genomics techniques and predictive single cell and spatial foundation models, we’re hiring! If you are interested, please reach out to me via direct message or email at [email protected]. We are excited for any potential collaborations along this line of research in Stanford, UCSF and Berkeley and other labs as well.

evo-devo

42,268 Aufrufe • vor 9 Monaten

Chapter 1 Pre-Alpha "Explore. Craft. Survive." will be dropping this quarter welcoming the very first inhabitants to #E2V1 & introduce early mechanics that will form the foundation of life inside #Earth2🌍✌️ Read below for important details: This update is no longer a simple avatar release but a significant update to the entire foundation of the #E2V1 world. When I first talked about dropping avatars I was eager to keep things moving and referred to a very simple system to test skins and movement, then to subsequently implement the full version after that. While this approach would have brought new content out quickly, it could have turned into weeks of extra development time with little upside for the wider community, apart from testing the movement and appearance of the avatars, there would have been little else to do until another subsequent update. Therefore I made the decision to focus solely on implementing the core part of the full system we plan to use long term on the platform. I apologise that this release will take a little longer than originally anticipated, having moved away from the initial lower spec'd plan, but I'd like to get this right & ready for a wider audience to start using and enjoying. As much as I'd love to push out updates faster, I will be taking this approach to #E2V1 development in the future as well. The plan will still be to release to our testing group first after our internal QA, test and then stabilise. If all looks good and we're happy with how everything is performing, we will push forward with a public release. There will inevitably be ongoing adjustments, additions and tweaks to these type of systems over time. It may seem like a long time, but #E2V1 was only released to our testing group 7 weeks ago! How crazy is that? I know for me it feels like many months already. Nonetheless, during this short period of time we have not only released a couple of updated versions, and an updated version to the Earth 2 Launcher, but the team has also been working incredibly hard on the #E2V1 and #BE mechanics for the Chapter 1 systems - and there is a lot involved in this upcoming release. We will be dropping an article later this week, which I believe will help our community understand the extensive list of features planned for this release and in turn explain why it is taking a little longer. I'd rather not spoil all of the details, but the article contains a lot of information about the various systems we're developing including rules of life, avatar synthesisation, death, saving, skins, day/night cycles, free to play model, avatar vitals, how land ownership fits in with all of this and much more. It is important to keep in mind that each of these systems needed to be designed and implemented, which takes time. The article will also begin to reveal some of the design plans I have been working on to increase utility for T1 land, which will also include various T1 classes. I also intend to follow through and expand utility on T1 land into other upcoming mechanics on the platform, some of which will be revealed in a separate article in the near future and are also part of the Chapter 1 release. Ideally, all going well, we will be testing early multiplayer at some stage this quarter and as mentioned earlier this year #Hordes in Q3, #EggHunt in Q4 & maybe some early buildings on properties. In addition to these goals, we have a couple of wild card features floating around in between as well. I know I reiterate it from time to time, but the long term goal for #Earth2 has not changed. We are building a #geolocational #metaverse - a platform! We still want #Players building cities, providing more individualised experiences, advertising, trading, shopping, e-commerce, socialising and so forth. And I still have plans to link #E2V1 into #AR and support #VR, but we cannot do all of this at once. I am also looking at ways we can expand the team and setup a studio, probably in East Asia, to speed up development. The current features we're focusing on plan to expand utility for land on a wide scale & introduce various experiences inside #E2V1, providing potential benefits for land owners & #Players alike on a broader level, while at the same time allowing our team to introduce mechanics that will be used throughout the future of the platform, test limitations and work our way around challenges as they arise. Thanks for reading and don't forget to drop a like and comment on the video! We appreciate the love and support and it also helps with the algorithm! Keep an eye out for the article later this week - a lot of details incoming! I want life to have meaning inside Earth 2 and I believe my approach could change the way #Players consider each action on the platform.. possibly alter the overall dynamics in a number of ways. #Metaverse #E2V1 #OpenWorld #earlyaccess #Earth2 $ESS
1:15

Chapter 1 Pre-Alpha "Explore. Craft. Survive." will be dropping this quarter welcoming the very first inhabitants to #E2V1 & introduce early mechanics that will form the foundation of life inside #Earth2🌍✌️ Read below for important details: This update is no longer a simple avatar release but a significant update to the entire foundation of the #E2V1 world. When I first talked about dropping avatars I was eager to keep things moving and referred to a very simple system to test skins and movement, then to subsequently implement the full version after that. While this approach would have brought new content out quickly, it could have turned into weeks of extra development time with little upside for the wider community, apart from testing the movement and appearance of the avatars, there would have been little else to do until another subsequent update. Therefore I made the decision to focus solely on implementing the core part of the full system we plan to use long term on the platform. I apologise that this release will take a little longer than originally anticipated, having moved away from the initial lower spec'd plan, but I'd like to get this right & ready for a wider audience to start using and enjoying. As much as I'd love to push out updates faster, I will be taking this approach to #E2V1 development in the future as well. The plan will still be to release to our testing group first after our internal QA, test and then stabilise. If all looks good and we're happy with how everything is performing, we will push forward with a public release. There will inevitably be ongoing adjustments, additions and tweaks to these type of systems over time. It may seem like a long time, but #E2V1 was only released to our testing group 7 weeks ago! How crazy is that? I know for me it feels like many months already. Nonetheless, during this short period of time we have not only released a couple of updated versions, and an updated version to the Earth 2 Launcher, but the team has also been working incredibly hard on the #E2V1 and #BE mechanics for the Chapter 1 systems - and there is a lot involved in this upcoming release. We will be dropping an article later this week, which I believe will help our community understand the extensive list of features planned for this release and in turn explain why it is taking a little longer. I'd rather not spoil all of the details, but the article contains a lot of information about the various systems we're developing including rules of life, avatar synthesisation, death, saving, skins, day/night cycles, free to play model, avatar vitals, how land ownership fits in with all of this and much more. It is important to keep in mind that each of these systems needed to be designed and implemented, which takes time. The article will also begin to reveal some of the design plans I have been working on to increase utility for T1 land, which will also include various T1 classes. I also intend to follow through and expand utility on T1 land into other upcoming mechanics on the platform, some of which will be revealed in a separate article in the near future and are also part of the Chapter 1 release. Ideally, all going well, we will be testing early multiplayer at some stage this quarter and as mentioned earlier this year #Hordes in Q3, #EggHunt in Q4 & maybe some early buildings on properties. In addition to these goals, we have a couple of wild card features floating around in between as well. I know I reiterate it from time to time, but the long term goal for #Earth2 has not changed. We are building a #geolocational #metaverse - a platform! We still want #Players building cities, providing more individualised experiences, advertising, trading, shopping, e-commerce, socialising and so forth. And I still have plans to link #E2V1 into #AR and support #VR, but we cannot do all of this at once. I am also looking at ways we can expand the team and setup a studio, probably in East Asia, to speed up development. The current features we're focusing on plan to expand utility for land on a wide scale & introduce various experiences inside #E2V1, providing potential benefits for land owners & #Players alike on a broader level, while at the same time allowing our team to introduce mechanics that will be used throughout the future of the platform, test limitations and work our way around challenges as they arise. Thanks for reading and don't forget to drop a like and comment on the video! We appreciate the love and support and it also helps with the algorithm! Keep an eye out for the article later this week - a lot of details incoming! I want life to have meaning inside Earth 2 and I believe my approach could change the way #Players consider each action on the platform.. possibly alter the overall dynamics in a number of ways. #Metaverse #E2V1 #OpenWorld #earlyaccess #Earth2 $ESS

Shane Isaac 🌍2️⃣

21,349 Aufrufe • vor 1 Jahr

Check out our latest #Earth2 #creation the #MotorwayMarauder🌍✌️From #ConceptArt to #gameready, gives me #MadMax & #Terminator vibes - love it! Some #IMPORTANT things to remember about #Earth2 #drops leading up to #E2V1👇 1⃣Acquiring #skins is completely optional 2⃣ Notably, the #skins we drop leading up to the #E2V1 release will more likely have a ‘Limited Supply’, something that rewards our early supporters with increased rarity. These Limited Supply drops will become less frequent in the future being replaced with drops of unlimited supply but restricted availability requiring Players to wait for a skin to become available in the skin store or hope to acquire it via game play inside #E2V1 3⃣ Our goal is not to have these early #skin drops sell out during their maiden release, but rather provide an opportunity for our early supporters to secure skins they like at a discounted price with the welcoming guarantee of an all time Limited Supply. 4⃣ Most of the early #skin drops will naturally allow #Players to resell them, but importantly many will also be assigned with a once ever discount rate. #Skins that have Future Price Reduction assigned with “No” means that if the #skin does not sell out on its maiden drop and subsequently gets listed in store again at a future date, #Earth2 will never resell that #skin at any discounted rate. 5⃣ #Earth2 is building a modular clothing system, meaning that most items of clothing, especially if sold separately, will be interchangeable on avatars making them more useful and giving the #Player more ways to mix up their outfits, look the part & express themselves. 6⃣ The version of the skin you see in store is the base version you receive with the skin, however, there will also be opportunities to tailor outfits in game allowing for the adjustment in fabrics, prints, colours and more (be prepared for a little legwork though, unless you want to take your clothing items to another #Player who specialises in tailoring). 7⃣ #Earth2 is proud to have uniquely designed the #skins you listed in our store & it is important to remember that these #skins are the end result of a long and stringent internal character pipeline we have developed. Each skin starts from concept art which is reviewed and iterated on numerous times before being approved. Once approval is given, it is handed to our 3D modelling team for sculpting into high poly, then low poly, while receiving numerous reviews and adhering to our pipeline protocols. When the low poly version is finished, the 3D team then finishes with UV mapping and texture baking and runs their own QA process. Once the entire 3D sculpting process is approved, the models enter the skinning, rigging and weight painting process and are then passed to the animation team for further testing with various animations and another set of QA such as minimising clipping and identifying any other potential flaws in the model. The model then needs to go through the final process with a game developer to ensure it passed the final tests and is game ready for #E2V1. 8⃣ The above steps only represent a light explanation of our character and clothing pipeline, but as you can see we’re not simply selling static concept art images or a high poly 3D model that lacks any kind of optimisations resulting in little chance of it ever being game ready in its advertised state - there is a lot of work that takes place in order to get these skins to the stage our #Players see them in the video footage. Each skin sold helps support different parts of the development of Earth 2 providing us with new opportunities to further expand our team, speed up development and keep stepping closer to our short and long term goals for #E2V1 and #Earth2. 9⃣ We understand that fashion will become a big part of the #Metaverse, but we want to remind people that while some #Players may decide to set rules for certain skins being required for access or a ticket to entry under certain circumstances, from #Earth2’s perspective these skins are for cosmetic purposes only. 🔟 These skin drops also give #Earth2 the chance to show off our ability and versatility in avatar creation and accessories so even if you’re not keen on picking up a skin, remember to have fun and simply enjoy the progress! Also take heart every time you see a #skindrop, if the #E2V1 pre-alpha was not launching this year I wouldn't be dropping skins. We want to have at least some diversity for #avatars inside #E2V1 from the early stages! #Earth2 #Metaverse #Skins #Fashion #Collectibles #3DWorld #WIP #prealpha2024
1:29

Check out our latest #Earth2 #creation the #MotorwayMarauder🌍✌️From #ConceptArt to #gameready, gives me #MadMax & #Terminator vibes - love it! Some #IMPORTANT things to remember about #Earth2 #drops leading up to #E2V1👇 1⃣Acquiring #skins is completely optional 2⃣ Notably, the #skins we drop leading up to the #E2V1 release will more likely have a ‘Limited Supply’, something that rewards our early supporters with increased rarity. These Limited Supply drops will become less frequent in the future being replaced with drops of unlimited supply but restricted availability requiring Players to wait for a skin to become available in the skin store or hope to acquire it via game play inside #E2V1 3⃣ Our goal is not to have these early #skin drops sell out during their maiden release, but rather provide an opportunity for our early supporters to secure skins they like at a discounted price with the welcoming guarantee of an all time Limited Supply. 4⃣ Most of the early #skin drops will naturally allow #Players to resell them, but importantly many will also be assigned with a once ever discount rate. #Skins that have Future Price Reduction assigned with “No” means that if the #skin does not sell out on its maiden drop and subsequently gets listed in store again at a future date, #Earth2 will never resell that #skin at any discounted rate. 5⃣ #Earth2 is building a modular clothing system, meaning that most items of clothing, especially if sold separately, will be interchangeable on avatars making them more useful and giving the #Player more ways to mix up their outfits, look the part & express themselves. 6⃣ The version of the skin you see in store is the base version you receive with the skin, however, there will also be opportunities to tailor outfits in game allowing for the adjustment in fabrics, prints, colours and more (be prepared for a little legwork though, unless you want to take your clothing items to another #Player who specialises in tailoring). 7⃣ #Earth2 is proud to have uniquely designed the #skins you listed in our store & it is important to remember that these #skins are the end result of a long and stringent internal character pipeline we have developed. Each skin starts from concept art which is reviewed and iterated on numerous times before being approved. Once approval is given, it is handed to our 3D modelling team for sculpting into high poly, then low poly, while receiving numerous reviews and adhering to our pipeline protocols. When the low poly version is finished, the 3D team then finishes with UV mapping and texture baking and runs their own QA process. Once the entire 3D sculpting process is approved, the models enter the skinning, rigging and weight painting process and are then passed to the animation team for further testing with various animations and another set of QA such as minimising clipping and identifying any other potential flaws in the model. The model then needs to go through the final process with a game developer to ensure it passed the final tests and is game ready for #E2V1. 8⃣ The above steps only represent a light explanation of our character and clothing pipeline, but as you can see we’re not simply selling static concept art images or a high poly 3D model that lacks any kind of optimisations resulting in little chance of it ever being game ready in its advertised state - there is a lot of work that takes place in order to get these skins to the stage our #Players see them in the video footage. Each skin sold helps support different parts of the development of Earth 2 providing us with new opportunities to further expand our team, speed up development and keep stepping closer to our short and long term goals for #E2V1 and #Earth2. 9⃣ We understand that fashion will become a big part of the #Metaverse, but we want to remind people that while some #Players may decide to set rules for certain skins being required for access or a ticket to entry under certain circumstances, from #Earth2’s perspective these skins are for cosmetic purposes only. 🔟 These skin drops also give #Earth2 the chance to show off our ability and versatility in avatar creation and accessories so even if you’re not keen on picking up a skin, remember to have fun and simply enjoy the progress! Also take heart every time you see a #skindrop, if the #E2V1 pre-alpha was not launching this year I wouldn't be dropping skins. We want to have at least some diversity for #avatars inside #E2V1 from the early stages! #Earth2 #Metaverse #Skins #Fashion #Collectibles #3DWorld #WIP #prealpha2024

Shane Isaac 🌍2️⃣

19,007 Aufrufe • vor 2 Jahren

NEW ROBOT BENCHMARK: If your robot can do Origami, it can do anything! Called The Robotic Origami Challenge, it is a dexterous-manipulation competition and benchmark held at IROS 2026, organized by 13 co-organizers with the Nippon Origami Association as judge and task curator -> pretty cool to have them on board imho. The evaluation consists of single task: a traditional Japanese paper airplane, in exactly six folds, from a 15×15 cm sheet of ≥60 gsm paper, on a competition-supplied standardized rig (bimanual arms + Sharpa Hands), both remotely and on-site. Teams bring policies, not hardware. An "Origami Grand Master" declares pass/fail on crease accuracy, structural fidelity, symmetry and paper integrity. Among passes, faster folds rank higher, with a 10-minute-per-attempt ceiling and flight explicitly irrelevant to the score. -> I find it interesting how they chose to evaluate the task. Quality is a binary pass or fail, therefore speed becomes the only thing graded here. Speed is currently the bottleneck in dexterous manipulation though, so this choice makes sense. I wonder whether there could be finer ways to grade the qaulity of such a creative task though. When it comes to data, registered teams get 500+ teleoperation episodes (six camera streams, 65-D joint state/action, 10-fingertip 6-axis tactile), an NVIDIA Isaac Sim environment with thin-shell paper physics (plastic creasing + fold memory), digital twins of every partner hand, and a remote eval lab (upload a policy, queue an eval, get scored). Still, I think it is a great dexterity benchmark the field badly needs, it supplies the hardware, an outside human judges, and the pass criterion is externally defined -> all three degrees of freedom are checked! Neutral measurement layer, here we go! The task is engineered to be un-gameable and to isolate pure dexterity. A known figure, exactly six folds, judged on creases, with flight explicitly declared irrelevant (the latter makes sense to me). Therefore, this underlines the goal to focus on dexterity, not task-selection or other strategies. I really like origami as an ideal controlled dexterity task: deterministic goal, deformable medium, sequential, bimanual, precision-bound. I am just not quite satisfied again by the binary pass or fail, I think quality of execution could be finely graded! But again, I understand this is not the goal yet. Also interesting to see the Sharpa Hands as de facto standard for everyone. Total land-grab that anoints Sharpa as the reference dexterous hand, also featured in Google's Gemini Robotics 2. By providing the hardware, the benchmark measures software while quietly making "good on Sharpa" the definition of good, and Sharpa gets real world data and feedback as a bonus. That's smart, the data flywheel starts spinning. The provided dataset is the richest tactile-manipulation corpus I have seen yet: 10-fingertip 6-axis tactile, plus plastic creasing and fold memory. 500+ teleop episodes with six camera streams, 65-D joint state/action, and ten fingertip 6-axis tactile sensors. The force/tactile channel are parts of the the benchmark's data, this is the first time I see this. Credits where it's due: organizers include Yang Gao, Noriaki Hirose, Steve Xie, Chris Paxton, Jiafei Duan, Michael Cho - Rbt/Acc, Michael Yuan, Haoquan Fang, and others.
0:36

NEW ROBOT BENCHMARK: If your robot can do Origami, it can do anything! Called The Robotic Origami Challenge, it is a dexterous-manipulation competition and benchmark held at IROS 2026, organized by 13 co-organizers with the Nippon Origami Association as judge and task curator -> pretty cool to have them on board imho. The evaluation consists of single task: a traditional Japanese paper airplane, in exactly six folds, from a 15×15 cm sheet of ≥60 gsm paper, on a competition-supplied standardized rig (bimanual arms + Sharpa Hands), both remotely and on-site. Teams bring policies, not hardware. An "Origami Grand Master" declares pass/fail on crease accuracy, structural fidelity, symmetry and paper integrity. Among passes, faster folds rank higher, with a 10-minute-per-attempt ceiling and flight explicitly irrelevant to the score. -> I find it interesting how they chose to evaluate the task. Quality is a binary pass or fail, therefore speed becomes the only thing graded here. Speed is currently the bottleneck in dexterous manipulation though, so this choice makes sense. I wonder whether there could be finer ways to grade the qaulity of such a creative task though. When it comes to data, registered teams get 500+ teleoperation episodes (six camera streams, 65-D joint state/action, 10-fingertip 6-axis tactile), an NVIDIA Isaac Sim environment with thin-shell paper physics (plastic creasing + fold memory), digital twins of every partner hand, and a remote eval lab (upload a policy, queue an eval, get scored). Still, I think it is a great dexterity benchmark the field badly needs, it supplies the hardware, an outside human judges, and the pass criterion is externally defined -> all three degrees of freedom are checked! Neutral measurement layer, here we go! The task is engineered to be un-gameable and to isolate pure dexterity. A known figure, exactly six folds, judged on creases, with flight explicitly declared irrelevant (the latter makes sense to me). Therefore, this underlines the goal to focus on dexterity, not task-selection or other strategies. I really like origami as an ideal controlled dexterity task: deterministic goal, deformable medium, sequential, bimanual, precision-bound. I am just not quite satisfied again by the binary pass or fail, I think quality of execution could be finely graded! But again, I understand this is not the goal yet. Also interesting to see the Sharpa Hands as de facto standard for everyone. Total land-grab that anoints Sharpa as the reference dexterous hand, also featured in Google's Gemini Robotics 2. By providing the hardware, the benchmark measures software while quietly making "good on Sharpa" the definition of good, and Sharpa gets real world data and feedback as a bonus. That's smart, the data flywheel starts spinning. The provided dataset is the richest tactile-manipulation corpus I have seen yet: 10-fingertip 6-axis tactile, plus plastic creasing and fold memory. 500+ teleop episodes with six camera streams, 65-D joint state/action, and ten fingertip 6-axis tactile sensors. The force/tactile channel are parts of the the benchmark's data, this is the first time I see this. Credits where it's due: organizers include Yang Gao, Noriaki Hirose, Steve Xie, Chris Paxton, Jiafei Duan, Michael Cho - Rbt/Acc, Michael Yuan, Haoquan Fang, and others.

Léo

22,827 Aufrufe • vor 8 Tagen