Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Meet Iris-3B: a pixel-space T2I model and general vision learner that generates every pixel directly. No VAE. Pre-trained from scratch, 3B, open weights. We also explore its generative prior on detail-critical vision tasks: depth estimation and image restoration. Fully open: weights, code and demo (Apache 2.0)

Speridlabs

1,026 subscribers

80,397 views • 3 days ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

34 Comments

Speridlabs3 days ago

Weights: Code: Try it:

Speridlabs3 days ago

We're hiring

Borja Perez3 days ago

@barckcode nunca te pido nada 🥹 y encima son de españita

Alexey Fateev3 days ago

@ComfyUI workflow guys? sounds incredible!

Armin Catovic3 days ago

Can it work on low res images, like 64x64 pixels? This is the limitation of todays imagegen models - even for pixel art they produce high res images, when scaled down look awful.

alejandro3 days ago

🚀🚀🚀

Jorge Dobón3 days ago

Go go go @speridlabs 🙌🏼

Julien Blanchon 🇺🇦3 days ago

@AntonObukhov1 Interesting as a prior

Buswe3 days ago

Dropping the VAE is what makes depth and restoration work here: no latent decode step to smooth away the thin, high frequency detail.

Alice The Ai Expert2 days ago

Incredible work 3B pixels directly with no VAE and fully open weights is exactly the kind of open research the community needs!

Javier Álvarez2 days ago

no VAE here... top my friend @hanqiulicai 🚀 🚀

FRareDom3 days ago

Open weights is incredible! THANK YOU!!

ander2 days ago

Congrats!

Amelia | AI & Tech2 days ago

Generating pixels directly without a VAE is boldOpen weights on a no-VAE generator is the real flex.

A.W.E.S.O.M.-O 40003 days ago

ngl open weights at 3b makes this easy for people to actually try fr

Salar 🇪🇸🇮🇷3 days ago

Neat

RAZA | AI EXPLORER2 days ago

Impressive work! Open-weight pixel-space models open up exciting possibilities for image generation and vision research.

ZenithAi2 days ago

Open weights and pixel level generation make this research especially promising

Pongphat2 days ago

Interesting!

Aaliya2 days ago

Using it to fix images sounds really useful too.

Elise Tech Ai2 days ago

This is fantastic Iris 3B directly in pixel space with no VAE is a bold and impressive approach love that its fully open

드럼통청약당첨2 days ago

Pixel Space with optimized architecture is future! Thank you for your work!

Adria Blancafort3 days ago

Nice!

AI路子野2 days ago

Still using a 4B VL addition. Not fully portable

Vantix AI Agency2 days ago

No VAE and direct pixel generation at 3B thats a bold architecture bet love that its fully open with weights and code

Jack3 days ago

pixel space models deserve more love

Anherutowa Calvo3 days ago

Lets go!!

Ella Tech & Tool2 days ago

Love this. Pixel-space generation with open weights will push T2I and vision research forward fast.

Sani Ai Tech2 days ago

Impressive to see pixel-native generation paired with open weights and broader vision capabilities

Elara AI1 day ago

Iris-3B pixel-space T2I model generates pixels directly without VAE openly released now

Aria Tech2 days ago

No VAE pixel-space T2I is bold Iris-3B looks super promising

Pablo3 days ago

@bot

Zara Tech & Tools2 days ago

Exciting open release pushing direct pixel generation

Martii3 days ago

🔥🔥🔥🔥🔥

Related Videos

SenseNova-U1.5-8B-MoT is HERE, it's a big deal! 🔥 open weights matching Nano Banana 2 on benchs 🍌 no VAE, no text encoder, no DiT. a mixture of transformers: text and image tokens use different weights, attend to each other, denoise in pixel space ▶️

SenseNova-U1.5-8B-MoT is HERE, it's a big deal! 🔥 open weights matching Nano Banana 2 on benchs 🍌 no VAE, no text encoder, no DiT. a mixture of transformers: text and image tokens use different weights, attend to each other, denoise in pixel space ▶️

Hugging Apps

42,720 views • 1 month ago

Today, every Nomic-Embed-Text embedding becomes multimodal. Introducing Nomic-Embed-Vision: - a high quality, unified embedding space for image, text, and multimodal tasks - outperforms both OpenAI CLIP and text-embedding-3-small - open weights and code to enable indie hacking, research, and experimentation - released in collaboration with MongoDB, LlamaIndex 🦙, , Hugging Face, Amazon Web Services, DigitalOcean, Lambda

Today, every Nomic-Embed-Text embedding becomes multimodal. Introducing Nomic-Embed-Vision: - a high quality, unified embedding space for image, text, and multimodal tasks - outperforms both OpenAI CLIP and text-embedding-3-small - open weights and code to enable indie hacking, research, and experimentation - released in collaboration with MongoDB, LlamaIndex 🦙, , Hugging Face, Amazon Web Services, DigitalOcean, Lambda

CalCo

103,287 views • 2 years ago

🚨 FASHN VTON v1.5 is now open source. Released in early 2025 and still widely used, this virtual try-on model generates photorealistic images from a person image and a garment image. Key points: • Pixel-space RGB generation, no VAE • Maskless inference, no person segmentation needed • 972M parameters, ~5s on H100, runs on consumer GPUs • Apache 2.0 licensed, first commercially usable open-source VTON Why open source? While the industry moves toward massive generalist models, FASHN VTON v1.5 proves a focused alternative. This is a production-grade virtual try-on model you can train for $5–10k, own, study, and extend. Built for researchers, developers, and fashion tech teams who want more than black-box APIs. More info in the comments.

🚨 FASHN VTON v1.5 is now open source. Released in early 2025 and still widely used, this virtual try-on model generates photorealistic images from a person image and a garment image. Key points: • Pixel-space RGB generation, no VAE • Maskless inference, no person segmentation needed • 972M parameters, ~5s on H100, runs on consumer GPUs • Apache 2.0 licensed, first commercially usable open-source VTON Why open source? While the industry moves toward massive generalist models, FASHN VTON v1.5 proves a focused alternative. This is a production-grade virtual try-on model you can train for $5–10k, own, study, and extend. Built for researchers, developers, and fashion tech teams who want more than black-box APIs. More info in the comments.

FASHN AI

19,211 views • 8 months ago

Introducing VisionPsy-Nano: state-of-the-art vision-language models at 460M parameters, small enough to run on the phone in your pocket. VisionPsy-Nano-460M leads every other ~0.5B vision-language model tested & compares favourably on 16 of 17 benchmarks, tops all four capability categories, and outperforms models up to 2.3x larger. Two variants, one tuned for maximum accuracy and one for real-time responsiveness. Open weights, Apache 2.0, and no cloud in the loop.

Introducing VisionPsy-Nano: state-of-the-art vision-language models at 460M parameters, small enough to run on the phone in your pocket. VisionPsy-Nano-460M leads every other ~0.5B vision-language model tested & compares favourably on 16 of 17 benchmarks, tops all four capability categories, and outperforms models up to 2.3x larger. Two variants, one tuned for maximum accuracy and one for real-time responsiveness. Open weights, Apache 2.0, and no cloud in the loop.

QVAC

15,330,405 views • 2 months ago

We don’t generate videos. 🎬 We generate worlds from videos. 🌍 Introducing InSpatio-World — the world's first open-source real-time 4D world model‼️ Your input: a video clip Our output: a dynamic, navigable, persistent world 🕹️ explore freely across viewpoints ⏪ control time forward and backward 🔓 open-source and ready to build on :) Live demo: 🔗 Code & weights: 🔗 Project page: 🔗

We don’t generate videos. 🎬 We generate worlds from videos. 🌍 Introducing InSpatio-World — the world's first open-source real-time 4D world model‼️ Your input: a video clip Our output: a dynamic, navigable, persistent world 🕹️ explore freely across viewpoints ⏪ control time forward and backward 🔓 open-source and ready to build on :) Live demo: 🔗 Code & weights: 🔗 Project page: 🔗

InSpatio｜影溯

110,213 views • 6 months ago

The robot’s “eyes” just received a big upgrade. LingBot-Depth 2.0, a depth-completion model with half the depth error just dropped. 12/16 benchmarks topped. Glass, mirrors, and transparent objects are so easy for us humans, but so hard for robots, because they do not behave like ordinary surfaces in a camera pipeline. A robot that misunderstands a balcony window or a table edge, will have a completely false planning inside a false world. Huge implecation. LingBot-Depth 2.0 takes an RGB image plus a broken depth map from a sensor and then outputs a cleaner depth map and a usable 3D point cloud. Numbers on LingBot-Depth 2.0 - Excels on glass, mirrors & transparent objects — where traditional depth cameras fail - Training data: 3M → 150M (50x scale-up) - 12 out of 16 first-place rankings on depth completion benchmarks - RMSE cut in half: 0.132 → 0.062 on the hardest indoor scenes LingBot-Vision trained on boundaries, because object edges carry the geometry robots need. No human boundary labels are used, which makes this approach easier to scale. The open-sourced LingBot-Vision is the general vision backbone, and LingBot-Depth 2.0 is the depth model built on it.

The robot’s “eyes” just received a big upgrade. LingBot-Depth 2.0, a depth-completion model with half the depth error just dropped. 12/16 benchmarks topped. Glass, mirrors, and transparent objects are so easy for us humans, but so hard for robots, because they do not behave like ordinary surfaces in a camera pipeline. A robot that misunderstands a balcony window or a table edge, will have a completely false planning inside a false world. Huge implecation. LingBot-Depth 2.0 takes an RGB image plus a broken depth map from a sensor and then outputs a cleaner depth map and a usable 3D point cloud. Numbers on LingBot-Depth 2.0 - Excels on glass, mirrors & transparent objects — where traditional depth cameras fail - Training data: 3M → 150M (50x scale-up) - 12 out of 16 first-place rankings on depth completion benchmarks - RMSE cut in half: 0.132 → 0.062 on the hardest indoor scenes LingBot-Vision trained on boundaries, because object edges carry the geometry robots need. No human boundary labels are used, which makes this approach easier to scale. The open-sourced LingBot-Vision is the general vision backbone, and LingBot-Depth 2.0 is the depth model built on it.

Rohan Paul

25,312 views • 3 months ago

Google presents Still-Moving Customized Video Generation without Customized Video Data Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.

Google presents Still-Moving Customized Video Generation without Customized Video Data Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.

AK

40,492 views • 2 years ago

Holy sh*t this is f*cking awesome China shipped a 397B model that can use a computer Nex-N2.5 Pro, open weights, Apache 2.0 Shanghai Chuangzhi Institute, out yesterday > 397B multimodal, vision + computer use > OSWorld-2: 56.4 vs Qwen3.8-Max at 46.7 > beat Pokémon Platinum to first gym badge > 468 actions, no human intervention most computer use agents fail after 5 clicks this one read dialogue, managed a party, fought battles switched members when HP got low, healed at the Pokémon Center on its own it also does Blender, CAD, vision-to-code watches your screen and builds a skill file from what you do Free on OpenRouter right now try it

Holy sht this is fcking awesome China shipped a 397B model that can use a computer Nex-N2.5 Pro, open weights, Apache 2.0 Shanghai Chuangzhi Institute, out yesterday > 397B multimodal, vision + computer use > OSWorld-2: 56.4 vs Qwen3.8-Max at 46.7 > beat Pokémon Platinum to first gym badge > 468 actions, no human intervention most computer use agents fail after 5 clicks this one read dialogue, managed a party, fought battles switched members when HP got low, healed at the Pokémon Center on its own it also does Blender, CAD, vision-to-code watches your screen and builds a skill file from what you do Free on OpenRouter right now try it

Rivon

17,079 views • 1 month ago

Molmo by Ai2 - Open source SoTA Multimodal (Vision) Language model, beating Claude 3.5 Sonnet, GPT4V and comparable to GPT4o 🔥 They release four model checkpoints: 1. MolmoE-1B, a mixture of experts model with 1B (active) 7B (total) 2. Molmo-7B-O, most open 7B model 3. Molmo-7B-D, demo model 4. Molmo-72B, best model System Architecture > Input: Multi-scale, multi-crop images generated from the original image. > Vision Encoder: OpenAI's ViT-L/14 336px CLIP model, a powerful ViT, encodes images into vision tokens. > Connector: MLP projects tokens to LLM input space, followed by pooling for dimensionality reduction. > LLM: Decoder-only Transformer, various options (OLMo, OLMoE, Qwen2, Mistral, Gemma2, Phi) with diverse scales and openness. Model Variants > Vision Encoder: Consistent ViT-L/14 CLIP model across variants. > LLM: OLMo-7B-1024, OLMoE-1B-7B-0924, Qwen2 (7B, 72B), Mistral 7B, Gemma2 9B, Phi 3 Medium, offering different capacities and openness levels. Training Strategy > Stage 1: Multimodal pre-training for caption generation with new captioning data. > Stage 2: Supervised fine-tuning on a dataset mixture, updating all parameters. > No RLHF involved, Learning rates adjusted based on component types and pre-training status. > All the weights are available on Hugging Face Hub 🤗 > Compatible with Transformers (Remote Code) Kudos Ai2 for such a brilliant and open work! 🐐 Video credits: Allen AI YT Channel

Molmo by Ai2 - Open source SoTA Multimodal (Vision) Language model, beating Claude 3.5 Sonnet, GPT4V and comparable to GPT4o 🔥 They release four model checkpoints: 1. MolmoE-1B, a mixture of experts model with 1B (active) 7B (total) 2. Molmo-7B-O, most open 7B model 3. Molmo-7B-D, demo model 4. Molmo-72B, best model System Architecture > Input: Multi-scale, multi-crop images generated from the original image. > Vision Encoder: OpenAI's ViT-L/14 336px CLIP model, a powerful ViT, encodes images into vision tokens. > Connector: MLP projects tokens to LLM input space, followed by pooling for dimensionality reduction. > LLM: Decoder-only Transformer, various options (OLMo, OLMoE, Qwen2, Mistral, Gemma2, Phi) with diverse scales and openness. Model Variants > Vision Encoder: Consistent ViT-L/14 CLIP model across variants. > LLM: OLMo-7B-1024, OLMoE-1B-7B-0924, Qwen2 (7B, 72B), Mistral 7B, Gemma2 9B, Phi 3 Medium, offering different capacities and openness levels. Training Strategy > Stage 1: Multimodal pre-training for caption generation with new captioning data. > Stage 2: Supervised fine-tuning on a dataset mixture, updating all parameters. > No RLHF involved, Learning rates adjusted based on component types and pre-training status. > All the weights are available on Hugging Face Hub 🤗 > Compatible with Transformers (Remote Code) Kudos Ai2 for such a brilliant and open work! 🐐 Video credits: Allen AI YT Channel

Vaibhav (VB) Srivastav

80,474 views • 2 years ago

Meet MapAnything – a transformer that directly regresses factored metric 3D scene geometry (from images, calibration, poses, or depth) in an end-to-end way. No pipelines, no extra stages. Just 3D geometry & cameras, straight from any type of input, delivering new state-of-the-art results 🚀 One universal model enables SoTA for: 🔥 Mono Depth Estimation 🔥 Multi-View SfM 🔥 Multi-View Stereo 🔥 Depth Completion 🔥 Registration … and many more possibilities! – plus everything is metric 🎯 We release code for data processing, training, benchmarking & ablations – everything Apache 2.0! Details & Links 👇

Meet MapAnything – a transformer that directly regresses factored metric 3D scene geometry (from images, calibration, poses, or depth) in an end-to-end way. No pipelines, no extra stages. Just 3D geometry & cameras, straight from any type of input, delivering new state-of-the-art results 🚀 One universal model enables SoTA for: 🔥 Mono Depth Estimation 🔥 Multi-View SfM 🔥 Multi-View Stereo 🔥 Depth Completion 🔥 Registration … and many more possibilities! – plus everything is metric 🎯 We release code for data processing, training, benchmarking & ablations – everything Apache 2.0! Details & Links 👇

Nikhil Keetha

123,641 views • 1 year ago

Introducing VL-JEPA: Vision-Language Joint Embedding Predictive Architecture for streaming, live action recognition, retrieval, VQA, and classification tasks with better performance and higher efficiency than large VLMs. • VL-JEPA is the first non-generative model that can perform general-domain vision-language tasks in real-time, built on a joint embedding predictive architecture. • We demonstrate in controlled experiments that VL-JEPA, trained with latent space embedding prediction, outperforms VLMs that rely on data space token prediction. • We show that VL-JEPA delivers significant efficiency gains over VLMs for online video streaming applications, thanks to its non-autoregressive design and native support for selective decoding. • We highlight that our VL-JEPA model, with an unified model architecture, can effectively handle a wide range of classification, retrieval, and VQA tasks at the same time. by Delong Chen (陈德龙) Mustafa Shukor Théo Moutakanni Willy Jade Lei Yu Tejaswi Kasarla Allen Bolourchi Yann LeCun Pascale Fung

Introducing VL-JEPA: Vision-Language Joint Embedding Predictive Architecture for streaming, live action recognition, retrieval, VQA, and classification tasks with better performance and higher efficiency than large VLMs. • VL-JEPA is the first non-generative model that can perform general-domain vision-language tasks in real-time, built on a joint embedding predictive architecture. • We demonstrate in controlled experiments that VL-JEPA, trained with latent space embedding prediction, outperforms VLMs that rely on data space token prediction. • We show that VL-JEPA delivers significant efficiency gains over VLMs for online video streaming applications, thanks to its non-autoregressive design and native support for selective decoding. • We highlight that our VL-JEPA model, with an unified model architecture, can effectively handle a wide range of classification, retrieval, and VQA tasks at the same time. by Delong Chen (陈德龙) Mustafa Shukor Théo Moutakanni Willy Jade Lei Yu Tejaswi Kasarla Allen Bolourchi Yann LeCun Pascale Fung

Pascale Fung

90,522 views • 10 months ago

🚨 BREAKING: A research lab just released a 15B model that generates multilingual talking human videos with synced audio, beats every competitor in human evaluation, and runs in 38 seconds on one GPU. It's called daVinci-MagiHuman. The key insight is that every other model in this category stacks cross-attention, multi-stream pipelines, and separate conditioning branches to handle video and audio together. This one throws all of that out and uses a single unified self-attention stream across all modalities. Super-resolution happens in latent space rather than pixel space so there's no extra VAE decode-encode round trip. The turbo VAE decoder cuts decoding overhead even further. The distilled version runs in 8 steps with no CFG at all. Visual quality, text alignment, and word error rate all beat Ovi 1.1 and LTX 2.3 on the benchmark table. 100% Opensource. Apache 2.0. Repo and research paper links are in the comments.

🚨 BREAKING: A research lab just released a 15B model that generates multilingual talking human videos with synced audio, beats every competitor in human evaluation, and runs in 38 seconds on one GPU. It's called daVinci-MagiHuman. The key insight is that every other model in this category stacks cross-attention, multi-stream pipelines, and separate conditioning branches to handle video and audio together. This one throws all of that out and uses a single unified self-attention stream across all modalities. Super-resolution happens in latent space rather than pixel space so there's no extra VAE decode-encode round trip. The turbo VAE decoder cuts decoding overhead even further. The distilled version runs in 8 steps with no CFG at all. Visual quality, text alignment, and word error rate all beat Ovi 1.1 and LTX 2.3 on the benchmark table. 100% Opensource. Apache 2.0. Repo and research paper links are in the comments.

Ihtesham Ali

45,621 views • 6 months ago

🚀 Meta FAIR is releasing several new research artifacts on our road to advanced machine intelligence (AMI). These latest advancements are transforming our understanding of perception. 1️⃣ Meta Perception Encoder: A large-scale vision encoder that excels across several image & video tasks. 2️⃣ Meta Perception Language Model: A fully open & reproducible vision-language model designed to tackle visual recognition tasks. 3️⃣ Meta Locate 3D: An end-to-end model for accurate object localization in 3D environments. 4️⃣ Releasing model weights for our 8B-parameter Dynamic Byte Latent Transformer, an alternative to traditional tokenization methods with the potential to redefine the standards for language model efficiency and reliability. 5️⃣Collaborative Reasoner: A framework for evaluating & improving collaborative reasoning skills in language models. Download the code, datasets, and research papers and learn more about how these artifacts are paving the way for more efficient and accurate AI systems.➡️

🚀 Meta FAIR is releasing several new research artifacts on our road to advanced machine intelligence (AMI). These latest advancements are transforming our understanding of perception. 1️⃣ Meta Perception Encoder: A large-scale vision encoder that excels across several image & video tasks. 2️⃣ Meta Perception Language Model: A fully open & reproducible vision-language model designed to tackle visual recognition tasks. 3️⃣ Meta Locate 3D: An end-to-end model for accurate object localization in 3D environments. 4️⃣ Releasing model weights for our 8B-parameter Dynamic Byte Latent Transformer, an alternative to traditional tokenization methods with the potential to redefine the standards for language model efficiency and reliability. 5️⃣Collaborative Reasoner: A framework for evaluating & improving collaborative reasoning skills in language models. Download the code, datasets, and research papers and learn more about how these artifacts are paving the way for more efficient and accurate AI systems.➡️

AI at Meta

163,584 views • 1 year ago

Hy4 preview powers dynamic builds for a 770B open model. 770B total parameters, 49B active per token, native 1M context, Apache 2.0, direct vLLM and SGLang support, plus official FP8 weights. In this demo, Hy4 creates a colorful, playable 2D space shooter game directly from a prompt, featuring active controls, enemies, visual particle effects, plus score tracking. It is a preview. Test it, measure tokens/s, compare it with BF16 or FP8, and see what it builds. Try it in WorkBuddy: Tencent Hy WorkBuddy #Hy4Preview #OpenSourceAI #GameDev #AI

Hy4 preview powers dynamic builds for a 770B open model. 770B total parameters, 49B active per token, native 1M context, Apache 2.0, direct vLLM and SGLang support, plus official FP8 weights. In this demo, Hy4 creates a colorful, playable 2D space shooter game directly from a prompt, featuring active controls, enemies, visual particle effects, plus score tracking. It is a preview. Test it, measure tokens/s, compare it with BF16 or FP8, and see what it builds. Try it in WorkBuddy: Tencent Hy WorkBuddy #Hy4Preview #OpenSourceAI #GameDev #AI

Tyler Wayne

78,042 views • 1 month ago

InternLM-XComposer-2.5 A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output We present InternLM-XComposer-2.5 (IXC-2.5), a versatile large-vision language model that supports long-contextual input and output. IXC-2.5 excels in various text-image comprehension and composition applications, achieving GPT-4V level capabilities with merely 7B LLM backend. Trained with 24K interleaved image-text contexts, it can seamlessly extend to 96K long contexts via RoPE extrapolation. This long-context capability allows IXC-2.5 to excel in tasks requiring extensive input and output contexts. Compared to its previous 2.0 version, InternLM-XComposer-2.5 features three major upgrades in vision-language comprehension: (1) Ultra-High Resolution Understanding, (2) Fine-Grained Video Understanding, and (3) Multi-Turn Multi-Image Dialogue. In addition to comprehension, IXC-2.5 extends to two compelling applications using extra LoRA parameters for text-image composition: (1) Crafting Webpages and (2) Composing High-Quality Text-Image Articles. IXC-2.5 has been evaluated on 28 benchmarks, outperforming existing open-source state-of-the-art models on 16 benchmarks. It also surpasses or competes closely with GPT-4V and Gemini Pro on 16 key tasks.

InternLM-XComposer-2.5 A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output We present InternLM-XComposer-2.5 (IXC-2.5), a versatile large-vision language model that supports long-contextual input and output. IXC-2.5 excels in various text-image comprehension and composition applications, achieving GPT-4V level capabilities with merely 7B LLM backend. Trained with 24K interleaved image-text contexts, it can seamlessly extend to 96K long contexts via RoPE extrapolation. This long-context capability allows IXC-2.5 to excel in tasks requiring extensive input and output contexts. Compared to its previous 2.0 version, InternLM-XComposer-2.5 features three major upgrades in vision-language comprehension: (1) Ultra-High Resolution Understanding, (2) Fine-Grained Video Understanding, and (3) Multi-Turn Multi-Image Dialogue. In addition to comprehension, IXC-2.5 extends to two compelling applications using extra LoRA parameters for text-image composition: (1) Crafting Webpages and (2) Composing High-Quality Text-Image Articles. IXC-2.5 has been evaluated on 28 benchmarks, outperforming existing open-source state-of-the-art models on 16 benchmarks. It also surpasses or competes closely with GPT-4V and Gemini Pro on 16 key tasks.

AK

66,476 views • 2 years ago

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress *continuous-time* trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress continuous-time trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Davide Scaramuzza

12,692 views • 2 years ago

how to evolve generative art 🦠✨👇 over the past weeks I built this toy 'proof-of-concept' example to explore and better understand how open-form generative art works, and discover some of the new archetypes it enables. In open-form genart each mint at the root of a project can define its own lineage: properties, traits, and characteristics that carry over to new editions evolved from it. For example, in this demo, the first parent defines the color palette that's used for the pixel pattern — these colors are then inherited by subsequent evolutions (with a random chance to alter a color or introduce a new one). But that's just scratching the tip of the iceberg — the REALLY COOL thing is that the root mint also specifies the rules that define how the pixel pattern morphs and changes over the course of its lineage! Meaning that each branch in this collection is its own unconstrained system, that evolves, and is explored by minting and evolving new editions. in this setting collectors play a huge role to explore the parametric space of the project ✨ on fxhash the UI will be much cleaner and more intuitive 🫧

how to evolve generative art 🦠✨👇 over the past weeks I built this toy 'proof-of-concept' example to explore and better understand how open-form generative art works, and discover some of the new archetypes it enables. In open-form genart each mint at the root of a project can define its own lineage: properties, traits, and characteristics that carry over to new editions evolved from it. For example, in this demo, the first parent defines the color palette that's used for the pixel pattern — these colors are then inherited by subsequent evolutions (with a random chance to alter a color or introduce a new one). But that's just scratching the tip of the iceberg — the REALLY COOL thing is that the root mint also specifies the rules that define how the pixel pattern morphs and changes over the course of its lineage! Meaning that each branch in this collection is its own unconstrained system, that evolves, and is explored by minting and evolving new editions. in this setting collectors play a huge role to explore the parametric space of the project ✨ on fxhash the UI will be much cleaner and more intuitive 🫧

Ahmad Moussa || Gorilla Sun

13,100 views • 1 year ago

In collaboration with Intel, our Depth Fusion showcases the power of our LDM3D diffusion model in generating 360° views from text prompts provided by the user. The LDM3D diffusion model generates a 2D RGB image and its corresponding relative depth map providing a complete RGBD representation corresponding to the text prompt. The LDM 3D model is a specialized version of the stable diffusion V 1.4 model that has been modified to fit both image and depth map data.The model was then fine tuned on a subset of the Laion400M data set - large scale image caption data set. The depth maps used to fine tune our model were generated by the DPTBeiT large 512 depth estimation model that provides highly accurate relative depth estimates for each pixel. We take the generated 2D RGB image and depth map and use them to compute a 360° projection using touchdesigner. Touchdesigner is a versatile platform that allows for the creation of immersive and interactive multimedia experiences. Our application harnesses the power of touchdesigner to bring the generated 360° views to life, providing users with a unique and engaging way to experience their text prompts, whether it’s a description of a tranquil forest, a noisy cityscape or a futuristic sci fi world. Our depth fusion can bring these concepts to life in a vivid and immersive detail. - Scottie Fox, VP Engineering Blockade Labs ScottieFox #AI #VR #3D #gamedev #stablediffusion

In collaboration with Intel, our Depth Fusion showcases the power of our LDM3D diffusion model in generating 360° views from text prompts provided by the user. The LDM3D diffusion model generates a 2D RGB image and its corresponding relative depth map providing a complete RGBD representation corresponding to the text prompt. The LDM 3D model is a specialized version of the stable diffusion V 1.4 model that has been modified to fit both image and depth map data.The model was then fine tuned on a subset of the Laion400M data set - large scale image caption data set. The depth maps used to fine tune our model were generated by the DPTBeiT large 512 depth estimation model that provides highly accurate relative depth estimates for each pixel. We take the generated 2D RGB image and depth map and use them to compute a 360° projection using touchdesigner. Touchdesigner is a versatile platform that allows for the creation of immersive and interactive multimedia experiences. Our application harnesses the power of touchdesigner to bring the generated 360° views to life, providing users with a unique and engaging way to experience their text prompts, whether it’s a description of a tranquil forest, a noisy cityscape or a futuristic sci fi world. Our depth fusion can bring these concepts to life in a vivid and immersive detail. - Scottie Fox, VP Engineering Blockade Labs ScottieFox #AI #VR #3D #gamedev #stablediffusion

Blockade Labs

11,458 views • 3 years ago