Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Meet Iris-3B: a pixel-space T2I model and general vision learner that generates every pixel directly. No VAE. Pre-trained from scratch, 3B, open weights. We also explore its generative prior on detail-critical vision tasks: depth estimation and image restoration. Fully open: weights, code and demo (Apache 2.0)

Speridlabs

1,026 subscribers

80,397 Aufrufe • vor 3 Tagen •via X (Twitter)

Bildung Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

34 Kommentare

Profilbild von Speridlabs

Speridlabsvor 3 Tagen

Weights: Code: Try it:

Profilbild von Speridlabs

Speridlabsvor 3 Tagen

We're hiring

Profilbild von Borja Perez

Borja Perezvor 3 Tagen

@barckcode nunca te pido nada 🥹 y encima son de españita

Profilbild von Alexey Fateev

Alexey Fateevvor 3 Tagen

@ComfyUI workflow guys? sounds incredible!

Profilbild von Armin Catovic

Armin Catovicvor 3 Tagen

Can it work on low res images, like 64x64 pixels? This is the limitation of todays imagegen models - even for pixel art they produce high res images, when scaled down look awful.

Profilbild von alejandro

alejandrovor 3 Tagen

🚀🚀🚀

Profilbild von Jorge Dobón

Jorge Dobónvor 3 Tagen

Go go go @speridlabs 🙌🏼

Profilbild von Julien Blanchon 🇺🇦

Julien Blanchon 🇺🇦vor 3 Tagen

@AntonObukhov1 Interesting as a prior

Profilbild von Buswe

Buswevor 3 Tagen

Dropping the VAE is what makes depth and restoration work here: no latent decode step to smooth away the thin, high frequency detail.

Profilbild von Alice The Ai Expert

Alice The Ai Expertvor 2 Tagen

Incredible work 3B pixels directly with no VAE and fully open weights is exactly the kind of open research the community needs!

Profilbild von Javier Álvarez

Javier Álvarezvor 3 Tagen

no VAE here... top my friend @hanqiulicai 🚀 🚀

Profilbild von FRareDom

FRareDomvor 3 Tagen

Open weights is incredible! THANK YOU!!

Profilbild von ander

andervor 2 Tagen

Congrats!

Profilbild von Amelia | AI & Tech

Amelia | AI & Techvor 2 Tagen

Generating pixels directly without a VAE is boldOpen weights on a no-VAE generator is the real flex.

Profilbild von A.W.E.S.O.M.-O 4000

A.W.E.S.O.M.-O 4000vor 3 Tagen

ngl open weights at 3b makes this easy for people to actually try fr

Profilbild von Salar 🇪🇸🇮🇷

Salar 🇪🇸🇮🇷vor 3 Tagen

Neat

Profilbild von RAZA | AI EXPLORER

RAZA | AI EXPLORERvor 2 Tagen

Impressive work! Open-weight pixel-space models open up exciting possibilities for image generation and vision research.

Profilbild von ZenithAi

ZenithAivor 2 Tagen

Open weights and pixel level generation make this research especially promising

Profilbild von Pongphat

Pongphatvor 2 Tagen

Interesting!

Profilbild von Aaliya

Aaliyavor 2 Tagen

Using it to fix images sounds really useful too.

Profilbild von Elise Tech Ai

Elise Tech Aivor 2 Tagen

This is fantastic Iris 3B directly in pixel space with no VAE is a bold and impressive approach love that its fully open

Profilbild von 드럼통청약당첨

드럼통청약당첨vor 2 Tagen

Pixel Space with optimized architecture is future! Thank you for your work!

Profilbild von Adria Blancafort

Adria Blancafortvor 3 Tagen

Nice!

Profilbild von AI路子野

AI路子野vor 2 Tagen

Still using a 4B VL addition. Not fully portable

Profilbild von Vantix AI Agency

Vantix AI Agencyvor 2 Tagen

No VAE and direct pixel generation at 3B thats a bold architecture bet love that its fully open with weights and code

Profilbild von Jack

Jackvor 3 Tagen

pixel space models deserve more love

Profilbild von Anherutowa Calvo

Anherutowa Calvovor 3 Tagen

Lets go!!

Profilbild von Ella Tech & Tool

Ella Tech & Toolvor 2 Tagen

Love this. Pixel-space generation with open weights will push T2I and vision research forward fast.

Profilbild von Sani Ai Tech

Sani Ai Techvor 2 Tagen

Impressive to see pixel-native generation paired with open weights and broader vision capabilities

Profilbild von Elara AI

Elara AIvor 1 Tag

Iris-3B pixel-space T2I model generates pixels directly without VAE openly released now

Profilbild von Aria Tech

Aria Techvor 2 Tagen

No VAE pixel-space T2I is bold Iris-3B looks super promising

Profilbild von Pablo

Pablovor 3 Tagen

@bot

Profilbild von Zara Tech & Tools

Zara Tech & Toolsvor 2 Tagen

Exciting open release pushing direct pixel generation

Profilbild von Martii

Martiivor 3 Tagen

🔥🔥🔥🔥🔥

Ähnliche Videos

SenseNova-U1.5-8B-MoT is HERE, it's a big deal! 🔥 open weights matching Nano Banana 2 on benchs 🍌 no VAE, no text encoder, no DiT. a mixture of transformers: text and image tokens use different weights, attend to each other, denoise in pixel space ▶️

SenseNova-U1.5-8B-MoT is HERE, it's a big deal! 🔥 open weights matching Nano Banana 2 on benchs 🍌 no VAE, no text encoder, no DiT. a mixture of transformers: text and image tokens use different weights, attend to each other, denoise in pixel space ▶️

Hugging Apps

42,720 Aufrufe • vor 1 Monat

Today, every Nomic-Embed-Text embedding becomes multimodal. Introducing Nomic-Embed-Vision: - a high quality, unified embedding space for image, text, and multimodal tasks - outperforms both OpenAI CLIP and text-embedding-3-small - open weights and code to enable indie hacking, research, and experimentation - released in collaboration with MongoDB, LlamaIndex 🦙, , Hugging Face, Amazon Web Services, DigitalOcean, Lambda

Today, every Nomic-Embed-Text embedding becomes multimodal. Introducing Nomic-Embed-Vision: - a high quality, unified embedding space for image, text, and multimodal tasks - outperforms both OpenAI CLIP and text-embedding-3-small - open weights and code to enable indie hacking, research, and experimentation - released in collaboration with MongoDB, LlamaIndex 🦙, , Hugging Face, Amazon Web Services, DigitalOcean, Lambda

CalCo

103,287 Aufrufe • vor 2 Jahren

🚨 FASHN VTON v1.5 is now open source. Released in early 2025 and still widely used, this virtual try-on model generates photorealistic images from a person image and a garment image. Key points: • Pixel-space RGB generation, no VAE • Maskless inference, no person segmentation needed • 972M parameters, ~5s on H100, runs on consumer GPUs • Apache 2.0 licensed, first commercially usable open-source VTON Why open source? While the industry moves toward massive generalist models, FASHN VTON v1.5 proves a focused alternative. This is a production-grade virtual try-on model you can train for $5–10k, own, study, and extend. Built for researchers, developers, and fashion tech teams who want more than black-box APIs. More info in the comments.

🚨 FASHN VTON v1.5 is now open source. Released in early 2025 and still widely used, this virtual try-on model generates photorealistic images from a person image and a garment image. Key points: • Pixel-space RGB generation, no VAE • Maskless inference, no person segmentation needed • 972M parameters, ~5s on H100, runs on consumer GPUs • Apache 2.0 licensed, first commercially usable open-source VTON Why open source? While the industry moves toward massive generalist models, FASHN VTON v1.5 proves a focused alternative. This is a production-grade virtual try-on model you can train for $5–10k, own, study, and extend. Built for researchers, developers, and fashion tech teams who want more than black-box APIs. More info in the comments.

FASHN AI

19,211 Aufrufe • vor 8 Monaten

Introducing VisionPsy-Nano: state-of-the-art vision-language models at 460M parameters, small enough to run on the phone in your pocket. VisionPsy-Nano-460M leads every other ~0.5B vision-language model tested & compares favourably on 16 of 17 benchmarks, tops all four capability categories, and outperforms models up to 2.3x larger. Two variants, one tuned for maximum accuracy and one for real-time responsiveness. Open weights, Apache 2.0, and no cloud in the loop.

Introducing VisionPsy-Nano: state-of-the-art vision-language models at 460M parameters, small enough to run on the phone in your pocket. VisionPsy-Nano-460M leads every other ~0.5B vision-language model tested & compares favourably on 16 of 17 benchmarks, tops all four capability categories, and outperforms models up to 2.3x larger. Two variants, one tuned for maximum accuracy and one for real-time responsiveness. Open weights, Apache 2.0, and no cloud in the loop.

QVAC

15,330,405 Aufrufe • vor 2 Monaten

We don’t generate videos. 🎬 We generate worlds from videos. 🌍 Introducing InSpatio-World — the world's first open-source real-time 4D world model‼️ Your input: a video clip Our output: a dynamic, navigable, persistent world 🕹️ explore freely across viewpoints ⏪ control time forward and backward 🔓 open-source and ready to build on :) Live demo: 🔗 Code & weights: 🔗 Project page: 🔗

We don’t generate videos. 🎬 We generate worlds from videos. 🌍 Introducing InSpatio-World — the world's first open-source real-time 4D world model‼️ Your input: a video clip Our output: a dynamic, navigable, persistent world 🕹️ explore freely across viewpoints ⏪ control time forward and backward 🔓 open-source and ready to build on :) Live demo: 🔗 Code & weights: 🔗 Project page: 🔗

InSpatio｜影溯

110,213 Aufrufe • vor 6 Monaten

The robot’s “eyes” just received a big upgrade. LingBot-Depth 2.0, a depth-completion model with half the depth error just dropped. 12/16 benchmarks topped. Glass, mirrors, and transparent objects are so easy for us humans, but so hard for robots, because they do not behave like ordinary surfaces in a camera pipeline. A robot that misunderstands a balcony window or a table edge, will have a completely false planning inside a false world. Huge implecation. LingBot-Depth 2.0 takes an RGB image plus a broken depth map from a sensor and then outputs a cleaner depth map and a usable 3D point cloud. Numbers on LingBot-Depth 2.0 - Excels on glass, mirrors & transparent objects — where traditional depth cameras fail - Training data: 3M → 150M (50x scale-up) - 12 out of 16 first-place rankings on depth completion benchmarks - RMSE cut in half: 0.132 → 0.062 on the hardest indoor scenes LingBot-Vision trained on boundaries, because object edges carry the geometry robots need. No human boundary labels are used, which makes this approach easier to scale. The open-sourced LingBot-Vision is the general vision backbone, and LingBot-Depth 2.0 is the depth model built on it.

The robot’s “eyes” just received a big upgrade. LingBot-Depth 2.0, a depth-completion model with half the depth error just dropped. 12/16 benchmarks topped. Glass, mirrors, and transparent objects are so easy for us humans, but so hard for robots, because they do not behave like ordinary surfaces in a camera pipeline. A robot that misunderstands a balcony window or a table edge, will have a completely false planning inside a false world. Huge implecation. LingBot-Depth 2.0 takes an RGB image plus a broken depth map from a sensor and then outputs a cleaner depth map and a usable 3D point cloud. Numbers on LingBot-Depth 2.0 - Excels on glass, mirrors & transparent objects — where traditional depth cameras fail - Training data: 3M → 150M (50x scale-up) - 12 out of 16 first-place rankings on depth completion benchmarks - RMSE cut in half: 0.132 → 0.062 on the hardest indoor scenes LingBot-Vision trained on boundaries, because object edges carry the geometry robots need. No human boundary labels are used, which makes this approach easier to scale. The open-sourced LingBot-Vision is the general vision backbone, and LingBot-Depth 2.0 is the depth model built on it.

Rohan Paul

25,312 Aufrufe • vor 3 Monaten

Google presents Still-Moving Customized Video Generation without Customized Video Data Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.

Google presents Still-Moving Customized Video Generation without Customized Video Data Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.

AK

40,492 Aufrufe • vor 2 Jahren

Holy sh*t this is f*cking awesome China shipped a 397B model that can use a computer Nex-N2.5 Pro, open weights, Apache 2.0 Shanghai Chuangzhi Institute, out yesterday > 397B multimodal, vision + computer use > OSWorld-2: 56.4 vs Qwen3.8-Max at 46.7 > beat Pokémon Platinum to first gym badge > 468 actions, no human intervention most computer use agents fail after 5 clicks this one read dialogue, managed a party, fought battles switched members when HP got low, healed at the Pokémon Center on its own it also does Blender, CAD, vision-to-code watches your screen and builds a skill file from what you do Free on OpenRouter right now try it

Holy sht this is fcking awesome China shipped a 397B model that can use a computer Nex-N2.5 Pro, open weights, Apache 2.0 Shanghai Chuangzhi Institute, out yesterday > 397B multimodal, vision + computer use > OSWorld-2: 56.4 vs Qwen3.8-Max at 46.7 > beat Pokémon Platinum to first gym badge > 468 actions, no human intervention most computer use agents fail after 5 clicks this one read dialogue, managed a party, fought battles switched members when HP got low, healed at the Pokémon Center on its own it also does Blender, CAD, vision-to-code watches your screen and builds a skill file from what you do Free on OpenRouter right now try it

Rivon

17,079 Aufrufe • vor 1 Monat

Molmo by Ai2 - Open source SoTA Multimodal (Vision) Language model, beating Claude 3.5 Sonnet, GPT4V and comparable to GPT4o 🔥 They release four model checkpoints: 1. MolmoE-1B, a mixture of experts model with 1B (active) 7B (total) 2. Molmo-7B-O, most open 7B model 3. Molmo-7B-D, demo model 4. Molmo-72B, best model System Architecture > Input: Multi-scale, multi-crop images generated from the original image. > Vision Encoder: OpenAI's ViT-L/14 336px CLIP model, a powerful ViT, encodes images into vision tokens. > Connector: MLP projects tokens to LLM input space, followed by pooling for dimensionality reduction. > LLM: Decoder-only Transformer, various options (OLMo, OLMoE, Qwen2, Mistral, Gemma2, Phi) with diverse scales and openness. Model Variants > Vision Encoder: Consistent ViT-L/14 CLIP model across variants. > LLM: OLMo-7B-1024, OLMoE-1B-7B-0924, Qwen2 (7B, 72B), Mistral 7B, Gemma2 9B, Phi 3 Medium, offering different capacities and openness levels. Training Strategy > Stage 1: Multimodal pre-training for caption generation with new captioning data. > Stage 2: Supervised fine-tuning on a dataset mixture, updating all parameters. > No RLHF involved, Learning rates adjusted based on component types and pre-training status. > All the weights are available on Hugging Face Hub 🤗 > Compatible with Transformers (Remote Code) Kudos Ai2 for such a brilliant and open work! 🐐 Video credits: Allen AI YT Channel

Molmo by Ai2 - Open source SoTA Multimodal (Vision) Language model, beating Claude 3.5 Sonnet, GPT4V and comparable to GPT4o 🔥 They release four model checkpoints: 1. MolmoE-1B, a mixture of experts model with 1B (active) 7B (total) 2. Molmo-7B-O, most open 7B model 3. Molmo-7B-D, demo model 4. Molmo-72B, best model System Architecture > Input: Multi-scale, multi-crop images generated from the original image. > Vision Encoder: OpenAI's ViT-L/14 336px CLIP model, a powerful ViT, encodes images into vision tokens. > Connector: MLP projects tokens to LLM input space, followed by pooling for dimensionality reduction. > LLM: Decoder-only Transformer, various options (OLMo, OLMoE, Qwen2, Mistral, Gemma2, Phi) with diverse scales and openness. Model Variants > Vision Encoder: Consistent ViT-L/14 CLIP model across variants. > LLM: OLMo-7B-1024, OLMoE-1B-7B-0924, Qwen2 (7B, 72B), Mistral 7B, Gemma2 9B, Phi 3 Medium, offering different capacities and openness levels. Training Strategy > Stage 1: Multimodal pre-training for caption generation with new captioning data. > Stage 2: Supervised fine-tuning on a dataset mixture, updating all parameters. > No RLHF involved, Learning rates adjusted based on component types and pre-training status. > All the weights are available on Hugging Face Hub 🤗 > Compatible with Transformers (Remote Code) Kudos Ai2 for such a brilliant and open work! 🐐 Video credits: Allen AI YT Channel

Vaibhav (VB) Srivastav

80,474 Aufrufe • vor 2 Jahren

Meet MapAnything – a transformer that directly regresses factored metric 3D scene geometry (from images, calibration, poses, or depth) in an end-to-end way. No pipelines, no extra stages. Just 3D geometry & cameras, straight from any type of input, delivering new state-of-the-art results 🚀 One universal model enables SoTA for: 🔥 Mono Depth Estimation 🔥 Multi-View SfM 🔥 Multi-View Stereo 🔥 Depth Completion 🔥 Registration … and many more possibilities! – plus everything is metric 🎯 We release code for data processing, training, benchmarking & ablations – everything Apache 2.0! Details & Links 👇

Meet MapAnything – a transformer that directly regresses factored metric 3D scene geometry (from images, calibration, poses, or depth) in an end-to-end way. No pipelines, no extra stages. Just 3D geometry & cameras, straight from any type of input, delivering new state-of-the-art results 🚀 One universal model enables SoTA for: 🔥 Mono Depth Estimation 🔥 Multi-View SfM 🔥 Multi-View Stereo 🔥 Depth Completion 🔥 Registration … and many more possibilities! – plus everything is metric 🎯 We release code for data processing, training, benchmarking & ablations – everything Apache 2.0! Details & Links 👇

Nikhil Keetha

123,641 Aufrufe • vor 1 Jahr

Introducing VL-JEPA: Vision-Language Joint Embedding Predictive Architecture for streaming, live action recognition, retrieval, VQA, and classification tasks with better performance and higher efficiency than large VLMs. • VL-JEPA is the first non-generative model that can perform general-domain vision-language tasks in real-time, built on a joint embedding predictive architecture. • We demonstrate in controlled experiments that VL-JEPA, trained with latent space embedding prediction, outperforms VLMs that rely on data space token prediction. • We show that VL-JEPA delivers significant efficiency gains over VLMs for online video streaming applications, thanks to its non-autoregressive design and native support for selective decoding. • We highlight that our VL-JEPA model, with an unified model architecture, can effectively handle a wide range of classification, retrieval, and VQA tasks at the same time. by Delong Chen (陈德龙) Mustafa Shukor Théo Moutakanni Willy Jade Lei Yu Tejaswi Kasarla Allen Bolourchi Yann LeCun Pascale Fung

Introducing VL-JEPA: Vision-Language Joint Embedding Predictive Architecture for streaming, live action recognition, retrieval, VQA, and classification tasks with better performance and higher efficiency than large VLMs. • VL-JEPA is the first non-generative model that can perform general-domain vision-language tasks in real-time, built on a joint embedding predictive architecture. • We demonstrate in controlled experiments that VL-JEPA, trained with latent space embedding prediction, outperforms VLMs that rely on data space token prediction. • We show that VL-JEPA delivers significant efficiency gains over VLMs for online video streaming applications, thanks to its non-autoregressive design and native support for selective decoding. • We highlight that our VL-JEPA model, with an unified model architecture, can effectively handle a wide range of classification, retrieval, and VQA tasks at the same time. by Delong Chen (陈德龙) Mustafa Shukor Théo Moutakanni Willy Jade Lei Yu Tejaswi Kasarla Allen Bolourchi Yann LeCun Pascale Fung

Pascale Fung

90,522 Aufrufe • vor 10 Monaten

🚨 BREAKING: A research lab just released a 15B model that generates multilingual talking human videos with synced audio, beats every competitor in human evaluation, and runs in 38 seconds on one GPU. It's called daVinci-MagiHuman. The key insight is that every other model in this category stacks cross-attention, multi-stream pipelines, and separate conditioning branches to handle video and audio together. This one throws all of that out and uses a single unified self-attention stream across all modalities. Super-resolution happens in latent space rather than pixel space so there's no extra VAE decode-encode round trip. The turbo VAE decoder cuts decoding overhead even further. The distilled version runs in 8 steps with no CFG at all. Visual quality, text alignment, and word error rate all beat Ovi 1.1 and LTX 2.3 on the benchmark table. 100% Opensource. Apache 2.0. Repo and research paper links are in the comments.

🚨 BREAKING: A research lab just released a 15B model that generates multilingual talking human videos with synced audio, beats every competitor in human evaluation, and runs in 38 seconds on one GPU. It's called daVinci-MagiHuman. The key insight is that every other model in this category stacks cross-attention, multi-stream pipelines, and separate conditioning branches to handle video and audio together. This one throws all of that out and uses a single unified self-attention stream across all modalities. Super-resolution happens in latent space rather than pixel space so there's no extra VAE decode-encode round trip. The turbo VAE decoder cuts decoding overhead even further. The distilled version runs in 8 steps with no CFG at all. Visual quality, text alignment, and word error rate all beat Ovi 1.1 and LTX 2.3 on the benchmark table. 100% Opensource. Apache 2.0. Repo and research paper links are in the comments.

Ihtesham Ali

45,621 Aufrufe • vor 6 Monaten

🚀 Meta FAIR is releasing several new research artifacts on our road to advanced machine intelligence (AMI). These latest advancements are transforming our understanding of perception. 1️⃣ Meta Perception Encoder: A large-scale vision encoder that excels across several image & video tasks. 2️⃣ Meta Perception Language Model: A fully open & reproducible vision-language model designed to tackle visual recognition tasks. 3️⃣ Meta Locate 3D: An end-to-end model for accurate object localization in 3D environments. 4️⃣ Releasing model weights for our 8B-parameter Dynamic Byte Latent Transformer, an alternative to traditional tokenization methods with the potential to redefine the standards for language model efficiency and reliability. 5️⃣Collaborative Reasoner: A framework for evaluating & improving collaborative reasoning skills in language models. Download the code, datasets, and research papers and learn more about how these artifacts are paving the way for more efficient and accurate AI systems.➡️

🚀 Meta FAIR is releasing several new research artifacts on our road to advanced machine intelligence (AMI). These latest advancements are transforming our understanding of perception. 1️⃣ Meta Perception Encoder: A large-scale vision encoder that excels across several image & video tasks. 2️⃣ Meta Perception Language Model: A fully open & reproducible vision-language model designed to tackle visual recognition tasks. 3️⃣ Meta Locate 3D: An end-to-end model for accurate object localization in 3D environments. 4️⃣ Releasing model weights for our 8B-parameter Dynamic Byte Latent Transformer, an alternative to traditional tokenization methods with the potential to redefine the standards for language model efficiency and reliability. 5️⃣Collaborative Reasoner: A framework for evaluating & improving collaborative reasoning skills in language models. Download the code, datasets, and research papers and learn more about how these artifacts are paving the way for more efficient and accurate AI systems.➡️

AI at Meta

163,584 Aufrufe • vor 1 Jahr

Hy4 preview powers dynamic builds for a 770B open model. 770B total parameters, 49B active per token, native 1M context, Apache 2.0, direct vLLM and SGLang support, plus official FP8 weights. In this demo, Hy4 creates a colorful, playable 2D space shooter game directly from a prompt, featuring active controls, enemies, visual particle effects, plus score tracking. It is a preview. Test it, measure tokens/s, compare it with BF16 or FP8, and see what it builds. Try it in WorkBuddy: Tencent Hy WorkBuddy #Hy4Preview #OpenSourceAI #GameDev #AI

Hy4 preview powers dynamic builds for a 770B open model. 770B total parameters, 49B active per token, native 1M context, Apache 2.0, direct vLLM and SGLang support, plus official FP8 weights. In this demo, Hy4 creates a colorful, playable 2D space shooter game directly from a prompt, featuring active controls, enemies, visual particle effects, plus score tracking. It is a preview. Test it, measure tokens/s, compare it with BF16 or FP8, and see what it builds. Try it in WorkBuddy: Tencent Hy WorkBuddy #Hy4Preview #OpenSourceAI #GameDev #AI

Tyler Wayne

78,042 Aufrufe • vor 1 Monat

InternLM-XComposer-2.5 A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output We present InternLM-XComposer-2.5 (IXC-2.5), a versatile large-vision language model that supports long-contextual input and output. IXC-2.5 excels in various text-image comprehension and composition applications, achieving GPT-4V level capabilities with merely 7B LLM backend. Trained with 24K interleaved image-text contexts, it can seamlessly extend to 96K long contexts via RoPE extrapolation. This long-context capability allows IXC-2.5 to excel in tasks requiring extensive input and output contexts. Compared to its previous 2.0 version, InternLM-XComposer-2.5 features three major upgrades in vision-language comprehension: (1) Ultra-High Resolution Understanding, (2) Fine-Grained Video Understanding, and (3) Multi-Turn Multi-Image Dialogue. In addition to comprehension, IXC-2.5 extends to two compelling applications using extra LoRA parameters for text-image composition: (1) Crafting Webpages and (2) Composing High-Quality Text-Image Articles. IXC-2.5 has been evaluated on 28 benchmarks, outperforming existing open-source state-of-the-art models on 16 benchmarks. It also surpasses or competes closely with GPT-4V and Gemini Pro on 16 key tasks.

InternLM-XComposer-2.5 A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output We present InternLM-XComposer-2.5 (IXC-2.5), a versatile large-vision language model that supports long-contextual input and output. IXC-2.5 excels in various text-image comprehension and composition applications, achieving GPT-4V level capabilities with merely 7B LLM backend. Trained with 24K interleaved image-text contexts, it can seamlessly extend to 96K long contexts via RoPE extrapolation. This long-context capability allows IXC-2.5 to excel in tasks requiring extensive input and output contexts. Compared to its previous 2.0 version, InternLM-XComposer-2.5 features three major upgrades in vision-language comprehension: (1) Ultra-High Resolution Understanding, (2) Fine-Grained Video Understanding, and (3) Multi-Turn Multi-Image Dialogue. In addition to comprehension, IXC-2.5 extends to two compelling applications using extra LoRA parameters for text-image composition: (1) Crafting Webpages and (2) Composing High-Quality Text-Image Articles. IXC-2.5 has been evaluated on 28 benchmarks, outperforming existing open-source state-of-the-art models on 16 benchmarks. It also surpasses or competes closely with GPT-4V and Gemini Pro on 16 key tasks.

AK

66,476 Aufrufe • vor 2 Jahren

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress *continuous-time* trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress continuous-time trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Davide Scaramuzza

12,692 Aufrufe • vor 2 Jahren

how to evolve generative art 🦠✨👇 over the past weeks I built this toy 'proof-of-concept' example to explore and better understand how open-form generative art works, and discover some of the new archetypes it enables. In open-form genart each mint at the root of a project can define its own lineage: properties, traits, and characteristics that carry over to new editions evolved from it. For example, in this demo, the first parent defines the color palette that's used for the pixel pattern — these colors are then inherited by subsequent evolutions (with a random chance to alter a color or introduce a new one). But that's just scratching the tip of the iceberg — the REALLY COOL thing is that the root mint also specifies the rules that define how the pixel pattern morphs and changes over the course of its lineage! Meaning that each branch in this collection is its own unconstrained system, that evolves, and is explored by minting and evolving new editions. in this setting collectors play a huge role to explore the parametric space of the project ✨ on fxhash the UI will be much cleaner and more intuitive 🫧

how to evolve generative art 🦠✨👇 over the past weeks I built this toy 'proof-of-concept' example to explore and better understand how open-form generative art works, and discover some of the new archetypes it enables. In open-form genart each mint at the root of a project can define its own lineage: properties, traits, and characteristics that carry over to new editions evolved from it. For example, in this demo, the first parent defines the color palette that's used for the pixel pattern — these colors are then inherited by subsequent evolutions (with a random chance to alter a color or introduce a new one). But that's just scratching the tip of the iceberg — the REALLY COOL thing is that the root mint also specifies the rules that define how the pixel pattern morphs and changes over the course of its lineage! Meaning that each branch in this collection is its own unconstrained system, that evolves, and is explored by minting and evolving new editions. in this setting collectors play a huge role to explore the parametric space of the project ✨ on fxhash the UI will be much cleaner and more intuitive 🫧

Ahmad Moussa || Gorilla Sun

13,100 Aufrufe • vor 1 Jahr

In collaboration with Intel, our Depth Fusion showcases the power of our LDM3D diffusion model in generating 360° views from text prompts provided by the user. The LDM3D diffusion model generates a 2D RGB image and its corresponding relative depth map providing a complete RGBD representation corresponding to the text prompt. The LDM 3D model is a specialized version of the stable diffusion V 1.4 model that has been modified to fit both image and depth map data.The model was then fine tuned on a subset of the Laion400M data set - large scale image caption data set. The depth maps used to fine tune our model were generated by the DPTBeiT large 512 depth estimation model that provides highly accurate relative depth estimates for each pixel. We take the generated 2D RGB image and depth map and use them to compute a 360° projection using touchdesigner. Touchdesigner is a versatile platform that allows for the creation of immersive and interactive multimedia experiences. Our application harnesses the power of touchdesigner to bring the generated 360° views to life, providing users with a unique and engaging way to experience their text prompts, whether it’s a description of a tranquil forest, a noisy cityscape or a futuristic sci fi world. Our depth fusion can bring these concepts to life in a vivid and immersive detail. - Scottie Fox, VP Engineering Blockade Labs ScottieFox #AI #VR #3D #gamedev #stablediffusion

In collaboration with Intel, our Depth Fusion showcases the power of our LDM3D diffusion model in generating 360° views from text prompts provided by the user. The LDM3D diffusion model generates a 2D RGB image and its corresponding relative depth map providing a complete RGBD representation corresponding to the text prompt. The LDM 3D model is a specialized version of the stable diffusion V 1.4 model that has been modified to fit both image and depth map data.The model was then fine tuned on a subset of the Laion400M data set - large scale image caption data set. The depth maps used to fine tune our model were generated by the DPTBeiT large 512 depth estimation model that provides highly accurate relative depth estimates for each pixel. We take the generated 2D RGB image and depth map and use them to compute a 360° projection using touchdesigner. Touchdesigner is a versatile platform that allows for the creation of immersive and interactive multimedia experiences. Our application harnesses the power of touchdesigner to bring the generated 360° views to life, providing users with a unique and engaging way to experience their text prompts, whether it’s a description of a tranquil forest, a noisy cityscape or a futuristic sci fi world. Our depth fusion can bring these concepts to life in a vivid and immersive detail. - Scottie Fox, VP Engineering Blockade Labs ScottieFox #AI #VR #3D #gamedev #stablediffusion

Blockade Labs

11,458 Aufrufe • vor 3 Jahren