Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

📢📢📢Introducing xGen-MM-Vid (BLIP-3-Video)! This highly efficient multimodal language model is laser-focused on video understanding. Compared to other models, xGen-MM-Vid represents a video with a fraction of the visual tokens (e.g., 32 vs. 4608 tokens). Paper: Website: Researcher’s 🧵:👇

Salesforce AI Research

19,737 subscribers

12,550 Aufrufe • vor 1 Jahr •via X (Twitter)

Wissenschaft & Technologie Nachrichten & Politik Bildung

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

📢Introducing Generated Reality📢 A world model for XR that turns your tracked hand and head poses into an interactive, generative video experience. Take world models to the next level by interacting with the world using your own body! 🔗 1/4

📢Introducing Generated Reality📢 A world model for XR that turns your tracked hand and head poses into an interactive, generative video experience. Take world models to the next level by interacting with the world using your own body! 🔗 1/4

Gordon Wetzstein

20,045 Aufrufe • vor 6 Monaten

Apparently SeeDance 2 is down for the time being on BytePlus but here's a young wizard with his dragon companion. It's a very cool video model. This video cost $0.72 USD to generate which seems reasonable. (300,000 tokens for a 15 secs at $0.0024 USD per 1000 tokens.)

Apparently SeeDance 2 is down for the time being on BytePlus but here's a young wizard with his dragon companion. It's a very cool video model. This video cost $0.72 USD to generate which seems reasonable. (300,000 tokens for a 15 secs at $0.0024 USD per 1000 tokens.)

TomLikesRobots🤖

50,451 Aufrufe • vor 7 Monaten

NVIDIA just released a very impressive text-to-video paper. Video Latent Diffusion Models (Video LDMs) use a diffusion model in a compressed latent space to generate high-resolution videos. Here's a brief overview of how it works: 1. Pre-train image LDM on a dataset of images. 2. Turn the image LDM into a Video LDM by adding temporal layers to model video frames. 3. Fine-tune the Video LDM on encoded video sequences to create a video generator. 4. Temporally align diffusion model upsamplers to generate high-resolution videos. 5. Validate Video LDM on real driving videos of 512x1024 resolution, achieving state-of-the-art performance. 6. Apply the approach in creative content creation with text-to-video modeling. Paper: Project:

NVIDIA just released a very impressive text-to-video paper. Video Latent Diffusion Models (Video LDMs) use a diffusion model in a compressed latent space to generate high-resolution videos. Here's a brief overview of how it works: 1. Pre-train image LDM on a dataset of images. 2. Turn the image LDM into a Video LDM by adding temporal layers to model video frames. 3. Fine-tune the Video LDM on encoded video sequences to create a video generator. 4. Temporally align diffusion model upsamplers to generate high-resolution videos. 5. Validate Video LDM on real driving videos of 512x1024 resolution, achieving state-of-the-art performance. 6. Apply the approach in creative content creation with text-to-video modeling. Paper: Project:

Lior Alexander

158,600 Aufrufe • vor 3 Jahren

The Hidden Language of Diffusion Models paper page: tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation

The Hidden Language of Diffusion Models paper page: tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation

AK

41,830 Aufrufe • vor 3 Jahren

introducing FLUX 3. beyond video, this is the first model to support interactions with the real world through a new action-prediction system. available now on Krea, more modalities coming soon.

introducing FLUX 3. beyond video, this is the first model to support interactions with the real world through a new action-prediction system. available now on Krea, more modalities coming soon.

Krea

32,550 Aufrufe • vor 1 Monat

📢 Good Monday BORG LOVE Morning, awesome BORGERS and crypto friends! Did you buy $BORG today? 📈💚 💰We put the prize on the wheel, which raised it for YOU to 2500 $BORG tokens because we're even more bullish this week! 🎉 Win a share of 2,500 $BORG tokens! Post a screenshot of your $BORG purchase today 👇 Tag a friend, tag $BORG & use #BORGLOVE 💚 🏆 3 lucky winners will be picked! 🚀

📢 Good Monday BORG LOVE Morning, awesome BORGERS and crypto friends! Did you buy $BORG today? 📈💚 💰We put the prize on the wheel, which raised it for YOU to 2500 $BORG tokens because we're even more bullish this week! 🎉 Win a share of 2,500 $BORG tokens! Post a screenshot of your $BORG purchase today 👇 Tag a friend, tag $BORG & use #BORGLOVE 💚 🏆 3 lucky winners will be picked! 🚀

SwissBorg

12,847 Aufrufe • vor 1 Jahr

🚀New paper out - We present Video-MSG (Multimodal Sketch Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵

🚀New paper out - We present Video-MSG (Multimodal Sketch Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵

Jialu Li

35,060 Aufrufe • vor 1 Jahr

Most recent diffusion language model research (that I’ve seen) seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to *potentially* giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.

Most recent diffusion language model research (that I’ve seen) seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to potentially giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.

nathan (in sf)

40,440 Aufrufe • vor 7 Monaten

Grok Imagine, the AI video model that doesn't want to be real. Since Grok launched its video capabilities, the discourse has been almost entirely content moderation - what will and won't generate. Spicy mode, regulatory investigations, the EUs reaction. It's loud, it's dramatic. And it's completely missing the point. Because buried underneath all of that noise is something genuinely fascinating for filmmakers and visual creators. Grok Imagine video is the only major AI video model that was built to honor imagination over physics. See how it performs in 6 styles versus other models along with detailed prompts 🧵👇

Grok Imagine, the AI video model that doesn't want to be real. Since Grok launched its video capabilities, the discourse has been almost entirely content moderation - what will and won't generate. Spicy mode, regulatory investigations, the EUs reaction. It's loud, it's dramatic. And it's completely missing the point. Because buried underneath all of that noise is something genuinely fascinating for filmmakers and visual creators. Grok Imagine video is the only major AI video model that was built to honor imagination over physics. See how it performs in 6 styles versus other models along with detailed prompts 🧵👇

Invideo

1,137,773 Aufrufe • vor 5 Monaten

A major question in multimodal modeling is how to leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵

A major question in multimodal modeling is how to leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵

Amir Zamir

23,927 Aufrufe • vor 2 Monaten

Added context to my tiny diffusion model to enable sequential generation of longer outputs! Currently the context is a quarter of the sequence length (seq_len=256, context_len=64). I have a theory that the less semantic-value-per-token, the worse the “curse of parallel decoding” is. With parallel decoding, we independently predict multiple tokens in one step. With the sentence “My poker hand was a ___ ___”, two valid predictions are “two pair” and “straight flush”. Because each token prediction is independent though, we can end up with a nonsensical output like “two flush”. This seems to be exacerbated with low semantic-value-per-token, as now you need more tokens to express the same concept. Instead of needing to independently predict two tokens, we might need to predict 10 instead (which is of course much harder). The model currently has noticeably worse output compared to nanogpt (similar size) and I believe this is a main reason. I’ll try adding confidence-aware parallel decoding (from NVIDIA’s Fast-dLLM paper) and other tricks and see how much they improve generation quality.

Added context to my tiny diffusion model to enable sequential generation of longer outputs! Currently the context is a quarter of the sequence length (seq_len=256, context_len=64). I have a theory that the less semantic-value-per-token, the worse the “curse of parallel decoding” is. With parallel decoding, we independently predict multiple tokens in one step. With the sentence “My poker hand was a _ _”, two valid predictions are “two pair” and “straight flush”. Because each token prediction is independent though, we can end up with a nonsensical output like “two flush”. This seems to be exacerbated with low semantic-value-per-token, as now you need more tokens to express the same concept. Instead of needing to independently predict two tokens, we might need to predict 10 instead (which is of course much harder). The model currently has noticeably worse output compared to nanogpt (similar size) and I believe this is a main reason. I’ll try adding confidence-aware parallel decoding (from NVIDIA’s Fast-dLLM paper) and other tricks and see how much they improve generation quality.

Nathan Barry

89,040 Aufrufe • vor 10 Monaten

LTX 2.3 is now live on Krea. this video model generates for 1/10th the cost of other top models and can learn any style or character - just upload some references and train a lora try it now 👇

LTX 2.3 is now live on Krea. this video model generates for 1/10th the cost of other top models and can learn any style or character - just upload some references and train a lora try it now 👇

Krea

21,845 Aufrufe • vor 4 Monaten

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,849 Aufrufe • vor 1 Jahr

Fine-tune DeepSeek-OCR on your own language! (100% local) DeepSeek-OCR is a 3B-parameter vision model that achieves 97% precision while using 10× fewer vision tokens than text-based LLMs. It handles tables, papers, and handwriting without killing your GPU or budget. Why it matters: Most vision models treat documents as massive sequences of tokens, making long-context processing expensive and slow. DeepSeek-OCR uses context optical compression to convert 2D layouts into vision tokens, enabling efficient processing of complex documents. The best part? You can easily fine-tune it for your specific use case on a single GPU. I used Unsloth to run this experiment on Persian text and saw an 88.26% improvement in character error rate. ↳ Base model: 149% character error rate (CER) ↳ Fine-tuned model: 60% CER (57% more accurate) ↳ Training time: 60 steps on a single GPU Persian was just the test case. You can swap in your own dataset for any language, document type, or specific domain you're working with. I've shared the complete guide in the next tweet - all the code, notebooks, and environment setup ready to run with a single click. Everything is 100% open-source!

Fine-tune DeepSeek-OCR on your own language! (100% local) DeepSeek-OCR is a 3B-parameter vision model that achieves 97% precision while using 10× fewer vision tokens than text-based LLMs. It handles tables, papers, and handwriting without killing your GPU or budget. Why it matters: Most vision models treat documents as massive sequences of tokens, making long-context processing expensive and slow. DeepSeek-OCR uses context optical compression to convert 2D layouts into vision tokens, enabling efficient processing of complex documents. The best part? You can easily fine-tune it for your specific use case on a single GPU. I used Unsloth to run this experiment on Persian text and saw an 88.26% improvement in character error rate. ↳ Base model: 149% character error rate (CER) ↳ Fine-tuned model: 60% CER (57% more accurate) ↳ Training time: 60 steps on a single GPU Persian was just the test case. You can swap in your own dataset for any language, document type, or specific domain you're working with. I've shared the complete guide in the next tweet - all the code, notebooks, and environment setup ready to run with a single click. Everything is 100% open-source!

Akshay 🚀

126,213 Aufrufe • vor 10 Monaten

📢 Our xData Godin reply library just got an update! 👇🏻Check out this video and reply below with the Godin button clicked to generate a variety of engaging responses. Let's make data on-chain more fun than ever! #xData #Godin #Web3Go🚀

📢 Our xData Godin reply library just got an update! 👇🏻Check out this video and reply below with the Godin button clicked to generate a variety of engaging responses. Let's make data on-chain more fun than ever! #xData #Godin #Web3Go🚀

DIN⏳

58,606 Aufrufe • vor 2 Jahren

✨ Every time the video models get better, the try on model on Photo AI also becomes a lot more useful, as a large % of my customers now are e-commerce store And showing clothes in a video is nice for sales! With AI this means stores don't need to do expensive shoots flying a model and entire camera and light crew around the world They can just upload a few photos of their models, then upload the clothes, and describe the setting (like a beach in Thailand) and in less than 10 seconds it's generated, for a video in less than a minute! Below is the input: a dress laid flat, and output: a full video shoot

✨ Every time the video models get better, the try on model on Photo AI also becomes a lot more useful, as a large % of my customers now are e-commerce store And showing clothes in a video is nice for sales! With AI this means stores don't need to do expensive shoots flying a model and entire camera and light crew around the world They can just upload a few photos of their models, then upload the clothes, and describe the setting (like a beach in Thailand) and in less than 10 seconds it's generated, for a video in less than a minute! Below is the input: a dress laid flat, and output: a full video shoot

@levelsio

334,317 Aufrufe • vor 1 Jahr

We got early access to the Gemini Omni API Google is calling this model "Nano Banana for video" 3 things we built with it👇 1. Landscaping proposal from customer video. Customer submits a video of the current lot. Hyperagent designs and renders the transformation. Perfect realism, no surreal changes to the surroundings. 2. Animated professor who explains your dashboards. Hyperagent runs analysis on a business question, then generates an explainer video to walkthrough the findings. 3. 8-bit morning briefing. Hyperagent builds morning briefings based on calendar, email/chat, and market intel. Then generates a sidescrolling platformer video showing the goals you'll clear today. Our take: > Video has been a vastly underused input for agents. This is the first model to make video directly malleable to our agents. > Get more imaginative with your outputs. How could a video artifact make the work more memorable or playful? coming soon to Hyperagent

We got early access to the Gemini Omni API Google is calling this model "Nano Banana for video" 3 things we built with it👇 1. Landscaping proposal from customer video. Customer submits a video of the current lot. Hyperagent designs and renders the transformation. Perfect realism, no surreal changes to the surroundings. 2. Animated professor who explains your dashboards. Hyperagent runs analysis on a business question, then generates an explainer video to walkthrough the findings. 3. 8-bit morning briefing. Hyperagent builds morning briefings based on calendar, email/chat, and market intel. Then generates a sidescrolling platformer video showing the goals you'll clear today. Our take: > Video has been a vastly underused input for agents. This is the first model to make video directly malleable to our agents. > Get more imaginative with your outputs. How could a video artifact make the work more memorable or playful? coming soon to Hyperagent

Hyperagent

325,111 Aufrufe • vor 2 Monaten

The recent Massachusetts Institute of Technology (MIT) CSAIL paper published on Recursive Language Models is a fascinating look into how AI systems reason in 2026. You can check it out here ➡️ → The paper notes that even frontier models suffer from “context rot” as inputs grow longer. More tokens don’t mean more understanding. Instead of compressing or summarising, RLMs “treat long prompts as part of an external environment” …and let the model programmatically inspect, decompose, and recursively requery itself over precise sections. This matters in Web3. Why? Smart contracts are long, stateful, and brittle. One missed assumption = unnecessary hassle. In the video below, we applied RLM principles to improve the prompt shown. The old prompt would try to force multi-step pauses. Therefore breaking the AI's workflow. Now it decomposes requirements, scores confidence for each component, verifies the logic, and then synthesises. Result = Production-ready contracts with flagged risks, not blind single-pass outputs. Try it out with our Smart Contract Generator today!

The recent Massachusetts Institute of Technology (MIT) CSAIL paper published on Recursive Language Models is a fascinating look into how AI systems reason in 2026. You can check it out here ➡️ → The paper notes that even frontier models suffer from “context rot” as inputs grow longer. More tokens don’t mean more understanding. Instead of compressing or summarising, RLMs “treat long prompts as part of an external environment” …and let the model programmatically inspect, decompose, and recursively requery itself over precise sections. This matters in Web3. Why? Smart contracts are long, stateful, and brittle. One missed assumption = unnecessary hassle. In the video below, we applied RLM principles to improve the prompt shown. The old prompt would try to force multi-step pauses. Therefore breaking the AI's workflow. Now it decomposes requirements, scores confidence for each component, verifies the logic, and then synthesises. Result = Production-ready contracts with flagged risks, not blind single-pass outputs. Try it out with our Smart Contract Generator today!

ChainGPT

82,385 Aufrufe • vor 7 Monaten