Loading video...

Video Failed to Load

Go Home

Super excited to introduce Pandora, a generative video World Model interactively controllable by language. #Sora and #GPT4 are both powerful. How about fusing them in a single model? 💥 Pandora gives a preview:🔭 > Build a General World Model (GWM) super efficiently by integrating pretrained autoregressive LLM and diffusion...

63,456 views • 2 years ago •via X (Twitter)

7 Comments

William Lamkin's profile picture
William Lamkin2 years ago

Very interesting work

Kun Zhou's profile picture
Kun Zhou2 years ago

Very cool work!

Xiuhan's profile picture
Xiuhan2 years ago

Amazing!

Benno Krojer's profile picture
Benno Krojer2 years ago

Super cool! What is your intuition on how easy it is to evaluate performance? Is human judgement the only way or did you study automatic metrics? Also, are there plans to release the model?

Infinity Orb's profile picture
Infinity Orb2 years ago

Pandora seems better at motion / consistency then current Video Gen and hopefully they will fully open source it. This could give open source a chance to compete with Google/ Veo and open AI/ Sora with video gen.

R-E's profile picture
R-E2 years ago

@ylecun what say you

Per Dyrnes's profile picture
Per Dyrnes2 years ago

Merge them like peanut butter & jelly, except with less crumbs. 🍞🥜

Related Videos

Tencent presents GameGen-O Open-world Video Game Generation We introduce GameGen-O, the first diffusion transformer model tailored for the generation of open-world video games. This model facilitates high-quality, open-domain generation by simulating a wide array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, thus allowing for the gameplay simulation. The development of GameGen-O involves a comprehensive data collection and processing effort from scratch. We collect and build the first Open-World Video Game Dataset (OGameData), amassed extensive data from over a hundred of next-generation open-world games, employing a proprietary data pipeline for efficient sorting, scoring, filtering, and decoupled captioning. This robust and extensive OGameData forms the foundation of our model's training process. GameGen-O undergoes a two-stage training process, consisting of foundation model pretraining and instruction tuning. In the first phase, the model is pre-trained on the OGameData via the text-to-video and video continuation, endowing GameGen-O with the capability for open-domain video game generation. In the second phase, the pre-trained model is frozen, and we fine-tuned using a trainable InstructNet, which enables the production of subsequent frames based on multimodal structural instructions. This whole training process imparts the model with the ability to generate and interactively control content. In summary, GameGen-O represents a notable initial step forward in the realm of open-world video game generation via generative models. It underscores the potential of generative models to serve as an alternative to rendering techniques, which can efficiently combine creative generation with interactive capabilities.

AK

367,000 views • 1 year ago

Brain signals and LLM embeddings converge for predicting every spoken or heard word. Beautiful research from Google AI They compared human brain activity during real conversations with internal embeddings from a speech-to-text LLM. Measured electrode signals in speech and language-related brain regions and matched them to the model’s word-level features. 🤖Key Highlights → Brain activity aligns linearly with LLM embeddings for real-life spoken conversations. → Sequence of comprehension: first speech sounds, then word meaning. → Sequence of production: planned meaning, then articulation, then hearing one’s own voice. → Consistent predictive coding (pre-onset anticipation, post-onset surprise) mirrors LLM next-word prediction. → Lower-tier auditory regions still show partial sensitivity to semantic information. 🤖 Model-Brain Alignment They observed a clear sequence: during comprehension, auditory cortex (superior temporal gyrus) showed strong correlation with speech embeddings, then language embeddings aligned with Broca’s area. During production, Broca’s area correlated with language embeddings before articulation, followed by motor cortex signals matching speech embeddings. This suggests that next-word prediction and higher-level meaning representation in the model parallel the brain’s approach. ⚙ So the study revealed a shared computational principle of predicting words in context. Even though the Transformer-based LLM processes words in parallel layers, the human brain processes them serially yet mirrors similar statistical regularities. This supports a “soft hierarchy” where both lower-level acoustic processing and higher-level semantic processing partially overlap in the brain.

Rohan Paul

15,213 views • 1 year ago

New short course Multimodal RAG: Chat with Videos, developed with Intel and taught by vasudevlal! In this course, you’ll work with LLaVA (Large Language and Vision Assistant), a Large Vision Language Model (LVLM) that can process both images and text. For example, given an image of a person doing a handstand on a skateboard at the beach, LLaVA doesn't just caption the scene, it’s able to predict possible outcomes, like the person losing balance or falling off. By understanding not just what's in a video frame, but what might happen next, your application can provide more insightful answers to questions about video. You'll build a full multimodal RAG pipeline that can chat about video content: - Use the BridgeTower model to create joint text-image embeddings in a 512-dimensional multimodal semantic space. - Learn video processing techniques to extract keyframes, generate transcripts using Whisper, and create captions. - Use the LanceDB vector database to store and retrieve high-dimensional multimodal embeddings. - Integrate the LLaVA model, combining CLIP's (Contrastive Language Image Pretraining) vision transformer with Llama, for advanced visual-textual reasoning. Your final system will ingest video data, generate embeddings for frames and text, perform similarity searches for relevant content, and use the retrieved multimodal context to inform LVLM-based response generation. The result is a system capable of answering nuanced questions about video content, effectively chatting about the video it has processed. Please sign up here!

Andrew Ng

107,548 views • 1 year ago