Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

🔥Excited to introduce CoDi-2! It follows complex multimodal-interleaved in-context instructions to generate any modalities (text, vision, audio) in zero/few-shot interactive way! Ziyi Yang Yang Liu Chenguang Zhu Mohit Bansal 🧵👇

Zineng Tang

1,653 subscribers

97,678 views • 2 years ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

10 Comments

Zineng Tang2 years ago

By aligning modalities with language for encoding and generation, CoDi-2 empowers Large Language Models (LLMs) to understand complex modality-interleaved instructions and in-context examples and conduct zero-shot/few-shot multimodal generation.

Zineng Tang2 years ago

Trained on a large-scale generation dataset encompassing in-context multi-modal instructions across text, vision, and audio, CoDi-2 can follow interleaved in-context text-audio-vision prompts and can zero-shot/few-shot jointly generate multiple modalities.

Zineng Tang2 years ago

CoDi-2 also demonstrates a wide range of zero-shot abilities for image generation like reasoning, compositionality, instruction editing, exemplar learning, and subject driven generation, etc.

Zineng Tang2 years ago

CoDi-2 also demonstrates zero-shot/few-shot abilities for audio generation with intricate prompting like instruction editing and exemplar learning.

Zineng Tang2 years ago

CoDi-2 surpasses previous domain-specific models on tasks such as subject-driven image generation, vision transformation, and audio editing.

Zineng Tang2 years ago

Overall, CoDi-2 signifies a substantial breakthrough in developing a comprehensive multimodal foundation model adept at interpreting in-context language-vision-audio interleaved instructions & producing multimodal outputs (in zero/few-shot way). @berkeley_ai @uncnlp @MSFTResearch

Zineng Tang2 years ago

As a reminder CoDi-1 will be presented at #NeurIPS2023, happy to chat about CoDi-1 and CoDi-2 in New Orleans! ->

Connor Shorten2 years ago

@yzy_ai @nlpyang @ChenguangZhu2 @mohitban47 generative-CoDi Weaviate module coming soon? @ZainHasan6 @antas_marcin 👀🔥

Wenmeng Zhou2 years ago

@yzy_ai @nlpyang @ChenguangZhu2 @mohitban47 cool! multi-modality in and out is the future I believe but it seems futue is coming now

SGM2 years ago

@yzy_ai @nlpyang @ChenguangZhu2 @mohitban47 @ZinengTang Will this product have a version where people can use it too i.e through a webui etc could be very helpful to a lot of people for sound effects perhaps you guys can release a version of it with freemium+paid plans

Related Videos

🖼️🎞️🔊📄Excited to introduce Composable Diffusion (CoDi), a new generative-AI foundation model that can take any combo of input modalities & generate any combo of output modalities (text, audio, image, video)! Ziyi Yang Chenguang Zhu Mohit Bansal 🧵👇 #CoDi

🖼️🎞️🔊📄Excited to introduce Composable Diffusion (CoDi), a new generative-AI foundation model that can take any combo of input modalities & generate any combo of output modalities (text, audio, image, video)! Ziyi Yang Chenguang Zhu Mohit Bansal 🧵👇 #CoDi

Zineng Tang

105,290 views • 3 years ago

Instead of asking a VLM to output progress, it reads the model’s internal belief directly from token logits. No in-context learning. No fine-tuning. No reward training. 📈 We introduce: TOPReward, a zero-shot reward modeling approach for robotics using token probabilities from pretrained video VLMs. The simplest way of doing reward modelling for robotics! Project: 🧵👇

Instead of asking a VLM to output progress, it reads the model’s internal belief directly from token logits. No in-context learning. No fine-tuning. No reward training. 📈 We introduce: TOPReward, a zero-shot reward modeling approach for robotics using token probabilities from pretrained video VLMs. The simplest way of doing reward modelling for robotics! Project: 🧵👇

Jiafei Duan

107,926 views • 6 months ago

Gemini-1.5 Pro has its spotlight stolen today, and people are poking fun at Sora vs Google memes. Well, I think it's the biggest boost in LLM capability so far in 2024. v1.5's 10M token context (1) excels at retrieval; (2) generalizes zero-shot to extremely long instructions like full tutorials and codebases; and (3) works across modalities such as text, audio, and video. Here's a stunning example: v1.5 learns to translate from English to Kalamang purely in context, following a full linguistic manual at inference time. Kalamang is a language spoken by fewer than 200 speakers in western New Guinea. Gemini has never seen this language during training and is only provided with 500 pages of linguistic documentation, a dictionary, and ~400 parallel sentences in context. It basically acquires a sophisticated new skill in the neural activations, instead of gradient finetuning. I talked about the Myth of Context Length many times before: don't get too excited by claims of 1M or even 1B context tokens. LSTMs already achieved literally infinite context length 25 yrs ago! What truly matters is how well the model actually uses the context to solve real-world problems, and Gemini-1.5 has surpassed the SOTA with flying colors. The paper is also well-written with lots of solid quantitative analysis on in-context memorization and generalization. Paper: “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context” Congrats to Jeff Dean Oriol Vinyals Sundar Pichai and team!

Gemini-1.5 Pro has its spotlight stolen today, and people are poking fun at Sora vs Google memes. Well, I think it's the biggest boost in LLM capability so far in 2024. v1.5's 10M token context (1) excels at retrieval; (2) generalizes zero-shot to extremely long instructions like full tutorials and codebases; and (3) works across modalities such as text, audio, and video. Here's a stunning example: v1.5 learns to translate from English to Kalamang purely in context, following a full linguistic manual at inference time. Kalamang is a language spoken by fewer than 200 speakers in western New Guinea. Gemini has never seen this language during training and is only provided with 500 pages of linguistic documentation, a dictionary, and ~400 parallel sentences in context. It basically acquires a sophisticated new skill in the neural activations, instead of gradient finetuning. I talked about the Myth of Context Length many times before: don't get too excited by claims of 1M or even 1B context tokens. LSTMs already achieved literally infinite context length 25 yrs ago! What truly matters is how well the model actually uses the context to solve real-world problems, and Gemini-1.5 has surpassed the SOTA with flying colors. The paper is also well-written with lots of solid quantitative analysis on in-context memorization and generalization. Paper: “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context” Congrats to Jeff Dean Oriol Vinyals Sundar Pichai and team!

Jim Fan

278,557 views • 2 years ago

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

AK

429,355 views • 3 years ago

MiniMax H3 is native in ComfyUI, day zero with the weights. Model highlights: → Text-to-video, prompt only → Image-to-video → First-and-last-frame - control the opening frame, the closing frame, or both → Reference-to-video - carry a subject, a motion, or a voice through the clip → In-place editing - modify a shot you already have instead of regenerating it The capability MiniMax (official) leads with is multimodal context understanding, and it's what collapses those five tasks into one model. H3 takes images, audio, and video together and resolves them against a prompt that explains how they relate. Download the workflows & learn more in our blog 👇

MiniMax H3 is native in ComfyUI, day zero with the weights. Model highlights: → Text-to-video, prompt only → Image-to-video → First-and-last-frame - control the opening frame, the closing frame, or both → Reference-to-video - carry a subject, a motion, or a voice through the clip → In-place editing - modify a shot you already have instead of regenerating it The capability MiniMax (official) leads with is multimodal context understanding, and it's what collapses those five tasks into one model. H3 takes images, audio, and video together and resolves them against a prompt that explains how they relate. Download the workflows & learn more in our blog 👇

ComfyUI

234,529 views • 1 month ago

There’s been a lot of buzz surrounding multimodal LLMs in medicine. At the Hiesinger Lab we’ve been building Almanac Chat for over 6 months alongside our collaborators and we’re excited to showcase the first preview! • Almanac Chat is the first truly multimodal medical LLM trained across specialties and modalities (i.e. images, text, video and audio) to serve as a generalist clinical copilot. • The entire stack (e.g. server, client, ML pipeline) is developed and hosted locally to guarantee privacy and flexibility at inference time. • We're looking forward to evaluating our model alongside domain experts at Stanford Medicine in Q2 2024. • We have a a lot more exciting things on the way - see if you can spot a few teasers in the video! 👥: We’re looking to scale our team, model, and hardware. If you're interested in a partnership or working alongside myself and our recent hire Joseph Cho please get in touch!

There’s been a lot of buzz surrounding multimodal LLMs in medicine. At the Hiesinger Lab we’ve been building Almanac Chat for over 6 months alongside our collaborators and we’re excited to showcase the first preview! • Almanac Chat is the first truly multimodal medical LLM trained across specialties and modalities (i.e. images, text, video and audio) to serve as a generalist clinical copilot. • The entire stack (e.g. server, client, ML pipeline) is developed and hosted locally to guarantee privacy and flexibility at inference time. • We're looking forward to evaluating our model alongside domain experts at Stanford Medicine in Q2 2024. • We have a a lot more exciting things on the way - see if you can spot a few teasers in the video! 👥: We’re looking to scale our team, model, and hardware. If you're interested in a partnership or working alongside myself and our recent hire Joseph Cho please get in touch!

Cyril Zakka, MD

85,980 views • 2 years ago

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

AK

290,517 views • 3 years ago

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)

Wenhu Chen

82,829 views • 1 year ago

Power to the Operators! Today at Dirac we’re excited to announce the launch of a new module for BuildOS: Operator+. Most manufacturing facilities in the U.S. still run on paper instructions — if they have them at all. They’re usually outdated, missing half the requirements, and don’t give you the ability to have traceability or a way for operators and technicians to give any structured feedback. Context is king. With BuildOS, we’ve already made it easy to generate clean, structured instructions straight from CAD — with animations, required tools, part callouts, and revision control built in. That alone is a huge unlock for manufacturing teams. But those instructions are only as good as what happens on the floor. Operator+ is our way of finally closing the gap. Operators now have a hand in the production planning process – they can suggest edits, automatically run time studies, log traceability information, flag non-conformance, stay fully in sync with the latest revision, and much more. Operator+ a massive step toward making the shop floor as modern and as connected as the design side. Today, we’re bringing Power to the Operator.

Power to the Operators! Today at Dirac we’re excited to announce the launch of a new module for BuildOS: Operator+. Most manufacturing facilities in the U.S. still run on paper instructions — if they have them at all. They’re usually outdated, missing half the requirements, and don’t give you the ability to have traceability or a way for operators and technicians to give any structured feedback. Context is king. With BuildOS, we’ve already made it easy to generate clean, structured instructions straight from CAD — with animations, required tools, part callouts, and revision control built in. That alone is a huge unlock for manufacturing teams. But those instructions are only as good as what happens on the floor. Operator+ is our way of finally closing the gap. Operators now have a hand in the production planning process – they can suggest edits, automatically run time studies, log traceability information, flag non-conformance, stay fully in sync with the latest revision, and much more. Operator+ a massive step toward making the shop floor as modern and as connected as the design side. Today, we’re bringing Power to the Operator.

Fil Aronshtein

11,283 views • 1 year ago

Fable 5.1 one-shot this Dark Souls meets Diablo game. All code, 222 KB total, zero asset files. Every texture, model and sound effect is generated procedurally. The whole sfx system is 7 KB of Web Audio synthesis. It nailed mechanics I only named in text: stamina, dodge rolls, barbarian skills, and a punishing Souls-style boss. The soul pickups and death animations came out way smoother than I expected. What gets me is how much it infers from a few genre references, with no screenshots or video to go on. Reaching this level used to take dozens of prompts in Sol or Opus 5. Anyone with taste can now make really fun games in the browser.

Fable 5.1 one-shot this Dark Souls meets Diablo game. All code, 222 KB total, zero asset files. Every texture, model and sound effect is generated procedurally. The whole sfx system is 7 KB of Web Audio synthesis. It nailed mechanics I only named in text: stamina, dodge rolls, barbarian skills, and a punishing Souls-style boss. The soul pickups and death animations came out way smoother than I expected. What gets me is how much it infers from a few genre references, with no screenshots or video to go on. Reaching this level used to take dozens of prompts in Sol or Opus 5. Anyone with taste can now make really fun games in the browser.

Meng To

148,512 views • 19 days ago

Text-to-speech is moving way too fast. Just a few days ago, I tweeted about PersonaPlex-7B, NVIDIA's new open source TTS ( And today, Qwen just open-sourced Qwen3-TTS 🤯 It’s a revolutionary text-to-speech model built for control. Not just about generating speech, but about shaping how it sounds directly from language. You can guide the pace, the tone, and the expressiveness straight from text, without touching audio graphs or hand-tuning parameters. That’s the real shift! What makes Qwen3-TTS stand out is how practical it already is: → voice cloning from just a few seconds of audio → voice creation without any reference sample → support for 10 languages out of the box → end-to-end latency down to ~97ms → works in both streaming and non-streaming setups The models come in two sizes (0.6B and 1.7B), so you can trade off quality and hardware cost depending on your setup. You can work with curated voices, designed voices, or cloned ones, and it integrates cleanly with vLLM for production use. It also ships as a simple Python package you can pip install. If you’re building real-time voice systems, this removes a lot of friction! 100% free and open source. I put the repo in the 🧵↓

Text-to-speech is moving way too fast. Just a few days ago, I tweeted about PersonaPlex-7B, NVIDIA's new open source TTS ( And today, Qwen just open-sourced Qwen3-TTS 🤯 It’s a revolutionary text-to-speech model built for control. Not just about generating speech, but about shaping how it sounds directly from language. You can guide the pace, the tone, and the expressiveness straight from text, without touching audio graphs or hand-tuning parameters. That’s the real shift! What makes Qwen3-TTS stand out is how practical it already is: → voice cloning from just a few seconds of audio → voice creation without any reference sample → support for 10 languages out of the box → end-to-end latency down to ~97ms → works in both streaming and non-streaming setups The models come in two sizes (0.6B and 1.7B), so you can trade off quality and hardware cost depending on your setup. You can work with curated voices, designed voices, or cloned ones, and it integrates cleanly with vLLM for production use. It also ships as a simple Python package you can pip install. If you’re building real-time voice systems, this removes a lot of friction! 100% free and open source. I put the repo in the 🧵↓

Charly Wargnier

59,144 views • 8 months ago

Today we’re excited to introduce Bland TTS, the first voice AI to cross the uncanny valley. Several months ago, our team solved one-shot style transfer of human speech. That means, from a single, brief MP3, you can clone any voice or remix another clone’s style (tone, cadence, or pronunciations, etc.). With Bland TTS: • Creatives can turn text into realistic sound effects or AI voice tracks with precise control over style and emotion. • Developers can plug in our TTS API to build voice apps fully customized to their liking. • Enterprises can launch AI-driven support lines that sound so natural — customers will save you as a contact (try talking to it on our site). 🧵Check out the thread below or visit our website to get started for free

Today we’re excited to introduce Bland TTS, the first voice AI to cross the uncanny valley. Several months ago, our team solved one-shot style transfer of human speech. That means, from a single, brief MP3, you can clone any voice or remix another clone’s style (tone, cadence, or pronunciations, etc.). With Bland TTS: • Creatives can turn text into realistic sound effects or AI voice tracks with precise control over style and emotion. • Developers can plug in our TTS API to build voice apps fully customized to their liking. • Enterprises can launch AI-driven support lines that sound so natural — customers will save you as a contact (try talking to it on our site). 🧵Check out the thread below or visit our website to get started for free

Bland

46,013,713 views • 1 year ago

NVIDIA just removed one of the biggest friction points in Voice AI. PersonaPlex-7B is an open-source, full-duplex conversational model. Free, open source (MIT), with open model weights on Hugging Face 🤗 Links to repo and weights in 🧵↓ The traditional ASR → LLM → TTS pipeline forces rigid turn-taking. It’s efficient, but it never feels natural. PersonaPlex-7B changes that. This NVIDIA model can listen and speak at the same time. It runs directly on continuous audio tokens with a dual-stream transformer, generating text and audio in parallel instead of passing control between components. That unlocks: → instant back-channel responses → interruptions that feel human → real conversational rhythm Persona control is fully zero-shot! If you’re building low-latency assistants or support agents, this is a big step forward 🔥

NVIDIA just removed one of the biggest friction points in Voice AI. PersonaPlex-7B is an open-source, full-duplex conversational model. Free, open source (MIT), with open model weights on Hugging Face 🤗 Links to repo and weights in 🧵↓ The traditional ASR → LLM → TTS pipeline forces rigid turn-taking. It’s efficient, but it never feels natural. PersonaPlex-7B changes that. This NVIDIA model can listen and speak at the same time. It runs directly on continuous audio tokens with a dual-stream transformer, generating text and audio in parallel instead of passing control between components. That unlocks: → instant back-channel responses → interruptions that feel human → real conversational rhythm Persona control is fully zero-shot! If you’re building low-latency assistants or support agents, this is a big step forward 🔥

Charly Wargnier ♨️

566,071 views • 8 months ago

Ok I've used Codex 5.4 Extra High for something that saved me so many hours and thousands of dollars and may interest you: -> how I automatically translate my game in 9 languages 1) I did the whole game in English, with no localization in mind 2) Then everybody told me that the reach is way higher by adding local languages 3) So I had a long chat with Codex to write a great spec to build a localization interface and super simple process for afterwards (I want to keep the simplicity) 4) Then translated the game in 9 languages! 5) Every localization is challenged a second time to improve 6) We are talking 580 keys per language - it took 1h35 to find all the hardcoded text and add keys - then it took 1h15 min to do all v1 translations 7) BONUS: it translates automatically all audio with ElevenLabs (using their text to speech skill) Total: Roughly 3 hours to implement a translation system and to translate the game Can any Japanese person tell me if the translation is good? There were 2 challenges: - translating once - keeping the game translated Now it's a blessing: I still work on the game like the usual, and if I add new text, it will automatically create the keys and translate in all languages just with basic instructions in Agents.md + a pre-commit hook to ensure it's done. I hope to use the tool for a professional or community-based human review in the future! Hope it helps:)

Ok I've used Codex 5.4 Extra High for something that saved me so many hours and thousands of dollars and may interest you: -> how I automatically translate my game in 9 languages 1) I did the whole game in English, with no localization in mind 2) Then everybody told me that the reach is way higher by adding local languages 3) So I had a long chat with Codex to write a great spec to build a localization interface and super simple process for afterwards (I want to keep the simplicity) 4) Then translated the game in 9 languages! 5) Every localization is challenged a second time to improve 6) We are talking 580 keys per language - it took 1h35 to find all the hardcoded text and add keys - then it took 1h15 min to do all v1 translations 7) BONUS: it translates automatically all audio with ElevenLabs (using their text to speech skill) Total: Roughly 3 hours to implement a translation system and to translate the game Can any Japanese person tell me if the translation is good? There were 2 challenges: - translating once - keeping the game translated Now it's a blessing: I still work on the game like the usual, and if I add new text, it will automatically create the keys and translate in all languages just with basic instructions in Agents.md + a pre-commit hook to ensure it's done. I hope to use the tool for a professional or community-based human review in the future! Hope it helps:)

Nicolas Zullo

13,418 views • 5 months ago

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.

Santiago

384,510 views • 3 years ago

Microsoft spent $13 billion and 3 years building an AI that knows your work context. Every time you open it, it still asks what you're working on. This developer set up a plain text file in 2 minutes. The file is called CLAUDE.md. It loads before every session. Before he types a single word. It already knows his name. It already knows his writing style. It already knows what he's building, who it's for, and what he never wants to see in a response. He doesn't introduce himself anymore. He doesn't explain his preferences anymore. He doesn't correct the same mistakes twice. He just works. No $30/month Copilot subscription. No Microsoft 365. No IT approval. No data sharing agreement. No onboarding. Just a plain text file, a free text editor, and 21 instructions a developer distilled from Andrej Karpathy's research. Those 21 instructions moved Claude's coding accuracy from 65% to 94%. The file hit #1 on GitHub with 82,000 stars. Most people using Claude right now have never heard of it. Microsoft has 221,000 employees, $13 billion invested in OpenAI, and a direct integration into every Windows laptop sold on the planet.. they built an AI assistant most companies pay $30/user/month for that still doesn't know your name. This developer has a laptop, a text file and a 2-minute setup.. he built something that knows more about how he works than any enterprise AI on the market. The $50 billion AI personalization industry just got embarrassed by a .md file. full breakdown down below

Microsoft spent $13 billion and 3 years building an AI that knows your work context. Every time you open it, it still asks what you're working on. This developer set up a plain text file in 2 minutes. The file is called CLAUDE.md. It loads before every session. Before he types a single word. It already knows his name. It already knows his writing style. It already knows what he's building, who it's for, and what he never wants to see in a response. He doesn't introduce himself anymore. He doesn't explain his preferences anymore. He doesn't correct the same mistakes twice. He just works. No $30/month Copilot subscription. No Microsoft 365. No IT approval. No data sharing agreement. No onboarding. Just a plain text file, a free text editor, and 21 instructions a developer distilled from Andrej Karpathy's research. Those 21 instructions moved Claude's coding accuracy from 65% to 94%. The file hit #1 on GitHub with 82,000 stars. Most people using Claude right now have never heard of it. Microsoft has 221,000 employees, $13 billion invested in OpenAI, and a direct integration into every Windows laptop sold on the planet.. they built an AI assistant most companies pay $30/user/month for that still doesn't know your name. This developer has a laptop, a text file and a 2-minute setup.. he built something that knows more about how he works than any enterprise AI on the market. The $50 billion AI personalization industry just got embarrassed by a .md file. full breakdown down below

Dep

14,179 views • 4 months ago

XAI EXTENDS GROK IMAGINE VIDEO GENERATION TO 10 SECONDS WITH QUALITY ENHANCEMENTS xAI has updated its Grok Imagine tool to produce videos lasting 10 seconds, doubling the prior limit. This change, along with refinements in visual and audio elements, expands the tool's utility for short-form content creation. xAI released the upgrade to Grok Imagine in early 2026. The company, founded by Elon Musk, announced the feature through posts on the X platform. This follows previous iterations where videos were capped at shorter durations, typically 5 seconds. Grok Imagine allows users to generate videos from text prompts, building on its image creation capabilities. The update addresses constraints in video length that limited expressive potential. Users can now input descriptions to create clips, such as animations or scenes, without needing initial images. This positions the tool within the broader landscape of AI-driven multimodal generation, where text-to-video systems are increasingly common. The core adjustment doubles the maximum video duration from 5 seconds to 10 seconds. Accompanying this are upgrades to video quality, including more stable visuals, richer details, and improved clarity. Audio has also been enhanced for better output, making the generated content more immersive. These changes were described as "big improvements across the board" in the announcement. No specific benchmarks or quantitative metrics for the quality improvements were detailed in the release statements. The feature rollout appears gradual, with some users accessing it via the Grok app or web interface. xAI has not introduced user controls for exact timing, though such options are mentioned as future possibilities. This development highlights xAI's emphasis on iterative enhancements in generative AI tools. By extending duration while refining output fidelity, it reflects engineering priorities aimed at balancing computational efficiency with user needs. The focus on audio and visual stability suggests attention to common pitfalls in early text-to-video models, such as inconsistencies or artifacts. The sources do not specify the underlying model architecture changes or training data adjustments enabling this upgrade. Performance in real-world scenarios, like handling complex prompts or maintaining consistency across clips, remains unquantified in the announcements. Interpretations of broader implications for AI video generation would require additional evidence beyond what's provided.

XAI EXTENDS GROK IMAGINE VIDEO GENERATION TO 10 SECONDS WITH QUALITY ENHANCEMENTS xAI has updated its Grok Imagine tool to produce videos lasting 10 seconds, doubling the prior limit. This change, along with refinements in visual and audio elements, expands the tool's utility for short-form content creation. xAI released the upgrade to Grok Imagine in early 2026. The company, founded by Elon Musk, announced the feature through posts on the X platform. This follows previous iterations where videos were capped at shorter durations, typically 5 seconds. Grok Imagine allows users to generate videos from text prompts, building on its image creation capabilities. The update addresses constraints in video length that limited expressive potential. Users can now input descriptions to create clips, such as animations or scenes, without needing initial images. This positions the tool within the broader landscape of AI-driven multimodal generation, where text-to-video systems are increasingly common. The core adjustment doubles the maximum video duration from 5 seconds to 10 seconds. Accompanying this are upgrades to video quality, including more stable visuals, richer details, and improved clarity. Audio has also been enhanced for better output, making the generated content more immersive. These changes were described as "big improvements across the board" in the announcement. No specific benchmarks or quantitative metrics for the quality improvements were detailed in the release statements. The feature rollout appears gradual, with some users accessing it via the Grok app or web interface. xAI has not introduced user controls for exact timing, though such options are mentioned as future possibilities. This development highlights xAI's emphasis on iterative enhancements in generative AI tools. By extending duration while refining output fidelity, it reflects engineering priorities aimed at balancing computational efficiency with user needs. The focus on audio and visual stability suggests attention to common pitfalls in early text-to-video models, such as inconsistencies or artifacts. The sources do not specify the underlying model architecture changes or training data adjustments enabling this upgrade. Performance in real-world scenarios, like handling complex prompts or maintaining consistency across clips, remains unquantified in the announcements. Interpretations of broader implications for AI video generation would require additional evidence beyond what's provided.

Lacey

28,881 views • 8 months ago