正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Chat with video using EmbeddingGemma 2 (100% local): EmbeddingGemma 2 by Google DeepMind maps text, images, video, and audio into a shared vector space. This makes it possible to search video frames with natural language. Devs have already built native Rust implementations for local inference. Unsloth also provides a... 176 MB, 4-bit GGUF checkpoint for text embeddings, although that version doesn't include the full multimodal pipeline. I tested the video-search use case by building a chat interface that finds matching frames and jumps to their timestamps. On my Apple M3 Pro, using the CPU: - Indexing took 29s for a 60s video - Search took 25 ms on average across 9 queries - The saved frame vectors occupied 180 KiB. The video is indexed once. Each new search embeds the query and compares it with the saved frame vectors. Processing stays entirely on the laptop. The video below depicts the entire flow in action. That said, model quantization reduces the memory needed for the weights. The retrieval index has its own storage cost, which grows as you add videos. To dive deeper into reducing that cost, I wrote a full article on binary quantization. It explains how storing vectors with one bit per dimension reduces their raw storage by 32x compared with float32, and includes a working retrieval implementation. Read it below.show more

Avi Chawla

80,308 subscribers

18,407 次观看 • 5 天前 •via X (Twitter)

教育科学技术

Anya Rossi• Live Now

Private livecam show

7 条评论

Pixeltable 的头像

Pixeltable4 天前

@GoogleDeepMind for a local video library, I'd keep the old index searchable while rebuilding with a new embedding model. then you can compare the two indexes on the same file and timestamp before switching over. that way a model upgrade doesn't take the library offline.

Ruslan | Building apps 的头像

Ruslan | Building apps5 天前

@GoogleDeepMind I’d try a query for something that appears for just one second. Sampling fewer frames saves space, but that’s where you find out what the smaller index actually misses.

K4k0 的头像

K4k05 天前

@GoogleDeepMind Most likely soon you just type smth like "the day my kid rode a bike alone" and it finds it in years of family videos, all on your own laptop, no cloud

Sergei Golimbievsky 的头像

Sergei Golimbievsky5 天前

@GoogleDeepMind This is perfect for my offline Google Photos project!

Mladen Lotar 的头像

Mladen Lotar5 天前

@GoogleDeepMind Embeddings on the ANE kept LLM decode from stalling here.

Aleksandar Janca 的头像

Aleksandar Janca5 天前

@GoogleDeepMind searching is the easy half on an m3 pro, how long did indexing an hour of video take on the cpu

Agentik 的头像

Agentik5 天前

@GoogleDeepMind Tried frame search like this on screen recordings once. Sampling every second blew the index up fast, so we only embedded frames where the scene actually changed. Does EmbeddingGemma 2 do any of that for you or is chunking still your problem?

相关视频

How do computers understand data? With semantic search! Instead of just matching keywords, it understands context using vector embeddings. Here’s how: 1) Convert data (text, images, etc.) into vectors (embeddings) 2) Store these vectors in a vector database 3) Search by meaning, not just the keywords Semantic search makes finding data across formats easier. Learn more in this blog post by leonie, my all-time favorite:

How do computers understand data? With semantic search! Instead of just matching keywords, it understands context using vector embeddings. Here’s how: 1) Convert data (text, images, etc.) into vectors (embeddings) 2) Store these vectors in a vector database 3) Search by meaning, not just the keywords Semantic search makes finding data across formats easier. Learn more in this blog post by leonie, my all-time favorite:

Femke Plantinga

23,926 次观看 • 2 年前

Read this if you make videos with Claude. I built a skill that cut my QA cost on this video from $127 to 16 cents. It was made with Claude in Remotion. It's called SageCut. It swaps the LLM out of frame-by-frame QA and uses a multimodal decision model, Sage (Levanto Labs), instead. I'm the only video guy at Levanto. I run 3 Claude plans and have been making Remotion videos since early this year, with BlenderMCP in the mix, so the output is pro-grade with very little babysitting. The problem: it burned through my Max plan, and a video could take 2 days. When I dug into the workflow, the reason was that Claude subagents were reviewing every single frame. That cost a ton and added 4+ hours per video. SageCut hands that step to Sage. Sage is great at spotting defective frames, and only those get passed to Opus 5.5 to fix. Across 6 A/B tests the output was exactly the same, at 1/10 the QA cost and 1/4 the QA time. On this video, Sage read all 3,153 frames for 16 cents. If you make Remotion videos, try it. The link to skill is below. How long does a Remotion video take you right now with Claude?

Read this if you make videos with Claude. I built a skill that cut my QA cost on this video from $127 to 16 cents. It was made with Claude in Remotion. It's called SageCut. It swaps the LLM out of frame-by-frame QA and uses a multimodal decision model, Sage (Levanto Labs), instead. I'm the only video guy at Levanto. I run 3 Claude plans and have been making Remotion videos since early this year, with BlenderMCP in the mix, so the output is pro-grade with very little babysitting. The problem: it burned through my Max plan, and a video could take 2 days. When I dug into the workflow, the reason was that Claude subagents were reviewing every single frame. That cost a ton and added 4+ hours per video. SageCut hands that step to Sage. Sage is great at spotting defective frames, and only those get passed to Opus 5.5 to fix. Across 6 A/B tests the output was exactly the same, at 1/10 the QA cost and 1/4 the QA time. On this video, Sage read all 3,153 frames for 16 cents. If you make Remotion videos, try it. The link to skill is below. How long does a Remotion video take you right now with Claude?

Big Iron Chris

85,241 次观看 • 1 个月前

I'm a bit confused... Google's Veo 2 is the best video model in text-to-video. But on the other hand... The newly released image-to-video for Veo 2 (on Freepik (now Magnific) and @FAL) feels underwhelming. Input images generated with Runway Frames. Here it is compared to Luma

I'm a bit confused... Google's Veo 2 is the best video model in text-to-video. But on the other hand... The newly released image-to-video for Veo 2 (on Freepik (now Magnific) and @FAL) feels underwhelming. Input images generated with Runway Frames. Here it is compared to Luma

Alex Patrascu

85,445 次观看 • 1 年前

This is how you build the next generation of AI applications. You need an idea, a good model architecture, and a great harness. That's what MUZIM did. It's an app that runs local multimodal and vision models offline to parse and index your images, videos, and documents frame by frame. • Super fast on-device processing • On-device video & image understanding • You can also bring your API keys for heavy reasoning • Your data stays where it is / nothing goes to the cloud You can do a couple of things with this: 1. Vibe searching for whatever you remember in photos or raw video clips. The app will jump straight to the exact frame and timestamp of the video. This is search as it should be. 2. Use local-running agents to perform tasks on your files. I show a couple of examples in my video. You can check it out here: Thanks to the MUZIM by OpenSoul team for showing me the tool and partnering with me on this post.

This is how you build the next generation of AI applications. You need an idea, a good model architecture, and a great harness. That's what MUZIM did. It's an app that runs local multimodal and vision models offline to parse and index your images, videos, and documents frame by frame. • Super fast on-device processing • On-device video & image understanding • You can also bring your API keys for heavy reasoning • Your data stays where it is / nothing goes to the cloud You can do a couple of things with this: 1. Vibe searching for whatever you remember in photos or raw video clips. The app will jump straight to the exact frame and timestamp of the video. This is search as it should be. 2. Use local-running agents to perform tasks on your files. I show a couple of examples in my video. You can check it out here: Thanks to the MUZIM by OpenSoul team for showing me the tool and partnering with me on this post.

Santiago

65,953 次观看 • 2 个月前

New short course Multimodal RAG: Chat with Videos, developed with Intel and taught by vasudevlal! In this course, you’ll work with LLaVA (Large Language and Vision Assistant), a Large Vision Language Model (LVLM) that can process both images and text. For example, given an image of a person doing a handstand on a skateboard at the beach, LLaVA doesn't just caption the scene, it’s able to predict possible outcomes, like the person losing balance or falling off. By understanding not just what's in a video frame, but what might happen next, your application can provide more insightful answers to questions about video. You'll build a full multimodal RAG pipeline that can chat about video content: - Use the BridgeTower model to create joint text-image embeddings in a 512-dimensional multimodal semantic space. - Learn video processing techniques to extract keyframes, generate transcripts using Whisper, and create captions. - Use the LanceDB vector database to store and retrieve high-dimensional multimodal embeddings. - Integrate the LLaVA model, combining CLIP's (Contrastive Language Image Pretraining) vision transformer with Llama, for advanced visual-textual reasoning. Your final system will ingest video data, generate embeddings for frames and text, perform similarity searches for relevant content, and use the retrieved multimodal context to inform LVLM-based response generation. The result is a system capable of answering nuanced questions about video content, effectively chatting about the video it has processed. Please sign up here!

New short course Multimodal RAG: Chat with Videos, developed with Intel and taught by vasudevlal! In this course, you’ll work with LLaVA (Large Language and Vision Assistant), a Large Vision Language Model (LVLM) that can process both images and text. For example, given an image of a person doing a handstand on a skateboard at the beach, LLaVA doesn't just caption the scene, it’s able to predict possible outcomes, like the person losing balance or falling off. By understanding not just what's in a video frame, but what might happen next, your application can provide more insightful answers to questions about video. You'll build a full multimodal RAG pipeline that can chat about video content: - Use the BridgeTower model to create joint text-image embeddings in a 512-dimensional multimodal semantic space. - Learn video processing techniques to extract keyframes, generate transcripts using Whisper, and create captions. - Use the LanceDB vector database to store and retrieve high-dimensional multimodal embeddings. - Integrate the LLaVA model, combining CLIP's (Contrastive Language Image Pretraining) vision transformer with Llama, for advanced visual-textual reasoning. Your final system will ingest video data, generate embeddings for frames and text, perform similarity searches for relevant content, and use the retrieved multimodal context to inform LVLM-based response generation. The result is a system capable of answering nuanced questions about video content, effectively chatting about the video it has processed. Please sign up here!

Andrew Ng

107,967 次观看 • 2 年前

Finally, a database can generate its own embeddings now. When you add semantic search to an app, the standard step is to wire up an external embedding service, sync the vectors to a separate store, and write glue to keep everything updated as data changes. While this is a standard setup, problems show up later when your data changes, but your pipeline has already run, and search quality starts degrading in ways that are hard to pin down because nothing is explicitly broken. MongoDB Atlas handles this with auto-embedding. You point an index at a text field, specify a Voyage AI model, and Atlas generates the vectors inside the database. When a document changes, it re-embeds itself automatically, so that the search always stays grounded to the current context. I tested this against MongoDB's sample movies collection, 21,000 plots indexed with one config. A plain English query about dystopian machine-controlled futures returned semantically relevant results, despite having no keyword overlap. You can try it yourself here: One index config replaced the embedding service, the vector store, and the sync layer you used to maintain separately. The video below depicts this in action, and I worked with the MongoDB team to put this together. I have also put together a full LLM engineering roadmap covering what it actually takes to build and ship production AI systems in 2026. Read it below.

Finally, a database can generate its own embeddings now. When you add semantic search to an app, the standard step is to wire up an external embedding service, sync the vectors to a separate store, and write glue to keep everything updated as data changes. While this is a standard setup, problems show up later when your data changes, but your pipeline has already run, and search quality starts degrading in ways that are hard to pin down because nothing is explicitly broken. MongoDB Atlas handles this with auto-embedding. You point an index at a text field, specify a Voyage AI model, and Atlas generates the vectors inside the database. When a document changes, it re-embeds itself automatically, so that the search always stays grounded to the current context. I tested this against MongoDB's sample movies collection, 21,000 plots indexed with one config. A plain English query about dystopian machine-controlled futures returned semantically relevant results, despite having no keyword overlap. You can try it yourself here: One index config replaced the embedding service, the vector store, and the sync layer you used to maintain separately. The video below depicts this in action, and I worked with the MongoDB team to put this together. I have also put together a full LLM engineering roadmap covering what it actually takes to build and ship production AI systems in 2026. Read it below.

Avi Chawla

14,463 次观看 • 3 个月前

People are finding really creative ways to use EmbeddingGemma 2 beyond search. Instead of running a full multimodal LLM, this community project trains a lightweight classifier on top of EmbeddingGemma 2 in real time. Because the model is compact and natively multimodal, it can learn new categories on the fly and classify inputs locally in ~20ms for text and ~100ms for images.

People are finding really creative ways to use EmbeddingGemma 2 beyond search. Instead of running a full multimodal LLM, this community project trains a lightweight classifier on top of EmbeddingGemma 2 in real time. Because the model is compact and natively multimodal, it can learn new categories on the fly and classify inputs locally in ~20ms for text and ~100ms for images.

Google Gemma

97,151 次观看 • 8 天前

Traditional (SQL) databases rely primarily on keyword-based searches to retrieve information. These searches match the exact words or phrases in your query to the text stored in the database. While effective for many applications, this method has limitations when it comes to understanding context or finding relevant information that doesn’t include the exact keywords. Hybrid search combines the strengths of traditional keyword-based BM25 search with the advanced capabilities of semantic search. To effectively implement a hybrid search, a vector database is essential. Vector databases go beyond just words; they understand the meaning behind the data. They transform data such as text, images, or audio into numerical representations called vectors. These vector embeddings enable the database to find similar items, even if they don't share exact keywords. When you integrate hybrid search with Retrieval-Augmented Generation (RAG) systems, you can achieve higher accuracy in retrieved context and better output in generated responses. Learn more about RAG systems in this video with Victoria Slocum:

Traditional (SQL) databases rely primarily on keyword-based searches to retrieve information. These searches match the exact words or phrases in your query to the text stored in the database. While effective for many applications, this method has limitations when it comes to understanding context or finding relevant information that doesn’t include the exact keywords. Hybrid search combines the strengths of traditional keyword-based BM25 search with the advanced capabilities of semantic search. To effectively implement a hybrid search, a vector database is essential. Vector databases go beyond just words; they understand the meaning behind the data. They transform data such as text, images, or audio into numerical representations called vectors. These vector embeddings enable the database to find similar items, even if they don't share exact keywords. When you integrate hybrid search with Retrieval-Augmented Generation (RAG) systems, you can achieve higher accuracy in retrieved context and better output in generated responses. Learn more about RAG systems in this video with Victoria Slocum:

Femke Plantinga

144,343 次观看 • 2 年前

MICROSOFT QUIETLY SHIPPED THE RAG PIPELINE EVERYONE KEEPS TRYING TO BUILD FROM SCRATCH it is called Kernel Memory, and it turns any pile of files into an AI memory that answers with citations. drop your data in and Microsoft's pipeline does the rest: > ingest PDFs, Word, PPT, Excel, web pages, even images > extract the text, chunk it, embed it into vectors > store the vectors with metadata and keep the originals in blob storage > ask a question and it pulls the right chunks with semantic plus hybrid search then it hands the answer back grounded, with links to the exact source and full token usage. and it plugs into whatever you already run: > Azure OpenAI, OpenAI, Ollama, or Anthropic for the models > Qdrant, pgvector, Redis, or Azure AI Search for the vectors > every step is a swappable handler, so you can replace any part with your own one heads up: Microsoft archived the repo in June 2026 and ships it as a reference blueprint, not production. but as a map for how memory and RAG are actually built, nothing is cleaner. run it in one command: docker run kernelmemory/service bookmark this. Microsoft already drew the blueprint, you just build on it.

MICROSOFT QUIETLY SHIPPED THE RAG PIPELINE EVERYONE KEEPS TRYING TO BUILD FROM SCRATCH it is called Kernel Memory, and it turns any pile of files into an AI memory that answers with citations. drop your data in and Microsoft's pipeline does the rest: > ingest PDFs, Word, PPT, Excel, web pages, even images > extract the text, chunk it, embed it into vectors > store the vectors with metadata and keep the originals in blob storage > ask a question and it pulls the right chunks with semantic plus hybrid search then it hands the answer back grounded, with links to the exact source and full token usage. and it plugs into whatever you already run: > Azure OpenAI, OpenAI, Ollama, or Anthropic for the models > Qdrant, pgvector, Redis, or Azure AI Search for the vectors > every step is a swappable handler, so you can replace any part with your own one heads up: Microsoft archived the repo in June 2026 and ships it as a reference blueprint, not production. but as a map for how memory and RAG are actually built, nothing is cleaner. run it in one command: docker run kernelmemory/service bookmark this. Microsoft already drew the blueprint, you just build on it.

NO1ennn

85,704 次观看 • 2 个月前

This is, by far, one of the best uses of modern AI. If you don't use embeddings when querying your database, you are definitely leaving a lot on the table. In this video, I'll show you how to run semantic searches using OpenAI and PostgreSQL. It's all thanks to Pgai, an open-source PostgreSQL extension: Here's what will happen: 1. We'll create a simple table with news articles 2. We'll generate embeddings for those articles 3. We'll run queries on top of those embeddings For this video, I generated the embeddings using a simple query, but pgai Vectorizer would do the same automatically as new information makes it into the database. This is awesome! If you have a PostgreSQL database with data you are searching over, you should start experimenting with semantic searches immediately. For most use cases, a combination of full-text search + semantic search is the best approach. If you don't have a PostgreSQL database around, you can try free for 30 days using Timescale: Thanks to the TimescaleDB (by Tiger Data) team for partnering with me on this post!

This is, by far, one of the best uses of modern AI. If you don't use embeddings when querying your database, you are definitely leaving a lot on the table. In this video, I'll show you how to run semantic searches using OpenAI and PostgreSQL. It's all thanks to Pgai, an open-source PostgreSQL extension: Here's what will happen: 1. We'll create a simple table with news articles 2. We'll generate embeddings for those articles 3. We'll run queries on top of those embeddings For this video, I generated the embeddings using a simple query, but pgai Vectorizer would do the same automatically as new information makes it into the database. This is awesome! If you have a PostgreSQL database with data you are searching over, you should start experimenting with semantic searches immediately. For most use cases, a combination of full-text search + semantic search is the best approach. If you don't have a PostgreSQL database around, you can try free for 30 days using Timescale: Thanks to the TimescaleDB (by Tiger Data) team for partnering with me on this post!

Santiago

110,133 次观看 • 1 年前

New short course on Building Applications with Vector Databases, taught by Pinecone’s Tim Tully! At the heart of a vector database is the ability to store a collection of vectors and then query against that, meaning input a new vector and find similar ones. This is useful for many AI applications. In this course, you'll learn how to use vector databases to build: (i) Semantic Search: Create a text search tool that goes beyond keyword matching, and instead focuses on the meaning of content. (ii) RAG (retrieval augmented generation): Enhance your LLM output by incorporating context from sources the model wasn't trained on. (iii) Recommender System: Combine semantic search and RAG to recommend topics, and demonstrate it with a news article recommender. (iv) Hybrid Search: Build an application that finds items using both images and descriptive text -- by combining both sparse and dense vector representations of the data -- using an eCommerce dataset as an example. (v) Image Similarity: Use image vector embeddings to create an app to compare facial features, using a database of public figures to determine the likeness between them. (vi) Anomaly Detection: Build an anomaly detection app that identifies unusual patterns in network communication logs. I hope you’ll enjoy learning how to build all these types of applications! Please sign up here:

New short course on Building Applications with Vector Databases, taught by Pinecone’s Tim Tully! At the heart of a vector database is the ability to store a collection of vectors and then query against that, meaning input a new vector and find similar ones. This is useful for many AI applications. In this course, you'll learn how to use vector databases to build: (i) Semantic Search: Create a text search tool that goes beyond keyword matching, and instead focuses on the meaning of content. (ii) RAG (retrieval augmented generation): Enhance your LLM output by incorporating context from sources the model wasn't trained on. (iii) Recommender System: Combine semantic search and RAG to recommend topics, and demonstrate it with a news article recommender. (iv) Hybrid Search: Build an application that finds items using both images and descriptive text -- by combining both sparse and dense vector representations of the data -- using an eCommerce dataset as an example. (v) Image Similarity: Use image vector embeddings to create an app to compare facial features, using a database of public figures to determine the likeness between them. (vi) Anomaly Detection: Build an anomaly detection app that identifies unusual patterns in network communication logs. I hope you’ll enjoy learning how to build all these types of applications! Please sign up here:

Andrew Ng

137,169 次观看 • 2 年前

MiniMax H3 is native in ComfyUI, day zero with the weights. Model highlights: → Text-to-video, prompt only → Image-to-video → First-and-last-frame - control the opening frame, the closing frame, or both → Reference-to-video - carry a subject, a motion, or a voice through the clip → In-place editing - modify a shot you already have instead of regenerating it The capability MiniMax (official) leads with is multimodal context understanding, and it's what collapses those five tasks into one model. H3 takes images, audio, and video together and resolves them against a prompt that explains how they relate. Download the workflows & learn more in our blog 👇

MiniMax H3 is native in ComfyUI, day zero with the weights. Model highlights: → Text-to-video, prompt only → Image-to-video → First-and-last-frame - control the opening frame, the closing frame, or both → Reference-to-video - carry a subject, a motion, or a voice through the clip → In-place editing - modify a shot you already have instead of regenerating it The capability MiniMax (official) leads with is multimodal context understanding, and it's what collapses those five tasks into one model. H3 takes images, audio, and video together and resolves them against a prompt that explains how they relate. Download the workflows & learn more in our blog 👇

ComfyUI

235,460 次观看 • 3 个月前

Tokenization -- turning text into a sequence of integers -- is a key part of generative AI, and most API providers charge per million tokens. How does tokenization work? Learn the details of tokenization and RAG optimization in Retrieval Optimization: From Tokenization to Vector Quantization, created in collaboration with Qdrant and taught by its Developer Relations Lead, Kacper Łukawski. This course focuses on Retrieval augmented generation (RAG), which has two steps: First, a retriever finds relevant information; then, the generator uses what’s retrieved as context to produce a response. You’ll learn to optimize the first step (the retriever) by understanding how tokenization works and how it impacts the relevance of your search. In addition, you will also learn to measure and improve retrieval quality, speed, and memory. In detail, you’ll: - Learn about the internal workings of the embedding models and how your text turns into vectors. - Understand how several tokenizers, such as Byte-Pair Encoding, WordPiece, Unigram, and SentencePiece work. - Explore common challenges with tokenizers, such as unknown tokens, domain-specific identifiers, and numerical values, that can negatively affect your vector search. - Understand how to measure the quality of your search across relevance, ranking, and score-related metrics. - Understand how the main parameters in "HNSW", a graph-based algorithm, affect the relevance and speed of vector search, and how to tune its parameters. - Experiment with the three major quantization methods – product, scalar, and binary – and learn how they impact memory requirements, search quality, and speed. By the end of this course, you’ll have a solid understanding of how tokenization functions and how to optimize vector search in your RAG systems. Please sign up here!

Tokenization -- turning text into a sequence of integers -- is a key part of generative AI, and most API providers charge per million tokens. How does tokenization work? Learn the details of tokenization and RAG optimization in Retrieval Optimization: From Tokenization to Vector Quantization, created in collaboration with Qdrant and taught by its Developer Relations Lead, Kacper Łukawski. This course focuses on Retrieval augmented generation (RAG), which has two steps: First, a retriever finds relevant information; then, the generator uses what’s retrieved as context to produce a response. You’ll learn to optimize the first step (the retriever) by understanding how tokenization works and how it impacts the relevance of your search. In addition, you will also learn to measure and improve retrieval quality, speed, and memory. In detail, you’ll: - Learn about the internal workings of the embedding models and how your text turns into vectors. - Understand how several tokenizers, such as Byte-Pair Encoding, WordPiece, Unigram, and SentencePiece work. - Explore common challenges with tokenizers, such as unknown tokens, domain-specific identifiers, and numerical values, that can negatively affect your vector search. - Understand how to measure the quality of your search across relevance, ranking, and score-related metrics. - Understand how the main parameters in "HNSW", a graph-based algorithm, affect the relevance and speed of vector search, and how to tune its parameters. - Experiment with the three major quantization methods – product, scalar, and binary – and learn how they impact memory requirements, search quality, and speed. By the end of this course, you’ll have a solid understanding of how tokenization functions and how to optimize vector search in your RAG systems. Please sign up here!

Andrew Ng

146,313 次观看 • 2 年前

Web scraping will never be the same. (100% open-source visual search at scale) PixelRAG is a retrieval system that skips HTML parsing completely. Instead of scraping a page into text and embedding chunks, it screenshots the page and retrieves the image. A vision-language model reads the answer straight off the pixels. Why that matters: parsing is where web RAG quietly loses information. - A single HTML-to-text parser can drop 40%+ of a page. - Tables, charts, and layout get flattened or thrown out. - Swapping parsers alone can move accuracy ~10 points on the same docs. PixelRAG indexes the page a person actually sees. The team built a visual index of all of Wikipedia, 30M+ screenshots, and it still beats the strongest text RAG baseline by 18.1% on text-only QA. The repo also ships a Claude Code plugin that gives Claude eyes. It lets Claude screenshot any URL and read the rendered page instead of scraping the DOM. So you can hand it a live page, an arXiv paper, or your local site and ask what it actually looks like. One setup script. No MCP server, no backend. How the pipeline works: - Renders each document (web, PDF, image) to image tiles. - Embeds them with Qwen3-VL-Embedding, LoRA fine-tuned on screenshots. - Builds a FAISS index and serves a search API. A stronger reader model lifts accuracy with no re-indexing, since the index is just pixels. Everything is open-source under Apache-2.0. GitHub repo: Talking about RAG, I recently wrote an article on a new approach that makes retrieval much more efficient by cutting corpus size by 40x, reducing tokens per query by 3x, and improving vector search relevance by 2.3x. The article is quoted below.

Web scraping will never be the same. (100% open-source visual search at scale) PixelRAG is a retrieval system that skips HTML parsing completely. Instead of scraping a page into text and embedding chunks, it screenshots the page and retrieves the image. A vision-language model reads the answer straight off the pixels. Why that matters: parsing is where web RAG quietly loses information. - A single HTML-to-text parser can drop 40%+ of a page. - Tables, charts, and layout get flattened or thrown out. - Swapping parsers alone can move accuracy ~10 points on the same docs. PixelRAG indexes the page a person actually sees. The team built a visual index of all of Wikipedia, 30M+ screenshots, and it still beats the strongest text RAG baseline by 18.1% on text-only QA. The repo also ships a Claude Code plugin that gives Claude eyes. It lets Claude screenshot any URL and read the rendered page instead of scraping the DOM. So you can hand it a live page, an arXiv paper, or your local site and ask what it actually looks like. One setup script. No MCP server, no backend. How the pipeline works: - Renders each document (web, PDF, image) to image tiles. - Embeds them with Qwen3-VL-Embedding, LoRA fine-tuned on screenshots. - Builds a FAISS index and serves a search API. A stronger reader model lifts accuracy with no re-indexing, since the index is just pixels. Everything is open-source under Apache-2.0. GitHub repo: Talking about RAG, I recently wrote an article on a new approach that makes retrieval much more efficient by cutting corpus size by 40x, reducing tokens per query by 3x, and improving vector search relevance by 2.3x. The article is quoted below.

Akshay 🚀

950,119 次观看 • 4 个月前

Andrew Ng + Google showed the foundation behind modern "RAG", "Graph RAG" and semantic retrieval: • 12:00 - turning text into embeddings that capture meaning • 24:00 - visualizing semantic relationships between vectors • 35:00 - using embeddings for classification, clustering and outlier detection • 50:00 - controlling LLM generation with temperature, top-k and top-p • 1:04:00 - semantic search with ScaNN • 1:15:00 - combining retrieval + an LLM into a RAG-style Q&A system - 84-minute deep dive, and one of the clearest explanations of what happens underneath retrieval systems The progression is simple: Text → Embeddings → Similarity → Semantic Search → Retrieval → LLM Answer Most people jump straight into RAG frameworks without understanding this layer Andrew Ng starts one level lower and shows why retrieval works in the first place Watch it today, then read the full "RAG → Context Graphs" roadmap below

Andrew Ng + Google showed the foundation behind modern "RAG", "Graph RAG" and semantic retrieval: • 12:00 - turning text into embeddings that capture meaning • 24:00 - visualizing semantic relationships between vectors • 35:00 - using embeddings for classification, clustering and outlier detection • 50:00 - controlling LLM generation with temperature, top-k and top-p • 1:04:00 - semantic search with ScaNN • 1:15:00 - combining retrieval + an LLM into a RAG-style Q&A system - 84-minute deep dive, and one of the clearest explanations of what happens underneath retrieval systems The progression is simple: Text → Embeddings → Similarity → Semantic Search → Retrieval → LLM Answer Most people jump straight into RAG frameworks without understanding this layer Andrew Ng starts one level lower and shows why retrieval works in the first place Watch it today, then read the full "RAG → Context Graphs" roadmap below

Morlex

34,790 次观看 • 3 个月前

HE MAKES MONEY IN REAL ESTATE WITHOUT BUYING, SELLING, OR EVEN SEEING A SINGLE HOUSE. HERE'S THE EXACT SETUP He never owns a property. He takes a single listing, turns it into a polished 30-second video, and sells that to the agent who posted it. Realtors need video for their feeds and almost none of them can make it. He sits in the middle and builds the whole thing once as a skill that runs on command Here is the exact process: 1. Pull the listing. Go to Zillow, open any listing, download the high-res images, and grab the property info. That is your raw material 2. Turn photos into video with Google Veo. Get a Google API key for Veo, the image-to-video model. It takes the listing photos and animates them into clean 30-second footage. This is the best one out right now 3. Add the voice with ElevenLabs. Get an ElevenLabs API key. Feed it the listing details and it returns a voiceover that sounds like a real human, not a robot. Lay it over the video with the text on screen 4. Send it with AgentMail. Get an AgentMail key so the system can send the finished email out on its own Then you wire it into one skill. Scrape the listing, send images to Veo, add the ElevenLabs voiceover and on-screen text, then send the email. Feed it each key one at a time and have it build each step Who you sell to: Pull realtors off Zillow and Realtor com whose listings have flat photos and zero video. That gap is your pitch. Send a free sample made from their own listing first, then charge a monthly rate for ongoing clips. One agent with ten listings is a recurring client, fully online Bookmark this

HE MAKES MONEY IN REAL ESTATE WITHOUT BUYING, SELLING, OR EVEN SEEING A SINGLE HOUSE. HERE'S THE EXACT SETUP He never owns a property. He takes a single listing, turns it into a polished 30-second video, and sells that to the agent who posted it. Realtors need video for their feeds and almost none of them can make it. He sits in the middle and builds the whole thing once as a skill that runs on command Here is the exact process: 1. Pull the listing. Go to Zillow, open any listing, download the high-res images, and grab the property info. That is your raw material 2. Turn photos into video with Google Veo. Get a Google API key for Veo, the image-to-video model. It takes the listing photos and animates them into clean 30-second footage. This is the best one out right now 3. Add the voice with ElevenLabs. Get an ElevenLabs API key. Feed it the listing details and it returns a voiceover that sounds like a real human, not a robot. Lay it over the video with the text on screen 4. Send it with AgentMail. Get an AgentMail key so the system can send the finished email out on its own Then you wire it into one skill. Scrape the listing, send images to Veo, add the ElevenLabs voiceover and on-screen text, then send the email. Feed it each key one at a time and have it build each step Who you sell to: Pull realtors off Zillow and Realtor com whose listings have flat photos and zero video. That gap is your pitch. Send a free sample made from their own listing first, then charge a monthly rate for ongoing clips. One agent with ten listings is a recurring client, fully online Bookmark this

Yarchi

106,174 次观看 • 5 个月前

Local. Open weights. Native 4K. LTX-2 is now a 100% Open Source AI video model, and I tested it on my rig! Installation, VRAM usage, and the prompts I used for this video, below 👇

Local. Open weights. Native 4K. LTX-2 is now a 100% Open Source AI video model, and I tested it on my rig! Installation, VRAM usage, and the prompts I used for this video, below 👇

TechHalla

35,380 次观看 • 10 个月前

Vector databases are ridiculous. How can you search millions of vectors at light speed? What's the magic making this possible? I wanted to understand how they work, and here is a full video with everything I learned. This is for nerds.

Vector databases are ridiculous. How can you search millions of vectors at light speed? What's the magic making this possible? I wanted to understand how they work, and here is a full video with everything I learned. This is for nerds.

Santiago

231,463 次观看 • 2 年前

I turned Google Gemma EmbeddingGemma 2, an embedding model, into a speech-to-text model, and it turned out to be one of the fastest STT models you can run on device. On a base M3 MacBook Air it transcribes an hour of audio in about 13 seconds: 288× real time. The idea came in March with Gemini Embedding 2. It puts text, images, audio and video into one shared space, so a spoken word lands right next to the written word. If that's true, you should be able to read a transcript straight out of the embedding model. I couldnt tried it then, but last week EmbeddingGemma 2 came out as an open model, so I tried it.

I turned Google Gemma EmbeddingGemma 2, an embedding model, into a speech-to-text model, and it turned out to be one of the fastest STT models you can run on device. On a base M3 MacBook Air it transcribes an hour of audio in about 13 seconds: 288× real time. The idea came in March with Gemini Embedding 2. It puts text, images, audio and video into one shared space, so a spoken word lands right next to the written word. If that's true, you should be able to read a transcript straight out of the embedding model. I couldnt tried it then, but last week EmbeddingGemma 2 came out as an open model, so I tried it.

Shahir

298,236 次观看 • 1 个月前

This is probably the most complex workflow I’ve ever built, only with open-source tools. It took my 4 days. It takes four inputs: author, title, and style; and generates a full visual animated story in one click in ComfyUI . I worked on it for four days. There are still some bugs, but here’s the first preview. Here’s a quick breakdown: - The four inputs are sent to LLMs with precise instructions to generate: first, prompts for images and image modifications; second, prompts for animations; third, prompts for generating music. - All voices are generated from the text and timed precisely, as they determine the length of each animation segment. - The first image and video are generated to serve as the title, but also as the guide for all other images created for the video. - Titles and subtitles are also added automatically in Comfy. - I also developed a lot of custom nodes for minor frame calculations, mostly to match audio and video. - The full system is a large loop that, for each line of text, generates an image and then a video from that image. The loop was the hardest part to build in this workflow, so it can process either a 20-second video or a 2-minute video with the same input. - There are multiple combinations of LLMs that try to understand the text in the best way to provide the best prompts for images and video. - The final video is assembled entirely within ComfyUI. - The music is generated based on the LLM output and matches the exact timing of the full animation. - Done! For reference, this workflow uses a lot of models and only works on an RTX 6000 Pro with plenty of RAM. My goal is not to replace humans, as I’ll try to explain later, this workflow is highly controlled and can be adapted or reworked at any point by real artists! My aim was to create a tool that can animate text in one go, allowing the AI some freedom while keeping a strict flow. I don’t know yet how I’ll share this workflow with people, I still need to polish it properly, but maybe through Patreon. Anyway, I hope you enjoy my research, and let’s always keep pushing further! :)

This is probably the most complex workflow I’ve ever built, only with open-source tools. It took my 4 days. It takes four inputs: author, title, and style; and generates a full visual animated story in one click in ComfyUI . I worked on it for four days. There are still some bugs, but here’s the first preview. Here’s a quick breakdown: - The four inputs are sent to LLMs with precise instructions to generate: first, prompts for images and image modifications; second, prompts for animations; third, prompts for generating music. - All voices are generated from the text and timed precisely, as they determine the length of each animation segment. - The first image and video are generated to serve as the title, but also as the guide for all other images created for the video. - Titles and subtitles are also added automatically in Comfy. - I also developed a lot of custom nodes for minor frame calculations, mostly to match audio and video. - The full system is a large loop that, for each line of text, generates an image and then a video from that image. The loop was the hardest part to build in this workflow, so it can process either a 20-second video or a 2-minute video with the same input. - There are multiple combinations of LLMs that try to understand the text in the best way to provide the best prompts for images and video. - The final video is assembled entirely within ComfyUI. - The music is generated based on the LLM output and matches the exact timing of the full animation. - Done! For reference, this workflow uses a lot of models and only works on an RTX 6000 Pro with plenty of RAM. My goal is not to replace humans, as I’ll try to explain later, this workflow is highly controlled and can be adapted or reworked at any point by real artists! My aim was to create a tool that can animate text in one go, allowing the AI some freedom while keeping a strict flow. I don’t know yet how I’ll share this workflow with people, I still need to polish it properly, but maybe through Patreon. Anyway, I hope you enjoy my research, and let’s always keep pushing further! :)

Lovis Odin

58,950 次观看 • 1 年前