正在加载视频...

视频加载失败

X bookmarks, but: - Vector + text search (username + name + content) - Sort + filter by engagement, post date, media

95,560 次观看 • 1 年前 •via X (Twitter)

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Big moment for Postgres! Search has always been Postgres' weak spot, and everyone just accepted it. If you needed a real relevance-ranked keyword search, the default answer was to spin up Elasticsearch or add Algolia and deal with the data sync headaches forever. The problem isn't that Postgres can't do text search. It can. But the built-in `ts_rank` function uses a basic term frequency algorithm that doesn't come close to what modern search engines deliver. So teams end up: - Running a separate Elasticsearch cluster just for search - Building sync pipelines that inevitably drift out of consistency - Paying for managed search services that charge per query - Accepting mediocre search relevance because "good enough" ships faster But this is actually a solvable problem. You can realistically bring industry-standard search ranking directly into Postgres, which eliminates the need for external infra entirely. This exact solution is now available with the newly open-sourced pg_textsearch by Tiger Data - Creators of TimescaleDB, a Postgres extension that brings true BM25 relevance ranking into the database. BM25 is the algorithm behind Elasticsearch, Lucene, and most modern search engines. Now it runs natively in Postgres. Here's what pg_textsearch enables: - True BM25 ranking with configurable parameters (the same algorithm powering production search systems) - Simple SQL syntax: `ORDER BY content 'search terms'` - Works with Postgres text search configurations for multiple languages - Pairs naturally with pgvector for hybrid keyword + semantic search That last point matters a lot for RAG apps. The video below shows this in action, and I worked with the team to put this together. You can now do hybrid retrieval (combining keyword matching with vector similarity) in a single database, without stitching together multiple systems. The syntax is clean enough that you can add relevance-ranked search to existing queries in minutes. pg_textsearch is fully open-source under the PostgreSQL license. You can find a link to their GitHub repo in the next tweet.

Akshay 🚀

215,344 次观看 • 5 个月前

Regarding the Eric Trump / Daniel Cormier “text scandal”, I am open to being convinced otherwise, but I do not believe for one second that Eric Trump would text something so stupid to Daniel Cormier. It’s laughably stupid on its face to think that Eric Trump would ask a UFC commentator if any of the fights are “rigged”. That he would ask it in writing. And that he would not, for example, just pick up the phone and call Dana White or Joe Rogan himself. That said, I just want to understand what happened. Daniel Cormier said he was hacked. Is that to say he was hacked at the text level, or the X post level? Did he in fact make the post, and delete it? If he made the post, did he do it as a joke to illustrate how absurd the text hack was, but realized how the post could be misinterpreted as real? Or did someone make the post from his X account he subsequently deleted it. I know there are some who are going to say he made the post in earnest, thinking the text messages were real, intending to call out the “corruption”, and he has since been threatened, deleted the post, and claimed “hack”, etc. I can tell you exactly why I do not believe, even if he made the post, he did it believing the text message was real, intending to call out the “corruption”. If Cormier seriously thought the text messages were real, he would understand that by posting it publicly, it would literally destroy the UFC, his livelihood and his legacy - in addition to the legacy of everyone who has ever competed in the UFC. There is no realm of the universe in which even if it were in fact a real text that he actually received from Eric Trump, he would post it publicly. Because it would mean the end of the UFC. I just want to know if he actually made the post from his X account, or if the entire post was the result of a hack (hence the subsequent deleting of the post). Serious and sincere questions for clarity that would benefit everyone Eric Trump

Viva Frei

30,167 次观看 • 29 天前

Vector Database by Hand ✍️ Vector databases are revolutionizing how we search and analyze complex data. They have become the backbone of Retrieval Augmented Generation (#RAG). How do vector databases work? [1] Given ↳ A dataset of three sentences, each has 3 words (or tokens) ↳ In practice, a dataset may contain millions or billions of sentences. The max number of tokens may be tens of thousands (e.g., 32,768 mistral-7b). Process "how are you" [2] 🟨 Word Embeddings ↳ For each word, look up corresponding word embedding vector from a table of 22 vectors, where 22 is the vocabulary size. ↳ In practice, the vocabulary size can be tens of thousands. The word embedding dimensions are in the thousands (e.g., 1024, 4096) [3] 🟩 Encoding ↳ Feed the sequence of word embeddings to an encoder to obtain a sequence of feature vectors, one per word. ↳ Here, the encoder is a simple one layer perceptron (linear layer + ReLU) ↳ In practice, the encoder is a transformer or one of its many variants. [4] 🟩 Mean Pooling ↳ Merge the sequence of feature vectors into a single vector using "mean pooling" which is to average across the columns. ↳ The result is a single vector. We often call it "text embeddings" or "sentence embeddings." ↳ Other pooling techniques are possible, such as CLS. But mean pooling is the most common. [5] 🟦 Indexing ↳ Reduce the dimensions of the text embedding vector by a projection matrix. The reduction rate is 50% (4->2). ↳ In practice, the values in this projection matrix is much more random. ↳ The purpose is similar to that of hashing, which is to obtain a short representation to allow faster comparison and retrieval. ↳ The resulting dimension-reduced index vector is saved in the vector storage. [6] Process "who are you" ↳ Repeat [2]-[5] [7] Process "who am I" ↳ Repeat [2]-[5] Now we have indexed our dataset in the vector database. [8] 🟥 Query: "am I you" ↳ Repeat [2]-[5] ↳ The result is a 2-d query vector. [9] 🟥 Dot Products ↳ Take dot product between the query vector and database vectors. They are all 2-d. ↳ The purpose is to use dot product to estimate similarity. ↳ By transposing the query vector, this step becomes a matrix multiplication. [10] 🟥 Nearest Neighbor ↳ Find the largest dot product by linear scan. ↳ The sentence with the highest dot product is "who am I" ↳ In practice, because scanning billions of vectors is slow, we use an Approximate Nearest Neighbor (ANN) algorithm like the Hierarchical Navigable Small Worlds (HNSW).

Tom Yeh

191,974 次观看 • 2 年前