Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

🚨Introducing a new database: PGX 🐘 Postgres for AI agents. In Rust. One runtime. Thousands of Postgres databases. ⚡ Up to 100× less idle memory ⚡ 1,000 databases in just 659 MB RAM ⚡ Clone a 674 MB database in 85 ms ⚡ ~0.7 MB per idle database RUST....

96,747 Aufrufe • vor 3 Tagen •via X (Twitter)

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Talking with someone the other day that estimated they had about 25,000 idle connections to Postgres. My actual response to them: "holy shit". They double checked, it was only about 12,000 Same day had a conversation with someone saying they didn't need pgbouncer because of activerecord's connection pooling. Let's dig into connection pooling in Postgres. Prior to Postgres 14 every connection to the database consumed memory, roughly 10MB, it may be slightly less but it still wasn't free. Even beyond Postgres 14 there is still various contention that happens when Postgres starts to use a connection. An application pooler maintains a set of connections and hands them out when needed on the application side. These are idle and real connections against the database that indeed do impact performance negatively. In contrast pgbouncer speaks the wire protocol, waits for the begin part of the transaction and then uses a connection. It more strictly manages how many idle ones it's having instead of per web server you're running. pgbouncer up until recently really needed to be run in transaction mode (which meant disabling prepared statements in your application framework). prepared_statement support in pgbouncer was added recently, and now you don't have to disable. Even when running with an older version of pgbouncer with prepared_statements disabled you'd still see a big performance gain. A quick check to know if you'd benefit from pgbouncer, run this query: SELECT count(*), state FROM pg_stat_activity GROUP BY 2; If you're idle account is high (yes this is dependent on your view, but to me if it's above 25-30 range, and especially if active is until half that) then you'd already start to benefit from pgbouncer. If it's at 10,000 then post haste get pgbouncer in place. Finally, you don't have to not use a framework pooler, they're fine, but don't think it replaces a native Postgres connection pooler.

Craig Kerstiens

12,411 Aufrufe • vor 2 Jahren

Big moment for Postgres! Search has always been Postgres' weak spot, and everyone just accepted it. If you needed a real relevance-ranked keyword search, the default answer was to spin up Elasticsearch or add Algolia and deal with the data sync headaches forever. The problem isn't that Postgres can't do text search. It can. But the built-in `ts_rank` function uses a basic term frequency algorithm that doesn't come close to what modern search engines deliver. So teams end up: - Running a separate Elasticsearch cluster just for search - Building sync pipelines that inevitably drift out of consistency - Paying for managed search services that charge per query - Accepting mediocre search relevance because "good enough" ships faster But this is actually a solvable problem. You can realistically bring industry-standard search ranking directly into Postgres, which eliminates the need for external infra entirely. This exact solution is now available with the newly open-sourced pg_textsearch by Tiger Data - Creators of TimescaleDB, a Postgres extension that brings true BM25 relevance ranking into the database. BM25 is the algorithm behind Elasticsearch, Lucene, and most modern search engines. Now it runs natively in Postgres. Here's what pg_textsearch enables: - True BM25 ranking with configurable parameters (the same algorithm powering production search systems) - Simple SQL syntax: `ORDER BY content 'search terms'` - Works with Postgres text search configurations for multiple languages - Pairs naturally with pgvector for hybrid keyword + semantic search That last point matters a lot for RAG apps. The video below shows this in action, and I worked with the team to put this together. You can now do hybrid retrieval (combining keyword matching with vector similarity) in a single database, without stitching together multiple systems. The syntax is clean enough that you can add relevance-ranked search to existing queries in minutes. pg_textsearch is fully open-source under the PostgreSQL license. You can find a link to their GitHub repo in the next tweet.

Akshay 🚀

215,785 Aufrufe • vor 9 Monaten

Holy shit... Keygraph just built an AI that hacks your web app before hackers do. It's called Shannon and it's a fully autonomous AI pentester that finds REAL exploits, not just alerts. 96.15% success rate on the hint-free XBOW Benchmark. Your team ships code every day with Claude Code and Cursor. Your pentest? Once a year. That's 364 days of shipping vulnerabilities to production. Shannon closes that gap. What it actually does: → Autonomously hunts attack vectors in your source code → Uses a built-in browser to execute real exploits → Handles 2FA/TOTP logins with zero intervention → Delivers copy-paste Proof-of-Concepts (no false positives) → Runs Nmap, Subfinder, WhatWeb, Schemathesis under the hood Real results on OWASP Juice Shop in a single run: → 20+ high-impact vulnerabilities found → Complete auth bypass + full database exfiltration → Privilege escalation to admin via registration bypass → SSRF enabling internal network recon → Systemic IDOR across user data The architecture is what makes it work. 4 phases: Recon → Vuln Analysis → Exploitation → Reporting Specialized agents run in parallel for Injection, XSS, SSRF, and Broken Auth. Strict "No Exploit, No Report" policy kills false positives at the source. Covers the critical OWASP classes: - Injection - XSS - SSRF - Broken Authentication & Authorization One command. ~1 hour runtime. ~$50 per full pentest with Claude Sonnet. Every Claude (coder) deserves their Shannon. The Red Team to your vibe-coding Blue Team. 100% Opensource (AGPL-3.0). 10.6k stars already. Repo in reply ↓

Guri Singh

20,591 Aufrufe • vor 6 Monaten

🚨ICE agents are illegally stopping U.S. citizens and demanding they get out of their cars to “prove” their citizenship… in Shakopee, Minnesota. In the video, multiple unmarked agent vehicles have a man surrounded while they run his information through a federal database. A woman drives by, sees what’s happening, and asks the man his name. He answers her… and says he’s a U.S. citizen. And yet, he’s still being treated like he has to justify existing in his own community. And just to clarify… Agents cannot just stop someone, demand identification, and hold them while they “check” someone’s status without reasonable suspicion of a crime. Driving while being a POC is not reasonable suspicion. Federal agents do not get to turn neighborhoods into checkpoints because they feel like it. When armed agents surround someone, run them through a database, and force them to prove who they are without explaining why they’re being detained… that is a violation of their constitutional rights. Rights don’t hinge on whether an agent feels suspicious that day. When law enforcement believes the Constitution is optional, enforcement turns into intimidation. Stops turn into fishing expeditions. Detentions turn into threats. Databases turn into weapons. Power expands to the edge of whatever no one pushes back on. And once that line moves, it does not move back on its own. That’s what people should be paying attention to.

Jesus Freakin Congress

156,099 Aufrufe • vor 8 Monaten

Vector Database by Hand ✍️ Vector databases are revolutionizing how we search and analyze complex data. They have become the backbone of Retrieval Augmented Generation (#RAG). How do vector databases work? [1] Given ↳ A dataset of three sentences, each has 3 words (or tokens) ↳ In practice, a dataset may contain millions or billions of sentences. The max number of tokens may be tens of thousands (e.g., 32,768 mistral-7b). Process "how are you" [2] 🟨 Word Embeddings ↳ For each word, look up corresponding word embedding vector from a table of 22 vectors, where 22 is the vocabulary size. ↳ In practice, the vocabulary size can be tens of thousands. The word embedding dimensions are in the thousands (e.g., 1024, 4096) [3] 🟩 Encoding ↳ Feed the sequence of word embeddings to an encoder to obtain a sequence of feature vectors, one per word. ↳ Here, the encoder is a simple one layer perceptron (linear layer + ReLU) ↳ In practice, the encoder is a transformer or one of its many variants. [4] 🟩 Mean Pooling ↳ Merge the sequence of feature vectors into a single vector using "mean pooling" which is to average across the columns. ↳ The result is a single vector. We often call it "text embeddings" or "sentence embeddings." ↳ Other pooling techniques are possible, such as CLS. But mean pooling is the most common. [5] 🟦 Indexing ↳ Reduce the dimensions of the text embedding vector by a projection matrix. The reduction rate is 50% (4->2). ↳ In practice, the values in this projection matrix is much more random. ↳ The purpose is similar to that of hashing, which is to obtain a short representation to allow faster comparison and retrieval. ↳ The resulting dimension-reduced index vector is saved in the vector storage. [6] Process "who are you" ↳ Repeat [2]-[5] [7] Process "who am I" ↳ Repeat [2]-[5] Now we have indexed our dataset in the vector database. [8] 🟥 Query: "am I you" ↳ Repeat [2]-[5] ↳ The result is a 2-d query vector. [9] 🟥 Dot Products ↳ Take dot product between the query vector and database vectors. They are all 2-d. ↳ The purpose is to use dot product to estimate similarity. ↳ By transposing the query vector, this step becomes a matrix multiplication. [10] 🟥 Nearest Neighbor ↳ Find the largest dot product by linear scan. ↳ The sentence with the highest dot product is "who am I" ↳ In practice, because scanning billions of vectors is slow, we use an Approximate Nearest Neighbor (ANN) algorithm like the Hierarchical Navigable Small Worlds (HNSW).

Tom Yeh

192,216 Aufrufe • vor 2 Jahren