Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Let Postgres own the Iceberg catalog and delegate analytics to DuckDB. The result => transactional lakehouse updates with fast analytical queries. This isn’t a concept. It’s exactly what pg_lake delivers today. pg_lake combines a set of extensions and components that let you query and modify Iceberg tables (and other...

denismagda's profile picture

Denis Magda

5,025 subscribers

28,461 Aufrufe • vor 4 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Your Postgres is 100x slower than traditional OLAP engines. A deceptively simple OSS extension fixes this. Here's an interview where we dive into the deep engineering around how this is achieved. Joining me (and leading the conversation) is Marco Slot: an engineer with an EXTENSIVE and impressive career history around PostgreSQL: 👉 Created pg_cron in 2017 (3.7k stars) - a tool to run cron-jobs in Postgres 👉 Built pg_incremental - fast, reliable, incremental batch processing inside PostgreSQL itself 👉 co-created pg_lake (after working on Crunchy Data's Warehouse, and getting acquired into Snowflake) 👉 Helped get pg_documentdb (MongoDB-on-Postgres) off the ground Marco Slot is a world-class expert in Postgres extensions. He seriously impressed me with his knowledge over the course of a private LinkedIn conversation, and now that I type out his resume - I understand where it came from. He should be on everyone's radar. So I brought him on the pod. In our full 2-hour deep-dive, we went over: • 🔥 how pg_lake makes analytics 100x faster (literally) • 🔥 perf internals like vectorized execution & CPU branching • 🤔 practical differences between OLTP and OLAP database development (and the age-old mission in uniting both) • 🤔 how (and why) pg_lake intercepts query plans and delegates parts of the query tree to DuckDB • 💡 why Postgres is architecturally terrible at analytical queries (and how vectorized execution fixes this) • 💡 Marco's hard-won experience through a decade+ career in Postgres • 🏆 Iceberg's role as the TCP/IP for tables • 🏆 what the real moat of PostgreSQL is Developments like pg_lake are a real reason why "Just Use Postgres" is much more than a meme, and it'll continue to dominate discourse. I promise you will learn a lot from this episode. Timestamps: (0:02) What is pg_lake? (2:23) Postgres' 100x slower problem and columnar storage experiments they had to make Postgres fast for analytics (6:00) practical examples and internals (16:20) perf internals - vectorized execution & CPU optimization (23:00) pg_lake architecture (why DuckDB isn't embedded) and the connection-per-process issue (29:16) how pg_lake intercepts the query plan tree and delegates parts to DuckDB (41:09) Iceberg catalogs (48:24) postgres to iceberg ingestion patterns (and pg_incremental) (53:40) Marco's (long) career: early AWS, Citus, Microsoft, Crunchy Data & Snowflake (1:04:20) Marco's observations around the merging between OLTP and OLAP (and the subtle dev differences there) (1:15:30) reverse ETL (1:33:08) Iceberg as the TCP/IP for tables (1:35:00) Marco's thoughts on the "Just Use Postgres" fever
1:39:30

Your Postgres is 100x slower than traditional OLAP engines. A deceptively simple OSS extension fixes this. Here's an interview where we dive into the deep engineering around how this is achieved. Joining me (and leading the conversation) is Marco Slot: an engineer with an EXTENSIVE and impressive career history around PostgreSQL: 👉 Created pg_cron in 2017 (3.7k stars) - a tool to run cron-jobs in Postgres 👉 Built pg_incremental - fast, reliable, incremental batch processing inside PostgreSQL itself 👉 co-created pg_lake (after working on Crunchy Data's Warehouse, and getting acquired into Snowflake) 👉 Helped get pg_documentdb (MongoDB-on-Postgres) off the ground Marco Slot is a world-class expert in Postgres extensions. He seriously impressed me with his knowledge over the course of a private LinkedIn conversation, and now that I type out his resume - I understand where it came from. He should be on everyone's radar. So I brought him on the pod. In our full 2-hour deep-dive, we went over: • 🔥 how pg_lake makes analytics 100x faster (literally) • 🔥 perf internals like vectorized execution & CPU branching • 🤔 practical differences between OLTP and OLAP database development (and the age-old mission in uniting both) • 🤔 how (and why) pg_lake intercepts query plans and delegates parts of the query tree to DuckDB • 💡 why Postgres is architecturally terrible at analytical queries (and how vectorized execution fixes this) • 💡 Marco's hard-won experience through a decade+ career in Postgres • 🏆 Iceberg's role as the TCP/IP for tables • 🏆 what the real moat of PostgreSQL is Developments like pg_lake are a real reason why "Just Use Postgres" is much more than a meme, and it'll continue to dominate discourse. I promise you will learn a lot from this episode. Timestamps: (0:02) What is pg_lake? (2:23) Postgres' 100x slower problem and columnar storage experiments they had to make Postgres fast for analytics (6:00) practical examples and internals (16:20) perf internals - vectorized execution & CPU optimization (23:00) pg_lake architecture (why DuckDB isn't embedded) and the connection-per-process issue (29:16) how pg_lake intercepts the query plan tree and delegates parts to DuckDB (41:09) Iceberg catalogs (48:24) postgres to iceberg ingestion patterns (and pg_incremental) (53:40) Marco's (long) career: early AWS, Citus, Microsoft, Crunchy Data & Snowflake (1:04:20) Marco's observations around the merging between OLTP and OLAP (and the subtle dev differences there) (1:15:30) reverse ETL (1:33:08) Iceberg as the TCP/IP for tables (1:35:00) Marco's thoughts on the "Just Use Postgres" fever

Stanislav Kozlovski

16,620 Aufrufe • vor 2 Monaten

We're excited to announce duckdb-skills, a DuckDB plugin for Claude Code! We think the embedded nature of DuckDB makes it a perfect companion for Claude in your local workflows. The skills supported include: + read-file and query – uses DuckDB's CLI to query data locally, unlocking easy access to any file that DuckDB can read. + read-memories – a clever idea to store your Claude memories in DuckDB and query them at blazing speed. These are powered by two additional skills: + attach-db – gives Claude a mechanism to manage DuckDB state through a .sql file linked to your project. + duckdb-docs – uses a remote DuckDB full-text search database to query the DuckDB docs and answer all of your (and Claude's own) questions.
2:16

We're excited to announce duckdb-skills, a DuckDB plugin for Claude Code! We think the embedded nature of DuckDB makes it a perfect companion for Claude in your local workflows. The skills supported include: + read-file and query – uses DuckDB's CLI to query data locally, unlocking easy access to any file that DuckDB can read. + read-memories – a clever idea to store your Claude memories in DuckDB and query them at blazing speed. These are powered by two additional skills: + attach-db – gives Claude a mechanism to manage DuckDB state through a .sql file linked to your project. + duckdb-docs – uses a remote DuckDB full-text search database to query the DuckDB docs and answer all of your (and Claude's own) questions.

DuckDB

60,551 Aufrufe • vor 2 Monaten

sqlit is a lazygit style SQL TUI for the terminal. It supports 10+ databases (Postgres, SQLite, Supabase, ClickHouse, DuckDB, etc), has query history, autocomplete and more. Peter Adams (Maxteabag on GitHub) made sqlit using textualize.io and is Terminal Tool of the Week! ⭐️
1:36

sqlit is a lazygit style SQL TUI for the terminal. It supports 10+ databases (Postgres, SQLite, Supabase, ClickHouse, DuckDB, etc), has query history, autocomplete and more. Peter Adams (Maxteabag on GitHub) made sqlit using textualize.io and is Terminal Tool of the Week! ⭐️

Terminal Trove

137,289 Aufrufe • vor 5 Monaten

Flamegraphs are a great way to narrow in on performance issues + broadly to explore program execution. I built a custom viewer for Postgres queries, inspired Jan Nidzwetzki's excellent blog on the subject. Highly recommend analysis tools like this for learning query execution.
0:45

Flamegraphs are a great way to narrow in on performance issues + broadly to explore program execution. I built a custom viewer for Postgres queries, inspired Jan Nidzwetzki's excellent blog on the subject. Highly recommend analysis tools like this for learning query execution.

Ben Dicken

47,763 Aufrufe • vor 2 Monaten

Just Use Postgres feat. Denis Magda Denis literally just published a book called “Just Use Postgres”. In this conversation, that turned out to be super fun, we talked about: • what MySQL got wrong w.r.t community • the recent explosion of Postgres development in the world (Supabase, Neon, Crunchy, ClickHouse) • when Postgres instead of Kafka • the significance of the meme “just use postgres” • Postgres for data analytics Chapters (00:00) Intro (04:26) The Just Use Postgres Movement (08:36) The Small Data Trend and Its Implications (15:49) The Shift from Distributed Systems to Simplicity (17:15) Updating your priors re: PG capabilities (20:28) MySQL's OSS community split & our experiences with such things (39:48) Postgres for Analytics (55:49) PG in the lake house (59:27) AI Marketing Slop (01:02:19) How you'd build a brand new app on Postgres (01:05:24) Speedrun through PG extensions (01:11:56) PG as a Message Queue
1:30:33

Just Use Postgres feat. Denis Magda Denis literally just published a book called “Just Use Postgres”. In this conversation, that turned out to be super fun, we talked about: • what MySQL got wrong w.r.t community • the recent explosion of Postgres development in the world (Supabase, Neon, Crunchy, ClickHouse) • when Postgres instead of Kafka • the significance of the meme “just use postgres” • Postgres for data analytics Chapters (00:00) Intro (04:26) The Just Use Postgres Movement (08:36) The Small Data Trend and Its Implications (15:49) The Shift from Distributed Systems to Simplicity (17:15) Updating your priors re: PG capabilities (20:28) MySQL's OSS community split & our experiences with such things (39:48) Postgres for Analytics (55:49) PG in the lake house (59:27) AI Marketing Slop (01:02:19) How you'd build a brand new app on Postgres (01:05:24) Speedrun through PG extensions (01:11:56) PG as a Message Queue

Stanislav Kozlovski

13,036 Aufrufe • vor 4 Monaten

How Postgres 17 improves IN queries In prior releases, postgres would loop and do multiple "primitive" index scans as many as your IN elements. In Postgres 17 it takes an array and eliminates the need for multiple scans. This only applies for constant values
3:39

How Postgres 17 improves IN queries In prior releases, postgres would loop and do multiple "primitive" index scans as many as your IN elements. In Postgres 17 it takes an array and eliminates the need for multiple scans. This only applies for constant values

Hussein Nasser

143,151 Aufrufe • vor 1 Jahr

Meet the new look of Neon Postgres. A total of ~1,200 hours of teamwork and five months from the initial kickoff to bring this design beauty to prod. Rebranding, design, dev, and motion. Ship faster with Postgres!
0:47

Meet the new look of Neon Postgres. A total of ~1,200 hours of teamwork and five months from the initial kickoff to bring this design beauty to prod. Rebranding, design, dev, and motion. Ship faster with Postgres!

Alex Barashkov

105,285 Aufrufe • vor 4 Monaten

Quick demo of Zero’s new background queries. Zero’s sync is query-based. Rather than specifying what data you want using rules or some other separate system, you just use queries. Right inside the client app, you do a query using a full sql-style language. You get filters, subqueries , limits, etc. Zero syncs the data backing these queries to the client. It’s important to realize Zero isn’t really a cache. It’s a replica. It’s eagerly replicating a precise snapshot of a slice of your database - the slice covered by the queries you have open. So there is never stale data in Zero. When you close a query we delete the rows uniquely returned by that query because we can no longer keep them up to date. Of course that kind of sucks for the common case of doing a query, navigating, then pressing “back”. Ideally we want that back nav to be fast. To address this Zero 0.17 adds background queries. You can add a ttl to a query and it will keep running and syncing in the background. This is much different than normal caching because this data stays up to date. If you make the same query again, the results will be *instantly* available *and already up to date with server*. If you make a different query the data from the background query is used client-side to answer the new query instantly if possible. This all happens completely automatically. Just by adding the ttl flag.
1:01

Quick demo of Zero’s new background queries. Zero’s sync is query-based. Rather than specifying what data you want using rules or some other separate system, you just use queries. Right inside the client app, you do a query using a full sql-style language. You get filters, subqueries , limits, etc. Zero syncs the data backing these queries to the client. It’s important to realize Zero isn’t really a cache. It’s a replica. It’s eagerly replicating a precise snapshot of a slice of your database - the slice covered by the queries you have open. So there is never stale data in Zero. When you close a query we delete the rows uniquely returned by that query because we can no longer keep them up to date. Of course that kind of sucks for the common case of doing a query, navigating, then pressing “back”. Ideally we want that back nav to be fast. To address this Zero 0.17 adds background queries. You can add a ttl to a query and it will keep running and syncing in the background. This is much different than normal caching because this data stays up to date. If you make the same query again, the results will be *instantly* available *and already up to date with server*. If you make a different query the data from the background query is used client-side to answer the new query instantly if possible. This all happens completely automatically. Just by adding the ttl flag.

Aaron Boodman

18,808 Aufrufe • vor 1 Jahr

Announcing the Supabase ETL! Supabase ETL is a change-data-capture pipeline that replicates your Postgres tables to analytical destinations in near real time! Available as private alpha starting today!
0:43

Announcing the Supabase ETL! Supabase ETL is a change-data-capture pipeline that replicates your Postgres tables to analytical destinations in near real time! Available as private alpha starting today!

Tyler Shukert

24,973 Aufrufe • vor 6 Monaten

Use Neon and ClickHouse to get the best of both worlds: the flexibility of Postgres for transactions + the performance of ClickHouse for analytics With PeerDB, data streaming is simple 👇
1:43

Use Neon and ClickHouse to get the best of both worlds: the flexibility of Postgres for transactions + the performance of ClickHouse for analytics With PeerDB, data streaming is simple 👇

Neon Postgres

11,422 Aufrufe • vor 1 Jahr

Big moment for Postgres! AI agents broke the idea of what a database is supposed to do. Traditional databases were built for humans, and Agents broke that model. - They branch endlessly. - They run ten experiments at once. - They need isolation, context, memory, structured reasoning, and safe sandboxes. Letting agents touch production systems is terrifying because the old model of Postgres was never built for this kind of behavior. Agentic Postgres is an agent-ready version of Postgres by TimescaleDB (by Tiger Data) that solves this. I think it is one of the biggest upgrades to the Agent stack this year and Tiger Data is working with me on this post to share what they did. Some key features: > It instantly creates branches of an entire database, which is perfect for parallel agent evals, safe experiments, migrations, or isolated testing. Forks take seconds and cost almost nothing. > It comes with a built-in MCP server, which agents can use to get schema guidance, best practices, and safe, structured access to Postgres. This is also helpful to run migrations with a real understanding. > It comes with actual hybrid search (vector search and BM25), so Agents can retrieve data directly inside the database. > The database is Memory native. This gives a persistent context for Agents to evolve. This is one of the first times I have seen Postgres feel ready for the AI native era.
0:26

Big moment for Postgres! AI agents broke the idea of what a database is supposed to do. Traditional databases were built for humans, and Agents broke that model. - They branch endlessly. - They run ten experiments at once. - They need isolation, context, memory, structured reasoning, and safe sandboxes. Letting agents touch production systems is terrifying because the old model of Postgres was never built for this kind of behavior. Agentic Postgres is an agent-ready version of Postgres by TimescaleDB (by Tiger Data) that solves this. I think it is one of the biggest upgrades to the Agent stack this year and Tiger Data is working with me on this post to share what they did. Some key features: > It instantly creates branches of an entire database, which is perfect for parallel agent evals, safe experiments, migrations, or isolated testing. Forks take seconds and cost almost nothing. > It comes with a built-in MCP server, which agents can use to get schema guidance, best practices, and safe, structured access to Postgres. This is also helpful to run migrations with a real understanding. > It comes with actual hybrid search (vector search and BM25), so Agents can retrieve data directly inside the database. > The database is Memory native. This gives a persistent context for Agents to evolve. This is one of the first times I have seen Postgres feel ready for the AI native era.

Avi Chawla

94,290 Aufrufe • vor 6 Monaten

With PGlite + Electric sync you can build an issue tracker handling 100k issues + 500k comments, synced from Postgres to PGlite in the browser! Super fast UI updates with all writes going though the local database, and Postgres FTS in PGlite for search 🔥
1:13

With PGlite + Electric sync you can build an issue tracker handling 100k issues + 500k comments, synced from Postgres to PGlite in the browser! Super fast UI updates with all writes going though the local database, and Postgres FTS in PGlite for search 🔥

Sam Willis

55,402 Aufrufe • vor 1 Jahr

Finish the week with a Postgres video 🌟 All about using JSON in Postgres: - when you should use it - how to use it efficiently - how JSONs actually get stored in postgres.
21:29

Finish the week with a Postgres video 🌟 All about using JSON in Postgres: - when you should use it - how to use it efficiently - how JSONs actually get stored in postgres.

Gwen (Chen) Shapira

27,568 Aufrufe • vor 1 Jahr

Your agents can't keep up with real-time data. Especially when it's scattered across dozens of sources. Most teams waste weeks building custom connectors for every database, API, and data warehouse. Then they build ETL pipelines to sync everything. By the time your agent retrieves the data, it's already outdated. Picture this: Your Postgres database updated 5 minutes ago. Your MongoDB collection changed 2 minutes ago. Your agent is still pulling from yesterday's snapshot. This is why most production RAG systems fail. There's a better approach: MindsDB is an open-source AI platform with a federated data engine that lets you query multiple data sources in real-time using SQL - without moving any data. Here's what makes it different: ↳ Your data stays in place. No ETL pipelines or data duplication ↳ Query Postgres, MongoDB, REST APIs, and more using consistent SQL ↳ JOIN across different sources in real-time with a unified interface ↳ Works with both structured and un-structured data And here's the best part: You don't even need to write SQL. Just describe what you want in plain English, and MindsDB converts it to SQL automatically. The system does all the heavy lifting. The breakthrough for AI agents is simple: When data updates at the source, your agent gets fresh results immediately. No sync delays. No stale embeddings. No custom code for each integration. You can literally write a SQL query that joins a Postgres table with a MongoDB collection and gets live results. This is what production AI applications need but rarely get. In this video, I give you a complete walkthrough of what we just discussed and how to actually do it. Make sure you watch this till the end. I've shared the link to MindsDB's GitHub repo in the next tweet!
9:28

Your agents can't keep up with real-time data. Especially when it's scattered across dozens of sources. Most teams waste weeks building custom connectors for every database, API, and data warehouse. Then they build ETL pipelines to sync everything. By the time your agent retrieves the data, it's already outdated. Picture this: Your Postgres database updated 5 minutes ago. Your MongoDB collection changed 2 minutes ago. Your agent is still pulling from yesterday's snapshot. This is why most production RAG systems fail. There's a better approach: MindsDB is an open-source AI platform with a federated data engine that lets you query multiple data sources in real-time using SQL - without moving any data. Here's what makes it different: ↳ Your data stays in place. No ETL pipelines or data duplication ↳ Query Postgres, MongoDB, REST APIs, and more using consistent SQL ↳ JOIN across different sources in real-time with a unified interface ↳ Works with both structured and un-structured data And here's the best part: You don't even need to write SQL. Just describe what you want in plain English, and MindsDB converts it to SQL automatically. The system does all the heavy lifting. The breakthrough for AI agents is simple: When data updates at the source, your agent gets fresh results immediately. No sync delays. No stale embeddings. No custom code for each integration. You can literally write a SQL query that joins a Postgres table with a MongoDB collection and gets live results. This is what production AI applications need but rarely get. In this video, I give you a complete walkthrough of what we just discussed and how to actually do it. Make sure you watch this till the end. I've shared the link to MindsDB's GitHub repo in the next tweet!

Akshay 🚀

65,672 Aufrufe • vor 7 Monaten

`bunx pg-here` global one-liner to run a standalone postgres instance with all data within the current folder throw-away postgres anytime anywhere. cli args for version/user/pass/db configs. docs: stop using docker just to launch a postgres!!
0:33

`bunx pg-here` global one-liner to run a standalone postgres instance with all data within the current folder throw-away postgres anytime anywhere. cli args for version/user/pass/db configs. docs: stop using docker just to launch a postgres!!

murat 🍥

50,537 Aufrufe • vor 3 Monaten

Introducing pg_strict for Postgres. Our new extension adds a safety net to Postgres, catching dangerous queries before they run.
3:38

Introducing pg_strict for Postgres. Our new extension adds a safety net to Postgres, catching dangerous queries before they run.

PlanetScale

114,683 Aufrufe • vor 4 Monaten

Even the most powerful artificial minds need infrastructure to understand the world. We've launched a radical project to change the way we build AI Agents. A graph layer that uses up to 34x less RAM than neo4j. To start: pgGraph (Full Open Souce, Rust), is a postgres extension that lets you run complex graph queries, like finding shortest paths using standard SQL. Directly in postgres. Coming next: Polygres, give your postgres a warp drive by combining pgGraph & pgVector into an all-in-one database. We're building the foundation of intelligence evokoa Links below ↓
0:29

Even the most powerful artificial minds need infrastructure to understand the world. We've launched a radical project to change the way we build AI Agents. A graph layer that uses up to 34x less RAM than neo4j. To start: pgGraph (Full Open Souce, Rust), is a postgres extension that lets you run complex graph queries, like finding shortest paths using standard SQL. Directly in postgres. Coming next: Polygres, give your postgres a warp drive by combining pgGraph & pgVector into an all-in-one database. We're building the foundation of intelligence evokoa Links below ↓

dale

14,138 Aufrufe • vor 8 Tagen

🚨 Tiger Data - Creators of TimescaleDB (Yes! the team behind TimescaleDB) is quietly revolutionizing how we build agents. The Old Way: Glueing together Postgres + Vector DBs + Search tools = complexity. The New Way: Tiger Data’s Agentic Postgres handles it all natively 🔥 What’s now possible 🧵 ↓ ✦ Hybrid retrieval → combine keyword filtering and vector similarity in a single query ✦ Forkable databases → retrieval and safe experimentation without external systems ✦ Native Postgres semantics → everything stays queryable, inspectable, and debuggable in SQL Why this matters: → Fewer moving parts in RAG pipelines → No extra infrastructure to stitch together → Agents can iterate and learn safely → Production data stays isolated and protected This is an awesome shift! Stop bolting workflows onto your database. Let your database be the workflow 🔥
0:38

🚨 Tiger Data - Creators of TimescaleDB (Yes! the team behind TimescaleDB) is quietly revolutionizing how we build agents. The Old Way: Glueing together Postgres + Vector DBs + Search tools = complexity. The New Way: Tiger Data’s Agentic Postgres handles it all natively 🔥 What’s now possible 🧵 ↓ ✦ Hybrid retrieval → combine keyword filtering and vector similarity in a single query ✦ Forkable databases → retrieval and safe experimentation without external systems ✦ Native Postgres semantics → everything stays queryable, inspectable, and debuggable in SQL Why this matters: → Fewer moving parts in RAG pipelines → No extra infrastructure to stitch together → Agents can iterate and learn safely → Production data stays isolated and protected This is an awesome shift! Stop bolting workflows onto your database. Let your database be the workflow 🔥

Charly Wargnier

20,181 Aufrufe • vor 3 Monaten

We re-architected Neon Auth to integrate it directly into the Neon platform. Auth now lives in your Postgres database and branches with your data. Built on Better Auth tech and available today 🧵
1:12

We re-architected Neon Auth to integrate it directly into the Neon platform. Auth now lives in your Postgres database and branches with your data. Built on Better Auth tech and available today 🧵

Neon Postgres

34,834 Aufrufe • vor 6 Monaten