Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Let Postgres own the Iceberg catalog and delegate analytics to DuckDB. The result => transactional lakehouse updates with fast analytical queries. This isn’t a concept. It’s exactly what pg_lake delivers today. pg_lake combines a set of extensions and components that let you query and modify Iceberg tables (and other...

28,503 Aufrufe • vor 7 Monaten •via X (Twitter)

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Your Postgres is 100x slower than traditional OLAP engines. A deceptively simple OSS extension fixes this. Here's an interview where we dive into the deep engineering around how this is achieved. Joining me (and leading the conversation) is Marco Slot: an engineer with an EXTENSIVE and impressive career history around PostgreSQL: 👉 Created pg_cron in 2017 (3.7k stars) - a tool to run cron-jobs in Postgres 👉 Built pg_incremental - fast, reliable, incremental batch processing inside PostgreSQL itself 👉 co-created pg_lake (after working on Crunchy Data's Warehouse, and getting acquired into Snowflake) 👉 Helped get pg_documentdb (MongoDB-on-Postgres) off the ground Marco Slot is a world-class expert in Postgres extensions. He seriously impressed me with his knowledge over the course of a private LinkedIn conversation, and now that I type out his resume - I understand where it came from. He should be on everyone's radar. So I brought him on the pod. In our full 2-hour deep-dive, we went over: • 🔥 how pg_lake makes analytics 100x faster (literally) • 🔥 perf internals like vectorized execution & CPU branching • 🤔 practical differences between OLTP and OLAP database development (and the age-old mission in uniting both) • 🤔 how (and why) pg_lake intercepts query plans and delegates parts of the query tree to DuckDB • 💡 why Postgres is architecturally terrible at analytical queries (and how vectorized execution fixes this) • 💡 Marco's hard-won experience through a decade+ career in Postgres • 🏆 Iceberg's role as the TCP/IP for tables • 🏆 what the real moat of PostgreSQL is Developments like pg_lake are a real reason why "Just Use Postgres" is much more than a meme, and it'll continue to dominate discourse. I promise you will learn a lot from this episode. Timestamps: (0:02) What is pg_lake? (2:23) Postgres' 100x slower problem and columnar storage experiments they had to make Postgres fast for analytics (6:00) practical examples and internals (16:20) perf internals - vectorized execution & CPU optimization (23:00) pg_lake architecture (why DuckDB isn't embedded) and the connection-per-process issue (29:16) how pg_lake intercepts the query plan tree and delegates parts to DuckDB (41:09) Iceberg catalogs (48:24) postgres to iceberg ingestion patterns (and pg_incremental) (53:40) Marco's (long) career: early AWS, Citus, Microsoft, Crunchy Data & Snowflake (1:04:20) Marco's observations around the merging between OLTP and OLAP (and the subtle dev differences there) (1:15:30) reverse ETL (1:33:08) Iceberg as the TCP/IP for tables (1:35:00) Marco's thoughts on the "Just Use Postgres" fever

Stanislav Kozlovski

16,857 Aufrufe • vor 5 Monaten

How Cloudflare cut query times by 35x without leaving Postgres: Their Postgres tables hit billions of rows, and every time-range query started getting slower. Plain Postgres scans the entire table on every query, regardless of the time window. They tried the manual route by building precomputed aggregates with cron jobs and evaluating ClickHouse, which needed a full ingestion pipeline just to handle their write pattern. Here's where most teams get stuck at exactly this point. They utilize manual partitioning, splitting data into child tables by day or month. Then they wire up cron jobs to refresh aggregate tables. Every schema change after that requires updating the cron logic and coordinating across teams. The infrastructure becomes the project. Tiger Cloud is managed TimescaleDB by Tiger Data - Creators of TimescaleDB on the Postgres you already know, with automatic time-based partitioning, continuous aggregates, and compression built in from the start. Cloudflare moved to TimescaleDB after exhausting the manual route and saw 5-35x query performance improvement on the same data. Here is how it works: → Hypertables partition data by timestamp automatically. Every time-range query hits only the relevant chunk, not the full table. → Continuous aggregates refresh incrementally in the background, with no cron jobs to maintain. To show what this looks like in practice, I built a real-time earthquake intelligence dashboard on a 3D globe using Claude Code and Tiger Cloud in a single session. The USGS earthquake catalog has 400,000+ events since 1900, each a timestamped row. The dashboard plots them as ripple animations sized by magnitude and colored by depth, with a time slider that scrubs through 120 years of seismic history. Every slider position fires a live query against the hypertable, and the side panel pulls from continuous aggregates. Claude Code connected to Tiger Cloud through the Tiger CLI MCP server, provisioned the database, pulled the USGS catalog, and assembled the full Next.js and Three.js frontend without leaving the session. The video below shows the final build in action, and I worked with the Tiger Data team to put this together. Tiger CLI is open-source (Apache 2.0) and works with Claude Code, Cursor, Codex, Gemini CLI, and VS Code. To try this yourself: → Sign up for Tiger Cloud here: New accounts get $1,000 in free credits, no credit card required. → Install Tiger CLI: curl -fsSL https(:)//cli(.)tigerdata(.)com | sh → Run tiger mcp install claude-code → Give Claude Code a prompt and let it build. I also wrote a full walkthrough on how you can turn any coding agent into a production-grade data engineer that can manage over a billion-row Postgres workloads. It covers everything from the database setup to the final build. Read it below.

Avi Chawla

22,094 Aufrufe • vor 26 Tagen

Your agents can't keep up with real-time data. Especially when it's scattered across dozens of sources. Most teams waste weeks building custom connectors for every database, API, and data warehouse. Then they build ETL pipelines to sync everything. By the time your agent retrieves the data, it's already outdated. Picture this: Your Postgres database updated 5 minutes ago. Your MongoDB collection changed 2 minutes ago. Your agent is still pulling from yesterday's snapshot. This is why most production RAG systems fail. There's a better approach: MindsDB is an open-source AI platform with a federated data engine that lets you query multiple data sources in real-time using SQL - without moving any data. Here's what makes it different: ↳ Your data stays in place. No ETL pipelines or data duplication ↳ Query Postgres, MongoDB, REST APIs, and more using consistent SQL ↳ JOIN across different sources in real-time with a unified interface ↳ Works with both structured and un-structured data And here's the best part: You don't even need to write SQL. Just describe what you want in plain English, and MindsDB converts it to SQL automatically. The system does all the heavy lifting. The breakthrough for AI agents is simple: When data updates at the source, your agent gets fresh results immediately. No sync delays. No stale embeddings. No custom code for each integration. You can literally write a SQL query that joins a Postgres table with a MongoDB collection and gets live results. This is what production AI applications need but rarely get. In this video, I give you a complete walkthrough of what we just discussed and how to actually do it. Make sure you watch this till the end. I've shared the link to MindsDB's GitHub repo in the next tweet!

Akshay 🚀

65,672 Aufrufe • vor 10 Monaten

Claude Code is now scary good at full-stack! I asked it to build a real-time weather intelligence dashboard with an interactive 3D globe and a forecasting layer that predicts weather 3 days ahead. It came back with a spinning globe that has a day/night cycle using NASA satellite imagery, city lights on the dark side, weather icons that switch between sun and moon based on local time, and a time travel slider that scrubs through 10 days of data. Claude Code built the whole thing in a single session, including the backend, database, data pipeline, and frontend. For the database, I needed something fast for time-series workloads since the app ingests hourly weather readings across many cities and serves time-range queries on every slider interaction. I used Tiger Cloud by Tiger Data - Creators of TimescaleDB, which gives you managed TimescaleDB on the Postgres you already know. Claude Code connected to it through the Tiger CLI MCP server and set up the entire backend directly: - Provisioned the database service - Created hypertables for time-partitioned weather storage - Set up continuous aggregates for pre-computed rollups - Built the data ingestion pipeline and the full NextJS + ThreeJS frontend The time travel slider queries thousands of rows on every position change. On a regular Postgres table, this would require manual partitioning and index tuning to stay fast as data grows. TimescaleDB partitions the data by timestamp automatically, so each query only hits the relevant time chunk. Continuous aggregates serve the trend charts and forecast layer from pre-computed rollups instead of rescanning raw data on every request. The video below shows the final build in action, and I worked with the Tiger Data team to put this together. Tiger CLI is open-source (Apache 2.0) and works with Claude Code, Cursor, Codex, Gemini CLI, and VS Code. To try this yourself: → Sign up for Tiger Cloud (I have shared the link in the replies). It gives you $1,000 free credits (no card needed) → Install Tiger CLI: curl -fsSL https(:)//cli(.)tigerdata(.)com | sh → Run tiger mcp install claude-code → Give Claude Code a prompt and let it build Find the sign-up link in the replies.

Avi Chawla

14,838 Aufrufe • vor 3 Monaten

Claude Code is now scary good at full-stack! I asked it to build a real-time weather intelligence dashboard with an interactive 3D globe, a forecasting layer that predicts weather 3 days ahead, and an anomaly detector that flags cities whose weather is behaving abnormally. It came back with a spinning globe that has a day/night cycle using NASA satellite imagery, city lights on the dark side, weather icons that switch between sun and moon based on local time, and a time travel slider that scrubs through 10 days of data. And when a city's weather breaks from its own normal, it pulses red (abnormally hot) or blue (abnormally cold) right on the globe, updating live and reflecting the anomaly state at any point you drag the slider to. Claude Code built the whole thing in a single session, including the backend, database, data pipeline, and frontend. For the database, I needed something fast for time-series workloads since the app ingests hourly weather readings across many cities and serves time-range queries on every slider interaction. I used Tiger Cloud by Tiger Data - Creators of TimescaleDB, which gives you managed TimescaleDB on the Postgres you already know. Claude Code connected to it through the Tiger CLI MCP server and set up the entire backend directly: - Provisioned the database service - Created hypertables for time-partitioned weather storage - Set up continuous aggregates for pre-computed rollups - Built the data ingestion pipeline and the full NextJS + ThreeJS frontend The time travel slider queries thousands of rows on every position change. On a regular Postgres table, this would require manual partitioning and index tuning to stay fast as data grows. TimescaleDB partitions the data by timestamp automatically, so each query only hits the relevant time chunk. Continuous aggregates serve the trend charts, the forecast layer, and the anomaly baselines from pre-computed rollups instead of rescanning raw data on every request. The video below shows the final build in action, and I worked with the Tiger Data team to put this together. Tiger CLI is open-source (Apache 2.0) and works with Claude Code, Cursor, Codex, Gemini CLI, and VS Code. To try this yourself: → Sign up for Tiger Cloud (I have shared the link in the replies). It gives you $1,000 free credits (no card needed) → Install Tiger CLI: curl -fsSL https(:)//cli(.)tigerdata(.)com | sh → Run tiger mcp install claude-code → Give Claude Code a prompt and let it build sign-up here: My co-founder also wrote a detailed article on this. The article is quoted below.

Akshay 🚀

37,269 Aufrufe • vor 22 Tagen