正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Flamegraphs are a great way to narrow in on performance issues + broadly to explore program execution. I built a custom viewer for Postgres queries, inspired Jan Nidzwetzki's excellent blog on the subject. Highly recommend analysis tools like this for learning query execution.

Ben Dicken

40,886 subscribers

47,895 次观看 • 3 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Let Postgres own the Iceberg catalog and delegate analytics to DuckDB. The result => transactional lakehouse updates with fast analytical queries. This isn’t a concept. It’s exactly what pg_lake delivers today. pg_lake combines a set of extensions and components that let you query and modify Iceberg tables (and other lakehouse formats) directly from Postgres. DuckDB is used to accelerate analytical queries and runs in a sidecar process called pgduck_server, which communicates with Postgres during query execution. How it works (diagram below): 1. An application sends a query to Postgres to calculate unrealized PnL (Profit and Loss) for the Disney ticker. 2. Postgres parses the query and identifies the part that computes the average price from historical lakehouse data. 3. That part is forwarded to pgduck_server for accelerated execution. 4. pgduck_server delegates execution to DuckDB, which queries the lakehouse (reusing cached data if available). 5. DuckDB computes the average price and returns it to Postgres. 6. Postgres joins the result with local portfolio data and computes the unrealized PnL. 7. The final result is returned to the application.

Let Postgres own the Iceberg catalog and delegate analytics to DuckDB. The result => transactional lakehouse updates with fast analytical queries. This isn’t a concept. It’s exactly what pg_lake delivers today. pg_lake combines a set of extensions and components that let you query and modify Iceberg tables (and other lakehouse formats) directly from Postgres. DuckDB is used to accelerate analytical queries and runs in a sidecar process called pgduck_server, which communicates with Postgres during query execution. How it works (diagram below): 1. An application sends a query to Postgres to calculate unrealized PnL (Profit and Loss) for the Disney ticker. 2. Postgres parses the query and identifies the part that computes the average price from historical lakehouse data. 3. That part is forwarded to pgduck_server for accelerated execution. 4. pgduck_server delegates execution to DuckDB, which queries the lakehouse (reusing cached data if available). 5. DuckDB computes the average price and returns it to Postgres. 6. Postgres joins the result with local portfolio data and computes the unrealized PnL. 7. The final result is returned to the application.

Denis Magda

28,503 次观看 • 5 个月前

I built a manufacturing-aware CAD interpretation engine that uses LLMs to query any step model's geometric data, saving $10k/Yr on analysis software. This web-based CAD analysis tool is a workbench for developing bespoke DFM tools for manufacturing services.

I built a manufacturing-aware CAD interpretation engine that uses LLMs to query any step model's geometric data, saving $10k/Yr on analysis software. This web-based CAD analysis tool is a workbench for developing bespoke DFM tools for manufacturing services.

Connor Kapoor

50,631 次观看 • 6 个月前

📝 AI/ML for Biology & Healthcare: A Learning Path This blog is highly inspired by my experience and exploration in the field of ML for biology and healthcare. Everything that I've been studying for the past years and a more structured way to learn the foundational knowledge to tackle problems in this impactful field. Here's what I covered in the blog: — Programming & Engineering — AI & Machine Learning — Mathematics for ML — Healthcare — Biology & Biolomedicine This is what I hoped to have when I first started learning all this. 🔗 link:

📝 AI/ML for Biology & Healthcare: A Learning Path This blog is highly inspired by my experience and exploration in the field of ML for biology and healthcare. Everything that I've been studying for the past years and a more structured way to learn the foundational knowledge to tackle problems in this impactful field. Here's what I covered in the blog: — Programming & Engineering — AI & Machine Learning — Mathematics for ML — Healthcare — Biology & Biolomedicine This is what I hoped to have when I first started learning all this. 🔗 link:

TK • 木下

26,889 次观看 • 8 个月前

Sometimes the best way to build a UI for an agent is to not build the UI at all. I built an AI agent starter that lives in iMessage. Built on Nitro + Vercel Workflow SDK + evlog - Durable execution - Auto-retry on errors - Full observability

Sometimes the best way to build a UI for an agent is to not build the UI at all. I built an AI agent starter that lives in iMessage. Built on Nitro + Vercel Workflow SDK + evlog - Durable execution - Auto-retry on errors - Full observability

Hugo

11,671 次观看 • 2 个月前

Reverse Engineer's Toolkit A collection of tools you may like if you are interested in #reverse engineering and/or malware analysis on x86 and x64 Windows systems. After installing this program, you'll have two ways to access the tools: • Double-click the retoolkit icon in the Desktop. • Right-click on a file, choose retoolkit. This way the selected file is passed as argument to the desired program. #cybersecurity #infosec

Reverse Engineer's Toolkit A collection of tools you may like if you are interested in #reverse engineering and/or malware analysis on x86 and x64 Windows systems. After installing this program, you'll have two ways to access the tools: • Double-click the retoolkit icon in the Desktop. • Right-click on a file, choose retoolkit. This way the selected file is passed as argument to the desired program. #cybersecurity #infosec

HackGit

138,096 次观看 • 2 年前

Your Postgres is 100x slower than traditional OLAP engines. A deceptively simple OSS extension fixes this. Here's an interview where we dive into the deep engineering around how this is achieved. Joining me (and leading the conversation) is Marco Slot: an engineer with an EXTENSIVE and impressive career history around PostgreSQL: 👉 Created pg_cron in 2017 (3.7k stars) - a tool to run cron-jobs in Postgres 👉 Built pg_incremental - fast, reliable, incremental batch processing inside PostgreSQL itself 👉 co-created pg_lake (after working on Crunchy Data's Warehouse, and getting acquired into Snowflake) 👉 Helped get pg_documentdb (MongoDB-on-Postgres) off the ground Marco Slot is a world-class expert in Postgres extensions. He seriously impressed me with his knowledge over the course of a private LinkedIn conversation, and now that I type out his resume - I understand where it came from. He should be on everyone's radar. So I brought him on the pod. In our full 2-hour deep-dive, we went over: • 🔥 how pg_lake makes analytics 100x faster (literally) • 🔥 perf internals like vectorized execution & CPU branching • 🤔 practical differences between OLTP and OLAP database development (and the age-old mission in uniting both) • 🤔 how (and why) pg_lake intercepts query plans and delegates parts of the query tree to DuckDB • 💡 why Postgres is architecturally terrible at analytical queries (and how vectorized execution fixes this) • 💡 Marco's hard-won experience through a decade+ career in Postgres • 🏆 Iceberg's role as the TCP/IP for tables • 🏆 what the real moat of PostgreSQL is Developments like pg_lake are a real reason why "Just Use Postgres" is much more than a meme, and it'll continue to dominate discourse. I promise you will learn a lot from this episode. Timestamps: (0:02) What is pg_lake? (2:23) Postgres' 100x slower problem and columnar storage experiments they had to make Postgres fast for analytics (6:00) practical examples and internals (16:20) perf internals - vectorized execution & CPU optimization (23:00) pg_lake architecture (why DuckDB isn't embedded) and the connection-per-process issue (29:16) how pg_lake intercepts the query plan tree and delegates parts to DuckDB (41:09) Iceberg catalogs (48:24) postgres to iceberg ingestion patterns (and pg_incremental) (53:40) Marco's (long) career: early AWS, Citus, Microsoft, Crunchy Data & Snowflake (1:04:20) Marco's observations around the merging between OLTP and OLAP (and the subtle dev differences there) (1:15:30) reverse ETL (1:33:08) Iceberg as the TCP/IP for tables (1:35:00) Marco's thoughts on the "Just Use Postgres" fever

Your Postgres is 100x slower than traditional OLAP engines. A deceptively simple OSS extension fixes this. Here's an interview where we dive into the deep engineering around how this is achieved. Joining me (and leading the conversation) is Marco Slot: an engineer with an EXTENSIVE and impressive career history around PostgreSQL: 👉 Created pg_cron in 2017 (3.7k stars) - a tool to run cron-jobs in Postgres 👉 Built pg_incremental - fast, reliable, incremental batch processing inside PostgreSQL itself 👉 co-created pg_lake (after working on Crunchy Data's Warehouse, and getting acquired into Snowflake) 👉 Helped get pg_documentdb (MongoDB-on-Postgres) off the ground Marco Slot is a world-class expert in Postgres extensions. He seriously impressed me with his knowledge over the course of a private LinkedIn conversation, and now that I type out his resume - I understand where it came from. He should be on everyone's radar. So I brought him on the pod. In our full 2-hour deep-dive, we went over: • 🔥 how pg_lake makes analytics 100x faster (literally) • 🔥 perf internals like vectorized execution & CPU branching • 🤔 practical differences between OLTP and OLAP database development (and the age-old mission in uniting both) • 🤔 how (and why) pg_lake intercepts query plans and delegates parts of the query tree to DuckDB • 💡 why Postgres is architecturally terrible at analytical queries (and how vectorized execution fixes this) • 💡 Marco's hard-won experience through a decade+ career in Postgres • 🏆 Iceberg's role as the TCP/IP for tables • 🏆 what the real moat of PostgreSQL is Developments like pg_lake are a real reason why "Just Use Postgres" is much more than a meme, and it'll continue to dominate discourse. I promise you will learn a lot from this episode. Timestamps: (0:02) What is pg_lake? (2:23) Postgres' 100x slower problem and columnar storage experiments they had to make Postgres fast for analytics (6:00) practical examples and internals (16:20) perf internals - vectorized execution & CPU optimization (23:00) pg_lake architecture (why DuckDB isn't embedded) and the connection-per-process issue (29:16) how pg_lake intercepts the query plan tree and delegates parts to DuckDB (41:09) Iceberg catalogs (48:24) postgres to iceberg ingestion patterns (and pg_incremental) (53:40) Marco's (long) career: early AWS, Citus, Microsoft, Crunchy Data & Snowflake (1:04:20) Marco's observations around the merging between OLTP and OLAP (and the subtle dev differences there) (1:15:30) reverse ETL (1:33:08) Iceberg as the TCP/IP for tables (1:35:00) Marco's thoughts on the "Just Use Postgres" fever

Stanislav Kozlovski

16,767 次观看 • 3 个月前

I spoke to this brave gentleman at the Iranian embassy protest in London on Sunday. He's under an 'execution order' in Iran for converting to Christianity.

I spoke to this brave gentleman at the Iranian embassy protest in London on Sunday. He's under an 'execution order' in Iran for converting to Christianity.

Tommy Robinson 🇬🇧

207,978 次观看 • 6 个月前

🆕 Introducing PrismaGPT, a query helper built on GPT-4 🎉 * Drop in your Prisma schema * Ask for what you want * Get back a Prisma Client query and raw SQL Great for when you know what you need but aren't quite sure how to construct the query 👍

🆕 Introducing PrismaGPT, a query helper built on GPT-4 🎉 * Drop in your Prisma schema * Ask for what you want * Get back a Prisma Client query and raw SQL Great for when you know what you need but aren't quite sure how to construct the query 👍

Ryan Chenkie

94,182 次观看 • 3 年前

Trade $thenickshirley on Base like a Pro - Trending Tokens - AI built for DeFi - Wallet Tracking - Limit Orders Onchain w TP/SL to set smart entries and exits - Fast Execution

Trade $thenickshirley on Base like a Pro - Trending Tokens - AI built for DeFi - Wallet Tracking - Limit Orders Onchain w TP/SL to set smart entries and exits - Fast Execution

Velvet | Onchain Terminal

14,628 次观看 • 6 个月前

USDJPY analysis I sent out to the private group last night; The amount of analysis that goes into my trades is a lot more than just looking at candles on the chart. Being a good analyst does not equate to being a good trader, analysis and execution are two separate skill sets.

USDJPY analysis I sent out to the private group last night; The amount of analysis that goes into my trades is a lot more than just looking at candles on the chart. Being a good analyst does not equate to being a good trader, analysis and execution are two separate skill sets.

The MMXM Trader

59,220 次观看 • 1 年前

I did an analysis of top 10 girl baby names in Kerala 1940 - 2007. More analysis in my blog: I've been working on this on-and-off for more than a year now. Finally relieved to publish this!

I did an analysis of top 10 girl baby names in Kerala 1940 - 2007. More analysis in my blog: I've been working on this on-and-off for more than a year now. Finally relieved to publish this!

Subin/സുബിൻ:/bin/su @[email protected]

78,730 次观看 • 5 个月前

Important to know that Postgres/MySQL on local SSD is *the* best way to get great performance. The reason it could get *even faster* boils down to abstraction boundaries. SSDs are built to be workload agnostic. They have their own controllers and garbage collectors meant to balance performance of many different I/O patterns. DBMS are written to support arbitrary backing I/O devices: SSD, HDD, or even a remote storage system. It's good that they're written to be broadly performant! The point of the paper was that, if you're willing to build something that specifically targets local SSDs (and even specific models of local SSDs) you can optimize the full DBMS -> OS -> hardware stack for the exact characteristics of a workload and hardware specifications. This is a classic tradeoff. You can usually get better performance as you narrow scope for specific hardware, architectures, etc. Maintainers of OSS must walk a careful balance of optimizing for performance while also maintaining wide compatibility. For the curious-minded, I wrote an interactive article on the history of I/O devices, and how that interplays with database performance:

Important to know that Postgres/MySQL on local SSD is the best way to get great performance. The reason it could get even faster boils down to abstraction boundaries. SSDs are built to be workload agnostic. They have their own controllers and garbage collectors meant to balance performance of many different I/O patterns. DBMS are written to support arbitrary backing I/O devices: SSD, HDD, or even a remote storage system. It's good that they're written to be broadly performant! The point of the paper was that, if you're willing to build something that specifically targets local SSDs (and even specific models of local SSDs) you can optimize the full DBMS -> OS -> hardware stack for the exact characteristics of a workload and hardware specifications. This is a classic tradeoff. You can usually get better performance as you narrow scope for specific hardware, architectures, etc. Maintainers of OSS must walk a careful balance of optimizing for performance while also maintaining wide compatibility. For the curious-minded, I wrote an interactive article on the history of I/O devices, and how that interplays with database performance:

Ben Dicken

79,406 次观看 • 2 个月前

Current Web3 systems are often slow and expensive to run. JAM takes a different approach, focusing on built-in interoperability and execution closer to the hardware to make things faster and more efficient.

Current Web3 systems are often slow and expensive to run. JAM takes a different approach, focusing on built-in interoperability and execution closer to the hardware to make things faster and more efficient.

Polkadot

18,986 次观看 • 3 个月前

The first launch from the Seedify Hackaton is on the way 🙌 Introducing @AgentXYZ_ai — The AI Trading Terminal built for traders of all levels to outperform the market. More than just tools — 28 AI agents work in sync to deliver a tactical edge with real-time precision: ▪️Real-time market, social & onchain insights ▪️One-click, AI-powered technical analysis setups ▪️Voice-activated trade execution ▪️X-based AI assistant for smart coin analysis & reports ▪️Instant sentiment breakdown — track social buzz, whale moves & smart money positioning at a glance The IDO is arriving at Seedify - stay on a lookout for registrations!

The first launch from the Seedify Hackaton is on the way 🙌 Introducing @AgentXYZ_ai — The AI Trading Terminal built for traders of all levels to outperform the market. More than just tools — 28 AI agents work in sync to deliver a tactical edge with real-time precision: ▪️Real-time market, social & onchain insights ▪️One-click, AI-powered technical analysis setups ▪️Voice-activated trade execution ▪️X-based AI assistant for smart coin analysis & reports ▪️Instant sentiment breakdown — track social buzz, whale moves & smart money positioning at a glance The IDO is arriving at Seedify - stay on a lookout for registrations!

Seedify

42,581 次观看 • 1 年前

Hey, fellow ELFs! If you are learning Korean like I am and want to practice listening, I highly recommend this podcast, run by a Korean teacher who has been an ELF for 20 years. 😉💙 This episode is about her life as a Super Junior fan from the beginning to the present,…⬇️

Hey, fellow ELFs! If you are learning Korean like I am and want to practice listening, I highly recommend this podcast, run by a Korean teacher who has been an ELF for 20 years. 😉💙 This episode is about her life as a Super Junior fan from the beginning to the present,…⬇️

MichHiiELF

21,493 次观看 • 1 年前

"I think we probably have all the tools we need to solve whatever we need to on the layer-1 scaling side...I'm not sure we need continuous little tweaking" - Rebecca Rettig on the Crypto In America podcast. The infrastructure wars are over. The rails have been built. Now, it's about execution. Markets Move Faster on Sei.

"I think we probably have all the tools we need to solve whatever we need to on the layer-1 scaling side...I'm not sure we need continuous little tweaking" - Rebecca Rettig on the Crypto In America podcast. The infrastructure wars are over. The rails have been built. Now, it's about execution. Markets Move Faster on Sei.

Sei

28,493 次观看 • 1 年前

today, on the DL, Notion released WORKERS. workers are a generic code execution environment that let you build interfaces within Notion the first interface we're exposing is the ability to make your own custom agent tools you can now make a tool for a custom agent that executes arbitrary code... and if you can execute arbitrary code, you can do pretty much anything!

today, on the DL, Notion released WORKERS. workers are a generic code execution environment that let you build interfaces within Notion the first interface we're exposing is the ability to make your own custom agent tools you can now make a tool for a custom agent that executes arbitrary code... and if you can execute arbitrary code, you can do pretty much anything!

Sam Catania

81,830 次观看 • 4 个月前