Excited to share Quail, our new open source AI-SQL engine (a collab with Modal)! By planning queries and LLM inference together, it reaches 1B+ input tokens/min on one H100 for one query 😱🚀 AI-powered data operators create a new, interesting inference workload👇

AI-SQL lets users write queries like: "Find movie reviews that discuss the ending and recommend the movie", as, e.g., SELECT * FROM reviews where AI.IF("the review discusses the ending") AND AI.IF("the review recommends the movie"). Queries are insanely expensive! An AI filter calls an LLM for *every row*; an AI join can call it for every candidate pair (m * n rows). One query can generate millions of LLM calls, needing ultra-high throughput!

We tried running all these LLM calls through hand-tuned vLLM baselines. On several queries, we observed lots of host overhead --- and, vLLM also discarded KV it needed later, causing it to process 50 million extra tokens. The query took 6.84 hours, when speed-of-light estimates suggested only 15 minutes...

AI-SQL is such a special case workload, where you know nearly all requests up front (and can plan KV a lot better), and you care more about throughput, or executing the *entire* query as quickly as possible. And, requests are entirely prefill. Not like the interactive request setting that general-purpose inference engines are optimized for. Turns out it makes sense to build a specialized inference engine here, motivating Quail.

Quail combines good query planning with good inference: e.g., it orders AI operators, pipelines rows so their KV stays in GPU HBM, and batches work to keep the GPU busy. For AI joins, we even use some fun tree attention tricks (c.f. SpecInfer, Hydragen) to reuse forward pass work when comparing one document with many others!

On 29 benchmark queries, Quail is 1.84x faster than hand-tuned vLLM baselines. On our largest medical reports query, it’s 14x faster (only 29 mins compared to 6.84 hours)! We also have considerably less KV regret.

We still have lots more to develop; for example, a use case where vLLM wins is on analyzing agent traces (that have matching prefixes across different rows). Its automatic prefix cache catches those matches; Quail doesn’t yet. vLLM is 2.32x faster on one such query. So automatic prefix caching is definitely on our roadmap.

Today, Quail supports AI filters and joins -- and we are adding much more. The code is MIT licensed. We'd love to hear what you try with it! This is a really fun collaboration with inference 🐐 @charles_irl from Modal; look out for their post :-) Our lab's blog post: Code: Playground demo:

this is amazing

Yes we should collab!! Really exciting to see what you are doing re LLMs in motherduck — I’ll send you an email soon :-)

1B+ input tokens/min on one H100 很夸张，但我更想看 planner 把哪些 row 判成“值得调用模型”。如果每个候选都进 LLM，吞吐再高也会被无效 token 和结果校验吃掉；这个分母怎么定义，感觉比峰值更能说明 Quail 的真实甜点区。

🔥🔥

Co-planning query execution and LLM inference is the key. That’s why hitting 1B+ tokens on one H100 feels less like magic and more like good engineering done right. Smart work by you and Modal! 🚀

@XyraSinclair

super cool

that's A LOT of tokens/min!

this is incredibly cool. and obvs love that you did it w team modal. so well done!

The database planner is the star here. Co-optimizing retrieval and inference could make natural-language filters feel less like a tax on every row. How much of the gain comes from batching, and how much comes from choosing which rows deserve an LLM call?

just ultra curious what is the real-world application of such a thing, like why would one want to LLM-process every single row even at an insane speed?

This is super cool!

The query planner is doing the unglamorous work that makes the headline number believable. Which part is hardest to keep efficient as query shapes get less predictable, operator ordering, KV reuse, or batching?

Hell yeah

1B tokens/min on one H100 is a serious receipt

the number i'd put next to the 1.84x is the cost of the re-run: one mid-query failure that re-issues completed operators eats the whole speedup. key operator outputs by the row that produced them so the plan doubles as the retry graph and a retry only re-runs what changed.

Planning inference and SQL together seems like the key shift here—the 1B+ tokens/min figure makes that especially interesting.

Does the planner push cheap SQL filters ahead of the AI.IF calls on its own? That ordering is usually most of the bill.

An AI-SQL engine that treats KV like a query-planning resource is a neat inversion. Does the planner expose a cost model developers can inspect, or is the magic still hidden behind the operator?

Congrats @sh_reya !

Very cool!! For a second I thought this was the problem/work Stonebraker was talking about here

operator ordering is where most of the savings live in practice. push every plain sql predicate ahead of the ai.if calls and most rows never reach the model at all.