What Postgres can’t do alone, it can tackle with... DuckDB. The pg_duckdb extension shows how two databases can work hand in hand by querying analytical data with DuckDB and joining it with transactional data in Postgres. The extension embeds DuckDB directly into Postgres. DuckDB literally runs inside a Postgres process. How it works (see diagram): 1. An app sends a query to Postgres joining portfolio data (Postgres) with historical market data (lakehouse) 2. pg_duckdb detects lakehouse access and delegates execution to DuckDB 3. DuckDB pulls required market data from the lakehouse, queries Postgres tables, and performs the join 4. Postgres returns the final result to the appshow more

First try with Vercel's new Postgres database! Was so... quick to set up 🤯 here I've got a collaborative form built with Liveblocks, and I'm having it automatically sync data to the database. More info 🧵show more

Big moment for Postgres! Search has always been Postgres'... weak spot, and everyone just accepted it. If you needed a real relevance-ranked keyword search, the default answer was to spin up Elasticsearch or add Algolia and deal with the data sync headaches forever. The problem isn't that Postgres can't do text search. It can. But the built-in `ts_rank` function uses a basic term frequency algorithm that doesn't come close to what modern search engines deliver. So teams end up: - Running a separate Elasticsearch cluster just for search - Building sync pipelines that inevitably drift out of consistency - Paying for managed search services that charge per query - Accepting mediocre search relevance because "good enough" ships faster But this is actually a solvable problem. You can realistically bring industry-standard search ranking directly into Postgres, which eliminates the need for external infra entirely. This exact solution is now available with the newly open-sourced pg_textsearch by Tiger Data - Creators of TimescaleDB, a Postgres extension that brings true BM25 relevance ranking into the database. BM25 is the algorithm behind Elasticsearch, Lucene, and most modern search engines. Now it runs natively in Postgres. Here's what pg_textsearch enables: - True BM25 ranking with configurable parameters (the same algorithm powering production search systems) - Simple SQL syntax: `ORDER BY content 'search terms'` - Works with Postgres text search configurations for multiple languages - Pairs naturally with pgvector for hybrid keyword + semantic search That last point matters a lot for RAG apps. The video below shows this in action, and I worked with the team to put this together. You can now do hybrid retrieval (combining keyword matching with vector similarity) in a single database, without stitching together multiple systems. The syntax is clean enough that you can add relevance-ranked search to existing queries in minutes. pg_textsearch is fully open-source under the PostgreSQL license. You can find a link to their GitHub repo in the next tweet.show more

Operational databases have long relied on tightly coupled compute... and storage. This architecture creates resource contention and pushes teams to manage infrastructure rather than build. As applications become more real time and automated, the transactional layer needs to adapt. Databricks Lakebase is built for that evolution: • Familiar Postgres semantics for app developers • Compute separated from durable state • Operational data running directly on the lakehouse • Serverless autoscaling (including scale to zero), branching, and recovery to match agent-driven workload Now generally available:show more

Talking with someone the other day that estimated they... had about 25,000 idle connections to Postgres. My actual response to them: "holy shit". They double checked, it was only about 12,000 Same day had a conversation with someone saying they didn't need pgbouncer because of activerecord's connection pooling. Let's dig into connection pooling in Postgres. Prior to Postgres 14 every connection to the database consumed memory, roughly 10MB, it may be slightly less but it still wasn't free. Even beyond Postgres 14 there is still various contention that happens when Postgres starts to use a connection. An application pooler maintains a set of connections and hands them out when needed on the application side. These are idle and real connections against the database that indeed do impact performance negatively. In contrast pgbouncer speaks the wire protocol, waits for the begin part of the transaction and then uses a connection. It more strictly manages how many idle ones it's having instead of per web server you're running. pgbouncer up until recently really needed to be run in transaction mode (which meant disabling prepared statements in your application framework). prepared_statement support in pgbouncer was added recently, and now you don't have to disable. Even when running with an older version of pgbouncer with prepared_statements disabled you'd still see a big performance gain. A quick check to know if you'd benefit from pgbouncer, run this query: SELECT count(*), state FROM pg_stat_activity GROUP BY 2; If you're idle account is high (yes this is dependent on your view, but to me if it's above 25-30 range, and especially if active is until half that) then you'd already start to benefit from pgbouncer. If it's at 10,000 then post haste get pgbouncer in place. Finally, you don't have to not use a framework pooler, they're fine, but don't think it replaces a native Postgres connection pooler.show more

Starting a new project today, building an end to... end system to forecast traffic of flights across cities (starting with Mumbai) The idea is to implement > ingestion service with kafka > data etl with polars > feast for feature store // mlflow for model registry > batch inferencing // dashboard > s3 // postgres for data storage All this orchestrated across multiple DAGs built with Airflow. This year has just been Agents and LLMs all along. Not a bad idea to keep revisiting the traditional format :) Will be posting more of this in the coming days, stay tunedshow more

OpenAI's Deep Research is getting a run for its... money. Deep Lake was just released, and it's a different take on an AI system that can do deep research on your own data. You can use Deep Lake to build AI search with reasoning on your private and public data. (Look at the attached videos to get an idea of how it works.) If you want to research proprietary and sensitive data, Deep Research won't help you because it's limited to public data. Deep Lake, however, will allow you to use your private data. On top of that, Deep Lake supports multi-modal retrieval from the ground up. It uses vision language models for data ingestion and retrieval so that you can connect any data (PDFs, images, videos, structured data, etc.) You can even use mixed-data queries! Deep Lake can search your data from S3, Dropbox, and GCP. It learns from your queries over time, making the results as relevant to your work as possible!show more

A good technical LLM interview question: Your RAG chatbot... is working as expected locally. You deploy it behind a load balancer with 3 replicas. Users report that it forgets what they just asked, and answers get worse with each restart. Why did this happen? (answer below) A local setup has one process that owns everything. - The vector index is a variable in memory. - Conversation history is a Python list. - The documents are on local disk. You never treat any of them as infrastructure, because restarting rebuilds all three in seconds and there is only ever one copy. The setup does not carry over to production directly. The vector index might disappear on restart, so the app re-embeds everything on boot and serves empty results until it finishes. Conversation history may belong to one replica, so a follow-up routed elsewhere has no memory of the previous turn. Documents could be on whichever container ingested them, so the three replicas hold three different corpora. None of this is evident with one user and one process. So the actual work in shipping RAG is not just the retrieval logic, but also storing the vector index, the conversation history, and the documents outside the app, where every replica reads and writes the same copy. Which comes down to three requirements: > The vector store needs persistence and has to be reachable from every replica. pgvector inside Postgres keeps embeddings next to the rest of the data instead of adding another system to operate. > Conversation state has to be checkpointed outside the app. LangGraph writes its state to Postgres, so any replica can pick up a thread mid-conversation. > Docs need shared object storage, so ingestion happens once instead of once per replica. If you get those three right, the retrieval logic you wrote in the notebook works unchanged. To learn how all of it is wired together, Akamai's GitHub has a working reference implementation. - rag-langgraph-k8s-quickstart is an airline policy Q&A assistant built with FastAPI, LangChain, and LangGraph. Terraform provisions the LKE cluster, a Postgres instance with pgvector for embeddings, a second Postgres for LangGraph checkpointing, and an object storage bucket for the policy documents, in one apply. - akamai-workshop-ai-inference covers the next step, running the model yourself instead of calling an API, with prefill and decode, KV cache tradeoffs, and continuous batching under real concurrency. Both are available on Akamai's new Developer Hub, alongside their tutorials and code samples. It also links to Edge Case, their Discord, where four developer advocates architect and deploy a production app live every other Wednesday. If you create a new Akamai Cloud account, you can also get $300 in credits for joining. Join here: That said, this post assumes the retrieval logic was right to begin with, and that is doing a lot of work. Most RAG systems fail earlier, at the point where a chunk gets treated as a self-contained unit of meaning. I wrote about the two skills that fix that gap, and why the chunk is usually the wrong thing to embed. Read it below. Thanks to Akamai Cloud for partnering today!show more

The Dawn of a New Era on $SUI (9)... Still in the festive spirit, let’s look at Tusky , Tusky is a storage service that's not controlled by one company. It uses something called WalrusProtocol to keep your data safe. Your data is encrypted from start to finish, so only you can see it. Instead of one place, your data goes to many different spots. This setup makes your data less likely to be lost or stolen. It helps keep your information safe and always available. Tusky gives you control over your files. You can easily manage them with the tools provided. You decide who gets to see your data. This makes it great for personal storage or working with others. It works with SuiNetwork for even more privacy. You can log in without sharing personal info, keeping everything more secure. This means only you can get to your data, with no third party involved. It is growing fast, with 100,000 uploads already. This indicates its increasing acceptance in the tech community focused on data sovereignty. It's good at managing lots of data safely. In tech, where you want to own your data, TuskyTools is popular. It gives users control over their information. This platform helps keep your data secure and gives you freedom.show more

The global industry standard oracle platform, Chainlink, is now... live on the Injective Mainnet. Time to market for dApp developers is critical and now with Injective’s EVM deployment, the iBuild AI dApp creator, onchain financial modules, and Chainlink data streams - developers on Injective can experience one of the fastest time to market anywhere in the industry. Chainlink brings real time Data Streams with sub-second latency to the only blockchain purpose-built for finance. This now gives Injective developers full access to Chainlink’s battle-tested data stream markets across the Injective ecosystem. Helix 🧬 will be the first Injective dApp to integrate Chainlink Data Streams set to power its crypto and RWA markets. Benefits for using Chainlink Data Streams on Injective: ✅ Low latency market data: up to sub-second delivery ✅ Institutional reliability: collaborating with institutional data giants like ICE and FTSE Russell, An LSEG Business ✅ Programmability: custom tailor formats, cadence, and fields to match a dApps exact needs. Injective markets now powered by Chainlink Data Streams. Ninjas are just getting started. 🥷show more

A new look for the new era of data... movement Over the past 12+ months we've been focused on building out mump2p, our Ethereum data propagation product, onboarding top validators to test it with, and educating our peers on the merits of decentralized coding. Now with the launch of mump2p on Ethereum mainnet firmly in our sights, it feels like the right time to give Optimum's visual identity a refresh-- something to match the sleek, powerful data acceleration network we're deploying. The advent of RLNC powered data movement frees blockchains from the networking bottleneck, so they can perform at the pace demanded by our ever-expanding digital economy. The new era begins soon with data propagation on Ethereum, more chains and use cases to follow. Time to raise the ceiling for blockchains, and do it in style.show more

Pretty human-like hand Beijing-based SynapX will unveil its tendon-driven... OctoH-Hand at WRC. Human-scale in size, the hand uses a hybrid architecture combining fully tendon-driven actuation with direct-drive motors in the forearm. It integrates 28 independently controllable actuators and 23 active DoF, along with tactile sensors embedded in the palm. Interestingly…SynapX is building more than just a dexterous hand. It has also developed a World model(SYNWorld), and a data collection system(OctoSense), creating a loop from data collection to world understanding and policy generation, and finally to real-world execution and feedback. Another physical AI bridge for humanoid robots.show more

What if crypto research was as easy as chatting... with ChatGPT, but powered by real market data👑 Introducing CMC AI, a powerful new tool from CoinMarketCap that combines the speed of AI with the depth of live crypto data. It delivers fast, data-backed answers to your questions: ✅ Want to know why Bitcoin's price is rising? ✅ Curious about the latest news on your favorite cryptocurrency? ✅ Need sentiment analysis? It pulls real-time data and explains it in seconds. But it goes far beyond basic Q&A. In the future, you’ll be able to ask anything! For example, you could ask it to: – Discover undervalued tokens based on volume, MC, and sentiment. – Compare Layer 1s or L2s by adoption, speed, and dev activity. – Detect rug-pull risk via wallet distribution and tokenomics red flags. – Break down your portfolio by risk, correlation, and potential return. – Explore new use cases in DeFi, AI, RWA and DePIN And much more! 🔗Try it here: CMC AI changes how you learn, think, and act in Web3🧠show more

I remember spending weeks digging through government sites, tracking... procurements, and building databases of China’s provincial leaders in my previous life as a journalist. The grind was real—slow and frustrating. I want to offer my friends in media and research with free structured web data with AgentQL . Need to track gov policies, procurement data, or economic trends? We just went live, and I’m excited to see how this can support the important work journalists do every day. Reach me out:show more

That’s why using Tails OS, Whonix, or Kodachi (this... looks like it might be the one- can anyone recognize the interface?) with automated data self-destruction is always a smart move… In the footage, two police officers can be seen “working” on the suspect’s computer. One pulls out a flash drive and plugs it in, only for the data to start self-destructing and wipe everything… As reported, the suspected owner of the deep-web marketplace escaped punishment thanks to a police officer’s mistake and was recently released.show more

Korea's NHN KCP runs a 2-second stablecoin payment pilot... on Avalanche NHN KCP confirmed today that it is running a commercial-feasibility pilot for stablecoin-based payments linked to its Payco easy-payment service. The test covers online gift certificate purchases inside the Payco app and offline payments at the cafe and cafeteria in the company's Seoul headquarters, with about 700 employees participating. The system runs on a payments-focused mainnet built in cooperation with Avalanche (Avalanche🔺). NHN KCP says it logged a 2-second processing time from QR scan to approval, and built what it calls the industry's first stablecoin payment admin page so merchants can track blockchain settlement data in real time without crypto expertise. NHN KCP plans to share the pilot data with financial-sector partners and large merchants to push toward commercialization. Another Korean payment major is moving stablecoins from product to infrastructure.show more

One of the best finance x AI workflows I've... ever built. It helped Claude print +$19,537 completely autonomously. This is my personal market Research Desk - and it took me weeks to build. Think of this as a fully local backtesting trading engine. You pick a market, a timeframe, and a strategy, and it runs that strategy against real historical price data to show you exactly how it would've performed. • It pulls real historical candles (crypto data straight from Binance, no API key needed, or your own CSV/stock data) • You pick from 400+ built-in strategies like moving-average crossover or RSI mean reversion, or bring your own logic • It shows a full interactive equity curve • Every single trade gets logged: entry, exit, P/L • It gives you the real performance metrics If you're not using AI in your trading, you're falling behind fast, and I think everyone should be building personal internal tools like this to elevate their portfolio.show more

What if a network could deliver AI results closer... to where data is created, instead of sending it to far‑off data centers? In collaboration with NVIDIA and Decart, we’re making that a reality by bringing GPU‑powered computing to the network edge — closer to homes and businesses — so AI applications respond in real-time. This is how we're laying the foundation for the next generation of AI‑driven services:show more