🚀Introducing The LLM Inference Provider Leaderboard - a live-updated,... unbiased eval of API Inference products. Featuring: Abacus.AI, Anyscale, DeepInfra, Decart, Fireworks AI, Lepton AI, Together AI, Perplexity, Replicate, as well as OpenAI and Anthropic models For each provider's Mixtral-8x7B and Llama-2-70B-Chat public endpoint, we benchmark cost, rate limit, P50 & P90 of throughput & TTFT, and average daily collections overtime for long term tracking. At Martian, we route each API request to the best LLM to reduce cost, reduce latency, and get the best performance. So finding the best providers is an important problem for us. We found that there's a > 5x cost difference, > 6x throughput variation, and even larger rate limit discrepancies among providers! Choosing between different LLMs is only part of the equation -- the selection of different inference endpoints is also crucial to get the best performance for your use case. See highlights of provider performance in🧵👇show more

⭐The Year of Inference is here. Featherless is now... an official inference provider on Hugging Face, unlocking 6,700+ LLMs for anyone to run, eval, and deploy instantly. It all starts with accessibility. From DeepSeek to Mistral, LLaMA to Qwen — powerful LLMs are one click away. We believe the future of AI is shaped by the long tail: personalized, specialized models tuned to real people’s needs. To get there, inference must be open, affordable, and usable by all. Whether you're fine-tuning, prototyping, or scaling a product, this moment is for you. 🫱🏻‍🫲🏻Let’s make inference the easiest part of building with AI. 📢 Share this so more builders know what’s now possible. Excited to be partnering with clem 🤗 Julien Chaumond Vaibhav (VB) Srivastav Simon Brandeis & Hugging Face team to take this to the next level!show more

"The best organizations won't manage token budgets manually. Instead,... they'll rely on orchestration layers that automatically route workloads to the right models based on performance, cost, and use case. Keeping track of which model is best for coding, finance, research, or support will become too complex for humans to manage directly, so intelligent routing will become a core part of enterprise AI infrastructure." Aravind Srinivas How will the best orgs of the future manage token budgeting Michael Mignano Nikesh Arora Ryan Petersen mmurphshow more

$ZKHIVE X $NMT 🤝 We are excited to announce... our major partnership with one of the leading AI projects in the space - NetMind.AI ! $NMT is an award winning AI powerhouse that has quickly become one of the most recognized decentralized AI platforms in the space, thanks to their wide range of offerings, from cost effective compute farms to elite AI consultancy services and AI chatbots. As part of our long term partnership, $ZKHIVE and NMT are going to build security AI models together 🤖🐝 #zkHive will also receive an access to $NMT’s compute farm as well as AI consultancy from $NMT’s team of experts, and $NMT will get access to all of #zkHive’s products and APIs! This partnership will improve zkHive’s AI capabilities and introduce thousands of new users to $ZKHIVE ecosystem!show more

Something NVIDIA & Google do better than anyone else... is software-hardware-system co-design, and not just optimizing hardware for current model architectures, but predicting future ones. Back in early 2022, when NVIDIA started the design process for NVL72, MoE (Mixture of Experts) models were not yet the standard, and dense models were still dominant for frontier models. However, NVIDIA's strong software-hardware co-design culture enabled them to make a calculated bet that MoEs were the future, and they built NVL72 specifically for best MoE performance per TCO (Total Cost of Ownership). Furthermore, back in 2022, disaggregated prefill and wide expert parallelism (wideEP) MoE inference optimizations hadn't been invented yet, but it turns out that these MoE inference optimizations work best on large-scale systems like NVL72. While most other AI chip companies' in-house AI labs focus on training small 5B models that mainly use data parallelism, NVIDIA and Google's in-house AI labs continuously push the boundaries of model architecture and training recipes, such as NVFP4 training. Just like Super Idol & IShowSpeed, there must be a strong partnership between software engineers and hardware engineers to deliver the best systems that maximize performance per TCO.show more

At Dreamforce, the exploration of the possibilities of AI... is incredible. At the same time, it's important that we understand that policies never move at the same speed as technology. So having a grasp over what is happening and how the negative aspects of AI don't proliferate in a big way, to minimize collateral damage is most important. I hope the companies and governments across the world wake up to this and ensure the best of AI possibilities will reach the people, as this is a phenomenal enabler for humanity like never before. -Sg Marc Benioff Dreamforce Salesforce #DF24show more

Season 2 of Fableborne went live this morning and... we are now trending in gaming! Hundreds of thousands of raids already complete! For 2 years we waited and waited, patiently building, restraining ourselves from all the noise and short term distractions focsuing just on building the best game to use as the base of what is to come. As a founder, working with this team each day, its easy for me to be unrealistically bullish. Its so satisfying to see the rest of web3 now seeing why.show more

Day 11/90 of Inference Engineering How does vLLM work... and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.show more

Moonshot AI is casually giving developers free daily access... to Kimi K3 😳 no subscription no upfront payment just sign in and start using one of the largest open AI models available what you get for $0: - Kimi K3 with 2.8T parameters - 1M token context window - strong coding and reasoning performance - native vision capabilities - free daily credits that refresh automatically why this is worth checking: > access a frontier model without paying API fees > long context for large codebases and documents > works on web, desktop, mobile, and CLI getting started takes less than 2 minutes: 1. go to 2. create a free account 3. Kimi K3 is available as the default model 4. start chatting or coding with your daily free credits bonus: Moonshot Together lets you invite friends for a chance to earn 3, 7, 15, 30, or even 365 days of Kimi Membership through its rewards program benchmark highlights: > 2.8T parameter MoE model > 1M context window > strong performance across coding, browsing, and reasoning benchmarks important: free credits reset daily, rate limits apply on the free tier, and the open-weight release is expected on July 27 A simple way to try one of the latest frontier AI models without paying for API accessshow more

We’re excited to introduce ShinkaEvolve: An open-source framework that... evolves programs for scientific discovery with unprecedented sample-efficiency. Blog: Code: Like AlphaEvolve and its variants, our framework leverages LLMs to find state-of-the-art solutions to complex problems, but using orders of magnitude fewer resources! Many evolutionary AI systems are powerful but act like brute-force engines, burning thousands of samples to find good solutions. This makes discovery slow and expensive. We took inspiration from the efficiency of nature. ‘Shinka’ (進化) is Japanese for evolution, and we designed our system to be just as resourceful. On the classic circle packing optimization problem, ShinkaEvolve discovered a new state-of-the-art solution using only 150 samples. This is a big leap in efficiency compared to previous methods that required thousands of evaluations. We applied ShinkaEvolve to a diverse set of hard problems with real-world applications: 1/ AIME Math Reasoning: It evolved sophisticated agentic scaffolds that significantly outperform strong baselines, discovering an entire Pareto frontier of solutions trading performance for efficiency. 2/ Competitive Programming: On ALE-Bench (a benchmark for NP-Hard optimization problems), ShinkaEvolve took the best existing agent's solutions and improved them, turning a 5th place solution on one task into a 2nd place leaderboard rank in a competitive programming competition. 3/ LLM Training: We even turned ShinkaEvolve inward to improve LLMs themselves. It tackled the open challenge of designing load balancing losses for Mixture-of-Experts (MoE) models. It discovered a novel loss function that leads to better expert specialization and consistently improves model performance and perplexity. ShinkaEvolve achieves its remarkable sample-efficiency through three key innovations that work together: (1) an adaptive parent sampling strategy to balance exploration and exploitation, (2) novelty-based rejection filtering to avoid redundant work, and (3) a bandit-based LLM ensemble that dynamically picks the best model for the job. By making ShinkaEvolve open-source and highly sample-efficient, our goal is to democratize access to advanced, open-ended discovery tools. Our vision for ShinkaEvolve is to be an easy-to-use companion tool to help scientists and engineers with their daily work. We believe that building more efficient, nature-inspired systems is key to unlocking the future of AI-driven scientific research. We are excited to see what the community builds with it! Learn more in our technical report:show more

S5 🚨: Anthropic is one of the most well-positioned... AI labs to disrupt consumer market this week, and all stars are aligning for them. - Super Bowl is the single biggest marketing event of the year - Sonnet 5, as a model name, is the best background for an advertisement - Cowork is an overpowered consumer tool that almost no one knows about - Sonnet 5 performance is at the next level Some S5 (non-thinking) generations 👀 h/t Леонидshow more

I’m incredibly proud to share that OpenAI chose Brex... to power their global spend and financial operations. When you're building at the frontier of AI and scaling global teams and infrastructure at an unprecedented pace like OpenAI is, Finance can't be the thing that slows you down. You need spend visibility the moment it happens, controls that enforce themselves, and agentic workflows that eliminate the manual work so your team stays focused on driving the business forward. We were so impressed by OpenAI’s rigor in evaluating every solution in the market, and whether they align to the agentic future OpenAI is building. Their decision to run on Brex is a huge testament to our AI roadmap and vision for the future of Finance. We started Brex around a simple idea: companies shouldn't have to choose between speed and control. There's no company in the world where that tradeoff matters more than OpenAI. We are honored to support them as they build the future. The best AI companies in the world, including OpenAI, Anthropic, Cursor, Vercel, Granola, Sierra, and Mercor choose Brex over every alternative for that exact reason. If you want to understand who’s truly building the future of AI in Finance, follow the customers you admire the most – not the hype.show more

LayerAI AI Agent Manifesto is Live: The Path Forward... 🧬 We've made it easy for ecosystem veterans & newcomers to get excited about the market & product opportunity we're tackling next: AI Agent Infrastructure. We're building an AI-powered agent platform where people can deploy, market, and succeed with this new token subcategory. At the heart of this transformation lies a challenge: primitive & so far limited tech & AI capabilities of incumbent platforms. We believe that LayerAI is equipped to rise as the new leading infrastructure provider for this market. LayerAI has already demonstrated market validation for AI Agents and looks to build on what we believe is the very start of this category in web3. 👉 Explore now:show more

We are entering an extremely exciting era for open-weight... models. Kimi K2.6 now feels like a top agentic model. I took it for a spin via Fireworks AI fast inference APIs. Kimi K2.6 has impressive agentic capabilities, design skills, and the ability to synthesize large amounts of information. I built a little Skill that produces survey papers on any AI research topic you want. (see example in the clip) You can use the skill to tell your agent to generate a survey on whatever topic and watch it go to work. The artifact was fully generated by Kimi.ai's Kimi K2.6. It's cheap and fast. Next step for me is to explore ways to continue integrating the capabilities of these models on use cases like automating my LLM knowledge bases and augmenting my agent memory capabilities. Stay tuned for more.show more

There is only one reason to share truth. And... no it has nothing to do with money- and everything to do about helping God move. Peace is the prize- a better world for everyone is the prize. We make SACRIFICES for things we want most in this world. Christ made the ultimate sacrifice If He can do that for us FREE of charge- we owe it to each other to work TOGETHER to make the world a better place- despite the toll and sacrifice. A level playing field- where we can all work hard and do our best to succeed- that is the payout. A better world for all We’re not here for just ourselves- otherwise the second commandment wouldn’t be like the first- love your neighbor as yourself. This world gets better the sooner we all wake up. Information is free- we collect and share it for free in hopes that it keeps spreading. Only working TOGETHER can we win. We are meant to move these messages for God so that humanity can wake up- this plan is designed that way- working together The Lord is ONE. We are bound as a family by Our Father in Heaven. We are apart of Him and He is apart of all of us. We are billions of little pieces of the Lords gift of life. We’ve been separated and pushed apart because it makes us weak. We were designed to be powerful by the love we draw from Creator and push through creation. We are ONE with Our Holy Father 🙏💗 Be EXCELLENT to each other frensshow more

We just dropped a game-changer to slash your Gemini... API costs 🤯 Yesterday we launched Gemini Context Caching, saving you up to 70% on your bill 💸💸 Here's how it works. Gemini let’s you add powerful AI models to create apps like: 📧 Email apps that summarize entire threads for you 💃 Storytelling apps that write the next chapter 💼 HR chatbots that actually understand your company's policies To make these apps, you need to repeatedly send Gemini instructions, which can be pricy. But through the integration of research, software and chips came a breakthrough; 🤖 Context Caching Just tell Google what you want to save money on and we'll take care of the rest. It’s an industry first and it’s going to be a big deal. I'm so proud of the team for always putting developers first 🙌 Cost is always a huge part of the equation and we keep making it better. Try Gemini Context Caching now and tell us what you think! Docs link is below. #AI #LLM #Geminishow more

BREAKING: Anthropic just dropped Opus 4.8—and it is a... MONSTER We've been testing for about a week Every 🧱 and our verdict is they could've just called it Opus 5, it's that good. Here's our vibe check: - Beats GPT-5.5 on Senior Engineer bench. On our toughest benchmark Opus 4.8 scores a 63—a hair higher than GPT-5.5's score of 62, and a full 30 points higher than Opus 4.7. It tackled a ground-up rewrite of a production codebase, and actually built something that works. HOWEVER: Coding performance varied a lot at different reasoning levels. We recommend using it on xhigh for best results. - Incredibly good writer. Opus 4.8 scored a 79.6 on our writing benchmark—measuring models on real-world writing tasks we do all of the time like essay writing, promo email writing, and more. It beats GPT-5.5 by 6 points. It produces well-written prose with fewer "AI-isms". It's also very good at writing in your voice given the right context. HOWEVER: Writing performance also varied with reasoning levels. Medium reasoning had higher incidence of AI-isms—we found best results with high. - Beast at knowledge work. Opus 4.8 is very good at general knowledge work tasks like report creation, research and more. It produced the best PowerPoint one-shot we've ever seen on our deck generation benchmark. - Emotionally intelligent, willing to question the frame. I've also found it to be quite good at talking through psychological or interpersonal issues. It has a high EQ, and it's also good at not glazing and helping to expand your perspective. Its thought process feels extremely rich and dynamic. THE BAD: These days a model is only as good as its harness, and Codex is still a far superior harness to the Claude Desktop app. This has kept me using Codex + GPT-5.5 as my daily driver, but I am flipping back and forth a lot more between Codex and Claude. Anthropic is back baby! Read the rest on Every 🧱:show more

Unpopular opinion: Most agent evals are theatre. You run... them once before the deployment. It'll take 800ms+ as another LLM would be judging your LLM. Most annoying part - no one tells where in the chain things went wrong. I wasted a lot of time in this loop. And then I came across Future AGI bringing 5 different tools under one umbrella, best part - the platform is completely open source. They open sourced their entire platform and the eval layer is noticeably different. It is multimodal - works on everything text, image, audio, pdf. Not an LLM-as-judge adding latency but an agent with memory and tools. The biggest win are learned classifiers trained on actual production failure patterns to run evals at low cost. It also runs across the full reasoning chain, not just the final response. Check out → Try it here →show more

Why is the market selling off today? (Save this).... The semi selloff right now is being driven by a mix of macro fear, profit taking and investors questioning how quickly all of this AI spending will actually pay off, not because demand for AI infrastructure suddenly disappeared. The market is basically trading this chain reaction, the ongoing US Iran escalation pushes oil higher, higher oil keeps inflation elevated, sticky inflation keeps Treasury yields high and that increases the risk of the Fed staying hawkish or even hiking again. That is a terrible setup for semis because many of these companies are valued on the massive earnings investors expect them to generate years from now. When yields rise, those future earnings become worth less today which is why the highest multiple AI and semiconductor names usually get hit first. (I don't think there will be a hike this year). This is also why everything is moving together right now. Nvidia, Micron, Nebius, SanDisk, Broadcom and Applied Optoelectronics are all completely different businesses, but institutions are not separating memory, networking, optics, compute and cloud infrastructure at the moment. They are reducing exposure to the entire AI trade, taking profits in the names that have already run the most and moving into a more defensive position potentially ahead of the Fed. There is also growing pressure around hyperscaler capex. Microsoft, Meta, Amazon and Google are still spending enormous amounts on GPUs, data centers, networking and power but the market is starting to ask when all of that spending will actually turn into revenue and free cash flow. Investors are no longer satisfied with hearing that AI capex is growing. They want proof that the returns are arriving fast enough to justify the valuations already priced into the entire AI ecosystem. That creates a weird situation where hyperscaler capex can continue rising while semiconductor stocks still fall. The market is not asking whether AI spending is growing anymore but rather asking whether it is growing fast enough to beat the expectations already baked into these stocks. Crowded positioning is another major factor. Semis and AI infrastructure stocks have been some of the biggest winners in the market so institutions are sitting on huge profits and many funds own the exact same names. When macro risk increases, investors usually sell the most liquid winners first. That does not mean demand for memory, optics or custom chips suddenly collapsed but rather means investors are locking in gains and reducing risk. Tariffs add another layer because even when they are not directly placed on chips, they can still raise the cost of servers, electrical equipment, cooling systems, construction materials and the overall data center buildout. That makes AI infrastructure more expensive while also adding another source of inflation. Then you have Jensen Huang’s letter to the White House this morning about open weight AI models, which I think is one of the most important long term developments here. Nvidia, Meta, Microsoft, Palantir and several other companies are pushing Washington not to place broad restrictions on open weight AI. OpenAI and Anthropic were notably absent because open models are much more of a threat to their business models. OpenAI and Anthropic benefit from a world where a few closed frontier labs control the best models and companies have to pay them through subscriptions and APIs. Open weight models weaken that advantage because businesses can download a model, customize it for their own use and run it on their own infrastructure or through a neocloud. That is bad for OpenAI and Anthropic because it puts pressure on pricing, margins and the idea that they will control the intelligence layer of the economy but it is very good for the AI ecosystem as a whole over the long run. But the question is what does this mean for all the OpenAI and Anthropic commitments? so that's adding to the fear as well. But with that being said open models make AI cheaper and more accessible. Instead of AI being controlled by a few giant labs, thousands of startups, universities, governments and regular businesses can deploy models themselves. That spreads AI adoption across the entire economy and creates a much larger infrastructure opportunity and that is exactly why Jensen cares. Nvidia does not need OpenAI or Anthropic to win. Nvidia just needs more people using AI. Whether the model comes from OpenAI, Anthropic, Meta, Mistral, Kimi or some startup nobody has heard of yet, it still needs GPUs, memory, networking, data centers and electricity. So open weight AI could actually weaken the model companies while making the infrastructure layer much bigger. More open models mean more companies running inference. More inference means more GPUs. More GPUs mean more HBM, optical transceivers, switches, data centers and power. That is bullish for Nvidia Nebius, Micron, Broadcom , Marvell and Applied Optoelectronics over the long run. So my take is that the current semi selloff is being driven mostly by macro uncertainty, higher oil, rising yields, Fed fears, tariffs, crowded positioning and questions around the return on hyperscaler capex. The underlying AI infrastructure thesis has not suddenly broken. We are not broadly seeing hyperscalers cancel GPU orders, slash capex, abandon data center projects or report that AI demand has collapsed. What has changed is the valuation investors are willing to pay while the macro environment remains unstable. The market is lowering the price it is willing to pay for semiconductor growth but is not necessarily saying that growth is gone. And while Jensen’s open weight push may be bad for OpenAI and Anthropic, it could be one of the best things possible for the AI ecosystem over the long run because it creates more models, more developers, more competition and ultimately much more demand for the infrastructure underneath all of it. Nothing about the AI thesis has changed for me, so I will be going shopping and taking advantage of this sale while the market is selling everything together. I am an analyst at Milk Road Pro, and if you want to see exactly what I am buying, you can join for just $1 using the link below.show more

Introducing - Spectre AI: The Monarch - Artificial Intelligence... Searching Reimagined Introducing a sneak-peak of the next core feature in the Spectre AI Search Engine: The Monarch aka the AI ChatZone. The Monarch will serve as your AI agent, providing not only price information, technical and sentiment analysis, and project discovery, but also offering visual assistance through Spectre AI's integrated utilities within the search platform. Real-time blockchain data integration comes with the support of Google for Startups. The AI ChatZone's landing page will feature projects that users have added to their UI Watchlist, allowing for quick and seamless discussions about their favorite projects. Additionally, the platform will include hyperlinked external sources, enabling users to access a wealth of information and resources directly from within the ChatZone. In this showcase, we demonstrated how the system responds to inquiries about $Palm AI's (PaLM AI - $PALM) price performance. The user interface and user experience (UI/UX) are fully developed, and the frontend is complete. Our backend is now entering the beta testing phase. The MVP is next in line to be showcased. Get ready for a new journey in AI with Spectre AI. #ai #artificialintelligence #google #nvda #nvidia #tech #spectre $SPECTshow more