LLMs require more GPU memory as they generate longer... responses. Can we make GPU memory constant without significantly sacrificing accuracy? IceCache is a new method for managing KV caches that leverages Dynamic Continuous Indexing (DCI) to efficiently group and retrieve tokens by semantics. Joint work w/ Yuzhen Mao, Qitong Wang and Martin Ester. For details, check out the links below.show more

Day 11/90 of Inference Engineering How does vLLM work... and how is it used in production? Before we discuss how vLLM works internally, it helps to understand what vLLM is. At a high level, vLLM is an inference engine that is designed to serve LLMs to thousands of concurrent users efficiently while managing scarce compute and memory. The goal for vLLM is to maximize throughput and minimize latency; optimizing for the best inference economics and experience for end users. With every request from the end user, it eventually ends up in the engine core, gets scheduled alongside other requests from other concurrent users, executes on the GPU, and updates the KV cache with the new key and value vectors, and streams the tokens back to the user. The Scheduler decides what requests should execute next while continuously batching requests together to maximize GPU utilization. Continuous batching is an inference optimization that allows new requests to join a running batch as other requests finish generating tokens. This helps with keeping the GPU utilization high instead of letting it sit idle waiting for an entire batch to complete generating. After the scheduler dispatches the selected batch to the Model Executor, the Model Executor prepares the tensors and metadata required for inference, retrieves each request’s block table from KV Cache Manager, launches the optimized transformer forward pass on the GPU, computes the logits, updates the KV cache with the new key and value vectors, and finally returns the results for sampling and streaming. The KV Cache Manager uses the PagedAttention memory layout to allocate fixed-size cache blocks on demand and maintains a Free Block Queue on the CPU that tracks which blocks in the GPU’s Paged KV Cache are currently free. When a request needs additional KV cache space, the KV Cache manager takes a free block from the queue and assigns it to that request, thus avoiding an expensive search through GPU memory for available cache blocks. All of these components form the core of vLLM’s inference engine. The Scheduler determines what requests are executed, the Model Executor determines how those requests are executed, the KV Cache Manager determines where each request’s KV cache lives using the PagedAttention Memory Layout. This architecture enables vLLM to serve thousands of concurrent requests with high throughput, low latency, and efficient GPU memory utilization. Heres a little animation that visualizes everything! - I've also completed the forward pass for my mnist.c project. I had a nice chat with shrey birmiwal, such a knowledgeable guy. Excited to learn more about vLLM and implement a tiny-vLLM one day.show more

Free NVIDIA GPU with 16 GB VRAM GPU for... Running Local LLMs! If you want to master local LLMs but you're waiting until you can afford a $1,500 GPU, you're honestly not going to make it. The open source AI ecosystem is moving way too fast for you to wait on your budget to catch up. Especially when you can build a bleeding edge inference engine from scratch right now, completely for free. You don't need a heavy local rig to start. Google is literally letting you use an enterprise grade NVIDIA Tesla T4 GPU for $0/hour. At standard cloud computing rates (~$0.20/hr), Google Colab’s 4 hour daily free tier hands you roughly $24 worth of data center tier GPU compute every single month. And most people just waste it. Let’s talk about the hardware you get access to for free. The NVIDIA Tesla T4 is an absolute workhorse: - Architecture: NVIDIA Turing (TU104) - VRAM: 16GB GDDR6 (320 GB/s bandwidth) - Compute: 320 Tensor Cores | 2560 CUDA Cores - Performance: 130 TOPS INT8 | 8.1 TFLOPS FP32 - Power: Sipping energy at a max 70W TDP This is the exact same hardware I used to run DeepMind's Gemma 4 26B A4B QAT MoE at a 250,000 context window without a single Out Of Memory (OOM) crash. If you have a web browser and 10 minutes, you have everything you need. I’ve put together a fully documented, cell by cell Google Colab notebook that teaches you exactly how to do this. Here is what the notebook actually teaches you: - How to provision an Ubuntu Linux environment with CUDA 13.0 and verify your driver stack. - How to pull the source code and compile the latest llama.cpp C++ binaries from scratch, specifically optimizing the build for your exact GPU using the -DCMAKE_CUDA_ARCHITECTURES=native flag. - How to directly download quantized local LLMs (GGUF format) straight from HuggingFace using the CLI. - How to manage 16GB VRAM limits, offload neural network layers to the GPU, and push massive context windows. Compile raw llama.cpp, ollama run a model, or spin up the LM Studio CLI. Pick whatever stack you are comfortable with. just start building. No hardware. No credit card. No excuses. Bookmark this post right now so you don't lose the tutorial. Even if you don't have time to run it today, you are going to want this workflow in your engineering toolkit. The link to the free Colab Notebook is in the comments below. Lemme know if you need more tutorials like this.show more

Introducing StreamingLLM. Imagine chatting with an AI assistant that... can contextually reference your conversations from weeks or months ago. Or summarizing reports that span thousands of pages. StreamingLLM makes this possible by enabling language models to smoothly handle endless texts without losing steam. Current LLMs are like students cramming for an exam - they can only memorize a limited context. StreamingLLM is the valedictorian with a photographic memory of everything you've ever discussed. It works by identifying and preserving the model's inherent "attention sinks" - initial tokens that anchored its reasoning. Combined with a rolling cache of recent tokens, StreamingLLM delivers up to 22x faster inference without any drop in accuracy. You know that irksome feeling when chatbots forget your earlier conversations? StreamingLLM abolishes that frustration. It remembers the touchdowns from your last game and your newborn's name without missing a beat. Monumental books, verbose contracts, drawn out debates - StreamingLLM takes them all in its stride. No shortcuts, no forgetfulness. It's like upgrading your assistant's RAM to handle heavier workloads flawlessly.show more

🚨 Anthropic committed up to 1M TPU chips for... Claude. Openai is leasing TPUs for chatgpt inference. Here's How kernels work on TPUs (deep dive 2/6 by emi) pallas is Google's answer to kernel writing. a python kernel SDK built on JAX. still very experimental (jax.experimental.pallas). on TPU it compiles through mosaic; on GPU it lowers to triton. if you know CUDA, the syntax will feel familiar but the execution model is completely different. in CUDA, grid=(4,4) launches 16 blocks running simultaneously across SMs. in pallas, those 16 iterations run one after another in lexicographic order. no threads. no warps. no blocks. no occupancy tuning. a TPU is a sequential machine with a very wide vector register — more like a CPU than a GPU. performance comes from width: a 128x128 systolic array doing matmul and an 8x128 SIMD vector unit doing everything else. maximum parallelism on chip: 2, one per TensorCore in megacore mode. three concepts replace CUDA's thread/block/grid hierarchy. Refs are mutable memory references. because execution is sequential, each iteration safely accumulates without atomics. in CUDA you'd need atomics or a separate reduction pass. the memory model is also very different from NVIDIA's. zero hardware caches. VMEM is 32-128 MiB of software-managed scratchpad — 500-1000x larger than GPU shared memory per SM. all data must be explicitly DMA'd from HBM to VMEM before any computation touches it. four levels: HBM → VMEM → VREGs → MXU/VPU, plus SMEM for scalar control data. every byte of data movement is your responsibility. this is like CUDA shared memory except it's 500x bigger and there's no cache fallback. pipelining is mandatory. without double-buffering HBM→VMEM transfers, the MXU just stalls waiting for data. this is the single most important optimization on TPU. and because grid execution is sequential and deterministic, consecutive iterations that need the same input block skip the redundant HBM transfer automatically, impossible on GPU where block execution order is undefined. the compilation pipeline is unlike anything in this series: python → jaxpr → stableHLO → XLA HLO (71+ optimization passes) → LLO (78+ passes) → 322-bit VLIW bundles. the compiler packs instructions for scalar, vector, matrix, and DMA units into a single 322-bit word. everything in that bundle executes in parallel, with no runtime scheduling.show more

$IREN "we haven't disclosed the specific amount of GPUs"... 1. 🤮 reminds me of $NBIS 2. Setting a terrible precedent here for future deals 3. Making it purposely difficult, to not let analysts properly value your 2027 revenue 4. Increasing the polarized view on IREN by the market However: "approximately 60MW of air-cooled Blackwells" 1. You typically don't talk about gross capacity in a deployment like this 2. If it would be gross capacity, the GPU hour rate at IT level would be crazy high (at PUE 1.2, $680m / 50 = 13.6m/MW) 3. At 60MW IT load, and ~14kW draw at DGX server level, we can get to ~4,286 DGX systems with 8 GPUs per. 4. Based on this we can conclude that 60MW of IT load can run approximately 34k DGX B300. 5. 34k DGX B300 at $680m/yr, would represent a GPU hour price of $2.28 Now this is the problem with not disclosing your GPU quantity. You purposely make your business model look bad, because by approach, you get to a GPU hour price that would imply a payback period of 4 years, where only the last year of the contract is 100% margin. But of course, we can also take "the glass is half full" approach. IREN has ordered 50K B300s from Dell. They have 2 purchase orders for this, 1 between Dell Canada and IE CA Leasing Ltd for 4 phases, and 1 between Dell USA and IE US Hardware 1 Inc (amended from IE US Hardware 4 Inc on April 27, 2026). The order for Canada is divided in 4 phases, and are going to Mackenzie for 80MW of gross capacity, which happens to be 4 buildings of 20MW. The order for Childress is divided in 2 phases, and are going to DC35 and DC36, (as depicted in the earnings presentation) and those are 50MW gross. The purchase price of the order for Childress was $1.2B, and for Canada it was $2.3B If we go with 50,000 B300s for a total of $3.5B then $1.2 would represent 34.285% of the 50,000 GPUs, or 17,140 B300s rounded down. For this calculation I will consider that $IREN will deploy 17,140 GPUs in 50MW gross capacity in DC35 and DC36 of block 3 in Childress.. That would imply at 1.2 PUE, IREN can run 17,140 B300s in 41.67MW IT load. Now by that ratio, they can run 24,680 GPUs in 60MW IT load — a massive difference with 34k units through the Nvidia DGX reference calculation. If common sense is applied, you can still get to 2 completely different outcomes, that show a difference of more than 9k GPUs. The GPU hour rate at 24.68k GPUs would be $3.145 per B300, as MASSIVE difference from the earlier calculated $2.28. Sure, the DGX system may be a factor here. And I'm sure that the reality is somewhere in the middle. But I personally hate this as an investor, to be unable to calculate profitability on unit economic basis. After all, contracts are signed on a $/GPU hour basis. Why hide this from your investors? Not being able to calculate payback periods, unable to calculate ROIC. And most importantly, we cannot properly assess the $NVDA deal on a contract basis. I really hope the payback period of this contract is not 4 years. I want the glass to be half full, but by starting to censor the purchases, IREN is taking a step in the wrong direction. Not a fan of this.show more

India should host the biggest vibe coding conference the... world has seen 🚀 Everything about Vibe coding(talks, panels, AI product building, hackathon, including the biggest names in the world) come here for an ultimate showdown(think 10k+ people in a Vibecoding conference) 🔥 And I have a plan Why? Because the number of learners, founders and professionals who are building via vibe coding in India and powering the global platforms is testament to the fact that if it should happen anywhere, it is here For the last 4+ years, I have enabled 3000+ people to build and continue to do so in a new avatar(announcement soon) and I believe if the entire ecosystem is willing to come together we can create a real spectacle We have the skills, the access and the experience to pull this off. The only thing it needs is for everyone seeing this to reach out and join hands(partners, sponsors & more) to make this as big as possible I am so hyped about it that the website is set, the name is set(Vibecon) and we can make an incredible run to make it happen. Reply to this post if you think we should do this, if we have >500 responses we will get to work 🥳 Reach out if you have ideas to partner to make the biggest Vibecoding conference a reality ♥️show more

TWIN FLAME LOVE IS NOT A SPARK THAT FADES... WITH ME. It is not sustained by excitement, romance, or constant closeness. It is sustained by truth. This is why it does not burn out. Twin flames share a frequency, not just emotions. Even when they are apart, the connection continues to exist because it is not dependent on words, actions, or physical presence. It lives in the nervous system, in memory, in the way awareness shifts after the meeting. Once activated, it does not return to what it was before. This love does not consume itself the way ordinary passion can. It matures. It moves through phases of intensity, silence, confusion, distance, and clarity, yet the core recognition remains unchanged. What changes is the capacity of each person to hold it without fear. When separation happens, the love does not disappear. It reorganizes. It turns inward and begins to work on unresolved wounds, attachment patterns, and old survival responses. The connection stays alive because it is no longer ted by chasing or longing, but by integration. Twin flame love endures because it is not trying to prove itself. It does not require constant reassurance. It is quiet when needed, intense when allowed, and steady beneath all cycles. Even in moments of doubt, something deeper continues to recognize the other as familiar, safe, and true. This is why twin flame love does not burn out. It is not fueled by emotion alone. It is carried by awareness. And awareness, once awakened, does not extinguish. ~ Twinflame Infinity ✨🙌🏽💫show more

Twin flame love is not a spark that fades... with time. It is not sustained by excitement, romance, or constant closeness. It is sustained by truth. This is why it does not burn out. Twin flames share a frequency, not just emotions. Even when they are apart, the connection continues to exist because it is not dependent on words, actions, or physical presence. It lives in the nervous system, in memory, in the way awareness shifts after the meeting. Once activated, it does not return to what it was before. This love does not consume itself the way ordinary passion can. It matures. It moves through phases of intensity, silence, confusion, distance, and clarity, yet the core recognition remains unchanged. What changes is the capacity of each person to hold it without fear. When separation happens, the love does not disappear. It reorganizes. It turns inward and begins to work on unresolved wounds, attachment patterns, and old survival responses. The connection stays alive because it is no longer fed by chasing or longing, but by integration. Twin flame love endures because it is not trying to prove itself. It does not require constant reassurance. It is quiet when needed, intense when allowed, and steady beneath all cycles. Even in moments of doubt, something deeper continues to recognize the other as familiar, safe, and true. This is why twin flame love does not burn out. It is not fueled by emotion alone. It is carried by awareness. And awareness, once awakened, does not extinguish. ~ Twinflames.Infinity ✨🙌🏿💫show more

Run Gemma 4 26B MoE on 8GB VRAM with... 250k context at 20+ tokens/sec If you own any 8GB VRAM graphics card, stop what you are doing. Local AI just had its absolute "Holy Shit" moment for budget hardware. Yesterday, I benchmarked Unsloth Gemma 4 12B Q4_K_XL on an 8GB card. The community went wild but immediately demanded more: "Can we run a 25B+ model on budget GPUs?" Today, I’m delivering exactly that. I am running a massive 26B parameter Mixture of Experts (MoE) model locally on a standard 8GB VRAM setup with 250k full native context!. If you own an RTX 3060, 3070, 4060, or any budget GPU with 8GB of VRAM, the local AI paradigm has completely changed. The performance metrics are astonishing: - 20 tokens/sec flat decode throughput. - Stable, flat decode speed even with massive prompts. - I threw a 60k token prompt at it, and it still clocked in at 20 TPS without dropping a single frame. # What about prefill? Yes, Time To First Token (TTFT) is slightly high when swallowing massive contexts. But with a solid 200 tokens/sec prefill speed, the wait is barely noticeable and highly usable. And this is running completely without Multi Token Prediction (MTP) active. How is this possible? It’s the magic of Google's new QAT (Quantization Aware Training) quants for Gemma 4. The model weight file (unsloth gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf) is only 13.2 GB, making it the ultimate local powerhouse. # The Test Setup: CPU: Intel Core i7 RAM: 16GB System RAM GPU: NVIDIA GeForce RTX 4060 Laptop GPU (8GB VRAM) # The Secret Sauce (The -cmoe Flag) To make this work properly on any 8GB card, you must use the -cmoe (CPU MoE) flag in llama.cpp. This flag isolates the heavy MoE expert weights directly to system memory (CPU/RAM) while letting your GPU focus strictly on the Attention layers and the KV Cache. It prevents VRAM spillage and holds the throughput rock solid. # The flags: -m "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf" -cmoe -c 248000 -v Once running, just open the UI on localhost and toggle the new reasoning lightbulb icon in the text input box to watch the model perform multi step thinking. Are you still running smaller models, or are you ready to scale up your budget local setups? Let's discuss in the repliesshow more

Dad’s home! 🏠 Because it’s one of the most... frequent questions I receive, here’s your periodic reminder that your baby doesn’t forget you when you go to work. There’s a developmental milestone known as object permanence and sometimes parents overthink it. In a nutshell it’s the understanding that things continue to exist even when they’re out of sight - and it typically develops somewhere between 6-9 months. Does that mean you cease to exist (to your baby) when you go to work or the grocery store? It can be kind of a scary thought to new parents. The answer, even before the development of object permanence, is no. But it may mean that (even for your toddler) you’re misattributing adult forms of memory to your little one. Infants and toddlers both live very much in the moment and are still developing the type of robust narrative memory we possess as adults. So while you’re gone, it’s not that you cease to exist. It’s more a case of “out of sight, out of mind.” Will they recognize you when you return? Will they be excited to see you? I’ll let this beautiful video from beingritty on IG show you the answer.show more

The West is not dying. It is being killed,... and the names of the traitors are known. They occupy our capitals, infest our courts, pollute our newsrooms, and preach in our churches. They open the gates, kneel before the foreigner, and smirk as their own blood is driven from the land. They mock the fallen, defile the heroic, and spit on the blood that raised every city worth defending. They are not misguided. They are not mistaken. They are the enemy. They must be treated as such. For too long, we have been ruled by cowards, “men without chests,” by merchants loyal to nothing but the dollar, by liars who speak of progress while presiding over decay. A new generation now rises, armed not with apologies but with the fire of remembrance, with the memory of what we once were and the will to become greater still. We do not ask permission. We do not seek approval. We will reclaim what is ours, because no one else will. Victory will not come through debate. It will come through discipline, through will, through the unbreakable decision to endure, to outlast, and to return to the excellence and greatness that befit our people. We do not need millions. We require only a vanguard: men of loyalty, endurance, and resolve, hardened by truth and unmoved by fear. I say this not for approval, nor is it offered in hope of a reply, but in the spirit of doing what must be done. It is a promise made in full knowledge of what must come. The time of submission draws to a close. The age of reconquest begins. Let the traitors tremble. Let the weak, the feckless, and the unworthy fall away. The future belongs to those with the strength and the daring to seize it.show more

Model-Free Reinforcement Learning (MFRL) has been alluring, especially with... supercharged compute with physics on GPU. However, the methods use 0-th order gradients, and are often not the best optimizers. Can we do better than PPO in continuous control for robotics? Turns out yes! 🥳 tl;dr: Faster, better RL than PPO in continuous control 💪 The answer lies in using more information from the simulation. We are juicing the simulation on GPU as it is, why not use it for gradients as well? This has been a driving question in a series of our works. We first studied this problem in ICLR 2022 paper on Short Horizon Actor Critic Naive gradient based methods are stuck in local minima and have exploding/vanishing gradients. SHAC solved this problem truncated rollouts and model based value estimation, where the model is Differentiable Sim. This boosted sample efficiency and wall-clock time immensely especially in high dimensional systems such as humanoids Yet, given enough compute PPO often caught up. Our follow up paper on on Adaptive Horizon Actor Critic at ICML 2024 discovers the cause and provides a fix. However, we find that even when given ground-truth dynamics, not all gradients are useful due to sample error. 1st-Order Model-Based Reinforcement Learning methods employing differentiable simulation provide gradients with reduced variance but are susceptible to bias in scenarios involving stiff dynamics, such as physical contact. We find that back-propagating through contact and long trajectories drastically reduces gradient accuracy. Using this insight, we propose AHAC to dynamically adapt its roll-out horizon to avoid differentiating through stiff contact. AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes (wall clock) and outperforms PPO by 40%, even in the limit of data provided to PPO. This work is led by Ignat Georgiev alongside Krishnan Srinivasan, Jie Xu, Eric Heiden and ample assistance from warp team at NVIDIA Robotics (Miles Macklin)show more

What if you kept asking an LLM to "make... it better"? In some recent work at FAIR, we investigate how we can efficiently use RL to fine-tune LLMs to iteratively self-improve on their previous solutions at inference-time. Training for iterated self-improvement can be costly. The naive approach to training for K self-improvement steps leads to K times the number of rollout steps per episode. We introduce Exploratory Iteration (ExIt), an RL-based automatic curriculum method that bootstraps diverse training distributions of self-improvement tasks by upcycling the LLM's own responses at previous turns as the starting points for both self-improvement and *self-divergence.* In order to decide what task to train on next, the curriculum prioritizes sampling of partial turn histories that led to higher return variance in its GRPO group (a learnability score that comes for free). This automatic curriculum over the bootstrapped task space teaches the model how to perform iterated self-improvement while only ever training the model on single-step self-improvement tasks. We look at ExIt's impact in both single-turn (contest math problems) and multi-turn (BFCLv3 multi-turn tasks), as well as MLE-bench, where the LLM is run in a search scaffold to produce solutions to real Kaggle competitions. Across these eval settings, we find ExIt produces models with greater capacity for inference-time self-improvement compared to GRPO. Notably, ExIt models can self-improve on test tasks for many more steps than the typical solution depth encountered during training, including a 22% improvement in MLE-bench performance compared to GRPO.show more

I had the same thought so I've been playing... with it in nanochat. E.g. here's 8 agents (4 claude, 4 codex), with 1 GPU each running nanochat experiments (trying to delete logit softcap without regression). The TLDR is that it doesn't work and it's a mess... but it's still very pretty to look at :) I tried a few setups: 8 independent solo researchers, 1 chief scientist giving work to 8 junior researchers, etc. Each research program is a git branch, each scientist forks it into a feature branch, git worktrees for isolation, simple files for comms, skip Docker/VMs for simplicity atm (I find that instructions are enough to prevent interference). Research org runs in tmux window grids of interactive sessions (like Teams) so that it's pretty to look at, see their individual work, and "take over" if needed, i.e. no -p. But ok the reason it doesn't work so far is that the agents' ideas are just pretty bad out of the box, even at highest intelligence. They don't think carefully though experiment design, they run a bit non-sensical variations, they don't create strong baselines and ablate things properly, they don't carefully control for runtime or flops. (just as an example, an agent yesterday "discovered" that increasing the hidden size of the network improves the validation loss, which is a totally spurious result given that a bigger network will have a lower validation loss in the infinite data regime, but then it also trains for a lot longer, it's not clear why I had to come in to point that out). They are very good at implementing any given well-scoped and described idea but they don't creatively generate them. But the goal is that you are now programming an organization (e.g. a "research org") and its individual agents, so the "source code" is the collection of prompts, skills, tools, etc. and processes that make it up. E.g. a daily standup in the morning is now part of the "org code". And optimizing nanochat pretraining is just one of the many tasks (almost like an eval). Then - given an arbitrary task, how quickly does your research org generate progress on it?show more

May the souls of our departed UNRWA colleagues rest... in peace, and may we draw strength from their example as we continue our vital work in the pursuit of a better world. Below is my full speech, as we gathered at World Health Organization (WHO) to honour our colleagues, who lost their lives, with a minute of silence: Dear colleagues, Today, we gather with heavy hearts to remember and honor our dear UN colleagues who tragically lost their lives in #Gaza. Their unwavering commitment, sacrifice, and bravery will forever be etched in our memories. These dedicated individuals embodied the spirit of the United Nations, standing on the frontlines of conflict zones to provide much-needed humanitarian assistance and support. They put their lives on the line to bring hope and relief to those affected by violence and despair. Their memory and the impact they made will forever remain. Their unwavering dedication to peace, justice, and the well-being of others serves as a guiding light and a reminder of the importance of our shared mission. In the face of immense challenges and danger, they remained steadfast in their commitment to make a difference in the lives of those they served. Their courage and selflessness touched the lives of countless individuals and communities. Today, we honor their memory and recommit ourselves to the pursuit of peace, justice, and the well-being of all people. We extend our deepest condolences and support to the families and loved ones left behind. Their loss is immeasurable, and we stand in solidarity with them during this difficult time. As we reflect on the lives of our fallen colleagues, let us also recognize the importance of unity and collaboration. Together, we can continue their important work and strive for a world where conflicts are resolved peacefully, where the rights of all individuals are respected, and where humanitarian aid reaches those in need without impediment. May the souls of our departed UN colleagues rest in peace, and may we draw strength from their example as we continue our vital work in the pursuit of a better world. May the path to lasting peace be paved for Palestinians and Israelis. May the wounds of the past be healed, and may Palestinians and Israelis find true peace, reconciliation, and coexistence, fostering a future of mutual respect, prosperity, and harmony. I now ask you to join me in a minute’s silence in honor of their memory and sacrifice, and for peace.show more

Increasingly, HTML Artifacts are becoming a core part of... how I work with AI agents. Long-horizon agent sessions need a better way to surface insights about what work it has done. This may not be obvious right now, but as you start to let your agent work on dynamic workflows, large codebases, long-running loops (e.g., using /goal), and deep research tasks, you need a good way to present results. Chat window is not it. You also don't want to just trust everything the agents do. Artifacts help provide an important verification layer, which in turn enables important decision-making. I like HTML artifacts because I can just ask the agent to produce as many of them (and in whatever form) as I need to verify the work and make sense out of everything. I even built a nice tab system for my artifacts. They are great for continual learning and research. I use HTML artifacts for logging, tracking experiments, brainstorming, managing my inbox, code reviews, agent session management, deep research, writing, reading, and so much more. I believe Andrej Karpathy wrote about this somewhere: As we move on to more advanced applications of AI agents and outputs get more complex, we will start to find the need for even more advanced forms of interactions with AI, including interactive neural videos/simulations.show more

The Sabotaging Practice of Over Supply and Sameness in... the NFT Space. The current zeitgeist of the NFT space is that the same artists are doing the same kind of work five times a year, with project after project leaving a trail of disappointment and discontent among collectors and all of us watching in disbelief as huge resources are extracted from the space over work that feels like it could be left as an "artist study." I understand that you can do what you want with your money as collectors, but we are killing the whole space with this incestuous practice. No artist is that prolific to be able to do 5 collections of 100+ pieces each every year and actually deliver innovation and some kind of creative evolution. Of course, they can pretend play that the work has something new, but there is no precedent nor proof that that has ever happened in the speed that it happens in the NFT space. Again, people are free to through away their resources on whatever they want but with this way of doing things, we more and more are going to start seeing the consequences. Oh! There are consequences? Yes. Maybe unintended, but there are. Let's see. Let's start with the loss of belief in the NFT space as somewhere where emerging artists can come and find support for their experiments. Why even bother to bring experiments, innovation, and new ways to think of art on the blockchain if the same people have all the collectors hypnotized with their magical flutes? Why even try to come to a space where taking risks and challenging the status quo (the mission of art!!!) is overlooked? This makes the NFT space a social club and not a space for art. I guess it is fine, but IMO it is a recipe for disaster. New collectors stay away because the art will slowly but surely become stale and un-challenging. Why even bother to come and see what is happening here if you can't, as a collector, see new weird and up-and-coming artists? The amount of noise emitted by the same artists doing the same art over and over, drowns out any new voices. Again. A recipe for disaster. The NFT space is becoming a space of disappointment and doubt. We think that collections going to zero one after the other, over and over, is not damaging? I feel we are kidding ourselves. Disappointment piles up, and again, the people who will hurt are the emerging artists, the new blood, the ones who are willing to risk the most and, in return, put fire in this cold space of sameness. I love this space—don't get me wrong—it has changed my life, and I believe it has a ton of potential, but things need to change for it to become a beacon of light in art. But we need to support new voices. We need to support new ideas. The challenge is huge. I hope to contribute all I can to this change. I hope more and more see how exciting it is to go out and try to discover what else is out there and move this space forward. But again, I understand the leaps of faith needed, but if there is a space that is based on that, it's the NFT space...so there is hope. We will see. 📺by Boldtronshow more

Europe is quietly becoming what the United States once... promised the world. More and more people are looking at their best years ahead and choosing a place where everyday life is designed to work. Where the future feels stable enough to plan for. Where safety is not a luxury product. Where you can build a good life without gambling your health, your family, or your dignity on one bad month. In much of Europe, the “dream” is not about becoming a billionaire. It is about becoming unafraid. It is the freedom of walking home at night without scanning every shadow. The comfort of knowing that if you get sick, you do not need to calculate whether you can afford to be treated. The relief of having a society that still believes children should carry backpacks, not trauma, and definitely not weapons. The calm of streets built for human beings, not just cars. The ability to take a holiday without feeling like you are committing career suicide. The basic decency of labor protections that assume you are a person first and a resource second. And then there is the part people underestimate until they live it: the texture of life. The cities are older and more beautiful than you expect. The distances are smaller. Weekends are real. Food is real. Public spaces are not just decorative, they are functional. Parks are full. Cafes are full. Trains take you somewhere, often across borders, without turning travel into a stress test. You can live in one country, work with another, and visit a third like it is normal because, in many places, it is. The European dream is also a quiet confidence in the social contract. That if you contribute, the system does not abandon you. That you can raise a family without feeling like you are one accident away from ruin. That “getting ahead” does not require burning out. That a good society is one where normal people can live normal lives and still feel proud of them. This is why more and more Americans are not just visiting Europe, but staying. Some come for studies and never leave. Some arrive for a job and realise the lifestyle is the real promotion. Some originally planned a one year experiment and then cannot imagine going back to a place where stress is treated as a personality trait and insecurity is marketed as freedom. Europe is not perfect. It has bureaucracy. It has politics. It has problems that deserve criticism. But in many European countries, life is still built around a simple idea: society should reduce fear, not monetise it. That is the new dream. And people can feel it the moment they arrive. If you could choose one thing to trade for a better life, what would it be: more income, or more security? And what do you think your country would have to change for people to stop leaving, and start staying? Stay connected, Follow Gandalv Gandalvshow more

you're paying $20/mo for something your $500 GPU can... already do. Gemma 4 26B A4B QAT MoE + Hermes Agent running on a single RTX 4060 (8GB VRAM). Built a vision capable, 100% free, 100% local, private AI assistant that lives in my Chrome browser. No API keys. No cloud. No subscriptions. 100% vibe coded. 0% handholding. It has full context of whatever's on my screen can answer questions, summarize pages, extract data, and see images. Same local model handles everything, no external calls, ever. keep reading for the model and hermes agent tips i learnt while building this locally. Here's the exact setup for anyone running local LLMs on 6-8 GB VRAM: llama.cpp server flags (on my NVIDIA RTX 4060 8gb VRAM): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --cache-type-k q8_0 --cache-type-v q8_0 -c 150000 --port 8080 Throughput with quantization: Prefill: 200-250 tokens/sec Decode: 20-25 tokens/sec reduce context if oom on 6 gb vram card. Key learnings: - Quantize KV cache to q8 for faster prefill/decode. Prefill goes from 100-150 (unquantized) to 200-250 tok/s (q8). - But watch out, once actual context grows past ~50k tokens on high entropy workloads, q8 KV quantization can cause hallucinations. Low entropy workloads are mostly unaffected. If you see it happening, drop the quantization. This is common across all local models. - In Hermes Agent settings -> Memory & Context, bump compression threshold from default 0.5 to 0.7. Default triggers way too frequent context compression and eats time. Up next: add persistent memory, web search, tool calling, streaming output and whatever you suggest. Running a 26B MoE with vision + 150k context window on 8GB VRAM would've sounded impossible 6 months ago. Works the same on the NVIDIA RTX 3060 Ti, 3070, 4060 Ti, 5060, 2080, or any 8GB card. VRAM is the only requirement. Local AI agents are closer than people think. You just need to know where the knobs are. Model's Unsloth quant hugging face link in the comments. Have you tried Hermes agent by Nous Research yet? What are you building with local LLMs? Drop it below, let's see what this community is shipping.show more