Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

your eval measured one system. production ran another. they start from the same idea and end somewhere else, which is why the number was right and the outcome was not. the code is the same until someone rewrites it for prod, because the eval lived in a notebook and...

45,239 görüntüleme • 12 gün önce •via X (Twitter)

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

I don't think most PMs realize the PRD is becoming obsolete. For the last decade, the PM's core artifact was a qualitative spec. Clear requirements, user stories, acceptance criteria. The engineering team interpreted it, built something close, and the PM spent two weeks reconciling what shipped with what they wrote. The best AI companies replaced that entire loop with evals. A set of inputs your product needs to handle. A task that generates outputs. A scoring function that produces a number between 0 and 1. No ambiguity. No interpretation gap. Ankur Goyal built the eval platform behind Vercel, Replit, Ramp, Notion, and Airtable. An $800M company. He walked through building an eval from zero on this episode and the score went from 0 to 0.75 in under 20 minutes. That's a PM shipping a measurable quality bar before a single line of product code exists. Here's the part that changes the PM role permanently. When the product passes the eval and users still hate it, the eval is wrong. That's on the PM. Evals make PM judgment quantifiable in a way PRDs never did. You can't hide behind "the spec was ambiguous." There's a number now. Six months ago, PM interviews asked "how do you use AI in your workflow." The next wave of interviews is going to ask you to write an eval. The PMs who can encode user intent as a scoring function are building the one skill that survives every model change, every framework swap, every agent rewrite. Write the eval.

Aakash Gupta

78,275 görüntüleme • 5 ay önce

You Can Learn AI Agent Harness & Loop Engineering In 19 Min, with LLM Ops, Eval, Tracing and RAG. They went viral not because they're complicated but because they're simple building blocks, and once you see them you can prompt your way to building real systems. 🎬YouTube: Here's the whole thing in one picture. An LLM is a powerful brain that knows everything about humanity and nothing about you or the software you're running. The harness is the set of tools you put on that horse so it runs where you want. Memory gives it context: who you are, what happened before, how to act. The loop lets it call tools again and again, with guardrails so it knows when to stop. Eval and LLM Ops trace every run, score it, and feed the fixes back in so the system keeps improving itself. Master these four and you can read almost any AI agent repo or paper and actually know what's going on. You Can Build Anything. You Can Learn Anything. 💪 Chapters: Intro: the 4 AI agent buzzwords What an AI agent run actually is The memory system: procedural, semantic, episodic What "harness" really means (the horse) Storing and updating memory (databases, skills, summarizer agent) Retrieval: RAG, SQL vs semantic search Tool calling and why agents loop Loop engineering and end-loop guardrails A Claude Code hooks example Eval and LLM Ops: why you need them Tracing every run (Langfuse, LangSmith) Evaluation: LLM as a judge Diagnosing what broke The gate: ship the fix or fix the bug Zoom out: the full system

Shen Sean Chen

15,952 görüntüleme • 1 ay önce

i watched gemma 4 12b build something genuinely impressive today, and then loop itself to death right in front of me. the full run is in the video, sped up but completely uncut, watch it to the end and you will catch the exact moment it stops building and starts looping right in the middle of the work. the task was clean, build a single file gravity simulator, n-body physics, orbits, collisions, running locally on one 3090 through an agent. and for ten minutes it was a joy to watch. it reached for a symplectic integrator on its own, the correct one, the kind that keeps orbits stable instead of spiralling out. real gravity with softening, proper orbital velocities, momentum conserved on collision. the physics was right. the thing actually worked. then on the very last step, writing a few tests to prove its own code, it fell into a loop. not a crash, a loop. it started repeating itself and would not stop. ten more minutes, thirty four thousand tokens into a single answer, the same fragments over and over, until i killed it myself. so it's not that gemma can't code. it did the hard part beautifully. it cannot finish. it cannot hold a long task together without unravelling, and finishing is the entire job in agentic work. here's the part that stings. i run this exact task, same harness, same card, on the chinese open models, qwen especially, and i never see this. they build it, they test it, they stop. every single time. google has the raw capability, you can see it sitting right there in the code, and then the model loops itself to death on a task a 27b from alibaba finishes clean. open weights, apache 2.0, so much to love on paper. i just need it to know when to stop talking.

Sudo su

39,574 görüntüleme • 2 ay önce

Elon Musk just told you the job is dying. Most people heard a prediction. A few heard a prison door opening. Musk: “In less than 20 years, working at all will be optional.” That is not a policy suggestion. That is a countdown. For three hundred years, the human blueprint has been identical. You are born. You move to the city. You rent a box near the office. You trade your body and your hours for the right to exist. You do this until you are old. Then you stop. Then you die. The entire model runs on one assumption. That human labor is the only engine. AI and robotics delete that assumption. When the machine handles production at a scale no human crew can match, the forced migration to the city evaporates. The commute evaporates. The cubicle evaporates. The alarm clock that owns your nervous system for forty years evaporates. Musk: “I think it won’t be the case that you have to be in a city for a job.” The city was never a choice. It was a requirement disguised as ambition. You moved to the noise and the concrete and the $4,000 rent because the paycheck lived there. Remove the paycheck from the equation and the geography changes overnight. You can live in the mountains. On the coast. In the silence of a town most people have never heard of. You can wake up to nothing but trees and cold air and the complete absence of anyone else’s schedule. That is not a fantasy. That is the math resolving. But here is where most people break. They hear “work is optional” and they see emptiness. A species with nothing to do. Billions of people staring at screens until their minds dissolve. That fear tells you everything about what the system has already done to us. We confused labor with purpose. The grind with meaning. The paycheck with proof that we matter. Musk: “In the same way that you could grow your own vegetables in your garden.” The analogy is precise. You do not grow tomatoes because the economy demands it. You grow them because something in you wants to build a thing with your hands and watch it come alive. That instinct does not disappear when the job does. It gets unleashed. The artist who spent twenty years doing accounting finally paints. The engineer who always wanted to build something of her own finally builds it. The kid in a small town who could never afford to take the risk finally takes it. Work does not vanish. Forced work vanishes. What replaces it is creation without a gun to your head. This is the part that keeps me up at night. We are standing at the edge of the largest liberation in human history. And the loudest voices in the room are begging to stay in the cell. They want the commute. They want the boss. They want the structure that tells them when to eat and when to sleep and when they are allowed to think about their own life. Because freedom without a template is terrifying. The next twenty years will not test our technology. The technology is already ahead of schedule. They will test whether the species can handle what it has been asking for since the beginning of civilization. Time. Space. Silence. And the unbearable weight of choosing what your life actually means when no one is forcing the answer. That is not a prediction. That is the final exam. And nobody is ready.

Dustin

111,792 görüntüleme • 4 ay önce

Jordan Peterson told a room full of university students the one thing their $200,000 degree was engineered to bury. Peterson: “If you can think, and speak, and write, you are absolutely deadly. Nothing can get in your way.” He did not say educated. Did not say credentialed. Did not say employed. Deadly. The ability to force your own reality into language is not a skill they forgot to teach you. It is the one skill the system cannot afford for you to have. Peterson: “No one ever tells students why they should write something.” Because the honest answer would collapse every transaction the institution runs. You were not taught to write. You were trained to replicate. Follow the rubric. Hit the word count. Reproduce the approved answer back to the grader. Pay six figures for the privilege. You do not think in ideas. You think in sentences. The precision of your language is the precision of your reality. Everything outside your vocabulary is not something you disagree with. It is something you cannot see. Every power structure in recorded history understood one equation. A population that cannot name what is being done to them will never fight what is being done to them. The modern version does not ban the weapon. It reclassifies it. Calls it coursework. Grades it. Strips it of everything dangerous and hands it back empty. Peterson: “It’s the most powerful weapon you can possibly provide someone with.” A person who can force one true sentence into existence without permission has already exited the system. That is the one graduate no institution was ever designed to produce. He said this as a tenured professor at the University of Toronto. Twenty years inside the machine. Students called his lectures life-changing. The institution pushed him out. The one professor who told you what the weapon actually does was removed for the crime of using it. That tells you everything. Not about him. About the machine you spent two decades inside and walked out unable to name.

Dustin

35,463 görüntüleme • 1 ay önce

Your trading strategy didn't break. The market it was built for quietly stopped existing. Read that twice. It's most of why 89% of retail finished 2025 in the red. There's now an app that does the entire job of a $400,000 quant. You type a trading idea in plain English. It writes the code, backtests 5 years in 12 seconds, runs thousands of simulations, and tells you cold whether your edge is dead or the regime just changed. No code. No Python. No $25,000 terminal. 20,000 already inside. Waitlist stops at 25,000: That distinction is the whole game, and you never had a way to see it. Every strategy is a bet that one thing stays true. Momentum bets trends continue. Mean reversion bets ranges hold. When the regime flips, the assumption dies and your strategy bleeds with nothing wrong in the code. You stare at the logic for a month and never find the bug, because there isn't one. So you delete it, or refit it to the last drawdown and build something that would have survived the pain you already felt and nothing coming next. The desks never had that problem. 92% of institutional volume is automated. Only 45% of retail is. They test 100 strategies for every 1 you test by hand, and kill 97 of them on purpose, because they can tell a dead edge from a normal drawdown. Now that exact loop costs $0. One hypothesis used to cost a fund $87,500 to test. With Horizon you get unlimited, in seconds, and a winner deploys live in 90 seconds and runs without your hands on it.

cvxv666

40,765 görüntüleme • 2 ay önce

Geoffrey Hinton just made every AI critic accidentally describe their own brain. Hinton: “They shouldn’t be called hallucinations. They should be called confabulations.” One word. The entire debate unravels. The tech industry sees AI produce a confident wrong answer and calls it a defect. A bug to patch. They are measuring intelligence against the standard of a filing cabinet. And exposing that they understand neither. Hinton: “It’s not that there’s a file stored somewhere in your brain, like in a filing cabinet or in a computer memory.” Your brain does not store memories. It rebuilds them from nothing every time you remember. Fills gaps it never discloses. Fabricates details you would stake your life on. Then hands it all to you as truth. Hinton: “If I ask you to remember something that happened a few years ago, you’ll construct something that seems very plausible to you. And some of the details will be right and some will be wrong.” The wrong parts feel identical to the right ones. No internal warning. No distinction between what was remembered and what was invented on the spot. You have argued over memories that were partially fiction. Told stories about your own life that your brain manufactured in real time. With total conviction. And never once suspected. This is not a defect in human cognition. This IS cognition. The mechanism that fabricates is the same one that reasons, creates, and makes connections no one taught it to make. Not a separate system. Same architecture. Same process. You cannot remove the confabulation without killing the intelligence. They are the same thing. Hinton: “Psychologists have been studying confabulation in people since at least the 1930s.” A century of evidence. No one called the human brain broken. The moment a machine runs on the same principle, the world calls it defective. The people demanding AI that never gets a single detail wrong are not asking for intelligence. They’re asking for a search engine that sounds articulate. What we built is something else entirely. A system that thinks the way thinking actually works. Not retrieval. Construction. The imperfection is not the cost of intelligence. It is the signature.

Dustin

16,260 görüntüleme • 1 ay önce

An entire empire was overthrown over a two percent tax on a breakfast beverage. Look at what you tolerate now. You are taxed when you earn it. Taxed when you spend it. Taxed when you save it. Taxed when you invest it. And when you die, they tax whatever is left. That is not a system. That is a harvest. You commute in a car you paid sales tax to buy. You drive it on roads you were already taxed to build. You fill it with gas taxed by the gallon. When you sell that car, the next buyer pays sales tax on it again. The same car. Taxed every time it changes hands. You arrive at a job where your salary is cut before it ever touches your hands. If you work for yourself, you pay both sides. Two people on paper. Neither one keeps what they earned. Then you go home. Every bill you open has a government standing behind it with its hand out. You buy a house with money they already took their share of. Then they charge you property tax on it every year for the rest of your life. You want to renovate your own kitchen. You need a permit. You want to build a deck on your own land. You need a permit. You pay for the property. Then you pay for permission to use it. Stop paying property tax and they seize your home. Not because you missed a mortgage payment. Because you missed a payment to the government for the privilege of keeping what is already yours. You do not own your home. You rent it from the state. If you leave something behind for your children, they are taxed on what you were already taxed to earn. The same wealth. Taxed at every stage of your life. Then taxed one final time because you had the audacity to die. They found a way to monetize your absence. We are told this is the price of civilization. It is not. It is architecture. The most effective prison ever built is the one where the inmates believe they are free. They did not take your freedom. They priced you out of it. If you kept the full value of your labor, you would be free within years. Not decades. Years. The system cannot allow that. A machine built on consumption needs a consumer that never stops. You did not sign a social contract. You were assigned one. Now pay attention. They spent decades perfecting the extraction of your productivity. Now they are building the technology to replace you. AI is not coming for your job because corporations are greedy. It is coming because a system that already takes half your output just realized it can take all of it. Without needing you in the equation. You were never the point of this arrangement. You were the input. And the moment they engineer a cheaper one, you become a rounding error on a quarterly earnings call. They did not build AI to free you. They built it to finish what the tax code started. It was never about the tea. It was about the precedent. Today we hand over half our waking lives and thank them for the potholes. You do not live in a free economy. You live in a subscription you never signed up for. And the penalty for canceling is everything you have.

Dustin

27,811 görüntüleme • 4 ay önce