Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

your eval measured one system. production ran another. they start from the same idea and end somewhere else, which is why the number was right and the outcome was not. the code is the same until someone rewrites it for prod, because the eval lived in a notebook and...

45,239 görüntüleme • 1 ay önce •via X (Twitter)

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

I don't think most PMs realize the PRD is becoming obsolete. For the last decade, the PM's core artifact was a qualitative spec. Clear requirements, user stories, acceptance criteria. The engineering team interpreted it, built something close, and the PM spent two weeks reconciling what shipped with what they wrote. The best AI companies replaced that entire loop with evals. A set of inputs your product needs to handle. A task that generates outputs. A scoring function that produces a number between 0 and 1. No ambiguity. No interpretation gap. Ankur Goyal built the eval platform behind Vercel, Replit, Ramp, Notion, and Airtable. An $800M company. He walked through building an eval from zero on this episode and the score went from 0 to 0.75 in under 20 minutes. That's a PM shipping a measurable quality bar before a single line of product code exists. Here's the part that changes the PM role permanently. When the product passes the eval and users still hate it, the eval is wrong. That's on the PM. Evals make PM judgment quantifiable in a way PRDs never did. You can't hide behind "the spec was ambiguous." There's a number now. Six months ago, PM interviews asked "how do you use AI in your workflow." The next wave of interviews is going to ask you to write an eval. The PMs who can encode user intent as a scoring function are building the one skill that survives every model change, every framework swap, every agent rewrite. Write the eval.

Aakash Gupta

78,361 görüntüleme • 6 ay önce

Workflows vs. Graphs, clearly explained! workflows are great, and almost everyone has one. here is the ceiling: a workflow decides every step before it runs. you drew eight boxes in March. six months later the same three fire, every single time, and the other five have never once been reached. then a case arrives that nobody drew, and it goes to the closest wrong box. quietly, with a green status, because from the inside that looks exactly like success. Graph engineering fixes this by moving the decision: not what the steps do, but when the steps get chosen. you need both, and here is the sentence that resolves the whole confusion: a workflow decides the steps before it runs. a graph decides them while it runs. ↳ drawn in advance: the boxes, the branches, the order, the error path ↳ decided at runtime: how many units exist, what each one is allowed to see, which ones get created at all Prompts → Context → Harness → Loops → Graphs branches do not make it a graph. the branches were drawn in advance too, which means every one of them is a case you already thought of. the trick is knowing which part is allowed to be fixed. the node kinds are fixed. a splitter is a splitter, a gate is a gate, a merge is code. what is not fixed is how many of them exist this run, and that is decided after something has been read. one thing to know before you scale it. a workflow fails in a way that never pages anyone. ↳ the wrong branch ran, every check inside it passed, and the output is well formed ↳ nothing errored, because routing to the wrong box is not an error, it is a route that last one catches careful people. you cannot test your way out of it either, because the test suite was written from the same diagram that has the gap in it. and the one that eats whole nights: a workflow that has never surprised you is not stable, it is narrow. if it has run four hundred times and produced the same three shapes, it is not handling your work. it is handling the part of your work that fits it, and you have quietly stopped sending it the rest. below i have quoted my full guide on graph engineering. it covers the three topologies, the verifier patterns, and where the gate should actually open. save this and read it below ↓

Hanako

16,810 görüntüleme • 23 gün önce

Elon Musk just told you the job is dying. Most people heard a prediction. A few heard a prison door opening. Musk: “In less than 20 years, working at all will be optional.” That is not a policy suggestion. That is a countdown. For three hundred years, the human blueprint has been identical. You are born. You move to the city. You rent a box near the office. You trade your body and your hours for the right to exist. You do this until you are old. Then you stop. Then you die. The entire model runs on one assumption. That human labor is the only engine. AI and robotics delete that assumption. When the machine handles production at a scale no human crew can match, the forced migration to the city evaporates. The commute evaporates. The cubicle evaporates. The alarm clock that owns your nervous system for forty years evaporates. Musk: “I think it won’t be the case that you have to be in a city for a job.” The city was never a choice. It was a requirement disguised as ambition. You moved to the noise and the concrete and the $4,000 rent because the paycheck lived there. Remove the paycheck from the equation and the geography changes overnight. You can live in the mountains. On the coast. In the silence of a town most people have never heard of. You can wake up to nothing but trees and cold air and the complete absence of anyone else’s schedule. That is not a fantasy. That is the math resolving. But here is where most people break. They hear “work is optional” and they see emptiness. A species with nothing to do. Billions of people staring at screens until their minds dissolve. That fear tells you everything about what the system has already done to us. We confused labor with purpose. The grind with meaning. The paycheck with proof that we matter. Musk: “In the same way that you could grow your own vegetables in your garden.” The analogy is precise. You do not grow tomatoes because the economy demands it. You grow them because something in you wants to build a thing with your hands and watch it come alive. That instinct does not disappear when the job does. It gets unleashed. The artist who spent twenty years doing accounting finally paints. The engineer who always wanted to build something of her own finally builds it. The kid in a small town who could never afford to take the risk finally takes it. Work does not vanish. Forced work vanishes. What replaces it is creation without a gun to your head. This is the part that keeps me up at night. We are standing at the edge of the largest liberation in human history. And the loudest voices in the room are begging to stay in the cell. They want the commute. They want the boss. They want the structure that tells them when to eat and when to sleep and when they are allowed to think about their own life. Because freedom without a template is terrifying. The next twenty years will not test our technology. The technology is already ahead of schedule. They will test whether the species can handle what it has been asking for since the beginning of civilization. Time. Space. Silence. And the unbearable weight of choosing what your life actually means when no one is forcing the answer. That is not a prediction. That is the final exam. And nobody is ready.

Dustin

111,869 görüntüleme • 6 ay önce

i watched gemma 4 12b build something genuinely impressive today, and then loop itself to death right in front of me. the full run is in the video, sped up but completely uncut, watch it to the end and you will catch the exact moment it stops building and starts looping right in the middle of the work. the task was clean, build a single file gravity simulator, n-body physics, orbits, collisions, running locally on one 3090 through an agent. and for ten minutes it was a joy to watch. it reached for a symplectic integrator on its own, the correct one, the kind that keeps orbits stable instead of spiralling out. real gravity with softening, proper orbital velocities, momentum conserved on collision. the physics was right. the thing actually worked. then on the very last step, writing a few tests to prove its own code, it fell into a loop. not a crash, a loop. it started repeating itself and would not stop. ten more minutes, thirty four thousand tokens into a single answer, the same fragments over and over, until i killed it myself. so it's not that gemma can't code. it did the hard part beautifully. it cannot finish. it cannot hold a long task together without unravelling, and finishing is the entire job in agentic work. here's the part that stings. i run this exact task, same harness, same card, on the chinese open models, qwen especially, and i never see this. they build it, they test it, they stop. every single time. google has the raw capability, you can see it sitting right there in the code, and then the model loops itself to death on a task a 27b from alibaba finishes clean. open weights, apache 2.0, so much to love on paper. i just need it to know when to stop talking.

Sudo su

39,764 görüntüleme • 4 ay önce