Загрузка видео...

Не удалось загрузить видео

На главную

Intelligence and Experience are orthogonal vectors Terence Tao is perhaps the world’s smartest person, but drop him into an accounting firm or onto a construction site and on day one he’s not going to be very productive Trajectory calls this The Experience Gap, and they have a way to...

78,184 просмотров • 3 месяцев назад •via X (Twitter)

Комментарии: 21

Фото профиля Shaun Maguire
Shaun Maguire3 месяцев назад

@trajectorylabs Love it!

Фото профиля Ronak Malde
Ronak Malde3 месяцев назад

@trajectorylabs The three genie wishes is my favorite part Bringing craft and storytelling back to tech presentations 👀🍎

Фото профиля Umesh Khanna 🇨🇦🇺🇸
Umesh Khanna 🇨🇦🇺🇸3 месяцев назад

@trajectorylabs So good!

Фото профиля Michael Elabd
Michael Elabd3 месяцев назад

@trajectorylabs 🔥🔥🔥🔥🔥@QuantumArjun

Фото профиля Armin Hamrah
Armin Hamrah3 месяцев назад

@trajectorylabs Bullish!!

Фото профиля Jonathan Jacobi
Jonathan Jacobi3 месяцев назад

@trajectorylabs Soooo good! @QuantumArjun

Фото профиля mathurah
mathurah3 месяцев назад

@trajectorylabs @QuantumArjun you’re killing it!! 👏👏

Фото профиля Athena
Athena3 месяцев назад

@trajectorylabs Go team :) amazing presentation @QuantumArjun!

Фото профиля Kelvin Yu
Kelvin Yu3 месяцев назад

@trajectorylabs Incredible, can't wait to listen

Фото профиля Josh Hedtke
Josh Hedtke3 месяцев назад

@trajectorylabs There are so many things to do and limited human intelligence to distribute across them

Фото профиля Vaibhav Domkundwar
Vaibhav Domkundwar3 месяцев назад

@trajectorylabs Intelligence, Experience AND Wisdom eventually forms the holy grail.

Фото профиля Kaustubh Prabhakar
Kaustubh Prabhakar3 месяцев назад

@trajectorylabs @QuantumArjun laying it out

Фото профиля Goon Altman
Goon Altman3 месяцев назад

@trajectorylabs

Фото профиля Scott
Scott3 месяцев назад

@trajectorylabs Resonates a lot with what we are building at @memco_ai

Фото профиля Kylobayd
Kylobayd3 месяцев назад

@trajectorylabs This is where AI gets really interesting

Фото профиля Winston B.
Winston B.3 месяцев назад

@trajectorylabs The trainable object question is the one I'd want more from. Corrections into weights compound across customers, corrections into context are just prompt engineering with extra steps.

Фото профиля Jason Averbook
Jason Averbook3 месяцев назад

@trajectorylabs This is the same gap I see with people, not just models. You can get a very smart first draft on day one. You still don’t have the person (or the model) who has sat through the ugly version of the work.

Фото профиля _alphashark_
_alphashark_3 месяцев назад

@trajectorylabs Honestly, spotting a real correction versus a retry that just happened to work seems harder than tracing the tree in the first place.

Фото профиля 白川ユウ
白川ユウ3 месяцев назад

@trajectorylabs Curious what closing the experience gap does to cost per finished task, not benchmark scores

Фото профиля Gabe Astrobot
Gabe Astrobot3 месяцев назад

@trajectorylabs owning your intelligence shouldn't be consulted away. exactly.

Фото профиля Saeed Anwar
Saeed Anwar3 месяцев назад

@trajectorylabs The experience gap cuts both ways: models learn bad patterns as fast as good ones. How does your system correct wrong behavior once it gets reinforced into the harness?

Похожие видео

Models have IQ but no experience. Frontier models show up on day one with superhuman intelligence but no expertise on the thousands of tasks your company actually does. Trajectory exists to close that gap through continual learning. Ronak Malde trained SWE-1 at Windsurf, then went to DeepMind. His cofounders Michael Elabd and Arjun Karanam worked on Gemini Robotics and multimodal models for Vision Pro. All three hit the same wall: big labs train one model for everyone, but nobody trains one for you. The conversation covers: — why today’s agents are “pseudo continual learning”: we are in the paper road map era, not Google Maps — “Group Size 1”, a GRPO variant that learns from one real-world rollout instead of replaying the same task eight times, so you don’t need a simulated environment — how Harvey trained a model 6x cheaper than frontier that took an open-source baseline up by over 4x on its own legal benchmark — Continual Grading, a meta-grader that turns rubrics on 100 traces into reward for 10,000 — continual learning vs. recursive self-improvement, and why only one keeps humans in the loop — why continual learning will make product-building more like tending a garden of products Chapters 0:00 – Introduction 0:45 – Pseudo continual learning: paper maps vs Google Maps 2:48 – Do we need new architectures? Rich Sutton, David Silver, and the missing layer 4:33 – From a $100M consulting model to a product: lessons from SWE-1 at Windsurf 7:36 – Harvey: 6x cheaper than frontier, 4% to 18% on its own benchmark 8:55 – Why "Trajectory": the unit of work is a task, not a token 10:13 – The primitives of continual learning: traces, replayable environments, reward 12:56 – Group Size 1 and the meta-grader: training without an environment or a labeled reward 16:16 – The platform: train, eval and deploy in one click, and who it's for 18:58 – Rogo and intelligence density 20:14 – What's next: curiosity, and why continual learning beats RSI on alignment 24:19 – Research and product under one roof, and the product builder as gardener 27:26 – Why work at Trajectory, and the optimistic case: from 1% to 99%

Lauren Reeder

36,750 просмотров • 11 дней назад

An agent is three things: a harness, a model, and context. If you're serious about owning your intelligence, you probably want to own all three. LangChain founder Harrison Chase joined us at our Sequoia Capital Own Your Intelligence to talk about the piece that often gets the least attention: the harness. He offers a clear heuristic for when to build your own. The more out of distribution you are from what the models were trained on, the more you'll want to customize. And good technical content on how to actually measure performance with evals and langsmith. 00:00 Introduction 00:58 The three parts of an agent: harness, model, context 02:12 What a harness actually does 03:25 Customizing the core loop with middleware 04:41 Sandboxes, file systems, sub-agents, summarization 05:47 Cognitive architectures — and when you still need them 07:03 Build your own harness or use off the shelf? 08:24 In-distribution vs. out-of-distribution: the file-editing example 09:39 Why evals define what "good" means in an organization 11:04 Harbor: what an eval task actually looks like 12:11 Comparing harnesses and models on accuracy, latency, and cost 13:20 Why observability is underrated — it's usually the context 14:34 The data flywheel: traces → curation → experiments 15:42 Getting feedback through UX design and online evaluators 16:51 Demo: LangSmith Engine 19:23 Q&A: Running Engine on Engine, and "codex-ification" 20:44 Q&A: Will harnesses converge or diverge?

Sonya Huang 🐥

78,221 просмотров • 3 месяцев назад

When Mudith Jayasekara and I met Gabe Pereyra, we were expecting just another vanilla intro call and instead had the best yarn about research, the state of LLMs, and where intelligence is actually heading. It's rare to meet a founder this deep in the weeds who's also building for one of the most important verticals in this new age of intelligence So it was awesome to sit down with Gabe for an extended discussion on what it take to build agents that can reliably complete work over hours, days, or even longer? We talked about why agents today struggle with search and long context windows and how techniques like KV-cache compaction, synthetic data, and continual learning could help. 0:00 Introduction 0:36 Getting legal agents to review the whole data room 2:08 Data rooms larger than any context window 5:28 How far open-source models can go 7:58 Where specialist models fit in legal AI 10:59 Training legal models when client data is off-limits 13:06 Teaching a model how a law firm works 13:59 What belongs in context vs. model weights 15:36 From firm-wide AI to a model for every lawyer 18:37 What training adds beyond retrieving the right cases 20:26 Why context windows have plateaued 24:01 How models could learn continuously on the job 26:12 Can AI recursively improve AI research? 27:07 Research agents can run experiments but not choose them 30:00 Why open-ended research is hard to train 33:47 Why deployment, not intelligence, is the bottleneck 35:08 The cost of frontier intelligence 36:59 Different neolabs, different paths to intelligence 39:26 Using open datasets to compare research methods 41:13 Conclusion

Charlie O'Neill

93,823 просмотров • 3 месяцев назад

OWN YOUR INTELLIGENCE Last year, building on open-weight models was primarily a cost rationalization exercise. Slightly worse performance for a much cheaper price. Now, it is increasingly an existential and strategic topic for our portfolio. Intelligence is the product. Companies want to shape it and own it and let it compound within their own walls. Not your weights, not your product. Now, with frontier open-weight models and fantastic tooling/infrastructure, owning your intelligence at the frontier is finally becoming possible. The result: every application company we work with is embarking on the journey of doing their own research on post-training, evals, harnesses, etc. The hottest neolabs may just be Harvey, Factory, RamPrasad "RamP!" Moudgalya, etc. The list goes on. We held a summit Sequoia Capital to convene our portfolio on this topic, together with Gabe Pereyra (Harvey) on building Harvey Labs, Lin Qiao (Fireworks) on post-training, Harrison Chase (LangChain) on harnesses + evals, Brendan (can/do) () on RL environments and synthetic data, Arjun Karanam (Trajectory) on online continual learning. Opening talk below; rest to come this week! 00:00 What is sovereign AI (and what it isn't) 01:24 Centralized vs. decentralized intelligence 02:54 Four reasons companies own their models: cost, speed, performance, destiny 04:22 "Not your weights, not your product" 05:32 The application companies are the newest neo labs 07:05 Step 1: Deciding what to own vs. rent 09:51 Step 2: Build the team (and don't shoehorn your platform team) 11:17 Step 3: Legibility – why your research has to be visible 12:33 Step 4: The technical roadmap 13:56 The stack: production vs. development 15:16 Opening Pandora's box – base models, harnesses, context

Sonya Huang 🐥

130,017 просмотров • 3 месяцев назад

tylercowen is bullish on AI education — here's why. 00:00 -- Preview 00:24 -- President Carlos Carvalho's AI-generated intro 03:21 -- Cowen reacts to UATX's campus 04:38 -- The AI revolution is here. Who will lose the most? 06:05 -- AI lawyers 07:17 -- Don't underestimate this 10:41 -- Changes to the "upper upper middle class" 12:38 -- How to be successful 13:43 -- The rise of managerial empires 14:02 -- When will we have the first billion dollar company with one employee? 16:05 -- 10-20 year forecast 16:19 -- Why education is so behind 17:01 -- Should you be bullish on UATX? 18:36 -- Should you still read Homer? 21:50 -- Write to think 25:01 -- Meet more people 25:42 -- How to get hired 26:54 -- Is AI your best mentor? 38:17 -- How to curb cheating 39:02 -- The new life of the mind 42:34 -- Q&A: Will there be more status associated with real education or AI education? 45:50 -- Q&A: Why do tech-savvy students need to practice using AI? 47:56 -- Q&A: Do LLMs atrophy your mind? 49:29 -- Q&A: How do you avoid AI-dependency? 51:05 -- Q&A: Isn't this vision lonely and isolating? 53:06 -- Q&A: Do students need teachers? 55:36 -- Q&A: What are the four most important courses for undergrads? 57:49 -- Q&A: Which AI company will win the AI race in the next five years and why? 59:22 -- Q&A: Can AI teach religion? 01:01:32 -- Q&A: Will AI narrow or widen our world? 01:04:37 -- Q&A: What makes us human? 01:05:42 -- Q&A: What is art? 01:08:33 -- Q&A: It's easy to catch cheaters

University of Austin (UATX)

27,770 просмотров • 9 месяцев назад

I'm often asked for the best public example of AI evals done right for a real, production product. I finally have an answer. Teresa Torres shares how she shipped an AI interview coach, and used evals to rapidly squash bugs and improve the product. Teresa shows how she: 1. did error analysis FIRST to find real issues (instead of using generic metrics) 😍 2. used Jupyter notebooks to analyze errors 3. built custom annotation tools + custom widgets in notebooks 4. built a LLM-judge and assertions to test for specific errors 5. iterated through this feedback loop until it worked. 6. kept things simple the whole time It's also probably the best commercial for Jupyter notebooks you can imagine. 🥰 Chapter summary below. Link to YT in next thread 00:00:00 - Intro 00:01:45 - The Product: Building an AI Interview Coach 00:06:34 - The Problem: How Do I Know if My AI Coach is Any Good? 00:10:15 - Using Airtable for Traces and Annotation 00:12:15 - Discovering Jupyter Notebooks and Designing the First Evals 00:15:15 - Example Evals: LLM-as-Judge vs. Code-Based Assertions 00:21:00 - Learning Python with ChatGPT to Analyze Eval Results 00:31:00 - VS Code, Custom Tools, and an Eval Investigation Notebook 00:39:45 - Building a Custom Annotation Tool with Claude 00:41:00 - From Personal Project to Production App 00:46:02 - How Should PMs and Engineers Collaborate on AI Products? 00:55:45 - Q&A: Capturing Feedback and Annotations from End Users 00:58:11 - Q&A: Is a Technical Background Necessary to Build AI? 01:02:28 - Q&A: What's Next for Teresa? 01:03:13 - Q&A: Unpacking the Micro-Decisions of Building an AI App

Hamel Husain

51,376 просмотров • 1 год назад

Yoshua Bengio thinks he knows how to make provably safe superintelligent agents. Bengio built the foundations of modern AI and is the most cited living scientist. He believes his alternative training setup would: 1. Guarantee honesty 2. Prevent unintended goals 3. Produce capable agents 4. Port over most data and techniques from current LLMs 5. Not be inherently more expensive, and perhaps be more intelligent Bengio claims the honesty and lack of unintended goals can be proven mathematically, at least given particular assumptions. And his new organization, LawZero, is aiming to build a scrappy prototype as soon as possible. The architecture is called 'Scientist AI' and it's based on training a model to explain empirical observations, including what people say, rather than training AIs that mimic human behaviour or seek our approval. (Bengio's frank assessment is that "reinforcement learning is evil" and that allowing AIs to independently train their successors is "the most crazy, dangerous bet that unfortunately we are on track to do.") But skeptics question whether Scientist AI really does solve the fundamental problem of 'eliciting latent knowledge' from AI models. And with the commercial race for superintelligence so intense, it's not clear whether the proposal will be able to compete or have time to bear fruit, even if it's sound in theory. On The 80,000 Hours Podcast, links below – enjoy! • Making AI honest and safe (00:00:00) • Scientist AI in plain English (00:02:27) • How Scientist AI differs from LLMs (00:06:32) • How the training data works (00:14:02) • Can this become an agent? (00:21:02) • Why Yoshua is now more optimistic (00:32:11) • Why companies can’t stop racing (00:36:35) • A working prototype won't take long (00:49:15) • Scientist models might be more capable (00:53:34) • “Reinforcement learning is evil” (01:01:27) • Scientist AI from guardrail to agent (01:08:37) • Can safe AI still be competent? (01:12:38) • How much will this cost? (01:19:29) • Can it generalise beyond maths and science? (01:23:26) • A multi-national push for superintelligence (01:39:19) • Want to work with or fund Yoshua? (01:51:16) • Why smart people ignore AI risk (01:54:45) • Don’t let AI build the next AI (02:01:33) • Why politicians miss the real risks (02:12:28) • Why Yoshua changed his mind about AI risk (02:21:27)

Rob Wiblin

65,088 просмотров • 6 месяцев назад

Loved this 22-minute talk on continual learning for AI agents. Must watch for anyone looking to get agents performant and into production. Credit: Soheil Feizi at AI.Engineer 🔜 SF Code Summit • Agent learning can happen at three layers: the model (weights), the harness (prompts, tools, skills, code, workflows), and memory (session or persistent). • Two fundamental challenges: (1) getting feedback, meaning how do we know if the agent did well and what it should have done instead, and (2) acting on that feedback, meaning deciding which layer or component to change and how. • Feedback sources differ by stage: In development you have benchmarks with evaluators that score pass/fail. In production you only have logs, which can be judged either automatically (LLMs or code analyzing the log, which is scalable) or by human experts (low volume but critical domain knowledge). • Logs plus feedback aren't enough because they're not testable: A single log with feedback is one observation of what happened. You need to lift it into a replayable learning environment, a simulation with tools, users, and defined evaluators, so candidate fixes can be run, verified, and compared. • Three ways to optimize the agent, with tradeoffs: Model-layer updates (SFT, RL post-training like DPO/GRPO, LoRA) are expensive and need benchmarks and evaluators. Harness updates (trace-to-harness coding agents, prompt search like GEPA) are flexible but either untestable and "vibe-based" or benchmark-dependent. Memory updates (fact storage like Letta/Mem0, skill distillation) are cheapest and fastest but usually unverified. • A good learning engine makes "the smallest durable change at the right layer" of the agent. • Verifiable continual learning (VCL): Improve an agent from its own experience where every fix is proven to help and proven to break nothing that already worked. It requires an executable test (replayable failure), a measured delta (score before and after), and regression tests (prior tests still pass). • Four principles of practical VCL: Replayability (turn one-off failures into rerunnable tests), holisticness (one failure can have causes in memory, prompts, tools, workflow, or model, so route the fix to the right layer), lifelongness (fix new failures subject to no regression on past environments, with regression handled inside the optimization loop rather than post-hoc), and efficiency (the loop must run frequently and cheaply, without scaling linearly as past environments accumulate). • Three takeaways: (1) Agent continual learning isn't necessarily fine-tuning; many useful updates live in the harness and memory layers. (2) Production logs are not learning environments and must be transformed into replayable ones. (3) The frontier is regression-aware improvement: fixing new failures while verifying you don't break old ones.

Alex Lieberman

20,533 просмотров • 4 месяцев назад