Загрузка видео...

Не удалось загрузить видео

На главную

in our latest 'backstage' conversation at Orchestra , our founding engineer Mathieu Ripert covers: why companies need evals why you can start with harness engineering and move to building training environments how orchestra ties all this together

lluismanrique's profile picture

Luis Manrique

1,267 subscribers

165,281 просмотров • 27 дней назад •via X (Twitter)

Наука и технологииФинансы
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 5

Фото профиля kartik bhardwaj
kartik bhardwaj27 дней назад

@orchestradotai @ripertmat Starting from the harness makes sense, the traces a good harness already logs are most of what you need to turn real failures into training environments later.

Фото профиля Chris Lu
Chris Lu27 дней назад

@orchestradotai @ripertmat Think eval engineering could be its own role?

Фото профиля Luis Manrique
Luis Manrique27 дней назад

@orchestradotai @ripertmat it may be already!

Фото профиля James Plowman
James Plowman21 дней назад

@orchestradotai @ripertmat great conversation here

Фото профиля Laura Lou
Laura Lou26 дней назад

@orchestradotai @ripertmat Hey Luis, starting with harness engineering and growing into training environments is a helpful way to frame evals. Great backstage episode. Lets connect! :)

Похожие видео

Anyone can build on The Rayls Public Chain. In our latest AMA, Alex Buelau discusses what's possible, how to get started, and why now is the time to start building. Watch the full conversation below 👇
1:30

Anyone can build on The Rayls Public Chain. In our latest AMA, Alex Buelau discusses what's possible, how to get started, and why now is the time to start building. Watch the full conversation below 👇

Rayls

11,102 просмотров • 3 месяцев назад

Playing “Buddy Bolden Blues” with Victor Goines backstage at Arts Centre Melbourne before tonight’s “All Rise” performance with Jazz at Lincoln Center Orchestra and Melbourne Symphony Orchestra conducted by @BenNorthey
0:56

Playing “Buddy Bolden Blues” with Victor Goines backstage at Arts Centre Melbourne before tonight’s “All Rise” performance with Jazz at Lincoln Center Orchestra and Melbourne Symphony Orchestra conducted by @BenNorthey

Wynton Marsalis

49,863 просмотров • 3 лет назад

Halluminate is building reinforcement learning environments and benchmarks to help AI models do knowledge work beyond coding, starting with finance. With a team of fewer than 10 people, the company works with four of the top five closed-source US AI labs and recently raised a $30M Series A. In this fireside, co-founders Jerry Wu and wyatt marshall sit down with YC's Jon Xu to share how testing browser agents led them to building the environments models now use to practice tasks like financial modeling. They explain why better verification, not just more training tasks, is critical to making models smarter, and how that work could eventually lead to simulated companies where teams of AI agents learn to work together. 00:00 — Halluminate’s Pivot From Evals to Training Environments 03:48 — Teaching AI to Do Financial Work 07:13 — Why Verification Matters More Than Volume 10:13 — From Finance to Simulated Companies 13:30 — AI Safety and Building the Team
16:33

Halluminate is building reinforcement learning environments and benchmarks to help AI models do knowledge work beyond coding, starting with finance. With a team of fewer than 10 people, the company works with four of the top five closed-source US AI labs and recently raised a $30M Series A. In this fireside, co-founders Jerry Wu and wyatt marshall sit down with YC's Jon Xu to share how testing browser agents led them to building the environments models now use to practice tasks like financial modeling. They explain why better verification, not just more training tasks, is critical to making models smarter, and how that work could eventually lead to simulated companies where teams of AI agents learn to work together. 00:00 — Halluminate’s Pivot From Evals to Training Environments 03:48 — Teaching AI to Do Financial Work 07:13 — Why Verification Matters More Than Volume 10:13 — From Finance to Simulated Companies 13:30 — AI Safety and Building the Team

Y Combinator

260,915 просмотров • 1 месяц назад

She explains why you need to start normalizing cutting ties with disloyal energy.
0:25

She explains why you need to start normalizing cutting ties with disloyal energy.

Art of Life 🦋

64,539 просмотров • 7 месяцев назад

Evals 101: a new course from Braintrust. Everything you need to know about evals, and how to do them yourself. Module one: Why are evals important? - the six most common problems developers face when shipping AI applications - why traditional software thinking doesn't apply to AI - how evals can fix these problems
1:51

Evals 101: a new course from Braintrust. Everything you need to know about evals, and how to do them yourself. Module one: Why are evals important? - the six most common problems developers face when shipping AI applications - why traditional software thinking doesn't apply to AI - how evals can fix these problems

Braintrust

37,043 просмотров • 6 месяцев назад

Anthropic engineer at Stanford: “80% of our engineers are using self-improving loops. Now everyone is building agentic graphs.” in 1-hour Stanford lecture, Anthropic engineer reveals how Claude thinks - and why Graphs are the future. This 1-hour watch will replace your $500 course on agentic engineering. Watch it today, then read how to become a Graph Engineer in the article below.
1:11:30

Anthropic engineer at Stanford: “80% of our engineers are using self-improving loops. Now everyone is building agentic graphs.” in 1-hour Stanford lecture, Anthropic engineer reveals how Claude thinks - and why Graphs are the future. This 1-hour watch will replace your $500 course on agentic engineering. Watch it today, then read how to become a Graph Engineer in the article below.

Codez

28,435 просмотров • 3 месяцев назад

My talk from AI Engineer is now live! It covers how we're automating parts of AI research and building systems to rapidly improve our models. I cover some of the work our team did to train Grok 4.5 together with SpaceXAI.
19:36

My talk from AI Engineer is now live! It covers how we're automating parts of AI research and building systems to rapidly improve our models. I cover some of the work our team did to train Grok 4.5 together with SpaceXAI.

Lee Robinson

128,924 просмотров • 3 месяцев назад

Our last Next Generation Leaders Training brought together 100 people from across the state to start building a new era of politics in New Jersey. Hear from some of them about what they learned and why they're choosing to step up. Sign up for our next training in South Jersey:
2:49

Our last Next Generation Leaders Training brought together 100 people from across the state to start building a new era of politics in New Jersey. Hear from some of them about what they learned and why they're choosing to step up. Sign up for our next training in South Jersey:

Andy Kim

10,268 просмотров • 10 месяцев назад

Performant reinforcement learning is a hard infrastructure and research problem. We’ve recently made on-demand multi-node clusters available and showed how to take advantageous of heterogeneous sampling compute. But there still exists the need for high-fidelity environments, clear and robust reward functions, and tuned training recipes that actually work. Most teams are actually nearly there! With an agent harness deployed in production and evals/agent traces collected from that deployment, you just need an easy way to start training your data. So we built a training library that gives you the building blocks, tuned defaults, and observability you need for easy, production-grade LLM training. It’s available now. All it takes is a prompt.
0:27

Performant reinforcement learning is a hard infrastructure and research problem. We’ve recently made on-demand multi-node clusters available and showed how to take advantageous of heterogeneous sampling compute. But there still exists the need for high-fidelity environments, clear and robust reward functions, and tuned training recipes that actually work. Most teams are actually nearly there! With an agent harness deployed in production and evals/agent traces collected from that deployment, you just need an easy way to start training your data. So we built a training library that gives you the building blocks, tuned defaults, and observability you need for easy, production-grade LLM training. It’s available now. All it takes is a prompt.

Modal

12,904 просмотров • 18 дней назад

If it’s made in Canada, you can trust it was made well. That’s part of our brand, and it’s why companies like Honda are building here.
0:50

If it’s made in Canada, you can trust it was made well. That’s part of our brand, and it’s why companies like Honda are building here.

Justin Trudeau

212,629 просмотров • 2 лет назад

1st time singing with our wonderful orchestra @pittsburghclo (You can see me furiously counting to try to stay with them, while remembering lyrics.) #intothewords
1:29

1st time singing with our wonderful orchestra @pittsburghclo (You can see me furiously counting to try to stay with them, while remembering lyrics.) #intothewords

Carolee Carmello

25,302 просмотров • 3 лет назад

Why settle for slow? Our latest interactive blog covers the history of IO devices, how storage works in the cloud, and why separation of storage and compute is bad for OLTP databases.
0:30

Why settle for slow? Our latest interactive blog covers the history of IO devices, how storage works in the cloud, and why separation of storage and compute is bad for OLTP databases.

PlanetScale

320,964 просмотров • 1 год назад

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:
1:12

So many people talking about AI evals. Hamel Husain and Shreya Shankar show me how to actually build one. Live. Hamel and Shreya teach the world’s most popular course on evals, and have long been at the forefront of this emerging and important new skill for AI product builders. I'm so excited to finally have them on the podcast. Learn: 🔸 WTF are evals 🔸 A step-by-step live walkthrough of how to create an eval 🔸 Why evals have become so important 🔸 The debate between “vibes” and systematic evals 🔸 Code-based evals vs. LLM-as-judge 🔸 Why you only need 4-7 evals (not hundreds) 🔸 Building an LLM judge in Google Sheets 🔸 Why you always want to start with error analysis 🔸 Much more Listen now 👇 • YouTube: • Spotify: • Apple: Thank you to our wonderful sponsors for supporting the podcast: 🏆 Fin — The #1 AI agent for customer service: 🏆 dscout — The UX platform to capture insights at every stage: from ideation to production: 🏆 Mercury — The art of simplified finances:

Lenny Rachitsky

125,296 просмотров • 1 год назад

.Andy Fang Boris Cherny on why engineering leaders need to experience the full process of building and shipping again. You have to build with AI to understand where engineering is going and avoid being out of step with your teams. That bias for action is a big part of our culture at DoorDash
0:46

.Andy Fang Boris Cherny on why engineering leaders need to experience the full process of building and shipping again. You have to build with AI to understand where engineering is going and avoid being out of step with your teams. That bias for action is a big part of our culture at DoorDash

Stanley Tang

42,661 просмотров • 2 месяцев назад

Most multimodal systems need data that combines every modality together. Hard to get, expensive to build. Our CTO Mathias Lechner, Mathias Lechner, sits down with Saniya Karwa, Saniya Karwa, from our multimodal research team to talk about building a mode that handles text, audio, and image, and why you might not need as much combined training data as you think.
4:13

Most multimodal systems need data that combines every modality together. Hard to get, expensive to build. Our CTO Mathias Lechner, Mathias Lechner, sits down with Saniya Karwa, Saniya Karwa, from our multimodal research team to talk about building a mode that handles text, audio, and image, and why you might not need as much combined training data as you think.

Liquid AI

15,488 просмотров • 5 месяцев назад

The sound of fall approaching is music to our ears, just like our latest installment in the Founder's Pack Series, performed by the Budapest Scoring Orchestra!
2:35

The sound of fall approaching is music to our ears, just like our latest installment in the Founder's Pack Series, performed by the Budapest Scoring Orchestra!

The Bazaar

27,631 просмотров • 2 лет назад

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):
0:30

I like to think of evals as something active, not passive -- it's a North Star that steers LLMs toward higher intelligence. Evals should drive your RL/SFT/post-training decisions. Internal evals at frontier labs make a huge difference -- and you can see it in how models behave differently (GPT seems better at one-shot tasks, Claude at multi-turn). If you want to learn more about building evals that actually improve your model in post-training, check out our AMD x DeeplearningAI course "Fine-tuning & RL for LLMs: Intro to Post-training" (content is free):

Sharon Zhou

16,903 просмотров • 9 месяцев назад

Latest Hold These Truths podcast covers everything you need to know about Flock cameras: how they work, how police use them, and why some people are literally shooting them with shotguns.
1:37

Latest Hold These Truths podcast covers everything you need to know about Flock cameras: how they work, how police use them, and why some people are literally shooting them with shotguns.

Dan Crenshaw

27,131 просмотров • 2 месяцев назад

📍🟡 “There is a global shortage of engineering talent. Our challenge is to retain them when Taiwanese and Singaporean companies are poaching them for SGD5000-8000.” Sidec CEO Yong Kai Ping discuss: •⁠ How Selangor managed to train 600 integrated circuit engineers in just 3 years •⁠ Why they spend half a million training an engineer •⁠ Why big companies like Samsung and TSMC are poaching Malaysia’s fresh graduates, and the unconventional way SIDEC is retaining talents Full interview ⬇️
2:37

📍🟡 “There is a global shortage of engineering talent. Our challenge is to retain them when Taiwanese and Singaporean companies are poaching them for SGD5000-8000.” Sidec CEO Yong Kai Ping discuss: •⁠ How Selangor managed to train 600 integrated circuit engineers in just 3 years •⁠ Why they spend half a million training an engineer •⁠ Why big companies like Samsung and TSMC are poaching Malaysia’s fresh graduates, and the unconventional way SIDEC is retaining talents Full interview ⬇️

BFM89.9

133,986 просмотров • 1 месяц назад