Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

We got 46% fewer errors than the single best LLM across the 16 most used benchmarks (TerminalBench, LiveCodeBench, etc). Here's how that's possible and what each model can achieve when used optimally (every benchmarks misses the majority of model capabilities) 👇 Interactive Site: Academic Paper:

Martian

3,808 subscribers

346,846 görüntüleme • 19 gün önce •via X (Twitter)

Bilim & Teknoloji Eğitim

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

We shipped a major upgrade to the Auto Router: It now routes based on what the market actually uses for each type of task. When the market migrates a workload to a new model, the router follows within days. Benchmarks & more 👇

We shipped a major upgrade to the Auto Router: It now routes based on what the market actually uses for each type of task. When the market migrates a workload to a new model, the router follows within days. Benchmarks & more 👇

OpenRouter

40,219 görüntüleme • 1 ay önce

Satya Nadella says he cares far less about AGI benchmarks than about real-world impact. The tech industry “became the place we were celebrating ourselves and I just hate it.” He says what matters is not who built the model, but who used it, and whether something changed.

Satya Nadella says he cares far less about AGI benchmarks than about real-world impact. The tech industry “became the place we were celebrating ourselves and I just hate it.” He says what matters is not who built the model, but who used it, and whether something changed.

vitrupo

78,718 görüntüleme • 1 yıl önce

Before founding Vals AI, Rayan Krishnan was a college radio host. We're thrilled to get him back in his element with Vivek Gopalan, this time talking all things benchmarks and evals. As AI capabilities compound, measuring them is becoming both harder and more important. What started as an academic exercise is quickly evolving into critical infrastructure for understanding what AI systems can actually do, and how they perform in the real world. Rayan and Vivek go deep on how benchmarks are evolving, measuring increasingly agentic systems, the weird and wonderful ways we test intelligence, the global AI race, and why evals sit on the critical path to ASI. 0:44 Importance and evolution of benchmarks 8:16 From academic artifact → real-world value 10:57 The next generation of benchmarks 13:34 Benchmarking models vs. agents 16:20 Building benchmarks for organizations 22:09 Weird and wonderful benchmarks 25:58 The AI arms race: US vs. China 31:02 Why evals are on the critical path to ASI 32:50 Vals Series A

Before founding Vals AI, Rayan Krishnan was a college radio host. We're thrilled to get him back in his element with Vivek Gopalan, this time talking all things benchmarks and evals. As AI capabilities compound, measuring them is becoming both harder and more important. What started as an academic exercise is quickly evolving into critical infrastructure for understanding what AI systems can actually do, and how they perform in the real world. Rayan and Vivek go deep on how benchmarks are evolving, measuring increasingly agentic systems, the weird and wonderful ways we test intelligence, the global AI race, and why evals sit on the critical path to ASI. 0:44 Importance and evolution of benchmarks 8:16 From academic artifact → real-world value 10:57 The next generation of benchmarks 13:34 Benchmarking models vs. agents 16:20 Building benchmarks for organizations 22:09 Weird and wonderful benchmarks 25:58 The AI arms race: US vs. China 31:02 Why evals are on the critical path to ASI 32:50 Vals Series A

8VC

105,320 görüntüleme • 1 ay önce

We are releasing Carbon: a crazy fast DNA model Carbon is 275x faster than the next best model. So fast you can process the whole human genome on a single GPU in <2 days. Here are the tricks we used: When modelling DNA sequences a lot of the performance comes down to tokenizing the sequences in a smart way. BPE tokenizer struggle because there are no whitespaces and character (called base in DNA) level tokenizers waste a lot of compute on too many tokens. Carbon is built with a unique tokenizer: we split sequences in chunks of 6 bases, but during both training and inference we can work with single base resolution. That's similar to having word tokens but resolving them at the character level. All possible thanks to the DNA tokens unique structure. The architecture combined with the tokenizer makes the model 275x faster than the previous SoTA (Evo2) at this size. We built an interactive demo so you can explore how the model can generate DNA sequences, investigate the structure of genes, predict the effect of mutations, generate and fold proteins and even reconstruct parts of the tree of life.

We are releasing Carbon: a crazy fast DNA model Carbon is 275x faster than the next best model. So fast you can process the whole human genome on a single GPU in <2 days. Here are the tricks we used: When modelling DNA sequences a lot of the performance comes down to tokenizing the sequences in a smart way. BPE tokenizer struggle because there are no whitespaces and character (called base in DNA) level tokenizers waste a lot of compute on too many tokens. Carbon is built with a unique tokenizer: we split sequences in chunks of 6 bases, but during both training and inference we can work with single base resolution. That's similar to having word tokens but resolving them at the character level. All possible thanks to the DNA tokens unique structure. The architecture combined with the tokenizer makes the model 275x faster than the previous SoTA (Evo2) at this size. We built an interactive demo so you can explore how the model can generate DNA sequences, investigate the structure of genes, predict the effect of mutations, generate and fold proteins and even reconstruct parts of the tree of life.

Leandro von Werra

405,709 görüntüleme • 4 ay önce

JUST IN: Google releases Gemini 1.5, a powerful MoE model. It's a huge breakthrough. The model has the longest context window ever seen: 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. You can can sign up in AI Studio to try it out.

JUST IN: Google releases Gemini 1.5, a powerful MoE model. It's a huge breakthrough. The model has the longest context window ever seen: 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. You can can sign up in AI Studio to try it out.

Lior Alexander

83,432 görüntüleme • 2 yıl önce

New Benchtalks with John Yang: on ProgramBench (0% frontier models at launch) and the lineage/future of coding benchmarks, from SWE-bench/InterCode to now 01:29 ProgramBench launch and reception 03:41 Why artifact-level evaluation, not code-level 06:03 Why models love Python 08:29 ProgramBench as a research tool 12:45 From SWE-bench & InterCode to ProgramBench 17:47 How to grade a coding model 21:53 The position paper & humans in the loop 25:01 Managing quality with agents-in-the-loop 28:40 Internet access and benchmark integrity 35:26 Where models may surpass human abilities 38:56 When a model hits 80% on ProgramBench 43:55 Benchmarks worth paying attention to 46:24 What benchmark do you wish existed 49:32 Will benchmarks still look like benchmarks in 5 years 52:02 How to contribute to ProgramBench

New Benchtalks with John Yang: on ProgramBench (0% frontier models at launch) and the lineage/future of coding benchmarks, from SWE-bench/InterCode to now 01:29 ProgramBench launch and reception 03:41 Why artifact-level evaluation, not code-level 06:03 Why models love Python 08:29 ProgramBench as a research tool 12:45 From SWE-bench & InterCode to ProgramBench 17:47 How to grade a coding model 21:53 The position paper & humans in the loop 25:01 Managing quality with agents-in-the-loop 28:40 Internet access and benchmark integrity 35:26 Where models may surpass human abilities 38:56 When a model hits 80% on ProgramBench 43:55 Benchmarks worth paying attention to 46:24 What benchmark do you wish existed 49:32 Will benchmarks still look like benchmarks in 5 years 52:02 How to contribute to ProgramBench

vincent sunn chen

27,008 görüntüleme • 3 ay önce

.Shyam Sankar says Palantir was able to make Nvidia's Nemotron Ultra model "better than frontier": "I literally almost felt gaslit when, within 24 hours of getting Nemotron up with no post-training, this is vanilla Nemotron Ultra, it did better than frontier." "If you just looked at the numbers, you would say, 'It's nowhere near the Frontier. That shouldn't even be possible.'" "But of course, the benchmarks are wrong. I mean, the benchmarks are right for what the benchmark's measuring, but that's not my business. Those are not the tasks my customers had that they were trying to solve."

.Shyam Sankar says Palantir was able to make Nvidia's Nemotron Ultra model "better than frontier": "I literally almost felt gaslit when, within 24 hours of getting Nemotron up with no post-training, this is vanilla Nemotron Ultra, it did better than frontier." "If you just looked at the numbers, you would say, 'It's nowhere near the Frontier. That shouldn't even be possible.'" "But of course, the benchmarks are wrong. I mean, the benchmarks are right for what the benchmark's measuring, but that's not my business. Those are not the tasks my customers had that they were trying to solve."

Jawwwn

372,411 görüntüleme • 1 ay önce

Today we’re announcing Ontology 1, our newest AI model. Perhaps surprisingly, it’s at least 2.7x more accurate than the world’s best ecommerce search engines, and handles queries that have never been possible before. We keep reaching for the edge of what it can do. We haven’t found it yet. Not only that, but it learns on its own with no retraining or fine-tuning. And it’s hallucination-free. It’s a successor architecture for search. Learn how we built it at and check out the benchmarks at Try it at

Today we’re announcing Ontology 1, our newest AI model. Perhaps surprisingly, it’s at least 2.7x more accurate than the world’s best ecommerce search engines, and handles queries that have never been possible before. We keep reaching for the edge of what it can do. We haven’t found it yet. Not only that, but it learns on its own with no retraining or fine-tuning. And it’s hallucination-free. It’s a successor architecture for search. Learn how we built it at and check out the benchmarks at Try it at

Onton

326,143 görüntüleme • 1 ay önce

Terminal-Bench 2.0 went from ~25% → 80% in four months and became the standard eval for frontier CLI agents. Now, TB3 is in the works. I talked to Alex Shaw about what happens when model capabilities climb faster than we can measure them. His answer: the benchmark factory (Harbor Framework)— infrastructure to develop hard, representative evals at the pace that the frontier moves. As Alex put it: "we need a thousand times more benchmarks than we have right now." 00:23 - How quickly models hill-climbed TB2 01:46 - What rapid progress reveals about benchmarks vs. real-world capability 03:28 - What made Terminal-Bench stick 04:58 - Why the terminal is the right abstraction for agentic AI 07:14 - How TB2 maintains task quality at scale 09:23 - Managing benchmark integrity in a benchmaxxing world 10:47 - Harbor: from experiment to benchmark factory 12:19 - What Harbor does that nothing else did 14:37 - The invariants: what won't change as agent evals evolve 16:55 - The benchmark Alex most wants to see built 18:18 - The ideal human-in-the-loop task creation flywheel 20:32 - How to contribute to Terminal-Bench 3.0

Terminal-Bench 2.0 went from ~25% → 80% in four months and became the standard eval for frontier CLI agents. Now, TB3 is in the works. I talked to Alex Shaw about what happens when model capabilities climb faster than we can measure them. His answer: the benchmark factory (Harbor Framework)— infrastructure to develop hard, representative evals at the pace that the frontier moves. As Alex put it: "we need a thousand times more benchmarks than we have right now." 00:23 - How quickly models hill-climbed TB2 01:46 - What rapid progress reveals about benchmarks vs. real-world capability 03:28 - What made Terminal-Bench stick 04:58 - Why the terminal is the right abstraction for agentic AI 07:14 - How TB2 maintains task quality at scale 09:23 - Managing benchmark integrity in a benchmaxxing world 10:47 - Harbor: from experiment to benchmark factory 12:19 - What Harbor does that nothing else did 14:37 - The invariants: what won't change as agent evals evolve 16:55 - The benchmark Alex most wants to see built 18:18 - The ideal human-in-the-loop task creation flywheel 20:32 - How to contribute to Terminal-Bench 3.0

vincent sunn chen

11,626 görüntüleme • 5 ay önce

Meet Reka Core, our best and most capable multimodal language model yet. 🔮 It’s been a busy few months training this model and we are glad to finally ship it! 💪 Core has a lot of capabilities, and one of them is understanding video --- let’s see what Core thinks of the 3 body trailer.👇

Meet Reka Core, our best and most capable multimodal language model yet. 🔮 It’s been a busy few months training this model and we are glad to finally ship it! 💪 Core has a lot of capabilities, and one of them is understanding video --- let’s see what Core thinks of the 3 body trailer.👇

Reka

758,105 görüntüleme • 2 yıl önce

Really fun to hang again with my friend 🃏 Noam Brown (OpenAI research scientist, our first guest ever on No Priors in early 2023) to talk about the implications of large test-time compute, and what happens when models are given $10M budgets to spend on a single task. Topics: 01:23 – Why Benchmarks Are Broken 04:19 – Compute Budgets and Projections 06:48 – How Long Should Models Think? 08:01 – Benchmarkmaxxing 09:48 – Noam's Evals 12:40 – Safety (When Model Capability Scales With Spend) 16:09 – Implications For the Model Release Cycle 18:34 – Latent Model Capability 22:27 – Limits on Recursive Self-Improvement 28:38 – Large-Scale Multi-Agent Coordination 30:39 – Competition at the Frontier 33:19 – Breaking the Benchmark Grid Equilibrium 34:57 – Why Benchmarks Should be Scaled by Cost

Really fun to hang again with my friend 🃏 Noam Brown (OpenAI research scientist, our first guest ever on No Priors in early 2023) to talk about the implications of large test-time compute, and what happens when models are given $10M budgets to spend on a single task. Topics: 01:23 – Why Benchmarks Are Broken 04:19 – Compute Budgets and Projections 06:48 – How Long Should Models Think? 08:01 – Benchmarkmaxxing 09:48 – Noam's Evals 12:40 – Safety (When Model Capability Scales With Spend) 16:09 – Implications For the Model Release Cycle 18:34 – Latent Model Capability 22:27 – Limits on Recursive Self-Improvement 28:38 – Large-Scale Multi-Agent Coordination 30:39 – Competition at the Frontier 33:19 – Breaking the Benchmark Grid Equilibrium 34:57 – Why Benchmarks Should be Scaled by Cost

sarah guo

87,850 görüntüleme • 2 ay önce

"More Agents Is All You Need" - Great idea in this Paper. ------- What is CoT-SC ? Simply put, CoT-SC generates multiple thought chains and picks the most self-consistent one as the final answer. 📌 CoT-SC leverages diverse Chain-of-Thought (CoT) prompts to extract various reasoning processes from a single Large Language Model (LLM) and selects the final answer through a majority voting mechanism. Essentially, it harnesses multiple reasoning pathways generated by prompting the LLM with different CoT prompts, evaluating these pathways, and determining the most self-consistent answer as the conclusion.

"More Agents Is All You Need" - Great idea in this Paper. ------- What is CoT-SC ? Simply put, CoT-SC generates multiple thought chains and picks the most self-consistent one as the final answer. 📌 CoT-SC leverages diverse Chain-of-Thought (CoT) prompts to extract various reasoning processes from a single Large Language Model (LLM) and selects the final answer through a majority voting mechanism. Essentially, it harnesses multiple reasoning pathways generated by prompting the LLM with different CoT prompts, evaluating these pathways, and determining the most self-consistent answer as the conclusion.

Rohan Paul

20,598 görüntüleme • 2 yıl önce

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

133,068 görüntüleme • 1 ay önce

Introducing Merge Gateway - Build Your Own Router. You're three sprints into your coding assistant. You pick the most hyped model, integrate, test, deploy. A month later, a new model drops. Now you re-test, re-integrate, re-deploy. Your product didn't change, but the benchmark did. That's how most AI teams operate. Chasing a "best" defined by people who've never seen their product. There is no best model. There's only the right one for your product, users, and use-cases. Build Your Own Router runs on your definition of good. Pick your benchmarks, weigh them, add your own evals. Merge routes every request to your winner. 👉$100 in credits to the first 200 people that comment

Introducing Merge Gateway - Build Your Own Router. You're three sprints into your coding assistant. You pick the most hyped model, integrate, test, deploy. A month later, a new model drops. Now you re-test, re-integrate, re-deploy. Your product didn't change, but the benchmark did. That's how most AI teams operate. Chasing a "best" defined by people who've never seen their product. There is no best model. There's only the right one for your product, users, and use-cases. Build Your Own Router runs on your definition of good. Pick your benchmarks, weigh them, add your own evals. Merge routes every request to your winner. 👉$100 in credits to the first 200 people that comment

Shensi Ding

3,668,024 görüntüleme • 3 ay önce

Some of our top customers are still choosing Llama 3.1 8B. For a while, we jumped to whatever hottest, latest model was taking up our twitter feed. 🙈 But as we are quickly realizing, to create a SOTA product, you need a model that fits your exact use case. Here’s what our customers tell us: > a lot of the legwork is actually around prompting > there’s an art to selecting and combining multiple models > benchmarks only show part of the picture. you have to understand the unique quirks of each model. Especially as model releases become more and more frequent, we need a clear way to evaluate new models. We have to break free of the naive trend to migrate to the ‘latest and greatest’. And you can easily achieve this using tools like Cerebras and Braintrust to swap models safely (without breaking production).

Some of our top customers are still choosing Llama 3.1 8B. For a while, we jumped to whatever hottest, latest model was taking up our twitter feed. 🙈 But as we are quickly realizing, to create a SOTA product, you need a model that fits your exact use case. Here’s what our customers tell us: > a lot of the legwork is actually around prompting > there’s an art to selecting and combining multiple models > benchmarks only show part of the picture. you have to understand the unique quirks of each model. Especially as model releases become more and more frequent, we need a clear way to evaluate new models. We have to break free of the naive trend to migrate to the ‘latest and greatest’. And you can easily achieve this using tools like Cerebras and Braintrust to swap models safely (without breaking production).

Cerebras

346,446 görüntüleme • 9 ay önce

Here is EVERY AI tool you need to be using in August 2026 I've personally used each for 100+ If you master them and use them all, you'll be a productivity MACHINE The list of the tools are down below. How to use each is in the video! • Best planning model: Fable 5 medium- smartest model out there. Use it to do all your ideation and planning • Best execution model: ChatGPT 5.6 sol medium- best bang for your buck. TONS of usage and incredibly smart • Best AI tool: ChatGPT Voice: the future of interfacing with technology • Best AI agent harness: ChatGPT Desktop app- can do anything any other AI agent can do, but does it better • Best model for building UI/UX/games: Claude Code w/ Claude Opus 5- this model gets a lot of hate. Some of it warranted, most of it not. NOTHING beats Opus 5 for UI • Local model: Qwen 3.6 27b on DGX Spark- best speed and intelligence ratio in local AI • Other must have tools: Buzz from Jack Dorsey. The future of AI is multiplayer, and this is the first true multiplayer AI platform out there

Here is EVERY AI tool you need to be using in August 2026 I've personally used each for 100+ If you master them and use them all, you'll be a productivity MACHINE The list of the tools are down below. How to use each is in the video! • Best planning model: Fable 5 medium- smartest model out there. Use it to do all your ideation and planning • Best execution model: ChatGPT 5.6 sol medium- best bang for your buck. TONS of usage and incredibly smart • Best AI tool: ChatGPT Voice: the future of interfacing with technology • Best AI agent harness: ChatGPT Desktop app- can do anything any other AI agent can do, but does it better • Best model for building UI/UX/games: Claude Code w/ Claude Opus 5- this model gets a lot of hate. Some of it warranted, most of it not. NOTHING beats Opus 5 for UI • Local model: Qwen 3.6 27b on DGX Spark- best speed and intelligence ratio in local AI • Other must have tools: Buzz from Jack Dorsey. The future of AI is multiplayer, and this is the first true multiplayer AI platform out there

Alex Finn

41,920 görüntüleme • 1 ay önce

SITUATION EXPLAINED: Why is the Bridgewater Thinking Machines paper a blueprint for every enterprise that has ever paid frontier model prices for generic outputs? 47fucb4r8curb4fc8f8r4bfic8r: "What Thinking Machines and Bridgewater did together is a blueprint for how a wide variety of stakeholders within the market can benefit tremendously from using LLMs and integrating them into a larger enterprise system." "They got a pretty old model. A Qwen3 model. Two hundred and thirty-five billion parameters. Not a new model. Not a very sophisticated or impressive state-of-the-art model. And they essentially fine-tuned it on Bridgewater's own proprietary data." "A financial analyst gets a ton of financial documents and has to filter which are worth looking at. Then evaluate which news pieces are relevant. They have a lot of tacit knowledge from doing the job on where to look. What they did was inject a lot of this tacit knowledge into the model through fine-tuning." "Stop obsessing over state-of-the-art models that are the most intelligent possible. Focus on targeting a very specific LLM specifically trained for a specific function to do that function better than the alternatives on the market." "This old non-state-of-the-art model got 84.7% accuracy versus 78% for all the state-of-the-art models. A massive improvement. And the kicker is they got a better performing model that is 14 times cheaper to use."

SITUATION EXPLAINED: Why is the Bridgewater Thinking Machines paper a blueprint for every enterprise that has ever paid frontier model prices for generic outputs? 47fucb4r8curb4fc8f8r4bfic8r: "What Thinking Machines and Bridgewater did together is a blueprint for how a wide variety of stakeholders within the market can benefit tremendously from using LLMs and integrating them into a larger enterprise system." "They got a pretty old model. A Qwen3 model. Two hundred and thirty-five billion parameters. Not a new model. Not a very sophisticated or impressive state-of-the-art model. And they essentially fine-tuned it on Bridgewater's own proprietary data." "A financial analyst gets a ton of financial documents and has to filter which are worth looking at. Then evaluate which news pieces are relevant. They have a lot of tacit knowledge from doing the job on where to look. What they did was inject a lot of this tacit knowledge into the model through fine-tuning." "Stop obsessing over state-of-the-art models that are the most intelligent possible. Focus on targeting a very specific LLM specifically trained for a specific function to do that function better than the alternatives on the market." "This old non-state-of-the-art model got 84.7% accuracy versus 78% for all the state-of-the-art models. A massive improvement. And the kicker is they got a better performing model that is 14 times cheaper to use."

MTS

93,185 görüntüleme • 2 ay önce