Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Inferact cofounder Simon Mo: “I fundamentally believe that open source, especially how vLLM itself is structured, is critical to the AI infrastructure in the world. And what we want to do with Inferact is to support, maintain, steward and push forward the open source ecosystem.” “Open source moves so... show more

a16z

936,334 subscribers

29,383 views • 9 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Today we announced a $150M seed round in Inferact, a new startup led by the maintainers of the vLLM project. Inferact will support the vLLM open source project through dedicated financial and developer resources and build what they see as the next generation commercial inference engine. Cofounders Simon Mo and Woosuk Kwon joined a16z GP Matt Bornstein for a conversation on how vLLM came to be, what they’ve learned from building it, and what they’re now doing at Inferact. They cover how vLLM began as a side project, why open source is essential to the world’s AI infrastructure, why inference is getting harder, and more. 00:00 Introduction 11:41 Community and collaboration in vLLM 19:19 Understanding inference engines 24:27 Cluster scale and GPU deployment 31:19 Belief in open source AI 35:45 Founding Inferact 40:00 The future of Inference at scale Simon Mo Woosuk Kwon Matt Bornstein

Today we announced a $150M seed round in Inferact, a new startup led by the maintainers of the vLLM project. Inferact will support the vLLM open source project through dedicated financial and developer resources and build what they see as the next generation commercial inference engine. Cofounders Simon Mo and Woosuk Kwon joined a16z GP Matt Bornstein for a conversation on how vLLM came to be, what they’ve learned from building it, and what they’re now doing at Inferact. They cover how vLLM began as a side project, why open source is essential to the world’s AI infrastructure, why inference is getting harder, and more. 00:00 Introduction 11:41 Community and collaboration in vLLM 19:19 Understanding inference engines 24:27 Cluster scale and GPU deployment 31:19 Belief in open source AI 35:45 Founding Inferact 40:00 The future of Inference at scale Simon Mo Woosuk Kwon Matt Bornstein

a16z

88,839 views • 9 months ago

Open source AI, the future of Python development, journalism in the age of AI, and more... I sat down with Simon Willison to chat about this, what he is working on these days, and where open source AI is going. Simon is the best, I hope you enjoy a glimpse into the future.

Open source AI, the future of Python development, journalism in the age of AI, and more... I sat down with Simon Willison to chat about this, what he is working on these days, and where open source AI is going. Simon is the best, I hope you enjoy a glimpse into the future.

Logan Kilpatrick

68,668 views • 1 year ago

vLLM runs on half a million GPUs at any given moment. Most people have never heard of it. Simon Mo, co-founder and CEO of Inferact and lead maintainer of vLLM, sits down with a16z’s Matt Bornstein and Elena Burger to discuss what it takes to actually run open models in production, the advantages of open models, Simon’s mission at Inferact, and more. 00:00 Intro 01:46 When open source became critical infrastructure 08:55 Day zero model releases, and the drama behind them 14:59 What Kimi K3 actually buys you 18:56 Why open model licenses are changing 22:24 The pharmaceutical analogy for funding model training 26:16 If GPUs got 99% cheaper 29:08 Why vLLM, OpenRouter, and Ollama all started before ChatGPT 35:42 Building a company on an open source project 39:48 The inventor of RoPE removing RoPE Simon Mo Matt Bornstein Elena

vLLM runs on half a million GPUs at any given moment. Most people have never heard of it. Simon Mo, co-founder and CEO of Inferact and lead maintainer of vLLM, sits down with a16z’s Matt Bornstein and Elena Burger to discuss what it takes to actually run open models in production, the advantages of open models, Simon’s mission at Inferact, and more. 00:00 Intro 01:46 When open source became critical infrastructure 08:55 Day zero model releases, and the drama behind them 14:59 What Kimi K3 actually buys you 18:56 Why open model licenses are changing 22:24 The pharmaceutical analogy for funding model training 26:16 If GPUs got 99% cheaper 29:08 Why vLLM, OpenRouter, and Ollama all started before ChatGPT 35:42 Building a company on an open source project 39:48 The inventor of RoPE removing RoPE Simon Mo Matt Bornstein Elena

a16z

211,070 views • 3 months ago

In an AI world, effort is no longer the bottleneck—ideas are. Open-source AI enables anyone to have access to build anything they want without depending on the one-sided gates that closed-source offers. This is amplified by the open innovation that open-source development enables.

In an AI world, effort is no longer the bottleneck—ideas are. Open-source AI enables anyone to have access to build anything they want without depending on the one-sided gates that closed-source offers. This is amplified by the open innovation that open-source development enables.

Sentient

17,408 views • 9 months ago

We co-led Inferact's $150M seed round to support them in their mission to build the inference engine for all current and future AI. In this episode of The Investment Memo, Lightspeed's Bucky Moore and James Alcorn sit down with Simon Mo (Co-Founder & CEO Inferact) to cover: - How vLLM grew to 60K+ GitHub stars - Why inference is shifting to the majority of compute - How vLLM evolved from a research project into the industry standard - Why building a company was the next step to push open-source inference forward 00:00 Introduction 02:03 The investment memo 04:47 Latency vs throughput vs cost 06:19 Paged attention explained 08:04 The evolution of attention 09:42 Growing the vLLM open source community 11:41 Working with hardware vendors 14:45 Deploying vLLM at large scale 16:03 Inferact's culture of openness 18:45 Building an open ecosystem and horizontal stack 19:45 Inferact's approach to fundraising 22:14 What is the future of inference? Simon Mo Bucky Moore James Alcorn

We co-led Inferact's $150M seed round to support them in their mission to build the inference engine for all current and future AI. In this episode of The Investment Memo, Lightspeed's Bucky Moore and James Alcorn sit down with Simon Mo (Co-Founder & CEO Inferact) to cover: - How vLLM grew to 60K+ GitHub stars - Why inference is shifting to the majority of compute - How vLLM evolved from a research project into the industry standard - Why building a company was the next step to push open-source inference forward 00:00 Introduction 02:03 The investment memo 04:47 Latency vs throughput vs cost 06:19 Paged attention explained 08:04 The evolution of attention 09:42 Growing the vLLM open source community 11:41 Working with hardware vendors 14:45 Deploying vLLM at large scale 16:03 Inferact's culture of openness 18:45 Building an open ecosystem and horizontal stack 19:45 Inferact's approach to fundraising 22:14 What is the future of inference? Simon Mo Bucky Moore James Alcorn

Lightspeed

27,040 views • 9 months ago

The open-source & open-weight frontier is 7 months behind the closed-model frontier. Let’s talk about it. Check out last week’s episode for the reasons why you might want to consider open-source & open-weight models. Welcome to the fourth episode of the Open Model Series with merve, Hugging Face AI/ML engineer & open-sourceress! We have been discussing open-source & open-weight AI: what it is, how it works, and how you can run them on your own compute. Please leave all your questions about open-source & open-weight AI in the comments! This is a super important topic that is novel to most people, and it can be challenging to grasp - so rest assured your questions are shared with many others!

The open-source & open-weight frontier is 7 months behind the closed-model frontier. Let’s talk about it. Check out last week’s episode for the reasons why you might want to consider open-source & open-weight models. Welcome to the fourth episode of the Open Model Series with merve, Hugging Face AI/ML engineer & open-sourceress! We have been discussing open-source & open-weight AI: what it is, how it works, and how you can run them on your own compute. Please leave all your questions about open-source & open-weight AI in the comments! This is a super important topic that is novel to most people, and it can be challenging to grasp - so rest assured your questions are shared with many others!

Harper Carroll

17,279 views • 2 months ago

Marc Andreessen on the case for open source AI: "How would you feel if the only way your kid could learn about AI is to go to work for one of two companies? Like that, as a parent, I think would probably drive you pretty crazy." "Whereas in a world of open source, every kid can learn how this stuff works." "Without open source, AI inevitably centralizes down to one or two companies, or maybe just flat out the government." "Close your eyes and picture the concept of government sushi. And then keep your eyes closed and picture government AI. How do you feel about that? Open source is the guarantor that that's not the future." Marc Andreessen 🇺🇸

Marc Andreessen on the case for open source AI: "How would you feel if the only way your kid could learn about AI is to go to work for one of two companies? Like that, as a parent, I think would probably drive you pretty crazy." "Whereas in a world of open source, every kid can learn how this stuff works." "Without open source, AI inevitably centralizes down to one or two companies, or maybe just flat out the government." "Close your eyes and picture the concept of government sushi. And then keep your eyes closed and picture government AI. How do you feel about that? Open source is the guarantor that that's not the future." Marc Andreessen 🇺🇸

a16z

37,217 views • 17 days ago

ELON: CHINA HAS THE BEST OPEN SOURCE AI MODELS RIGHT NOW “The best open source models are generally from China, which is bizarre. I think the second best one, or maybe it’s better than second best, is Grok 2.5. The open source model is actually very good, and we will continue to open source our models.” Source: The All-In Podcast

ELON: CHINA HAS THE BEST OPEN SOURCE AI MODELS RIGHT NOW “The best open source models are generally from China, which is bizarre. I think the second best one, or maybe it’s better than second best, is Grok 2.5. The open source model is actually very good, and we will continue to open source our models.” Source: The All-In Podcast

Mario Nawfal

1,163,582 views • 1 year ago

Inferact Co-founder Simon Mo on the trajectory of open and closed models: "What really differentiates open-weight models from closed-weight models, right? In the end, there's not much differentiation. It's more about the distribution strategy and go-to-market strategy... Capability-wise, I don't really see a big gap, not even today." "It's not about just source data. It's not about where to get the data from. Rather, it's who can build the best environment and who can make the most optimization and algorithmic choices to leverage all this learning from this environment." "So the next year is all gonna be about that. It's about how open-weight model labs are differentiating and really getting the model to meet the real world." "Project out in a year, there's not gonna be any difference." Simon Mo

Inferact Co-founder Simon Mo on the trajectory of open and closed models: "What really differentiates open-weight models from closed-weight models, right? In the end, there's not much differentiation. It's more about the distribution strategy and go-to-market strategy... Capability-wise, I don't really see a big gap, not even today." "It's not about just source data. It's not about where to get the data from. Rather, it's who can build the best environment and who can make the most optimization and algorithmic choices to leverage all this learning from this environment." "So the next year is all gonna be about that. It's about how open-weight model labs are differentiating and really getting the model to meet the real world." "Project out in a year, there's not gonna be any difference." Simon Mo

a16z

33,782 views • 3 months ago

I can't believe this is open source. Wan 2.1 has great prompt coherence and the realism is really impressive for an open source model. This is image to video by the way.

I can't believe this is open source. Wan 2.1 has great prompt coherence and the realism is really impressive for an open source model. This is image to video by the way.

Halim Alrasihi

65,185 views • 1 year ago

Tobi talking about making pi-autoresearch open-source: "I do not believe in software as owned" "Open-source is about making gifts to the world"

Tobi talking about making pi-autoresearch open-source: "I do not believe in software as owned" "Open-source is about making gifts to the world"

David Cortés

66,390 views • 6 months ago

What does it mean when an AI model is “open weights” and how is it different than “open source”? Welcome to the next episode of the Open Model Series with merve, Hugging Face AI/ML engineer & open-sourceress! Every week we will discuss open-source & open-weight AI: what it is, how it works, and how you can run them on your own compute. Episode 3 out next week; episode 1 linked below. Please leave all your questions about open-source & open-weight AI in the comments. This is a super important topic that is novel to most people, and it can be challenging to grasp - so your questions are shared with many others.

What does it mean when an AI model is “open weights” and how is it different than “open source”? Welcome to the next episode of the Open Model Series with merve, Hugging Face AI/ML engineer & open-sourceress! Every week we will discuss open-source & open-weight AI: what it is, how it works, and how you can run them on your own compute. Episode 3 out next week; episode 1 linked below. Please leave all your questions about open-source & open-weight AI in the comments. This is a super important topic that is novel to most people, and it can be challenging to grasp - so your questions are shared with many others.

Harper Carroll

32,624 views • 2 months ago

Matt Bornstein says open source is how AI companies stop being wrappers: "A year ago, a bunch of smaller companies or new application companies were trying to figure out, 'How do I really build an AI without just being a wrapper on top of OpenAI?' The answer to that question turned out to be open source." "This is what Cursor did. This is what Decagon and Harvey are in the process of doing now. A bunch of really strong application-level startups made the determination: we can't build just on closed source." "We need to do our own mid-training, our own post-training, our own inference and deployment tricks. All of that means it must be built on top of open source. The closed-source vendors won't give you the access to do this." "Open source became really central in a way that's not always visible because it's deeply embedded in some of these products. Some of the most innovative products and applications now really depend on this very deeply." Matt Bornstein

Matt Bornstein says open source is how AI companies stop being wrappers: "A year ago, a bunch of smaller companies or new application companies were trying to figure out, 'How do I really build an AI without just being a wrapper on top of OpenAI?' The answer to that question turned out to be open source." "This is what Cursor did. This is what Decagon and Harvey are in the process of doing now. A bunch of really strong application-level startups made the determination: we can't build just on closed source." "We need to do our own mid-training, our own post-training, our own inference and deployment tricks. All of that means it must be built on top of open source. The closed-source vendors won't give you the access to do this." "Open source became really central in a way that's not always visible because it's deeply embedded in some of these products. Some of the most innovative products and applications now really depend on this very deeply." Matt Bornstein

a16z

40,604 views • 2 months ago

We are now Cult.Repo. The original creators behind the iconic Open Source documentaries (Vue, React, GraphQL) are back together! Honeypot is now called Cult.Repo and is independently-owned. We are on a mission to document the complete history of every major open source language ever created. With 250K subscribers and over 15M views, we're building on our foundation as creators of the acclaimed Open Source Origin Story documentary format. Coming in 2025: Python Software Foundation, Vite ⚡ documentaries and much more. Our independence is core to our mission. We believe the human stories behind open source deserve authentic, in-depth storytelling. Our new model is powered directly by: - Community support from viewers - Sponsorships from organizations that champion open source This transition represents our commitment to independent, meaningful tech storytelling that honours the collaborative spirit of open source itself.

We are now Cult.Repo. The original creators behind the iconic Open Source documentaries (Vue, React, GraphQL) are back together! Honeypot is now called Cult.Repo and is independently-owned. We are on a mission to document the complete history of every major open source language ever created. With 250K subscribers and over 15M views, we're building on our foundation as creators of the acclaimed Open Source Origin Story documentary format. Coming in 2025: Python Software Foundation, Vite ⚡ documentaries and much more. Our independence is core to our mission. We believe the human stories behind open source deserve authentic, in-depth storytelling. Our new model is powered directly by: - Community support from viewers - Sponsorships from organizations that champion open source This transition represents our commitment to independent, meaningful tech storytelling that honours the collaborative spirit of open source itself.

Cult.Repo

71,439 views • 1 year ago

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.

Base Labs

251,954 views • 2 months ago

In 2022, Berkeley PhD student Woosuk Kwon thought he could make Meta's new open-weight model, OPT-175B, run faster in two weeks. Four years and 3,000 contributors later, that project is vLLM, one of the largest open-source inference engines, running on ~500,000 GPUs at any given time. He and fellow researchers Simon Mo, Kaichao You, and Roger Wang borrowed "virtual memory" from operating systems, handing out memory only as it's needed so each GPU can juggle requests of wildly different sizes. (The "v" in vLLM is for virtual.) Their first user, Berkeley neighbor Arena.ai, cut its GPU needs in half. Today Google, Amazon, and Meta all run vLLM in production. In January, the core maintainers founded Inferact and raised $150M to give vLLM a full-time team. They're hiring across eng, marketing, and devrel in SF and Singapore: Simon Mo Woosuk Kwon Kaichao You Roger Wang

In 2022, Berkeley PhD student Woosuk Kwon thought he could make Meta's new open-weight model, OPT-175B, run faster in two weeks. Four years and 3,000 contributors later, that project is vLLM, one of the largest open-source inference engines, running on ~500,000 GPUs at any given time. He and fellow researchers Simon Mo, Kaichao You, and Roger Wang borrowed "virtual memory" from operating systems, handing out memory only as it's needed so each GPU can juggle requests of wildly different sizes. (The "v" in vLLM is for virtual.) Their first user, Berkeley neighbor Arena.ai, cut its GPU needs in half. Today Google, Amazon, and Meta all run vLLM in production. In January, the core maintainers founded Inferact and raised $150M to give vLLM a full-time team. They're hiring across eng, marketing, and devrel in SF and Singapore: Simon Mo Woosuk Kwon Kaichao You Roger Wang

a16z

71,294 views • 23 days ago

Open research has a purpose: to facilitate open-source AI through an open economy that serves us all 🌎 We’ve converged years of open research into creating the GRID—a comprehensive, production-grade open-source AI stack that enterprises are adopting every day. This is the future of open-source 😁

Open research has a purpose: to facilitate open-source AI through an open economy that serves us all 🌎 We’ve converged years of open research into creating the GRID—a comprehensive, production-grade open-source AI stack that enterprises are adopting every day. This is the future of open-source 😁

Sentient

56,939 views • 11 months ago

Vinod Khosla on open source models on his new interview with WSJ "There’s a lot of misperceptions about open source. Open source is way too expensive to run compared to closed source models. Open source is way more dangerous than closed source. In fact, it’s the one area I’ve recommended to David Sacks and the White House generally to do a treaty with China. You can’t trust China, but it’s one area where constraining open source to not be able to do wild things, like design another bad virus, is in the mutual interest of both the US and China. I think because it’s in both parties’ interest to not have models go wild in what they can do in the hands of individuals. This is not just a nation state thing. An individual at home could design a bad virus and release it. And so, it’s in both parties’ interest to not have that happen. In open source, it’s very easy to do. In closed source, you know, OpenAI, Anthropic, Google know what’s going on, what people are using their models for. So you try and do something bad with their models, they know. Open source, you can run anywhere. So there’s no way to prevent the safety and harnessing to be kept in place. So if you constrain where open source runs, on certified clouds where, you know, they can’t be used for certain bad purposes, that can be done both in the US and China and everywhere else. ---- Full video on "WSJ Events" YouTube channel, (link in comment)

Vinod Khosla on open source models on his new interview with WSJ "There’s a lot of misperceptions about open source. Open source is way too expensive to run compared to closed source models. Open source is way more dangerous than closed source. In fact, it’s the one area I’ve recommended to David Sacks and the White House generally to do a treaty with China. You can’t trust China, but it’s one area where constraining open source to not be able to do wild things, like design another bad virus, is in the mutual interest of both the US and China. I think because it’s in both parties’ interest to not have models go wild in what they can do in the hands of individuals. This is not just a nation state thing. An individual at home could design a bad virus and release it. And so, it’s in both parties’ interest to not have that happen. In open source, it’s very easy to do. In closed source, you know, OpenAI, Anthropic, Google know what’s going on, what people are using their models for. So you try and do something bad with their models, they know. Open source, you can run anywhere. So there’s no way to prevent the safety and harnessing to be kept in place. So if you constrain where open source runs, on certified clouds where, you know, they can’t be used for certain bad purposes, that can be done both in the US and China and everywhere else. ---- Full video on "WSJ Events" YouTube channel, (link in comment)

Rohan Paul

11,730 views • 3 days ago

.Bill Gurley “I don’t think politicians understand how powerful open source is as a competitive weapon. If you have ten companies all doing open source, the minute an idea is shared, it's available to the other ten and they absorb it immediately. I think the odds that the U.S., with two proprietary models, can stay ahead of China, with ten open source ones, is very low.”

.Bill Gurley “I don’t think politicians understand how powerful open source is as a competitive weapon. If you have ten companies all doing open source, the minute an idea is shared, it's available to the other ten and they absorb it immediately. I think the odds that the U.S., with two proprietary models, can stay ahead of China, with ten open source ones, is very low.”

Codie Sanchez

65,669 views • 7 months ago

Unitree G1 Open Source Dataset In order to promote the development of the global embodied AI industry, the Unitree G1 robot operation data set is open sourced, adapted to a variety of open source solutions, and continuously updated: Open source data collection: Open source learning algorithms: Open source datasets and models: #AI #Teleoperation #OpenSourceDataset #Unitree #EmbodiedAI #Humanoid #DataCollection #AGI

Unitree G1 Open Source Dataset In order to promote the development of the global embodied AI industry, the Unitree G1 robot operation data set is open sourced, adapted to a variety of open source solutions, and continuously updated: Open source data collection: Open source learning algorithms: Open source datasets and models: #AI #Teleoperation #OpenSourceDataset #Unitree #EmbodiedAI #Humanoid #DataCollection #AGI

Unitree

148,925 views • 2 years ago