Loading video...

Video Failed to Load

Go Home

Today, weโ€™re introducing new tools and model updates to help you build, deploy, and scale Voice AI applications.๐ŸŽ™๏ธ ๐Ÿ†• Speech Understanding: Turn transcripts into actionable data with speaker identification, custom formatting & translation ๐Ÿ†• LLM Gateway: One API for your voice-to-intelligence pipeline, with GPT, Claude, Gemini & more ๐Ÿ†•...

18,157 views โ€ข 11 months ago โ€ขvia X (Twitter)

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Learn to build conversational AI voice agents in "Building AI Voice Agents for Production", created in collaboration with LiveKit and RealAvatar, and taught by dsa (Co-founder & CEO of LiveKit), Shayne (Developer Advocate, LiveKit), and Nedelina Teneva (Head of AI at RealAvatar, an AI Fund portfolio company). Voice agents combine speech and reasoning capabilities to enable real-time conversations. They're already being used to support customer service, to improve accessibility in healthcare, for entertainment applications, and for talk therapy. In this course, youโ€™ll learn to build voice agents that listen, reason, and respond naturally. Youโ€™ll follow the architecture used to create the "AI Andrew" Avatar, a collaborative project between and RealAvatar that responds to users in what sounds like my voice. Youโ€™ll build a voice agent from scratch and deploy it to the cloud, enabling support for many simultaneous users. What youโ€™ll learn: - Understand the fundamentals of voice agents, including key components like speech-to-text (STT), text-to-speech (TTS), and LLMs, and how latency is introduced at each layer. - Explore voice agent architectures and the trade-offs between modular pipelines and speech-to-speech APIs. - Explore how platforms like LiveKit mitigate latency issues with optimized networking infrastructure and low-latency communication protocols. - Learn how to connect client devices to voice agents using WebRTCโ€”and why it outperforms HTTP and WebSocket for low-latency audio streaming. - Incorporate voice activity detection (VAD), end-of-turn detection, and context management to detect turns, handle interruptions, and manage conversational flow. - Understand the trade-offs between latency, quality, and cost in an example in which you build a voice agent and change its voice. - Equip your agent with metrics to measure latency at each stage of the voice pipeline and learn the key levers you can pull to make your agent faster and more responsive. The voice agents built in this course also incorporate voice technology from , a supporting contributor to the project. By the end of this course, you'll have learned the components of an AI voice agent pipeline, combined them into a system with low-latency communication, and deployed them on cloud infrastructure so it scales to many users. Iโ€™m looking forward to seeing what voice agents you build from this course! Please sign up here:

Andrew Ng

87,868 views โ€ข 1 year ago

New course: MCP: Build Rich-Context AI Apps with Anthropic. Learn to build AI apps that access tools, data, and prompts using the Model Context Protocol in this short course, created in partnership with Anthropic Anthropic and taught by Elie Schoppik Elie Schoppik, its Head of Technical Education. Connecting AI applications to external systems that bring rich context to LLM-based applications has often meant writing custom integrations for each use case. MCP is an open protocol that standardizes how LLMs access tools, data, and prompts from external sources, and simplifies how you provide context to your LLM-based applications. For example, you can provide context via third-party tools that let your LLM make API calls to search the web, access data from local docs, retrieve code from a GitHub repo, and so on. MCP, developed by Anthropic, is based on a client-server architecture that defines the communication details between an MCP client, hosted inside the AI application, and an MCP server that exposes tools, resources, and prompt templates. The server can be a subprocess launched by the client that runs locally or an independent process running remotely. In this hands-on course, you'll learn the core architecture behind MCP. Youโ€™ll create an MCP-compatible chatbot, build and deploy an MCP server, and connect the chatbot to your MCP server and other open-source servers. Hereโ€™s what youโ€™ll do: - Understand why MCP makes AI development less fragmented and standardizes connections between AI applications and external data sources - Learn the core components of the client-server architecture of MCP and the underlying communication mechanism - Build a chatbot with custom tools for searching academic papers, and transform it into an MCP-compatible application - Build a local MCP server that exposes tools, resources, and prompt templates using FastMCP, and test it using MCP Inspector - Create an MCP client inside your chatbot to dynamically connect to your server - Connect your chatbot to reference servers built by Anthropicโ€™s MCP team, such as filesystem, which implements filesystem operations, and fetch, which extracts contents from the web as markdown - Configure Claude Desktop to connect to your server and others, and explore how it abstracts away the low-level logic of MCP clients - Deploy your MCP server remotely and test it with the Inspector or other MCP-compatible applications - Learn about the roadmap for future MCP development, such as multi-agent architecture, MCP registry API, server discovery, authorization, and authentication MCP is an exciting and important technology that lets you build rich-context AI applications that connect to a growing ecosystem of MCP servers, with minimal integration work. Please sign up here!

Andrew Ng

142,256 views โ€ข 1 year ago

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

kwindla

331,698 views โ€ข 29 days ago

99% of AI applications are cool-looking demos. Impressive, but don't get fooled by the hype. It takes a lot to build enterprise-grade products that deliver real value. I have at least three weekly conversations with companies that want to use a Large Language Model with their data. The demand is huge! Here is one idea about what you can do to help. The use cases that most of these companies want to solve are similar: They have an extensive knowledge base and want to build a simple application that uses that information to answer questions. In other words, they need help building Retrieval Augmented Generation (RAG) applications they can use in many different scenarios: 1. To train new employees 2. To help their support team 3. To search old meetings and documents 4. To help with their research However, building these systems is not straightforward. Yes, there's a lot of information online, but there aren't enough people who know how to create solutions that work. Here is the idea: Today, you can build an enterprise-grade RAG application without writing code. A couple of MIT PhDs with 10+ years of experience building AI applications created . It's a no-code platform for building applications using Large Language Models. They are partnering with me on this post. You can use Stack AI to create, test, and deploy an end-to-end production-ready AI system. It's SOC-2, HIPAA, and GDPR compliant and offers SSO, role management, access control, and on-premise deployments. Of course, you can use the platform with any LLM on the market now. It's the whole nine yards for building AI applications. Check them out here: 2023 was about models. 2024 is about the tools using these models to build production-ready applications. That's where I'd start.

Santiago

197,702 views โ€ข 2 years ago

New short course: Serverless Agentic Workflows with Amazon Bedrock. Learn to build and deploy serverless agents in this course created with Amazon Web Services and taught by Mike G Chambers, a Senior Developer Advocate at AWS specializing in GenAI. (Disclosure: I serve on Amazon's board.) Generative AI applications are becoming more complex, sophisticated, and agentic. Agentic applications have workloads that can be hard to predict in advance -- for example, what tools will it decide to call? -- and a serverless architecture helps you efficiently providing on-demand resources. This course teaches you to build and deploy a serverless agentic application. Youโ€™ll learn to create agents with tools, code execution, and guardrails, and build responsible agents for business use cases: - Build a customer service bot for a fictional tea mug business that can answering questions, retrieve information, and process orders. - Connect your customer service agent to a CRM to get customer info and log support tickets in real-time. - Explore how you invoke the agent, and see the trace to review the agentโ€™s thought process and observation loop until it reaches its final output. - Attach a code interpreter to your agent, giving it the ability to perform accurate calculations by writing and running its own Python code. - Implement guardrails to prevent your agent from revealing sensitive information or using inappropriate language. By the end, you will have built a sophisticated AI agent capable of handling real-world customer support scenarios. Please sign up here!

Andrew Ng

81,048 views โ€ข 1 year ago