Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

New video from Shreya Shankar on data processing with LLMs at scale, an underrated topic! Shreya starts with a real use case: public defenders analyzing case files for racial bias (4:08). Hundreds of pages per defendant. Court transcripts, police reports, news articles. Running GPT-5 on everything costs a fortune....

HamelHusain's profile picture

Hamel Husain

48,621 subscribers

35,510 Aufrufe • vor 6 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.
4:50

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.

Santiago

384,495 Aufrufe • vor 3 Jahren

This is a very clever idea to use an LLM with your SQL data. SQL + AI has been tried before, but one of the best parts of this solution is getting the same exact OpenAI's completion API. In other words: You are now talking to an LLM that knows everything about your database and you don't even notice it! I recorded a 2-min video to show you how it works. Thanks to the mindshub team for the collaboration, listening to my feedback, and helping me understand what they built. Go to to try this yourself.
2:07

This is a very clever idea to use an LLM with your SQL data. SQL + AI has been tried before, but one of the best parts of this solution is getting the same exact OpenAI's completion API. In other words: You are now talking to an LLM that knows everything about your database and you don't even notice it! I recorded a 2-min video to show you how it works. Thanks to the mindshub team for the collaboration, listening to my feedback, and helping me understand what they built. Go to to try this yourself.

Santiago

230,457 Aufrufe • vor 1 Jahr

Your agents can't keep up with real-time data. Especially when it's scattered across dozens of sources. Most teams waste weeks building custom connectors for every database, API, and data warehouse. Then they build ETL pipelines to sync everything. By the time your agent retrieves the data, it's already outdated. Picture this: Your Postgres database updated 5 minutes ago. Your MongoDB collection changed 2 minutes ago. Your agent is still pulling from yesterday's snapshot. This is why most production RAG systems fail. There's a better approach: MindsDB is an open-source AI platform with a federated data engine that lets you query multiple data sources in real-time using SQL - without moving any data. Here's what makes it different: ↳ Your data stays in place. No ETL pipelines or data duplication ↳ Query Postgres, MongoDB, REST APIs, and more using consistent SQL ↳ JOIN across different sources in real-time with a unified interface ↳ Works with both structured and un-structured data And here's the best part: You don't even need to write SQL. Just describe what you want in plain English, and MindsDB converts it to SQL automatically. The system does all the heavy lifting. The breakthrough for AI agents is simple: When data updates at the source, your agent gets fresh results immediately. No sync delays. No stale embeddings. No custom code for each integration. You can literally write a SQL query that joins a Postgres table with a MongoDB collection and gets live results. This is what production AI applications need but rarely get. In this video, I give you a complete walkthrough of what we just discussed and how to actually do it. Make sure you watch this till the end. I've shared the link to MindsDB's GitHub repo in the next tweet!
9:28

Your agents can't keep up with real-time data. Especially when it's scattered across dozens of sources. Most teams waste weeks building custom connectors for every database, API, and data warehouse. Then they build ETL pipelines to sync everything. By the time your agent retrieves the data, it's already outdated. Picture this: Your Postgres database updated 5 minutes ago. Your MongoDB collection changed 2 minutes ago. Your agent is still pulling from yesterday's snapshot. This is why most production RAG systems fail. There's a better approach: MindsDB is an open-source AI platform with a federated data engine that lets you query multiple data sources in real-time using SQL - without moving any data. Here's what makes it different: ↳ Your data stays in place. No ETL pipelines or data duplication ↳ Query Postgres, MongoDB, REST APIs, and more using consistent SQL ↳ JOIN across different sources in real-time with a unified interface ↳ Works with both structured and un-structured data And here's the best part: You don't even need to write SQL. Just describe what you want in plain English, and MindsDB converts it to SQL automatically. The system does all the heavy lifting. The breakthrough for AI agents is simple: When data updates at the source, your agent gets fresh results immediately. No sync delays. No stale embeddings. No custom code for each integration. You can literally write a SQL query that joins a Postgres table with a MongoDB collection and gets live results. This is what production AI applications need but rarely get. In this video, I give you a complete walkthrough of what we just discussed and how to actually do it. Make sure you watch this till the end. I've shared the link to MindsDB's GitHub repo in the next tweet!

Akshay 🚀

65,672 Aufrufe • vor 8 Monaten

Google open-sourced MCP Toolbox for Databases. I gave it access to everything else. For context, Google's MCP Toolbox for Databases is an open-source server that lets AI agents securely query structured databases like PostgreSQL and MySQL through the MCP protocol However, most enterprise knowledge doesn't actually live in databases. It's scattered across emails, Slack threads, GitHub repos, Salesforce records, customer reviews, and internal docs. So Agents can't see any of it, which means they're working with a fraction of the context they need. I fixed that using MindsDB. It acts as a universal SQL layer that sits on top of all your data sources: structured, semi-structured, and unstructured. This means you can query Salesforce, Gmail, GitHub, S3 files, Jira, and 200+ more sources using SQL syntax. The clever part is how it connects to the MCP Toolbox. MindsDB exposes everything through MySQL, so from the Agent's perspective, it's just running SQL and getting context back. It doesn't know or care that the data came from five different sources behind the scenes. This setup unlocks some powerful capabilities: → One SQL interface for dozens of enterprise sources → Cross-datasource joins (combine GitHub and CRM data in a single query) → Built-in ML capabilities for working with unstructured data → Simple MCP tools that now have massively expanded reach In the video below, the Agent queries GitHub data and a customer review database in one SQL query. So what used to require ETL pipelines and weeks of engineering effort now happens instantly. At the end of the day, AI agents are only as useful as the data they can access. This gives them a lot more to work with. I have shared the GitHub repo in the replies, where you can find more details about this.
0:22

Google open-sourced MCP Toolbox for Databases. I gave it access to everything else. For context, Google's MCP Toolbox for Databases is an open-source server that lets AI agents securely query structured databases like PostgreSQL and MySQL through the MCP protocol However, most enterprise knowledge doesn't actually live in databases. It's scattered across emails, Slack threads, GitHub repos, Salesforce records, customer reviews, and internal docs. So Agents can't see any of it, which means they're working with a fraction of the context they need. I fixed that using MindsDB. It acts as a universal SQL layer that sits on top of all your data sources: structured, semi-structured, and unstructured. This means you can query Salesforce, Gmail, GitHub, S3 files, Jira, and 200+ more sources using SQL syntax. The clever part is how it connects to the MCP Toolbox. MindsDB exposes everything through MySQL, so from the Agent's perspective, it's just running SQL and getting context back. It doesn't know or care that the data came from five different sources behind the scenes. This setup unlocks some powerful capabilities: → One SQL interface for dozens of enterprise sources → Cross-datasource joins (combine GitHub and CRM data in a single query) → Built-in ML capabilities for working with unstructured data → Simple MCP tools that now have massively expanded reach In the video below, the Agent queries GitHub data and a customer review database in one SQL query. So what used to require ETL pipelines and weeks of engineering effort now happens instantly. At the end of the day, AI agents are only as useful as the data they can access. This gives them a lot more to work with. I have shared the GitHub repo in the replies, where you can find more details about this.

Akshay 🚀

39,331 Aufrufe • vor 4 Monaten

This is, by far, one of the best uses of modern AI. If you don't use embeddings when querying your database, you are definitely leaving a lot on the table. In this video, I'll show you how to run semantic searches using OpenAI and PostgreSQL. It's all thanks to Pgai, an open-source PostgreSQL extension: Here's what will happen: 1. We'll create a simple table with news articles 2. We'll generate embeddings for those articles 3. We'll run queries on top of those embeddings For this video, I generated the embeddings using a simple query, but pgai Vectorizer would do the same automatically as new information makes it into the database. This is awesome! If you have a PostgreSQL database with data you are searching over, you should start experimenting with semantic searches immediately. For most use cases, a combination of full-text search + semantic search is the best approach. If you don't have a PostgreSQL database around, you can try free for 30 days using Timescale: Thanks to the Timescale (now TigerData) team for partnering with me on this post!
19:19

This is, by far, one of the best uses of modern AI. If you don't use embeddings when querying your database, you are definitely leaving a lot on the table. In this video, I'll show you how to run semantic searches using OpenAI and PostgreSQL. It's all thanks to Pgai, an open-source PostgreSQL extension: Here's what will happen: 1. We'll create a simple table with news articles 2. We'll generate embeddings for those articles 3. We'll run queries on top of those embeddings For this video, I generated the embeddings using a simple query, but pgai Vectorizer would do the same automatically as new information makes it into the database. This is awesome! If you have a PostgreSQL database with data you are searching over, you should start experimenting with semantic searches immediately. For most use cases, a combination of full-text search + semantic search is the best approach. If you don't have a PostgreSQL database around, you can try free for 30 days using Timescale: Thanks to the Timescale (now TigerData) team for partnering with me on this post!

Santiago

109,517 Aufrufe • vor 1 Jahr

Everyone is focused on tracking the ways LLMs are getting better. And they are. But we know there are still things that LLMs can’t do well—the tasks where you can feel the architecture fighting the problem. So I was excited to chat with Eve Bodnia (@eve_bodnia), who is developing an alternative AI model to LLMs, on Every 📧's AI & I. Eve's argument: energy-based models (EBMs), which map possible outcomes onto a mathematical landscape, will lead to the next AI phase shift. We get into: - How energy-based models work. Likely outcomes sit in valleys, and unlikely ones sit on peaks. Whereas LLMs process one token at a time, an EBM scans the full terrain to find the lowest point, or the most probable answer. - Language-based versus data-native models. LLMs are language-dependent even when the problem has nothing to do with language. "If your data is numbers, relationships, and functions, and you try to map those rules into words and then search for the next word, you're losing a lot of information," Bodnia says. EBMs work directly with the underlying data structure, including numbers and spatial coordinates. - Sequential versus panoramic reasoning. An LLM is like driving through San Francisco without a map. Each turn constrains the next, and if you go down the wrong street, you can't reverse course. An EBM has the bird's-eye view—it can evaluate multiple routes at once and course-correct before hitting a dead end. - The LLM plateau no one wants to talk about. LLMs are getting incrementally better, step-change improvements aren’t coming, Eve argues. To achieve that, we need new solutions that compensate for what LLMs are inherently bad at, like non-language reasoning, verification, and real-time data analysis. This is a must-watch for anyone who's curious what might come after the LLM. Watch below! Timestamps: Introduction: 00:00:51 Why correctness and verifiability matter in AI: 00:02:09 What an energy-based model is: 00:09:33 How EBMs construct energy landscapes to understand data: 00:14:21 Why modeling intelligence through language alone is a flawed approach: 00:19:00 What it means for a model to "understand" data: 00:26:54 How EBMs solve the vibe coding problem and enable formally verified code: 00:37:21 Why LLM progress is plateauing: 00:43:21 Mission-critical industries haven't adopted LLMs, and why EBMs can fill that gap: 00:49:54
53:37

Everyone is focused on tracking the ways LLMs are getting better. And they are. But we know there are still things that LLMs can’t do well—the tasks where you can feel the architecture fighting the problem. So I was excited to chat with Eve Bodnia (@eve_bodnia), who is developing an alternative AI model to LLMs, on Every 📧's AI & I. Eve's argument: energy-based models (EBMs), which map possible outcomes onto a mathematical landscape, will lead to the next AI phase shift. We get into: - How energy-based models work. Likely outcomes sit in valleys, and unlikely ones sit on peaks. Whereas LLMs process one token at a time, an EBM scans the full terrain to find the lowest point, or the most probable answer. - Language-based versus data-native models. LLMs are language-dependent even when the problem has nothing to do with language. "If your data is numbers, relationships, and functions, and you try to map those rules into words and then search for the next word, you're losing a lot of information," Bodnia says. EBMs work directly with the underlying data structure, including numbers and spatial coordinates. - Sequential versus panoramic reasoning. An LLM is like driving through San Francisco without a map. Each turn constrains the next, and if you go down the wrong street, you can't reverse course. An EBM has the bird's-eye view—it can evaluate multiple routes at once and course-correct before hitting a dead end. - The LLM plateau no one wants to talk about. LLMs are getting incrementally better, step-change improvements aren’t coming, Eve argues. To achieve that, we need new solutions that compensate for what LLMs are inherently bad at, like non-language reasoning, verification, and real-time data analysis. This is a must-watch for anyone who's curious what might come after the LLM. Watch below! Timestamps: Introduction: 00:00:51 Why correctness and verifiability matter in AI: 00:02:09 What an energy-based model is: 00:09:33 How EBMs construct energy landscapes to understand data: 00:14:21 Why modeling intelligence through language alone is a flawed approach: 00:19:00 What it means for a model to "understand" data: 00:26:54 How EBMs solve the vibe coding problem and enable formally verified code: 00:37:21 Why LLM progress is plateauing: 00:43:21 Mission-critical industries haven't adopted LLMs, and why EBMs can fill that gap: 00:49:54

Dan Shipper 📧

26,805 Aufrufe • vor 2 Monaten

Traditional data pipelines don't work for RAG applications. There are 3 issues with them: ​ 1. Traditional data engineering solutions are optimized to handle structured data. RAG applications rely primarily on unstructured data. ​ 2. The connector ecosystem to load data from unstructured data sources is very immature. ​ 3. Traditional solutions do not offer any way to transform unstructured data into an optimized vector search index. ​ The goal of a RAG Pipeline is to solve these problems. ​ The number one objective is to create a reliable vector search index using factual knowledge and relevant context. This sounds easy, but it's one of the biggest challenges we face when building RAG applications. ​ At a high level, there are four different stages in the architecture of a RAG pipeline: ​ 1. Ingestion: Here is where the pipeline loads the information from the data source. ​ 2. Extraction: Where the pipeline processes the input data and decides how to retrieve the text contained inside them. ​ 3. Transform: Where the pipeline chunks the data and generates document embeddings. ​ 4. Load: Where the pipeline creates a search index in a vector database and loads the document embeddings. ​ There are different rabbit holes at each one of these stages. Here are three of them: ​ 1. Ingesting data once is simple. The hard part is refreshing the vector database whenever the original data source changes. ​ 2. Extracting the content of a plain text document is simple. The hard part is to extract content from complex documents containing tables, images, or cross-references. ​ 3. A simple continual chunking strategy with an overlap is simple. The hard part is to find the optimal strategy for your specific knowledge base and the way you are planning to query it. ​ In the attached video, I'll show you how you can build an enterprise-grade RAG Pipeline that solves every one of the above problems. ​ I'll use Vectorize. They partnered with me on this post. You can use them to build RAG pipelines optimized for accurate context retrieval. ​ ​ If you have a few documents lying around, set up a free account and give it a try.
3:33

Traditional data pipelines don't work for RAG applications. There are 3 issues with them: ​ 1. Traditional data engineering solutions are optimized to handle structured data. RAG applications rely primarily on unstructured data. ​ 2. The connector ecosystem to load data from unstructured data sources is very immature. ​ 3. Traditional solutions do not offer any way to transform unstructured data into an optimized vector search index. ​ The goal of a RAG Pipeline is to solve these problems. ​ The number one objective is to create a reliable vector search index using factual knowledge and relevant context. This sounds easy, but it's one of the biggest challenges we face when building RAG applications. ​ At a high level, there are four different stages in the architecture of a RAG pipeline: ​ 1. Ingestion: Here is where the pipeline loads the information from the data source. ​ 2. Extraction: Where the pipeline processes the input data and decides how to retrieve the text contained inside them. ​ 3. Transform: Where the pipeline chunks the data and generates document embeddings. ​ 4. Load: Where the pipeline creates a search index in a vector database and loads the document embeddings. ​ There are different rabbit holes at each one of these stages. Here are three of them: ​ 1. Ingesting data once is simple. The hard part is refreshing the vector database whenever the original data source changes. ​ 2. Extracting the content of a plain text document is simple. The hard part is to extract content from complex documents containing tables, images, or cross-references. ​ 3. A simple continual chunking strategy with an overlap is simple. The hard part is to find the optimal strategy for your specific knowledge base and the way you are planning to query it. ​ In the attached video, I'll show you how you can build an enterprise-grade RAG Pipeline that solves every one of the above problems. ​ I'll use Vectorize. They partnered with me on this post. You can use them to build RAG pipelines optimized for accurate context retrieval. ​ ​ If you have a few documents lying around, set up a free account and give it a try.

Santiago

40,441 Aufrufe • vor 1 Jahr

Andrej Karpathy beautifully explains the fundamental difference of learning between a human and an LLM. > “The book I’m reading is a set of prompts for me to do synthetic data generation. It's by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs; they don't really do that.” many of the best minds in this field think LLMs are not really learning anything, and therefore are incapable of surpassing human-level intelligence.
7:01

Andrej Karpathy beautifully explains the fundamental difference of learning between a human and an LLM. > “The book I’m reading is a set of prompts for me to do synthetic data generation. It's by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs; they don't really do that.” many of the best minds in this field think LLMs are not really learning anything, and therefore are incapable of surpassing human-level intelligence.

ℏεsam

895,122 Aufrufe • vor 8 Monaten

It is very easy to make mistakes when creating evals for your AI product. Shreya Shankar and I run through the most common mistakes in this talk (with memes 🌶️!) . Chapter summaries below: 00:51 Foundation model benchmarks are not the same as your application evals 03:00 Generic Evals Are Useless 04:00 Do not outsource labeling & prompting to non domain experts 09:28 You should make your own data annotation app 12:40 Your LLM prompts should be specific and grounded in error analysis 15:25 Use binary labels 18:57 Look at your data 23:41 Be careful of overfitting to test data 25:40 Do online tests Links more resources in the reply
28:15

It is very easy to make mistakes when creating evals for your AI product. Shreya Shankar and I run through the most common mistakes in this talk (with memes 🌶️!) . Chapter summaries below: 00:51 Foundation model benchmarks are not the same as your application evals 03:00 Generic Evals Are Useless 04:00 Do not outsource labeling & prompting to non domain experts 09:28 You should make your own data annotation app 12:40 Your LLM prompts should be specific and grounded in error analysis 15:25 Use binary labels 18:57 Look at your data 23:41 Be careful of overfitting to test data 25:40 Do online tests Links more resources in the reply

Hamel Husain

46,085 Aufrufe • vor 1 Jahr

$BTC Statistical Study using Claude - A Beginner's Workflow Here's an example of a z-score study on $BTC - still tinkering so don't take this as overtly useful information but the creation of a dashboard for the visualization of statistical data is phenomenal. my current workflow: > import $BTC time data - .csv file (can get this information from multiple venues - Binance is where I got mine) > creating bins - if you have a larger data set - you can use recent data to prevent overt bias in the long direction or filter consolidation and trending regime data into separate bins for statistical analysis - however, you will have to define thresholds and determine what that entails. > defining metrics in Claude that you want to use for statistical analysis e.g. for z-score what is it based on and what type of calculation? make sure you understand the calculations being performed for any metrics that you are doing a study for and modify them accordingly. > prompting Claude to do a statistical analysis with specific instructions and then tell it to create visualization for this. I've been messing around with this and I'm seriously impressed by the output.
0:33

$BTC Statistical Study using Claude - A Beginner's Workflow Here's an example of a z-score study on $BTC - still tinkering so don't take this as overtly useful information but the creation of a dashboard for the visualization of statistical data is phenomenal. my current workflow: > import $BTC time data - .csv file (can get this information from multiple venues - Binance is where I got mine) > creating bins - if you have a larger data set - you can use recent data to prevent overt bias in the long direction or filter consolidation and trending regime data into separate bins for statistical analysis - however, you will have to define thresholds and determine what that entails. > defining metrics in Claude that you want to use for statistical analysis e.g. for z-score what is it based on and what type of calculation? make sure you understand the calculations being performed for any metrics that you are doing a study for and modify them accordingly. > prompting Claude to do a statistical analysis with specific instructions and then tell it to create visualization for this. I've been messing around with this and I'm seriously impressed by the output.

Stoic

19,757 Aufrufe • vor 1 Jahr

This workflow combining Loom’s AI features + a custom ChatGPT GPT is saving me hours. Instead of creating onboarding Docs for new team members, I film a Loom → generate SOP → train a GPT to answer questions Game changer for businesses to delegate faster. Here's how to do it: First, I record a video of whatever task I want to delegate to the new team member with Loom. The more in-depth, the better, but I just used a 7-minute video. Then, I use Loom's new AI 'Write a document' feature. Upgrading to Loom AI from the standard Loom plan cost me $2. Loom AI can generate an entire SOP, PR description, step-by-step guide, QA, and more from a simple Loom video in <5 seconds. In the past, I’ve spent 2 hours+ hand-writing each one of these docs to onboard new team members, so Loom AI is already a massive timesaver. But it gets even better! Next, we can take that data from the SOP document, and we use it as 'Knowledge' to train a Custom GPT that can answer the new team members' questions. The more SOP docs/Knowledge you feed the GPT, the better. But one is fine if that's all you have because the GPT will pull any unknown answers from the web or its training data. Here are the prompt Instructions you want to put into the Custom GPT (copy and paste this): You are an expert CEO, specialized in onboarding and training new team members. Using the Knowledge provided, you will help new team members with any questions or stipulations they may have about their new role. Stick as true to the data provided as possible, but if= they ask any questions that the Knowledge base does not have a specific answer for, you are permitted to use your pre-trained data and/or web browsing capabilities. That's it! It can't replace you entirely, but it'll save you 90% of the time you would've wasted on writing an SOP doc and answering questions. Simple AI workflows here and there really add up. There's also a workflow to help with the job screening process, but I'll save that for another day :^)
0:54

This workflow combining Loom’s AI features + a custom ChatGPT GPT is saving me hours. Instead of creating onboarding Docs for new team members, I film a Loom → generate SOP → train a GPT to answer questions Game changer for businesses to delegate faster. Here's how to do it: First, I record a video of whatever task I want to delegate to the new team member with Loom. The more in-depth, the better, but I just used a 7-minute video. Then, I use Loom's new AI 'Write a document' feature. Upgrading to Loom AI from the standard Loom plan cost me $2. Loom AI can generate an entire SOP, PR description, step-by-step guide, QA, and more from a simple Loom video in <5 seconds. In the past, I’ve spent 2 hours+ hand-writing each one of these docs to onboard new team members, so Loom AI is already a massive timesaver. But it gets even better! Next, we can take that data from the SOP document, and we use it as 'Knowledge' to train a Custom GPT that can answer the new team members' questions. The more SOP docs/Knowledge you feed the GPT, the better. But one is fine if that's all you have because the GPT will pull any unknown answers from the web or its training data. Here are the prompt Instructions you want to put into the Custom GPT (copy and paste this): You are an expert CEO, specialized in onboarding and training new team members. Using the Knowledge provided, you will help new team members with any questions or stipulations they may have about their new role. Stick as true to the data provided as possible, but if= they ask any questions that the Knowledge base does not have a specific answer for, you are permitted to use your pre-trained data and/or web browsing capabilities. That's it! It can't replace you entirely, but it'll save you 90% of the time you would've wasted on writing an SOP doc and answering questions. Simple AI workflows here and there really add up. There's also a workflow to help with the job screening process, but I'll save that for another day :^)

Rowan Cheung

129,424 Aufrufe • vor 2 Jahren

Major program launch: Data Analytics Professional Certificate! This large, five-course sequence takes you all the way to being job-ready as a data analyst, and shows how to use Generative AI as a thought partner to enhance your work in this role. Offered by on Coursera, this is taught by Sean Barnes, Ph.D., a Data Science & Engineering Leader at Netflix. Analyzing data remains one of the most important skills in where the world is going with AI. This comprehensive certificate takes you all the way to being job-ready. Each course comes with practical projects demonstrated in real-world contexts, such as analyzing sales data for a Korean bakery, video game sales trends across different regions, or identifying factors impacting customer retention for a communications company. You'll also work on estimating fire distribution for forest fire prevention, analyzing how a diamond's properties affect its market value, and developing predictive models for retail sales analysis, carbon emissions, and coral reef conservation. Here's some of what you'll learn: - How to define data and categorize it into its many types such as discrete & continuous numerical, structured & unstructured, time series, categorical, and know what insights can be derived from the different types of data categories. - How to differentiate between data-related job roles and their responsibilities, and how data flows through an organization from the moment of capture to decision-making. - How to perform data processing functions and apply conditional formatting in spreadsheets to extract business value from your data using statistical calculations and best practices for visualizing and interpreting data. - How to use LLMs for stakeholder analysis, data exploration, and data visualization. - Best practices for using LLMs for as a thought partner to data analysis work By the end of this professional certificate program, you will have learned core statistical concepts, analysis techniques, and visualization methodologies that will serve as the foundation for working as a data analyst. The world needs more data analysts, especially ones who know how to use modern generative AI. With data science roles projected to grow 36% by 2033, the skills taught in this program create new professional opportunities in data. Sign up here!
2:24

Major program launch: Data Analytics Professional Certificate! This large, five-course sequence takes you all the way to being job-ready as a data analyst, and shows how to use Generative AI as a thought partner to enhance your work in this role. Offered by on Coursera, this is taught by Sean Barnes, Ph.D., a Data Science & Engineering Leader at Netflix. Analyzing data remains one of the most important skills in where the world is going with AI. This comprehensive certificate takes you all the way to being job-ready. Each course comes with practical projects demonstrated in real-world contexts, such as analyzing sales data for a Korean bakery, video game sales trends across different regions, or identifying factors impacting customer retention for a communications company. You'll also work on estimating fire distribution for forest fire prevention, analyzing how a diamond's properties affect its market value, and developing predictive models for retail sales analysis, carbon emissions, and coral reef conservation. Here's some of what you'll learn: - How to define data and categorize it into its many types such as discrete & continuous numerical, structured & unstructured, time series, categorical, and know what insights can be derived from the different types of data categories. - How to differentiate between data-related job roles and their responsibilities, and how data flows through an organization from the moment of capture to decision-making. - How to perform data processing functions and apply conditional formatting in spreadsheets to extract business value from your data using statistical calculations and best practices for visualizing and interpreting data. - How to use LLMs for stakeholder analysis, data exploration, and data visualization. - Best practices for using LLMs for as a thought partner to data analysis work By the end of this professional certificate program, you will have learned core statistical concepts, analysis techniques, and visualization methodologies that will serve as the foundation for working as a data analyst. The world needs more data analysts, especially ones who know how to use modern generative AI. With data science roles projected to grow 36% by 2033, the skills taught in this program create new professional opportunities in data. Sign up here!

Andrew Ng

84,686 Aufrufe • vor 1 Jahr

Sam Altman in a new interview says Orbital Data Centers are not going to matter at scale this decade: “I honestly think the idea, with the current landscape of putting data centers in space is ridiculous. If you do the very rough math of launch costs relative to the cost of power on earth and to say nothing how you are going to fix a broken GPU in space— they do break a lot unfortunately— we are not there yet. There will be a time, space is great for a lot of things, Orbital Data Centers are not something that is going to matter at scale this decade.” This clip isn’t going to age well for Sam.
0:39

Sam Altman in a new interview says Orbital Data Centers are not going to matter at scale this decade: “I honestly think the idea, with the current landscape of putting data centers in space is ridiculous. If you do the very rough math of launch costs relative to the cost of power on earth and to say nothing how you are going to fix a broken GPU in space— they do break a lot unfortunately— we are not there yet. There will be a time, space is great for a lot of things, Orbital Data Centers are not something that is going to matter at scale this decade.” This clip isn’t going to age well for Sam.

Nic Cruz Patane

72,344 Aufrufe • vor 4 Monaten

If you use LLM-as-judge, this one is for you. (bookmark it) Most teams validate their agent's outputs by calling a frontier model as the judge. It works, until it doesn't. Three problems stack up fast: → Cost: you're hitting a frontier API on every turn, every tool call, every response. In production that burns millions. → Latency: bigger models, remote calls, slow reasoning on every check. → Blind spots: frontier models don't actually know your domain. In finance, insurance, or healthcare, they miss the keywords and principles your work depends on. So I walk through a different approach: train your own small LLM judge. Instead of a giant model, you start with a small one and let the system generate the training data for you. It decomposes your domain, samples synthetic examples, runs them through a debate arena where judges reach consensus, then trains on the refined set. The result is a judge that's cheaper, faster, and more accurate on your data than Gemini, Claude, or GPT, with an OpenAI-compatible endpoint you can even deploy on-prem. I show the whole thing end to end, using a Claude Code plugin and a web interface, with a real insurance RAG grounding evaluator as the example. You can get the plugin here: Here's the full breakdown: 00:00 - Intro 00:12 - Three problems with using frontier LLMs as judges 01:05 - A different approach: train your own small judge 01:31 - How it works (synthetic data and a debate arena) 02:50 - Installing the Claude Code plugin 04:03 - Defining your task with /eval 04:34 - Example: an insurance RAG grounding evaluator 05:51 - Kicking it off and giving early feedback 06:26 - Choosing labels, domain, and strictness 08:30 - The web interface and dashboard 09:52 - Bringing your own example data (optional) 10:26 - The finished model: endpoint, accuracy, and speed 11:16 - Control, on-prem deployment, and interpretability 11:57 - Benchmarks vs frontier models and the GitHub repo 12:30 - Outro I worked with the Plurai team on this. Thanks for sponsoring the video.
12:37

If you use LLM-as-judge, this one is for you. (bookmark it) Most teams validate their agent's outputs by calling a frontier model as the judge. It works, until it doesn't. Three problems stack up fast: → Cost: you're hitting a frontier API on every turn, every tool call, every response. In production that burns millions. → Latency: bigger models, remote calls, slow reasoning on every check. → Blind spots: frontier models don't actually know your domain. In finance, insurance, or healthcare, they miss the keywords and principles your work depends on. So I walk through a different approach: train your own small LLM judge. Instead of a giant model, you start with a small one and let the system generate the training data for you. It decomposes your domain, samples synthetic examples, runs them through a debate arena where judges reach consensus, then trains on the refined set. The result is a judge that's cheaper, faster, and more accurate on your data than Gemini, Claude, or GPT, with an OpenAI-compatible endpoint you can even deploy on-prem. I show the whole thing end to end, using a Claude Code plugin and a web interface, with a real insurance RAG grounding evaluator as the example. You can get the plugin here: Here's the full breakdown: 00:00 - Intro 00:12 - Three problems with using frontier LLMs as judges 01:05 - A different approach: train your own small judge 01:31 - How it works (synthetic data and a debate arena) 02:50 - Installing the Claude Code plugin 04:03 - Defining your task with /eval 04:34 - Example: an insurance RAG grounding evaluator 05:51 - Kicking it off and giving early feedback 06:26 - Choosing labels, domain, and strictness 08:30 - The web interface and dashboard 09:52 - Bringing your own example data (optional) 10:26 - The finished model: endpoint, accuracy, and speed 11:16 - Control, on-prem deployment, and interpretability 11:57 - Benchmarks vs frontier models and the GitHub repo 12:30 - Outro I worked with the Plurai team on this. Thanks for sponsoring the video.

Akshay 🚀

29,586 Aufrufe • vor 11 Tagen

The most interesting part for me is where Andrej Karpathy describes why LLMs aren't able to learn like humans. As you would expect, he comes up with a wonderfully evocative phrase to describe RL: “sucking supervision bits through a straw.” A single end reward gets broadcast across every token in a successful trajectory, upweighting even wrong or irrelevant turns that lead to the right answer. > “Humans don't use reinforcement learning, as I've said before. I think they do something different. Reinforcement learning is a lot worse than the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before is much worse.” So what do humans do instead? > “The book I’m reading is a set of prompts for me to do synthetic data generation. It's by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs; they don't really do that.” > “I'd love to see during pretraining some kind of a stage where the model thinks through the material and tries to reconcile it with what it already knows. There's no equivalent of any of this. This is all research.” Why can’t we just add this training to LLMs today? > “There are very subtle, hard to understand reasons why it's not trivial. If I just give synthetic generation of the model thinking about a book, you look at it and you're like, 'This looks great. Why can't I train on it?' You could try, but the model will actually get much worse if you continue trying.” > “Say we have a chapter of a book and I ask an LLM to think about it. It will give you something that looks very reasonable. But if I ask it 10 times, you'll notice that all of them are the same.” > “You're not getting the richness and the diversity and the entropy from these models as you would get from humans. How do you get synthetic data generation to work despite the collapse and while maintaining the entropy? It is a research problem.” How do humans get around model collapse? > “These analogies are surprisingly good. Humans collapse during the course of their lives. Children haven't overfit yet. They will say stuff that will shock you. Because they're not yet collapsed. But we [adults] are collapsed. We end up revisiting the same thoughts, we end up saying more and more of the same stuff, the learning rates go down, the collapse continues to get worse, and then everything deteriorates.” In fact, there’s an interesting paper arguing that dreaming evolved to assist generalization, and resist overfitting to daily learning - look up The Overfitted Brain by Erik Hoel. I asked Karpathy: Isn’t it interesting that humans learn best at a part of their lives (childhood) whose actual details they completely forget, adults still learn really well but have terrible memory about the particulars of the things they read or watch, and LLMs can memorize arbitrary details about text that no human could but are currently pretty bad at generalization? > “[Fallible human memory] is a feature, not a bug, because it forces you to only learn the generalizable components. LLMs are distracted by all the memory that they have of the pre-trained documents. That's why when I talk about the cognitive core, I actually want to remove the memory. I'd love to have them have less memory so that they have to look things up and they only maintain the algorithms for thought, and the idea of an experiment, and all this cognitive glue for acting.”
7:01

The most interesting part for me is where Andrej Karpathy describes why LLMs aren't able to learn like humans. As you would expect, he comes up with a wonderfully evocative phrase to describe RL: “sucking supervision bits through a straw.” A single end reward gets broadcast across every token in a successful trajectory, upweighting even wrong or irrelevant turns that lead to the right answer. > “Humans don't use reinforcement learning, as I've said before. I think they do something different. Reinforcement learning is a lot worse than the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before is much worse.” So what do humans do instead? > “The book I’m reading is a set of prompts for me to do synthetic data generation. It's by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs; they don't really do that.” > “I'd love to see during pretraining some kind of a stage where the model thinks through the material and tries to reconcile it with what it already knows. There's no equivalent of any of this. This is all research.” Why can’t we just add this training to LLMs today? > “There are very subtle, hard to understand reasons why it's not trivial. If I just give synthetic generation of the model thinking about a book, you look at it and you're like, 'This looks great. Why can't I train on it?' You could try, but the model will actually get much worse if you continue trying.” > “Say we have a chapter of a book and I ask an LLM to think about it. It will give you something that looks very reasonable. But if I ask it 10 times, you'll notice that all of them are the same.” > “You're not getting the richness and the diversity and the entropy from these models as you would get from humans. How do you get synthetic data generation to work despite the collapse and while maintaining the entropy? It is a research problem.” How do humans get around model collapse? > “These analogies are surprisingly good. Humans collapse during the course of their lives. Children haven't overfit yet. They will say stuff that will shock you. Because they're not yet collapsed. But we [adults] are collapsed. We end up revisiting the same thoughts, we end up saying more and more of the same stuff, the learning rates go down, the collapse continues to get worse, and then everything deteriorates.” In fact, there’s an interesting paper arguing that dreaming evolved to assist generalization, and resist overfitting to daily learning - look up The Overfitted Brain by Erik Hoel. I asked Karpathy: Isn’t it interesting that humans learn best at a part of their lives (childhood) whose actual details they completely forget, adults still learn really well but have terrible memory about the particulars of the things they read or watch, and LLMs can memorize arbitrary details about text that no human could but are currently pretty bad at generalization? > “[Fallible human memory] is a feature, not a bug, because it forces you to only learn the generalizable components. LLMs are distracted by all the memory that they have of the pre-trained documents. That's why when I talk about the cognitive core, I actually want to remove the memory. I'd love to have them have less memory so that they have to look things up and they only maintain the algorithms for thought, and the idea of an experiment, and all this cognitive glue for acting.”

Dwarkesh Patel

1,050,747 Aufrufe • vor 8 Monaten

RLM is the most import foundation of my Pi Harness (other than Pi of course). It's seeded with late interaction retrieval results (thanks to @lightonai for pylate). The Agent initiates it with query then.. 𝐒𝐞𝐭𝐮𝐩 A python REPL is created and seeded with: 1. Late interaction search to pre-filter. Instead of doing top 3/5/10, it's top hundreds of documents. This is set into a `context` variable. 2. Python functions are loaded in to do more searches if `context` variable isn't enough. And to make llm calls with cheaper models in parallel batches. 𝐈𝐭𝐞𝐫𝐚𝐭𝐢𝐨𝐧 𝐋𝐨𝐨𝐩 From there, an LLM iterates in the REPL based on the query. It's just like exploring in a jupyter notebook. The LLM writes prose (like a markdown cell) and code to be run in the REPL each turn. This allows the LLM to sort, filter, and synthesize information. It can fan out and ask smaller models to summarize, combine, contrast, or do anything else to documents to help it understand the data. After several turns the LLM reponds with the final answer. Either because it found the answer, or hit the budget limit. Context as a Python variable, LLM as the programmer, REPL as the runtime. 𝐖𝐡𝐲 𝐃𝐨𝐞𝐬 𝐓𝐡𝐢𝐬 𝐖𝐨𝐫𝐤 1. Richer Shell. Agents (and subagents) work by intermixing code and prose/thinking. But they use static scripts or bash that run and exit and start over each tool call. That's not ideal for exploration and synthesis of data. For that, state is useful to continue building and exploring the data as you learn more. There's a reason jupyter notebooks have been popular with data scientists. 2. Keeps main agent context clean. The better context you have the better the agent will perform (duh!). This means three thing: better human input, less missing search results, and less incorrect search results. Letting the agent iterate allows it to synthesize just what is needed and nothing else. All bad paths or peeks at something that turns out to be irrelevant stays out of main agent context. 3. Stack the good ideas! People often compare late interaction search vs RLM. Or static vs dynamic languages. Or agentic search vs semantic search. But...You can just use them all together for what they're each good at. Use them all for the area they're really great for. Read the full post which has more detail about how and why.
3:43

RLM is the most import foundation of my Pi Harness (other than Pi of course). It's seeded with late interaction retrieval results (thanks to @lightonai for pylate). The Agent initiates it with query then.. 𝐒𝐞𝐭𝐮𝐩 A python REPL is created and seeded with: 1. Late interaction search to pre-filter. Instead of doing top 3/5/10, it's top hundreds of documents. This is set into a `context` variable. 2. Python functions are loaded in to do more searches if `context` variable isn't enough. And to make llm calls with cheaper models in parallel batches. 𝐈𝐭𝐞𝐫𝐚𝐭𝐢𝐨𝐧 𝐋𝐨𝐨𝐩 From there, an LLM iterates in the REPL based on the query. It's just like exploring in a jupyter notebook. The LLM writes prose (like a markdown cell) and code to be run in the REPL each turn. This allows the LLM to sort, filter, and synthesize information. It can fan out and ask smaller models to summarize, combine, contrast, or do anything else to documents to help it understand the data. After several turns the LLM reponds with the final answer. Either because it found the answer, or hit the budget limit. Context as a Python variable, LLM as the programmer, REPL as the runtime. 𝐖𝐡𝐲 𝐃𝐨𝐞𝐬 𝐓𝐡𝐢𝐬 𝐖𝐨𝐫𝐤 1. Richer Shell. Agents (and subagents) work by intermixing code and prose/thinking. But they use static scripts or bash that run and exit and start over each tool call. That's not ideal for exploration and synthesis of data. For that, state is useful to continue building and exploring the data as you learn more. There's a reason jupyter notebooks have been popular with data scientists. 2. Keeps main agent context clean. The better context you have the better the agent will perform (duh!). This means three thing: better human input, less missing search results, and less incorrect search results. Letting the agent iterate allows it to synthesize just what is needed and nothing else. All bad paths or peeks at something that turns out to be irrelevant stays out of main agent context. 3. Stack the good ideas! People often compare late interaction search vs RLM. Or static vs dynamic languages. Or agentic search vs semantic search. But...You can just use them all together for what they're each good at. Use them all for the area they're really great for. Read the full post which has more detail about how and why.

Isaac Flath

40,212 Aufrufe • vor 2 Monaten

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:
2:07

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and , Director of Product. I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error. This course shows you how to structure your evals to assess the performance of each component of an agent and its end-to-end performance. For each component, you select the appropriate evaluators, test examples, and performance metrics. This helps you identify areas for improvement both during development and in production. (If you're familiar with error analysis in supervised learning, think of this as adapting those ideas to agentic workflows.) In this course, you'll build an AI agent, and add observability to visualize and debug its steps. You’ll learn about code-based evals, in which you write code explicitly to test a certain step, as well as LLM-as-a-Judge evals, in which you prompt an LLM to efficiently come up with ways to evaluate more open-ended outputs. In detail, you’ll: - Understand key differences between evaluating LLM-based systems and traditional software testing. - Add observability to an agent by collecting traces of the steps taken by the agent and visualizing them - Choose the appropriate evaluator - code-based, LLM-as-a-Judge, human-annotation based - for each component. - Compute a convergence score to evaluate if your agent can respond to a query in an efficient number of steps. - Run structured experiments to improve the agent’s performance by exploring changes to the prompt, LLM model, or the agent’s logic. - Understand how to deploy these evaluation techniques to monitor the agent’s performance in production. By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. Please sign up here:

Andrew Ng

126,406 Aufrufe • vor 1 Jahr

🔎🤖LangSmith Insights Agent Really excited to launch our first in-product agent This agent lives inside LangSmith and combs through traces, giving you insights into: 🧑‍🤝‍🧑how users are using your agent ⁉️how your agent may be messing up 🛃{your custom insight here} The problem we saw was that people were launching agents... and didn't know how their users were actually using them! You put a chat box in front of people, and they may ask it anything - the surface area for agents is often super wide In addition - agents would fail silently. They could give a bad response - this wouldn't show up in error logs, but its good to know. If you know what look for, you can set up LLM as a judge evaluators. But what if you don't? (most people don't initially) The best way to figure this out - as Hamel Husain says - "look at your data". But LLMs are really good at looking at your data! So can they do it for you? This is exactly what insights agent attempts to do. It's live in LangSmith today. You can read more about it here:
0:40

🔎🤖LangSmith Insights Agent Really excited to launch our first in-product agent This agent lives inside LangSmith and combs through traces, giving you insights into: 🧑‍🤝‍🧑how users are using your agent ⁉️how your agent may be messing up 🛃{your custom insight here} The problem we saw was that people were launching agents... and didn't know how their users were actually using them! You put a chat box in front of people, and they may ask it anything - the surface area for agents is often super wide In addition - agents would fail silently. They could give a bad response - this wouldn't show up in error logs, but its good to know. If you know what look for, you can set up LLM as a judge evaluators. But what if you don't? (most people don't initially) The best way to figure this out - as Hamel Husain says - "look at your data". But LLMs are really good at looking at your data! So can they do it for you? This is exactly what insights agent attempts to do. It's live in LangSmith today. You can read more about it here:

Harrison Chase

98,520 Aufrufe • vor 8 Monaten

Building real-time data pipelines and stream processing is one of the best-paid skills in the market. ​ Apache Kafka and Flink are the industry standards, but they are Java-based and have a Python wrapper. ​ If you are a Python developer, there is an open-source alternative! ​ Watch this video. It's a full tutorial solving a real-world problem using the Quix Streams library. ​ Quix Streams is all Python, and it's open-source! ​ The video shows you how to process GitHub's Firehose API, a constant stream of raw activity. You can use this stream to identify real-time trends with code, issues, and the popularity of public repositories. ​ Here is what the video covers: ​ • How to tap into the GitHub Firehose API using Python and server-sent events (SSE) ​ • How to efficiently stream that data into Kafka ​ • How to optimize Kafka producer settings like batch size and compression for maximum throughput ​ A few other notes about Quix Streams: ​ • You don't need to know any Java • It has a dead-simple API • It integrates seamlessly with your Python stack • It's designed for parallel processing at high velocity ​ You can use Quix Streams to build complete stream processing pipelines in Python, including feature engineering, pre-computations, inference, and real-time machine learning. ​ Thanks to the Quix team for collaborating with me on this post.
27:43

Building real-time data pipelines and stream processing is one of the best-paid skills in the market. ​ Apache Kafka and Flink are the industry standards, but they are Java-based and have a Python wrapper. ​ If you are a Python developer, there is an open-source alternative! ​ Watch this video. It's a full tutorial solving a real-world problem using the Quix Streams library. ​ Quix Streams is all Python, and it's open-source! ​ The video shows you how to process GitHub's Firehose API, a constant stream of raw activity. You can use this stream to identify real-time trends with code, issues, and the popularity of public repositories. ​ Here is what the video covers: ​ • How to tap into the GitHub Firehose API using Python and server-sent events (SSE) ​ • How to efficiently stream that data into Kafka ​ • How to optimize Kafka producer settings like batch size and compression for maximum throughput ​ A few other notes about Quix Streams: ​ • You don't need to know any Java • It has a dead-simple API • It integrates seamlessly with your Python stack • It's designed for parallel processing at high velocity ​ You can use Quix Streams to build complete stream processing pipelines in Python, including feature engineering, pre-computations, inference, and real-time machine learning. ​ Thanks to the Quix team for collaborating with me on this post.

Santiago

89,230 Aufrufe • vor 2 Jahren

Quick demo of Zero’s new background queries. Zero’s sync is query-based. Rather than specifying what data you want using rules or some other separate system, you just use queries. Right inside the client app, you do a query using a full sql-style language. You get filters, subqueries , limits, etc. Zero syncs the data backing these queries to the client. It’s important to realize Zero isn’t really a cache. It’s a replica. It’s eagerly replicating a precise snapshot of a slice of your database - the slice covered by the queries you have open. So there is never stale data in Zero. When you close a query we delete the rows uniquely returned by that query because we can no longer keep them up to date. Of course that kind of sucks for the common case of doing a query, navigating, then pressing “back”. Ideally we want that back nav to be fast. To address this Zero 0.17 adds background queries. You can add a ttl to a query and it will keep running and syncing in the background. This is much different than normal caching because this data stays up to date. If you make the same query again, the results will be *instantly* available *and already up to date with server*. If you make a different query the data from the background query is used client-side to answer the new query instantly if possible. This all happens completely automatically. Just by adding the ttl flag.
1:01

Quick demo of Zero’s new background queries. Zero’s sync is query-based. Rather than specifying what data you want using rules or some other separate system, you just use queries. Right inside the client app, you do a query using a full sql-style language. You get filters, subqueries , limits, etc. Zero syncs the data backing these queries to the client. It’s important to realize Zero isn’t really a cache. It’s a replica. It’s eagerly replicating a precise snapshot of a slice of your database - the slice covered by the queries you have open. So there is never stale data in Zero. When you close a query we delete the rows uniquely returned by that query because we can no longer keep them up to date. Of course that kind of sucks for the common case of doing a query, navigating, then pressing “back”. Ideally we want that back nav to be fast. To address this Zero 0.17 adds background queries. You can add a ttl to a query and it will keep running and syncing in the background. This is much different than normal caching because this data stays up to date. If you make the same query again, the results will be *instantly* available *and already up to date with server*. If you make a different query the data from the background query is used client-side to answer the new query instantly if possible. This all happens completely automatically. Just by adding the ttl flag.

Aaron Boodman

18,808 Aufrufe • vor 1 Jahr