Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

I can search a database with 32 million records in less than 50 milliseconds. This speed is insane! There's no reason your search is slow. There's no reason you aren't using this library when building applications with a lot of data. Take a look at typesense... |. It's an open-source search engine. It's crazy fast and easy to use. They are sponsoring this post. Here is their GitHub repository: You can use the library in Python, JavaScript, PHP, Ruby, Swift, and pretty much any other programming language out there. Here are some of the things you can do with Typesense: • Vector search over embeddings • Conversational search for RAG applications • Image similarity search • Voice search And it does all of this extremely fast. A few interesting notes about the library: • Optimized for developer happiness • Works out of the box for most use cases • Very intuitive API • Easy to deploy and scale Probably the best way to compare them is against ElasticSearch. The latter is much more complex to configure and run with a steep learning curve. Check their GitHub repository. They have examples, tutorials, and everything you need.show more

Santiago

449,428 subscribers

160,154 Aufrufe • vor 2 Jahren •via X (Twitter)

Wissenschaft & Technologie Bildung

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von AkivaSk

AkivaSkvor 2 Jahren

@typesense I guess Windows have never heard about them.

Profilbild von Santiago

Santiagovor 2 Jahren

@typesense Apple hasn't either.

Profilbild von adrien

adrienvor 2 Jahren

@typesense Thanks for sharing @svpino What if my data is long form content such as a lot of transcripts of 3 hours videos? Do you still recommend using this? Also your description said it’s vector search but your demo seems to be keyword search? Thanks.

Profilbild von Santiago

Santiagovor 2 Jahren

@typesense It supports vector search.

Profilbild von Darko Medin

Darko Medinvor 2 Jahren

@typesense Really impressive example @svpino. Speed is essential in many business applications today.

Profilbild von Marcin Wojnarski

Marcin Wojnarskivor 2 Jahren

@typesense In-memory database, so its scalability and resilience is limited compared to ElasticSearch which can be distributed. Comparing apples to oranges. 32M records is nothing if you only search across several metadata fields without full-text content.

Profilbild von Muratcan Koylan

Muratcan Koylanvor 2 Jahren

@typesense This seems like keyword search tho

Profilbild von Santiago

Santiagovor 2 Jahren

@typesense The demo is a keyword search with tolerance for typos. Check their features:

Profilbild von Lionel Ngendakuriyo

Lionel Ngendakuriyovor 2 Jahren

@typesense What is the box specs ?

Profilbild von Santiago

Santiagovor 2 Jahren

@typesense I didn't find the box specs for this specific example, but I did find specs for similar examples:

Ähnliche Videos

Searching 32 million records in 30 milliseconds is insane! I don't know how large your database is, but there's no reason your search results are slower than this. typesense | is a crazy fast open-source search engine. They are like ElasticSearch but way easier to set up, configure, and use. They are sponsoring this post. You can do all of this with Typesense: • Fuzzy search: Typo-tolerant fuzzy search-as-you-type • Vector search: Return records that are similar to each other • Semantic search: Return records that are conceptually related • Geo Search: Return records that are close to a given lat/long They have client libraries in Python, JavaScript, PHP, Ruby, Go, Rust, Swift, .Net, Java, and pretty much any other programming language out there. You can start searching in less than a minute: 1. Run their Docker image (or one of their native binaries) 2. Import your data 3. Start searching! GitHub repository: I recorded a quick video to show you a demo. I promise you haven't seen something this fast! This is the best upgrade you'll make to your application this year. Guaranteed.

Searching 32 million records in 30 milliseconds is insane! I don't know how large your database is, but there's no reason your search results are slower than this. typesense | is a crazy fast open-source search engine. They are like ElasticSearch but way easier to set up, configure, and use. They are sponsoring this post. You can do all of this with Typesense: • Fuzzy search: Typo-tolerant fuzzy search-as-you-type • Vector search: Return records that are similar to each other • Semantic search: Return records that are conceptually related • Geo Search: Return records that are close to a given lat/long They have client libraries in Python, JavaScript, PHP, Ruby, Go, Rust, Swift, .Net, Java, and pretty much any other programming language out there. You can start searching in less than a minute: 1. Run their Docker image (or one of their native binaries) 2. Import your data 3. Start searching! GitHub repository: I recorded a quick video to show you a demo. I promise you haven't seen something this fast! This is the best upgrade you'll make to your application this year. Guaranteed.

Santiago

172,755 Aufrufe • vor 1 Jahr

Knowledge graphs for representing information are unbeatable. After this, you will never build a RAG system without knowledge graphs. It will take you five lines of code to build a knowledge graph with your data. I recorded a video to show you how you can do this. I used Cognee, an open-source library that outperforms any basic vector search approach in terms of retrieval relevance. They are collaborating with me on this post. Cognee is: • Easy to use • Reduces hallucinations • Open-source Here is a link to the repository: They also offer a comprehensive platform and UI with Python notebooks you can utilize to manage your data. Here is the link:

Knowledge graphs for representing information are unbeatable. After this, you will never build a RAG system without knowledge graphs. It will take you five lines of code to build a knowledge graph with your data. I recorded a video to show you how you can do this. I used Cognee, an open-source library that outperforms any basic vector search approach in terms of retrieval relevance. They are collaborating with me on this post. Cognee is: • Easy to use • Reduces hallucinations • Open-source Here is a link to the repository: They also offer a comprehensive platform and UI with Python notebooks you can utilize to manage your data. Here is the link:

Santiago

126,337 Aufrufe • vor 10 Monaten

108 workflow templates you can use to build AI applications without writing any code. You can use these templates with n8n. I recorded the attached video to show you how it works. n8n is the workhorse behind an open-source, self-hosted AI starter kit you can install on your computer. They are sponsoring this post. Here is the link to the starter kit repository: And here is the spreadsheet with the 108 templates: Whatever idea you have, search for something similar in the list of templates, and you'll save a ton of time. Lately, I've talked to many non-coders who want to start using AI more seriously to build things. n8n is perfect for that.

108 workflow templates you can use to build AI applications without writing any code. You can use these templates with n8n. I recorded the attached video to show you how it works. n8n is the workhorse behind an open-source, self-hosted AI starter kit you can install on your computer. They are sponsoring this post. Here is the link to the starter kit repository: And here is the spreadsheet with the 108 templates: Whatever idea you have, search for something similar in the list of templates, and you'll save a ton of time. Lately, I've talked to many non-coders who want to start using AI more seriously to build things. n8n is perfect for that.

Santiago

78,133 Aufrufe • vor 1 Jahr

It takes 5 minutes to create a world-class web research agent that runs 24/7 and returns structured search results. 1. You can limit the sources it's allowed to use 2. You can specify the goals it needs to complete 3. You can design the format structure of the results 4. You can configure how much computing power it will use I show you how to set up a Web Search Agent in the video. A few benefits here: 1. This is great for complex research tasks 2. It's really easy to add this like a LEGO piece to a more complex workflow 3. Create an agent once, and reuse it with different inputs 4. The agent adapts to your use case as you work with it By the way, Nimble published a whole library of templates and research agents you can reuse and modify. You can start sending queries to one of these existing agents, and it will begin to adapt to Go here to try it out: Thanks to the Nimble team for partnering with me on this post.

It takes 5 minutes to create a world-class web research agent that runs 24/7 and returns structured search results. 1. You can limit the sources it's allowed to use 2. You can specify the goals it needs to complete 3. You can design the format structure of the results 4. You can configure how much computing power it will use I show you how to set up a Web Search Agent in the video. A few benefits here: 1. This is great for complex research tasks 2. It's really easy to add this like a LEGO piece to a more complex workflow 3. Create an agent once, and reuse it with different inputs 4. The agent adapts to your use case as you work with it By the way, Nimble published a whole library of templates and research agents you can reuse and modify. You can start sending queries to one of these existing agents, and it will begin to adapt to Go here to try it out: Thanks to the Nimble team for partnering with me on this post.

Santiago

20,943 Aufrufe • vor 11 Tagen

This is, by far, one of the best uses of modern AI. If you don't use embeddings when querying your database, you are definitely leaving a lot on the table. In this video, I'll show you how to run semantic searches using OpenAI and PostgreSQL. It's all thanks to Pgai, an open-source PostgreSQL extension: Here's what will happen: 1. We'll create a simple table with news articles 2. We'll generate embeddings for those articles 3. We'll run queries on top of those embeddings For this video, I generated the embeddings using a simple query, but pgai Vectorizer would do the same automatically as new information makes it into the database. This is awesome! If you have a PostgreSQL database with data you are searching over, you should start experimenting with semantic searches immediately. For most use cases, a combination of full-text search + semantic search is the best approach. If you don't have a PostgreSQL database around, you can try free for 30 days using Timescale: Thanks to the TimescaleDB (by Tiger Data) team for partnering with me on this post!

This is, by far, one of the best uses of modern AI. If you don't use embeddings when querying your database, you are definitely leaving a lot on the table. In this video, I'll show you how to run semantic searches using OpenAI and PostgreSQL. It's all thanks to Pgai, an open-source PostgreSQL extension: Here's what will happen: 1. We'll create a simple table with news articles 2. We'll generate embeddings for those articles 3. We'll run queries on top of those embeddings For this video, I generated the embeddings using a simple query, but pgai Vectorizer would do the same automatically as new information makes it into the database. This is awesome! If you have a PostgreSQL database with data you are searching over, you should start experimenting with semantic searches immediately. For most use cases, a combination of full-text search + semantic search is the best approach. If you don't have a PostgreSQL database around, you can try free for 30 days using Timescale: Thanks to the TimescaleDB (by Tiger Data) team for partnering with me on this post!

Santiago

110,133 Aufrufe • vor 1 Jahr

Building with AI gets easier every day. Here is an open-source library that makes integrating AI into an application extremely easy: Star the repository! This library alone can make React the best front-end framework out there! There are a bunch of cool things I like about CopilotKit. Here are 3 of them: 1. It allows you to take any -powered agent and bring it into your application. (This is a brand-new feature!) 2. You can build an AI-powered chatbot in your application. The chatbot will have access to your context and can act on the application. 3. You can build a RAG workflow to process and answer questions from a real-time knowledge base. I recorded a video to show you how simple it is to make some of this happen. A few lines of code, and you are in business. Here is a link to the sample application: CopilotKit is open-source. You can self-host it. You can use it with any LLM. Thanks to the team for showing me their tool and collaborating with me on this post!

Santiago

108,835 Aufrufe • vor 2 Jahren

99% of AI applications are cool-looking demos. Impressive, but don't get fooled by the hype. It takes a lot to build enterprise-grade products that deliver real value. I have at least three weekly conversations with companies that want to use a Large Language Model with their data. The demand is huge! Here is one idea about what you can do to help. The use cases that most of these companies want to solve are similar: They have an extensive knowledge base and want to build a simple application that uses that information to answer questions. In other words, they need help building Retrieval Augmented Generation (RAG) applications they can use in many different scenarios: 1. To train new employees 2. To help their support team 3. To search old meetings and documents 4. To help with their research However, building these systems is not straightforward. Yes, there's a lot of information online, but there aren't enough people who know how to create solutions that work. Here is the idea: Today, you can build an enterprise-grade RAG application without writing code. A couple of MIT PhDs with 10+ years of experience building AI applications created . It's a no-code platform for building applications using Large Language Models. They are partnering with me on this post. You can use Stack AI to create, test, and deploy an end-to-end production-ready AI system. It's SOC-2, HIPAA, and GDPR compliant and offers SSO, role management, access control, and on-premise deployments. Of course, you can use the platform with any LLM on the market now. It's the whole nine yards for building AI applications. Check them out here: 2023 was about models. 2024 is about the tools using these models to build production-ready applications. That's where I'd start.

99% of AI applications are cool-looking demos. Impressive, but don't get fooled by the hype. It takes a lot to build enterprise-grade products that deliver real value. I have at least three weekly conversations with companies that want to use a Large Language Model with their data. The demand is huge! Here is one idea about what you can do to help. The use cases that most of these companies want to solve are similar: They have an extensive knowledge base and want to build a simple application that uses that information to answer questions. In other words, they need help building Retrieval Augmented Generation (RAG) applications they can use in many different scenarios: 1. To train new employees 2. To help their support team 3. To search old meetings and documents 4. To help with their research However, building these systems is not straightforward. Yes, there's a lot of information online, but there aren't enough people who know how to create solutions that work. Here is the idea: Today, you can build an enterprise-grade RAG application without writing code. A couple of MIT PhDs with 10+ years of experience building AI applications created . It's a no-code platform for building applications using Large Language Models. They are partnering with me on this post. You can use Stack AI to create, test, and deploy an end-to-end production-ready AI system. It's SOC-2, HIPAA, and GDPR compliant and offers SSO, role management, access control, and on-premise deployments. Of course, you can use the platform with any LLM on the market now. It's the whole nine yards for building AI applications. Check them out here: 2023 was about models. 2024 is about the tools using these models to build production-ready applications. That's where I'd start.

Santiago

197,702 Aufrufe • vor 2 Jahren

Today I’m excited to introduce Copilot Search in Bing. Copilot Search blends the best of traditional and generative search to help you find what you need. Whether it’s a navigational search result, a quick straightforward answer, or a complex query that leads you on a journey of discovery, Bing is your AI-powered search and answer engine. Copilot Search is rolling out today to everyone. To get started, go to and start exploring. This is a meaningful next step in our evolution of search, building on our learnings from Bing Chat, Copilot, and Bing Generative Search to provide our users the best search experience while supporting and building a healthy web ecosystem. Learn more in today’s announcement:

Today I’m excited to introduce Copilot Search in Bing. Copilot Search blends the best of traditional and generative search to help you find what you need. Whether it’s a navigational search result, a quick straightforward answer, or a complex query that leads you on a journey of discovery, Bing is your AI-powered search and answer engine. Copilot Search is rolling out today to everyone. To get started, go to and start exploring. This is a meaningful next step in our evolution of search, building on our learnings from Bing Chat, Copilot, and Bing Generative Search to provide our users the best search experience while supporting and building a healthy web ecosystem. Learn more in today’s announcement:

Jordi Ribas

16,729 Aufrufe • vor 1 Jahr

Tune Studio is an end-to-end platform for developing applications using Large Language Models. So far, I haven't seen any other platform like this one. You can do everything here: 1. You can curate your data. 2. Use the playground to play with different models and try your ideas. 3. Fine-tune an open-source model on your data. 4. Deploy the model when you are done. This is awesome for anyone building generative AI applications. You can use Tune Studio to work with any of the open-source models out there. They were one of the few companies to host Llama 2 and Llama 3 before anyone else. Here is a link to check it out: One of their main selling points is that Tune Studio scales! You don't have to worry about serving your model to lots of users. They also have built-in user management, authentication, on-prem support, user context management, and pretty much everything you need to build generative AI applications. Thanks to the Tune team for collaborating with me on this post. We are living through the best years of development tools for AI developers. The field is unstoppable.

Tune Studio is an end-to-end platform for developing applications using Large Language Models. So far, I haven't seen any other platform like this one. You can do everything here: 1. You can curate your data. 2. Use the playground to play with different models and try your ideas. 3. Fine-tune an open-source model on your data. 4. Deploy the model when you are done. This is awesome for anyone building generative AI applications. You can use Tune Studio to work with any of the open-source models out there. They were one of the few companies to host Llama 2 and Llama 3 before anyone else. Here is a link to check it out: One of their main selling points is that Tune Studio scales! You don't have to worry about serving your model to lots of users. They also have built-in user management, authentication, on-prem support, user context management, and pretty much everything you need to build generative AI applications. Thanks to the Tune team for collaborating with me on this post. We are living through the best years of development tools for AI developers. The field is unstoppable.

Santiago

39,101 Aufrufe • vor 2 Jahren

Vector databases are ridiculous. How can you search millions of vectors at light speed? What's the magic making this possible? I wanted to understand how they work, and here is a full video with everything I learned. This is for nerds.

Vector databases are ridiculous. How can you search millions of vectors at light speed? What's the magic making this possible? I wanted to understand how they work, and here is a full video with everything I learned. This is for nerds.

Santiago

231,296 Aufrufe • vor 2 Jahren

Here is an open-source tool to generate a complete dataset. 1. Describe the data you want 2. An orchestrator agent searches the web 3. Sub-agents run in parallel to fetch the data 4. You get a structured dataset you can download For example, you can run Bigset with the query "all leica lenses being sold on amazon", or "leica stores in kyoto with their opening hours and ratings". Bigset uses TinyFish's free Search and Fetch APIs in the background. You can configure it to refresh the data on a schedule. You can self-host it with your own keys. Here is the GitHub repository: You can get free TinyFish API keys here: Thanks to the TinyFish team for partnering with me on this post.

Here is an open-source tool to generate a complete dataset. 1. Describe the data you want 2. An orchestrator agent searches the web 3. Sub-agents run in parallel to fetch the data 4. You get a structured dataset you can download For example, you can run Bigset with the query "all leica lenses being sold on amazon", or "leica stores in kyoto with their opening hours and ratings". Bigset uses TinyFish's free Search and Fetch APIs in the background. You can configure it to refresh the data on a schedule. You can self-host it with your own keys. Here is the GitHub repository: You can get free TinyFish API keys here: Thanks to the TinyFish team for partnering with me on this post.

Santiago

20,805 Aufrufe • vor 2 Monaten

Forget about HTML, JavaScript, and CSS. You can build AI and data applications using Python alone! Python is the best programming language out there. You can now use it across the stack. Take a look at Taipy, an open-source Python library to build end-to-end production applications. Star their repository: I recorded a quick video showing how to build a simple chat interface to talk to OpenAI GPT-3.5 using Taipy alone. Something important to keep in mind: Taipy's goal is not to replace web developers but to provide an alternative to those who need to build applications without web experience. If you are a data scientist or someone dealing with data, Taipy will simplify your life considerably. Thanks to the team behind Taipy for collaborating with me on this post. Here is a link to the full demo. Download this code and run it. It takes 30 seconds:

Forget about HTML, JavaScript, and CSS. You can build AI and data applications using Python alone! Python is the best programming language out there. You can now use it across the stack. Take a look at Taipy, an open-source Python library to build end-to-end production applications. Star their repository: I recorded a quick video showing how to build a simple chat interface to talk to OpenAI GPT-3.5 using Taipy alone. Something important to keep in mind: Taipy's goal is not to replace web developers but to provide an alternative to those who need to build applications without web experience. If you are a data scientist or someone dealing with data, Taipy will simplify your life considerably. Thanks to the team behind Taipy for collaborating with me on this post. Here is a link to the full demo. Download this code and run it. It takes 30 seconds:

Santiago

233,902 Aufrufe • vor 2 Jahren

New short course on Building Applications with Vector Databases, taught by Pinecone’s Tim Tully! At the heart of a vector database is the ability to store a collection of vectors and then query against that, meaning input a new vector and find similar ones. This is useful for many AI applications. In this course, you'll learn how to use vector databases to build: (i) Semantic Search: Create a text search tool that goes beyond keyword matching, and instead focuses on the meaning of content. (ii) RAG (retrieval augmented generation): Enhance your LLM output by incorporating context from sources the model wasn't trained on. (iii) Recommender System: Combine semantic search and RAG to recommend topics, and demonstrate it with a news article recommender. (iv) Hybrid Search: Build an application that finds items using both images and descriptive text -- by combining both sparse and dense vector representations of the data -- using an eCommerce dataset as an example. (v) Image Similarity: Use image vector embeddings to create an app to compare facial features, using a database of public figures to determine the likeness between them. (vi) Anomaly Detection: Build an anomaly detection app that identifies unusual patterns in network communication logs. I hope you’ll enjoy learning how to build all these types of applications! Please sign up here:

New short course on Building Applications with Vector Databases, taught by Pinecone’s Tim Tully! At the heart of a vector database is the ability to store a collection of vectors and then query against that, meaning input a new vector and find similar ones. This is useful for many AI applications. In this course, you'll learn how to use vector databases to build: (i) Semantic Search: Create a text search tool that goes beyond keyword matching, and instead focuses on the meaning of content. (ii) RAG (retrieval augmented generation): Enhance your LLM output by incorporating context from sources the model wasn't trained on. (iii) Recommender System: Combine semantic search and RAG to recommend topics, and demonstrate it with a news article recommender. (iv) Hybrid Search: Build an application that finds items using both images and descriptive text -- by combining both sparse and dense vector representations of the data -- using an eCommerce dataset as an example. (v) Image Similarity: Use image vector embeddings to create an app to compare facial features, using a database of public figures to determine the likeness between them. (vi) Anomaly Detection: Build an anomaly detection app that identifies unusual patterns in network communication logs. I hope you’ll enjoy learning how to build all these types of applications! Please sign up here:

Andrew Ng

137,091 Aufrufe • vor 2 Jahren

This is the fastest text-to-speech and speech-to-text API out there. It's under 250 ms to first-byte latency. That's what we need to deploy conversational AI applications everywhere! (I can't wait for the death of automated voice machines.) Look at the attached video. For the first time, I could have a conversation with an AI agent in real time without awkward pauses. This technology is Aura from Deepgram. It's the fastest text-to-speech option I've seen. Here is a GitHub repository with the demo in the video: You can use it to build a conversational AI application using natural voice. You'll need a Deepgram API Key and an OpenAI API Key. Thanks to Deepgram for giving me access to Aura and collaborating with me on this post. If you want to learn more about Aura, here is the announcement post:

This is the fastest text-to-speech and speech-to-text API out there. It's under 250 ms to first-byte latency. That's what we need to deploy conversational AI applications everywhere! (I can't wait for the death of automated voice machines.) Look at the attached video. For the first time, I could have a conversation with an AI agent in real time without awkward pauses. This technology is Aura from Deepgram. It's the fastest text-to-speech option I've seen. Here is a GitHub repository with the demo in the video: You can use it to build a conversational AI application using natural voice. You'll need a Deepgram API Key and an OpenAI API Key. Thanks to Deepgram for giving me access to Aura and collaborating with me on this post. If you want to learn more about Aura, here is the announcement post:

Santiago

185,640 Aufrufe • vor 2 Jahren

Traditional (SQL) databases rely primarily on keyword-based searches to retrieve information. These searches match the exact words or phrases in your query to the text stored in the database. While effective for many applications, this method has limitations when it comes to understanding context or finding relevant information that doesn’t include the exact keywords. Hybrid search combines the strengths of traditional keyword-based BM25 search with the advanced capabilities of semantic search. To effectively implement a hybrid search, a vector database is essential. Vector databases go beyond just words; they understand the meaning behind the data. They transform data such as text, images, or audio into numerical representations called vectors. These vector embeddings enable the database to find similar items, even if they don't share exact keywords. When you integrate hybrid search with Retrieval-Augmented Generation (RAG) systems, you can achieve higher accuracy in retrieved context and better output in generated responses. Learn more about RAG systems in this video with Victoria Slocum:

Traditional (SQL) databases rely primarily on keyword-based searches to retrieve information. These searches match the exact words or phrases in your query to the text stored in the database. While effective for many applications, this method has limitations when it comes to understanding context or finding relevant information that doesn’t include the exact keywords. Hybrid search combines the strengths of traditional keyword-based BM25 search with the advanced capabilities of semantic search. To effectively implement a hybrid search, a vector database is essential. Vector databases go beyond just words; they understand the meaning behind the data. They transform data such as text, images, or audio into numerical representations called vectors. These vector embeddings enable the database to find similar items, even if they don't share exact keywords. When you integrate hybrid search with Retrieval-Augmented Generation (RAG) systems, you can achieve higher accuracy in retrieved context and better output in generated responses. Learn more about RAG systems in this video with Victoria Slocum:

Femke Plantinga

140,618 Aufrufe • vor 2 Jahren

Building real-time data pipelines and stream processing is one of the best-paid skills in the market. Apache Kafka and Flink are the industry standards, but they are Java-based and have a Python wrapper. If you are a Python developer, there is an open-source alternative! Watch this video. It's a full tutorial solving a real-world problem using the Quix Streams library. Quix Streams is all Python, and it's open-source! The video shows you how to process GitHub's Firehose API, a constant stream of raw activity. You can use this stream to identify real-time trends with code, issues, and the popularity of public repositories. Here is what the video covers: • How to tap into the GitHub Firehose API using Python and server-sent events (SSE) • How to efficiently stream that data into Kafka • How to optimize Kafka producer settings like batch size and compression for maximum throughput A few other notes about Quix Streams: • You don't need to know any Java • It has a dead-simple API • It integrates seamlessly with your Python stack • It's designed for parallel processing at high velocity You can use Quix Streams to build complete stream processing pipelines in Python, including feature engineering, pre-computations, inference, and real-time machine learning. Thanks to the Quix team for collaborating with me on this post.

Building real-time data pipelines and stream processing is one of the best-paid skills in the market. Apache Kafka and Flink are the industry standards, but they are Java-based and have a Python wrapper. If you are a Python developer, there is an open-source alternative! Watch this video. It's a full tutorial solving a real-world problem using the Quix Streams library. Quix Streams is all Python, and it's open-source! The video shows you how to process GitHub's Firehose API, a constant stream of raw activity. You can use this stream to identify real-time trends with code, issues, and the popularity of public repositories. Here is what the video covers: • How to tap into the GitHub Firehose API using Python and server-sent events (SSE) • How to efficiently stream that data into Kafka • How to optimize Kafka producer settings like batch size and compression for maximum throughput A few other notes about Quix Streams: • You don't need to know any Java • It has a dead-simple API • It integrates seamlessly with your Python stack • It's designed for parallel processing at high velocity You can use Quix Streams to build complete stream processing pipelines in Python, including feature engineering, pre-computations, inference, and real-time machine learning. Thanks to the Quix team for collaborating with me on this post.

Santiago

89,230 Aufrufe • vor 2 Jahren

Web scraping is a critical skill, and yet nobody talks about it. How do you think companies are training their Large Language Models? Where do you think the data comes from? But web scraping goes beyond all of that. Imagine giving an AI agent access to any public online data in real time! I like to call this "web-scrapping on demand", and I'm pretty sure it's going to unlock unlimited power for AI applications. I recorded a quick video to show you how you can do this using Apify. I've talked about them before, and they are collaborating with me on this post. They have one of the best open-source web scraping and browser automation libraries out there: But it gets much better than this! You can use MCP to connect your AI Agents and applications to the Apify platform and use any specialized actor on demand to scrape and process online data. In the video, I used Cursor to scrape LinkedIn posts with the words "Machine Learning" in real time. Worked like a charm with no code needed! Here is a link to the platform: Think about this: You can now feed your AI applications with any public data on demand! We aren't ready for what's coming.

Web scraping is a critical skill, and yet nobody talks about it. How do you think companies are training their Large Language Models? Where do you think the data comes from? But web scraping goes beyond all of that. Imagine giving an AI agent access to any public online data in real time! I like to call this "web-scrapping on demand", and I'm pretty sure it's going to unlock unlimited power for AI applications. I recorded a quick video to show you how you can do this using Apify. I've talked about them before, and they are collaborating with me on this post. They have one of the best open-source web scraping and browser automation libraries out there: But it gets much better than this! You can use MCP to connect your AI Agents and applications to the Apify platform and use any specialized actor on demand to scrape and process online data. In the video, I used Cursor to scrape LinkedIn posts with the words "Machine Learning" in real time. Worked like a charm with no code needed! Here is a link to the platform: Think about this: You can now feed your AI applications with any public data on demand! We aren't ready for what's coming.

Santiago

101,473 Aufrufe • vor 1 Jahr

AI search for papers is here Search anything you want with papers Here are a few examples of what you can do: Semantically search by sentences & phrases. Example: evaluating factuality in LLM outputs Search by concepts. Example: latent diffusion Search by authors names. Example: Ilya Sutskever Hugging Face just crossed 50,000 papers (all the majors ones of the past two years should be covered), with dozens added every day & just enabled semantic search. It's becoming the best place to do collaborative research.

AI search for papers is here Search anything you want with papers Here are a few examples of what you can do: Semantically search by sentences & phrases. Example: evaluating factuality in LLM outputs Search by concepts. Example: latent diffusion Search by authors names. Example: Ilya Sutskever Hugging Face just crossed 50,000 papers (all the majors ones of the past two years should be covered), with dozens added every day & just enabled semantic search. It's becoming the best place to do collaborative research.

AK

35,896 Aufrufe • vor 1 Jahr