Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake... show more

Zach Wilson

51,515 subscribers

20,412 views • 1 year ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

5 Comments

Nina Taft1 year ago

Great advise, I would also encourage folks who just started and setting up own infra to setup billing alerts to a)don’t get unexpected bills b)get a legit biz sense in the price of data manipulation

Mobile Scanner1 year ago

Scan any documents, convert images into text, PDF files, etc. 👍

prodigal son1 year ago

Going to incorporate this with my cloud dev studies to add some level of functionality to my pipelines

Row Skilli-Vitzky1 year ago

What of doing all this with Airbyte , Snowflake and dbtFusion for Ingestion, Storage and Orchestration respectively I’ve been curious to know your thoughts on the future of loading data with python especially as these tools get cheaper

Fran1 year ago

Gonna try this

Related Videos

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:

Zach Wilson

29,968 views • 1 year ago

We just launched a major new Data Engineering Professional Certificate on Coursera! Data underlies all modern AI systems, and engineers who know how to build systems to store and serve it are in high demand. If you're interested in learning this skill, please check out this 4-course sequence, which is designed to make you job-ready to be a Data Engineer. This is a new specialization taught by Joe Reis, the co-author of the best-selling book “Fundamentals of Data Engineering," in collaboration with AWS. (Disclosure, I serve on Amazon's board.) For many AI systems, data engineering is 80% of the work, and modeling is 20%. But people’s attention on these two topics is often flipped. This makes the job of the data engineer particularly important. In this professional certificate, you'll learn foundational data engineering skills while implementing modern data architectures using open-source tools: - Learn the key steps of the data lifecycle, to generate, ingest, store, transform, and serve data. - Learn to align with organizational goals to design the data pipeline right for your business' needs. - Understand how to make necessary trade-offs between speed, scalability, security, and cost. Joe has distilled into this specialization decades of experience helping startups and large companies with data infrastructure. He is also joined by 17 other industry leaders in the data field, who will help you learn in-demand skills for the growing field of data engineering. Please sign up here:

We just launched a major new Data Engineering Professional Certificate on Coursera! Data underlies all modern AI systems, and engineers who know how to build systems to store and serve it are in high demand. If you're interested in learning this skill, please check out this 4-course sequence, which is designed to make you job-ready to be a Data Engineer. This is a new specialization taught by Joe Reis, the co-author of the best-selling book “Fundamentals of Data Engineering," in collaboration with AWS. (Disclosure, I serve on Amazon's board.) For many AI systems, data engineering is 80% of the work, and modeling is 20%. But people’s attention on these two topics is often flipped. This makes the job of the data engineer particularly important. In this professional certificate, you'll learn foundational data engineering skills while implementing modern data architectures using open-source tools: - Learn the key steps of the data lifecycle, to generate, ingest, store, transform, and serve data. - Learn to align with organizational goals to design the data pipeline right for your business' needs. - Understand how to make necessary trade-offs between speed, scalability, security, and cost. Joe has distilled into this specialization decades of experience helping startups and large companies with data infrastructure. He is also joined by 17 other industry leaders in the data field, who will help you learn in-demand skills for the growing field of data engineering. Please sign up here:

Andrew Ng

120,581 views • 2 years ago

Perplexity Computer can now use Snowflake as a data source and work as a personal Data Science team if you are on Snowflake And the Data Analytics team as well 👀

Perplexity Computer can now use Snowflake as a data source and work as a personal Data Science team if you are on Snowflake And the Data Analytics team as well 👀

🚨 AI News | TestingCatalog

20,048 views • 5 months ago

Stop feeding your AI and analytics stale data. Tableau Prep keeps your flows current with automated data refreshes. Paired with Composable Data Sources in Desktop, Prep enables you to power your models with a complete pipeline of trusted data. Learn more:

Stop feeding your AI and analytics stale data. Tableau Prep keeps your flows current with automated data refreshes. Paired with Composable Data Sources in Desktop, Prep enables you to power your models with a complete pipeline of trusted data. Learn more:

Tableau

33,468 views • 2 months ago

Balancing a 9-5 job with dreams of a Data Science career? 📊 Our Part-time Data Science program has you covered! Learn Python for Data Science in just 35 weeks. Enroll now for our February 2024 class. Your journey into the world of Data begins with us! Apply Now!🚀 #DataScience

Balancing a 9-5 job with dreams of a Data Science career? 📊 Our Part-time Data Science program has you covered! Learn Python for Data Science in just 35 weeks. Enroll now for our February 2024 class. Your journey into the world of Data begins with us! Apply Now!🚀 #DataScience

Moringa School

4,530,436 views • 2 years ago

Today, we’re excited to announce and launch Reflex Cloud on Product Hunt! Reflex is an open-source framework for building and deploying data and AI web apps in pure Python. Frontend and Backend in Pure Python: No JavaScript required! With Reflex Cloud, you can now deploy, manage, and scale your Python apps with just a single command! If you're a Python developer, an upvote or a share would mean a lot to us :)💪❤️

Today, we’re excited to announce and launch Reflex Cloud on Product Hunt! Reflex is an open-source framework for building and deploying data and AI web apps in pure Python. Frontend and Backend in Pure Python: No JavaScript required! With Reflex Cloud, you can now deploy, manage, and scale your Python apps with just a single command! If you're a Python developer, an upvote or a share would mean a lot to us :)💪❤️

Reflex

55,100 views • 1 year ago

I got a smart meter recently and saw you can download a HDF file with the data so I had the idea of writing a script that could parse that and show the data in a useful manner. However I discovered that someone has already done this and done a really good job on it. The video explains how it works. In simple terms it uses the ESB smart meter data and shows a breakdown of how much data you are using and when and also recommends plans and estimates what each one would cost based on the data. The tool is available at easier to read on Laptop or Tablet or then your phone to Landscape.

I got a smart meter recently and saw you can download a HDF file with the data so I had the idea of writing a script that could parse that and show the data in a useful manner. However I discovered that someone has already done this and done a really good job on it. The video explains how it works. In simple terms it uses the ESB smart meter data and shows a breakdown of how much data you are using and when and also recommends plans and estimates what each one would cost based on the data. The tool is available at easier to read on Laptop or Tablet or then your phone to Landscape.

Carlow Weather

298,565 views • 2 years ago

there is a game called "data center" on steam which let's you build and manage your own data center. this is lowkey genius, the best way to educate people on a new trait. hyperscalers should learn a thing or two from "edutainment".

there is a game called "data center" on steam which let's you build and manage your own data center. this is lowkey genius, the best way to educate people on a new trait. hyperscalers should learn a thing or two from "edutainment".

P.M

7,343,946 views • 8 months ago

I made an AI data science team with Python. I'm excited to share my new project, the AI Data Science Team, a Python package for creating AI co-pilots and agents to assist in common data science tasks. I walk you through the GitHub page, showcasing the progress and potential of this project. I'll explain how these agents can streamline tasks like code generation and modeling pipeline creation, making data science projects more efficient. Introduction to AI Data Science Team Project Integrating Human-in-the-Loop Example: Accessing Agents for Feature Engineering Setting up Feature Engineering Agent Code On GitHub: Python AI-Tips Newsletter (stay updated as I build this project in the open):

I made an AI data science team with Python. I'm excited to share my new project, the AI Data Science Team, a Python package for creating AI co-pilots and agents to assist in common data science tasks. I walk you through the GitHub page, showcasing the progress and potential of this project. I'll explain how these agents can streamline tasks like code generation and modeling pipeline creation, making data science projects more efficient. Introduction to AI Data Science Team Project Integrating Human-in-the-Loop Example: Accessing Agents for Feature Engineering Setting up Feature Engineering Agent Code On GitHub: Python AI-Tips Newsletter (stay updated as I build this project in the open):

Matt Dancho (Business Science)

11,799 views • 1 year ago

v0’s Snowflake integration is now in public preview. Prompt v0 to use Snowflake, connect your account, and watch as it generates polished dashboards with your data.

v0’s Snowflake integration is now in public preview. Prompt v0 to use Snowflake, connect your account, and watch as it generates polished dashboards with your data.

v0

122,775 views • 4 months ago

I'm excited to introduce my FREE AI Pandas Data Analyst Copilot which created a data analysis report with dozens of charts from my questions in under 30 seconds. Today, I'll share with you how to automate data analysis with my Pandas AI Agent + Copilot, which is available on GitHub. I'll guide you through setting up the Copilot app, creating dozens of data analysis charts from any CSV or Excel file, and interacting with your data live. This AI is a BIG help! Table of Contents: 00:00 Introduction to the App 02:24 Setting Up the App 04:25 Using the App 08:58 Understanding the Python Code Github to AI Data Science Team (app is in the apps folder): Get the Code and Future Updates by Joining my Python AI/ML Tips Newsletter: === Want to learn how to build AI projects companies actually want? (live Python Code) On Wednesday, April 9th, I'm sharing one of my best AI Projects: Time Series Forecasting with AI Register here (500 Seats):

I'm excited to introduce my FREE AI Pandas Data Analyst Copilot which created a data analysis report with dozens of charts from my questions in under 30 seconds. Today, I'll share with you how to automate data analysis with my Pandas AI Agent + Copilot, which is available on GitHub. I'll guide you through setting up the Copilot app, creating dozens of data analysis charts from any CSV or Excel file, and interacting with your data live. This AI is a BIG help! Table of Contents: 00:00 Introduction to the App 02:24 Setting Up the App 04:25 Using the App 08:58 Understanding the Python Code Github to AI Data Science Team (app is in the apps folder): Get the Code and Future Updates by Joining my Python AI/ML Tips Newsletter: === Want to learn how to build AI projects companies actually want? (live Python Code) On Wednesday, April 9th, I'm sharing one of my best AI Projects: Time Series Forecasting with AI Register here (500 Seats):

Matt Dancho (Business Science)

12,736 views • 1 year ago

How to Create a Spinning Globe 🌍 with Your Own Data Layers (No Coding Required) Learn how to create a spinning 3D globe map without writing any code using a simple web-based tool. In this step-by-step geospatial tutorial, you'll see how to build an interactive spinning globe, overlay raster and vector data layers, and visualize datasets directly in your browser or in a Jupyter Notebook environment. Web-based demo: Jupyter notebook demo: Video tutorial: #geospatial #opensource #python #jupyter #maplibre

How to Create a Spinning Globe 🌍 with Your Own Data Layers (No Coding Required) Learn how to create a spinning 3D globe map without writing any code using a simple web-based tool. In this step-by-step geospatial tutorial, you'll see how to build an interactive spinning globe, overlay raster and vector data layers, and visualize datasets directly in your browser or in a Jupyter Notebook environment. Web-based demo: Jupyter notebook demo: Video tutorial: #geospatial #opensource #python #jupyter #maplibre

Qiusheng Wu

12,745 views • 7 months ago

Making Data and AI Lovable. Lovable now integrates with Databricks, providing a natural language interface that allows anyone—regardless of technical skills—to build live data apps can read and write data stored in Databricks. Bridge the gap between complex data engineering and beautiful, functional front-ends.

Making Data and AI Lovable. Lovable now integrates with Databricks, providing a natural language interface that allows anyone—regardless of technical skills—to build live data apps can read and write data stored in Databricks. Bridge the gap between complex data engineering and beautiful, functional front-ends.

Databricks

34,315 views • 6 months ago

New JavaScript short course: Build a full-stack web application that uses RAG in JavaScript RAG Web Apps with LlamaIndex, taught by Laurie Voss, VP of Developer Relations at LlamaIndex 🦙 and npm co-founder. - Build a RAG application for querying your own data - Develop tools to interact with multiple data sources using an agent that intelligently selects the right tool for your queries - Create a full-stack web app that can chat with your data - Dig further into production-ready techniques, like how to persist your data so you aren’t constantly reindexing, and try the create-llama command line tool from LlamaIndex You can sign up here:

New JavaScript short course: Build a full-stack web application that uses RAG in JavaScript RAG Web Apps with LlamaIndex, taught by Laurie Voss, VP of Developer Relations at LlamaIndex 🦙 and npm co-founder. - Build a RAG application for querying your own data - Develop tools to interact with multiple data sources using an agent that intelligently selects the right tool for your queries - Create a full-stack web app that can chat with your data - Dig further into production-ready techniques, like how to persist your data so you aren’t constantly reindexing, and try the create-llama command line tool from LlamaIndex You can sign up here:

Andrew Ng

218,703 views • 2 years ago

Knowledge graphs for representing information are unbeatable. After this, you will never build a RAG system without knowledge graphs. It will take you five lines of code to build a knowledge graph with your data. I recorded a video to show you how you can do this. I used Cognee, an open-source library that outperforms any basic vector search approach in terms of retrieval relevance. They are collaborating with me on this post. Cognee is: • Easy to use • Reduces hallucinations • Open-source Here is a link to the repository: They also offer a comprehensive platform and UI with Python notebooks you can utilize to manage your data. Here is the link:

Knowledge graphs for representing information are unbeatable. After this, you will never build a RAG system without knowledge graphs. It will take you five lines of code to build a knowledge graph with your data. I recorded a video to show you how you can do this. I used Cognee, an open-source library that outperforms any basic vector search approach in terms of retrieval relevance. They are collaborating with me on this post. Cognee is: • Easy to use • Reduces hallucinations • Open-source Here is a link to the repository: They also offer a comprehensive platform and UI with Python notebooks you can utilize to manage your data. Here is the link:

Santiago

126,565 views • 1 year ago

Solana works with Google Cloud to enable high-performance use cases with efficiency. Dan Albert from the Solana Foundation talks about simplifying blockchain data with tools like #BigQuery and BigTable and how they enable developers to move and build faster ↓

Solana works with Google Cloud to enable high-performance use cases with efficiency. Dan Albert from the Solana Foundation talks about simplifying blockchain data with tools like #BigQuery and BigTable and how they enable developers to move and build faster ↓

Google Cloud

254,178 views • 3 years ago

Now you can analyze financial data in Rogo too! 🤯 Years of historical time series data on companies (stock prices, revenue, profit, etc.) 💸 Totally free at 🚀 (Just switch the data source to Finance with the toggle in the top left)

Now you can analyze financial data in Rogo too! 🤯 Years of historical time series data on companies (stock prices, revenue, profit, etc.) 💸 Totally free at 🚀 (Just switch the data source to Finance with the toggle in the top left)

John Willett

30,807 views • 3 years ago