Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Data engineering can be completely open source! You don’t need to rely on many platforms aside from maybe S3! - compute: Spark - data lake format: iceberg - orchestrator: Airflow, Mage or Dagster You can get far with just these three!

Zach Morris Wilson

41,857 subscribers

15,628 views • 1 year ago •via X (Twitter)

Science & Technology Education

Anya Rossi• Live Now

Private livecam show

10 Comments

VJ1 year ago

@mage_ai @dagster For a minute, I thought you were rapping.

The Information1 year ago

OpenAI is betting on a little-known startup to stay ahead of Elon Musk in the supercomputer race.

Data Bass Engineer1 year ago

@mage_ai @dagster I’d rather use open source tools so I can spend my money on Fat chainz

Zach Morris Wilson1 year ago

@mage_ai @dagster Spoken like a true Data Bass Engineer

November1 year ago

@mage_ai @dagster Love the chain bro 😎

Alex Noonan1 year ago

@mage_ai @dagster Dagster

bhavit sharma1 year ago

@mage_ai @dagster nice jacket dude

DataEngDude1 year ago

@mage_ai @dagster death to #iceberg long live #deltalake

Tom1 year ago

@mage_ai @dagster Spark is not compute!

👾1 year ago

@mage_ai @dagster Do you like Dagster?

Related Videos

Friendly reminder that you can run Qwen 3.5 (and other open-source models) on your phone And the app (PocketPal) is: - Completely free - 100% open-source - Available on iOS and Android You can then chat even if you're offline or just don't want to share personal data. Links below

Friendly reminder that you can run Qwen 3.5 (and other open-source models) on your phone And the app (PocketPal) is: - Completely free - 100% open-source - Available on iOS and Android You can then chat even if you're offline or just don't want to share personal data. Links below

Paul Couvert

25,997 views • 3 months ago

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:

Zach Wilson

29,205 views • 1 year ago

🥁 Sync any data source. Whatever context you need can now live in Notion. Our new database sync (powered by Workers) runs on our infrastructure and lets you sync any data source with an API into a Notion database. Sync customer data from Salesforce to build detailed reports. Or connect Strava and Spotify data to curate the perfect running playlist.

🥁 Sync any data source. Whatever context you need can now live in Notion. Our new database sync (powered by Workers) runs on our infrastructure and lets you sync any data source with an API into a Notion database. Sync customer data from Salesforce to build detailed reports. Or connect Strava and Spotify data to curate the perfect running playlist.

Notion

87,608 views • 1 month ago

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:

Karan Vaidya

40,455 views • 6 months ago

With you can create an unlimited number of databases using AI! You can create dummy data or import data from CSV files. You can now deploy the database you built on to Supabase with a click of a button!

With you can create an unlimited number of databases using AI! You can create dummy data or import data from CSV files. You can now deploy the database you built on to Supabase with a click of a button!

Tyler Shukert

94,941 views • 1 year ago

Getting a big tech data engineer job in 2016: - do you know SQL? - yes - here’s $500k Getting a big tech data engineer job in 2024: - do you know Spark, Kafka, Iceberg? - yes - did you shake hands with Bill Inmon when he invented the data warehouse? - no - rejected and ghosted Here’s why! Follow to stay up to date on data engineering!

Getting a big tech data engineer job in 2016: - do you know SQL? - yes - here’s $500k Getting a big tech data engineer job in 2024: - do you know Spark, Kafka, Iceberg? - yes - did you shake hands with Bill Inmon when he invented the data warehouse? - no - rejected and ghosted Here’s why! Follow to stay up to date on data engineering!

Zach Wilson

43,547 views • 1 year ago

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake or Google BigQuery. - update your Python script to load a table in Snowflake/BigQuery - schedule your script with CRON in the cloud with a service like Heroku. - build aggregations and visualizations on top of your ingested data Only thing this misses is data quality and complex job orchestration which you can learn later! How would you learn data engineering nowadays?

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake or Google BigQuery. - update your Python script to load a table in Snowflake/BigQuery - schedule your script with CRON in the cloud with a service like Heroku. - build aggregations and visualizations on top of your ingested data Only thing this misses is data quality and complex job orchestration which you can learn later! How would you learn data engineering nowadays?

Zach Wilson

20,368 views • 1 year ago

❌ Data Science ✅ Data Engineering is the sexiest job of the 21st century Top 7 skills you can master for Data Engineering /🧵/

❌ Data Science ✅ Data Engineering is the sexiest job of the 21st century Top 7 skills you can master for Data Engineering /🧵/

Darshil | Data Engineer👨🏻‍🔧

69,092 views • 3 years ago

On data centers and energy usage, "Data centers don't need the grid, you can put the data center where the power source is." Khosla Ventures’ Vinod Khosla #BloombergGreen ⏯️

On data centers and energy usage, "Data centers don't need the grid, you can put the data center where the power source is." Khosla Ventures’ Vinod Khosla #BloombergGreen ⏯️

Bloomberg Live

38,466 views • 11 months ago

You can be old school, that's completely fine. Maybe just don’t crap on others for living differently. 🤷

You can be old school, that's completely fine. Maybe just don’t crap on others for living differently. 🤷

Larkin Love

16,794 views • 1 year ago

YC W25's superglue (YC W25) translates data into the exact format you need. It's an open source proxy that automatically extracts, maps & transforms data so devs don't have to deal with complex integration code. Congrats on the launch, adina goerres + Stefan Faistenauer!

YC W25's superglue (YC W25) translates data into the exact format you need. It's an open source proxy that automatically extracts, maps & transforms data so devs don't have to deal with complex integration code. Congrats on the launch, adina goerres + Stefan Faistenauer!

Y Combinator

103,140 views • 1 year ago

Let’s be real: Nobody knows what you want, but Smart’s got what you need so you can LIVE FOR REAL, LIVE MORE TODAY 💚 Enjoy the 5G network with Smart! Plus, get all-access data + Unli TikTok with Power All, or experience no expiry data with Magic Data! Get your Smart eSIM now! #LiveForRealWithSmart Belle Mariano Donny

Let’s be real: Nobody knows what you want, but Smart’s got what you need so you can LIVE FOR REAL, LIVE MORE TODAY 💚 Enjoy the 5G network with Smart! Plus, get all-access data + Unli TikTok with Power All, or experience no expiry data with Magic Data! Get your Smart eSIM now! #LiveForRealWithSmart Belle Mariano Donny

SMART

1,679,324 views • 2 years ago

You don't need to be an expert to help light up the world. 💡 Missing electrical grid data hinders everything from building renewables to disaster response. The MapYourGrid open source project lets anyone contribute to an open map of the world's energy grid. This is just one way developers can take climate action. Get the full plan.👇

You don't need to be an expert to help light up the world. 💡 Missing electrical grid data hinders everything from building renewables to disaster response. The MapYourGrid open source project lets anyone contribute to an open map of the world's energy grid. This is just one way developers can take climate action. Get the full plan.👇

GitHub

25,745 views • 9 months ago

Announcing Fire Enrich V2 - our open-source Clay-like data enrichment platform you can run with any LLM! Upload a CSV, spawn sub-agents using @Firecrawl_dev's /search and /scrape to find real-time company data, funding, or any data point. Fork the example today 👇

Announcing Fire Enrich V2 - our open-source Clay-like data enrichment platform you can run with any LLM! Upload a CSV, spawn sub-agents using @Firecrawl_dev's /search and /scrape to find real-time company data, funding, or any data point. Fork the example today 👇

Nicolas Camara

43,766 views • 8 months ago

Lilac (Lilac) is an open-source tool that ensures your data scientists always have enough GPUs for their work. They seamlessly connect compute from any source, on-prem or cloud.

Lilac (Lilac) is an open-source tool that ensures your data scientists always have enough GPUs for their work. They seamlessly connect compute from any source, on-prem or cloud.

Y Combinator

38,348 views • 10 months ago

This is the first example I've seen where subscribing to an AI agent gets you access to proprietary, paid data. SimilarWeb offers just 1 month of data on a free plan. On a Manus plan (free or paid), you can now get 12 months of SimilarWeb data.

This is the first example I've seen where subscribing to an AI agent gets you access to proprietary, paid data. SimilarWeb offers just 1 month of data on a free plan. On a Manus plan (free or paid), you can now get 12 months of SimilarWeb data.

Olivia Moore

287,885 views • 5 months ago

We're currently rebranding APItoolkit to Monoscope (monoscope ). This is because the scope of the platform was beyond API monitoring (Even if it still can do that; 3rd party or own API monitoring). But Monoscope is now a general open telemetry vendor. So, anything that can be instrumented via OpenTelemetry can be monitored using Monoscope. You can also create all sorts of dashboards (and yes, you can use AI to make the charts). And you can have all your data be stored in your own s3 bucket (you own the data. We don't charge for retention), so you can query years of data via the platform. No extra cost. Let me know what you think. Cheers.

We're currently rebranding APItoolkit to Monoscope (monoscope ). This is because the scope of the platform was beyond API monitoring (Even if it still can do that; 3rd party or own API monitoring). But Monoscope is now a general open telemetry vendor. So, anything that can be instrumented via OpenTelemetry can be monitored using Monoscope. You can also create all sorts of dashboards (and yes, you can use AI to make the charts). And you can have all your data be stored in your own s3 bucket (you own the data. We don't charge for retention), so you can query years of data via the platform. No extra cost. Let me know what you think. Cheers.

Anthony Alaribe {···}

17,825 views • 8 months ago

New configurator, fully built on data binding in Rive - set any values (even from external data), calculate the final price, and test image binding (early access) Currently exploring ways to export all configured data, so it can be integrated wherever you need to receive or store it. Built this with Uliana Kalashnikova, supported the data binding side. Try it:

New configurator, fully built on data binding in Rive - set any values (even from external data), calculate the final price, and test image binding (early access) Currently exploring ways to export all configured data, so it can be integrated wherever you need to receive or store it. Built this with Uliana Kalashnikova, supported the data binding side. Try it:

Daria Nevezhyna

63,813 views • 11 months ago

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!

César A. Hidalgo

12,925 views • 1 year ago