Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Data engineering can be completely open source! You don’t need to rely on many platforms aside from maybe S3! - compute: Spark - data lake format: iceberg - orchestrator: Airflow, Mage or Dagster You can get far with just these three!

EcZachly's profile picture

Zach Morris Wilson

41,857 subscribers

15,628 Aufrufe • vor 1 Jahr •via X (Twitter)

Wissenschaft & TechnologieBildung
LIVE

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von VJ
VJvor 1 Jahr

@mage_ai @dagster For a minute, I thought you were rapping.

Profilbild von The Information
The Informationvor 1 Jahr

OpenAI is betting on a little-known startup to stay ahead of Elon Musk in the supercomputer race.

Profilbild von Data Bass Engineer
Data Bass Engineervor 1 Jahr

@mage_ai @dagster I’d rather use open source tools so I can spend my money on Fat chainz

Profilbild von Zach Morris Wilson
Zach Morris Wilsonvor 1 Jahr

@mage_ai @dagster Spoken like a true Data Bass Engineer

Profilbild von November
Novembervor 1 Jahr

@mage_ai @dagster Love the chain bro 😎

Profilbild von Alex Noonan
Alex Noonanvor 1 Jahr

@mage_ai @dagster Dagster

Profilbild von bhavit sharma
bhavit sharmavor 1 Jahr

@mage_ai @dagster nice jacket dude

Profilbild von DataEngDude
DataEngDudevor 1 Jahr

@mage_ai @dagster death to #iceberg long live #deltalake

Profilbild von Tom
Tomvor 1 Jahr

@mage_ai @dagster Spark is not compute!

Profilbild von 👾
👾vor 1 Jahr

@mage_ai @dagster Do you like Dagster?

Ähnliche Videos

Friendly reminder that you can run Qwen 3.5 (and other open-source models) on your phone And the app (PocketPal) is: - Completely free - 100% open-source - Available on iOS and Android You can then chat even if you're offline or just don't want to share personal data. Links below
0:40

Friendly reminder that you can run Qwen 3.5 (and other open-source models) on your phone And the app (PocketPal) is: - Completely free - 100% open-source - Available on iOS and Android You can then chat even if you're offline or just don't want to share personal data. Links below

Paul Couvert

25,997 Aufrufe • vor 3 Monaten

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:
1:19

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:

Zach Wilson

29,164 Aufrufe • vor 11 Monaten

🥁 Sync any data source. Whatever context you need can now live in Notion. Our new database sync (powered by Workers) runs on our infrastructure and lets you sync any data source with an API into a Notion database. Sync customer data from Salesforce to build detailed reports. Or connect Strava and Spotify data to curate the perfect running playlist.
1:41

🥁 Sync any data source. Whatever context you need can now live in Notion. Our new database sync (powered by Workers) runs on our infrastructure and lets you sync any data source with an API into a Notion database. Sync customer data from Salesforce to build detailed reports. Or connect Strava and Spotify data to curate the perfect running playlist.

Notion

87,283 Aufrufe • vor 28 Tagen

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:
1:00

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:

Karan Vaidya

40,455 Aufrufe • vor 5 Monaten

With you can create an unlimited number of databases using AI! You can create dummy data or import data from CSV files. You can now deploy the database you built on to Supabase with a click of a button!
1:16

With you can create an unlimited number of databases using AI! You can create dummy data or import data from CSV files. You can now deploy the database you built on to Supabase with a click of a button!

Tyler Shukert

94,941 Aufrufe • vor 1 Jahr

Getting a big tech data engineer job in 2016: - do you know SQL? - yes - here’s $500k Getting a big tech data engineer job in 2024: - do you know Spark, Kafka, Iceberg? - yes - did you shake hands with Bill Inmon when he invented the data warehouse? - no - rejected and ghosted Here’s why! Follow to stay up to date on data engineering!
1:22

Getting a big tech data engineer job in 2016: - do you know SQL? - yes - here’s $500k Getting a big tech data engineer job in 2024: - do you know Spark, Kafka, Iceberg? - yes - did you shake hands with Bill Inmon when he invented the data warehouse? - no - rejected and ghosted Here’s why! Follow to stay up to date on data engineering!

Zach Wilson

43,547 Aufrufe • vor 1 Jahr

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake or Google BigQuery. - update your Python script to load a table in Snowflake/BigQuery - schedule your script with CRON in the cloud with a service like Heroku. - build aggregations and visualizations on top of your ingested data Only thing this misses is data quality and complex job orchestration which you can learn later! How would you learn data engineering nowadays?
0:51

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake or Google BigQuery. - update your Python script to load a table in Snowflake/BigQuery - schedule your script with CRON in the cloud with a service like Heroku. - build aggregations and visualizations on top of your ingested data Only thing this misses is data quality and complex job orchestration which you can learn later! How would you learn data engineering nowadays?

Zach Wilson

20,363 Aufrufe • vor 11 Monaten

❌ Data Science ✅ Data Engineering is the sexiest job of the 21st century Top 7 skills you can master for Data Engineering /🧵/
0:27

❌ Data Science ✅ Data Engineering is the sexiest job of the 21st century Top 7 skills you can master for Data Engineering /🧵/

Darshil | Data Engineer👨🏻‍🔧

69,092 Aufrufe • vor 3 Jahren

You can be old school, that's completely fine. Maybe just don’t crap on others for living differently. 🤷
2:15

You can be old school, that's completely fine. Maybe just don’t crap on others for living differently. 🤷

Larkin Love

16,794 Aufrufe • vor 11 Monaten

On data centers and energy usage, "Data centers don't need the grid, you can put the data center where the power source is." Khosla Ventures’ Vinod Khosla #BloombergGreen ⏯️
0:48

On data centers and energy usage, "Data centers don't need the grid, you can put the data center where the power source is." Khosla Ventures’ Vinod Khosla #BloombergGreen ⏯️

Bloomberg Live

38,466 Aufrufe • vor 11 Monaten

Let’s be real: Nobody knows what you want, but Smart’s got what you need so you can LIVE FOR REAL, LIVE MORE TODAY 💚 Enjoy the 5G network with Smart! Plus, get all-access data + Unli TikTok with Power All, or experience no expiry data with Magic Data! Get your Smart eSIM now! #LiveForRealWithSmart Belle Mariano Donny
1:37

Let’s be real: Nobody knows what you want, but Smart’s got what you need so you can LIVE FOR REAL, LIVE MORE TODAY 💚 Enjoy the 5G network with Smart! Plus, get all-access data + Unli TikTok with Power All, or experience no expiry data with Magic Data! Get your Smart eSIM now! #LiveForRealWithSmart Belle Mariano Donny

SMART

1,679,127 Aufrufe • vor 2 Jahren

YC W25's superglue (YC W25) translates data into the exact format you need. It's an open source proxy that automatically extracts, maps & transforms data so devs don't have to deal with complex integration code. Congrats on the launch, adina goerres + Stefan Faistenauer!
1:06

YC W25's superglue (YC W25) translates data into the exact format you need. It's an open source proxy that automatically extracts, maps & transforms data so devs don't have to deal with complex integration code. Congrats on the launch, adina goerres + Stefan Faistenauer!

Y Combinator

103,140 Aufrufe • vor 1 Jahr

You don't need to be an expert to help light up the world. 💡 Missing electrical grid data hinders everything from building renewables to disaster response. The MapYourGrid open source project lets anyone contribute to an open map of the world's energy grid. This is just one way developers can take climate action. Get the full plan.👇
0:40

You don't need to be an expert to help light up the world. 💡 Missing electrical grid data hinders everything from building renewables to disaster response. The MapYourGrid open source project lets anyone contribute to an open map of the world's energy grid. This is just one way developers can take climate action. Get the full plan.👇

GitHub

25,745 Aufrufe • vor 8 Monaten

Announcing Fire Enrich V2 - our open-source Clay-like data enrichment platform you can run with any LLM! Upload a CSV, spawn sub-agents using @Firecrawl_dev's /search and /scrape to find real-time company data, funding, or any data point. Fork the example today 👇
0:30

Announcing Fire Enrich V2 - our open-source Clay-like data enrichment platform you can run with any LLM! Upload a CSV, spawn sub-agents using @Firecrawl_dev's /search and /scrape to find real-time company data, funding, or any data point. Fork the example today 👇

Nicolas Camara

43,766 Aufrufe • vor 8 Monaten

Lilac (Lilac) is an open-source tool that ensures your data scientists always have enough GPUs for their work. They seamlessly connect compute from any source, on-prem or cloud.
1:38

Lilac (Lilac) is an open-source tool that ensures your data scientists always have enough GPUs for their work. They seamlessly connect compute from any source, on-prem or cloud.

Y Combinator

38,348 Aufrufe • vor 10 Monaten

This is the first example I've seen where subscribing to an AI agent gets you access to proprietary, paid data. SimilarWeb offers just 1 month of data on a free plan. On a Manus plan (free or paid), you can now get 12 months of SimilarWeb data.
0:22

This is the first example I've seen where subscribing to an AI agent gets you access to proprietary, paid data. SimilarWeb offers just 1 month of data on a free plan. On a Manus plan (free or paid), you can now get 12 months of SimilarWeb data.

Olivia Moore

287,807 Aufrufe • vor 4 Monaten

We're currently rebranding APItoolkit to Monoscope (monoscope ). This is because the scope of the platform was beyond API monitoring (Even if it still can do that; 3rd party or own API monitoring). But Monoscope is now a general open telemetry vendor. So, anything that can be instrumented via OpenTelemetry can be monitored using Monoscope. You can also create all sorts of dashboards (and yes, you can use AI to make the charts). And you can have all your data be stored in your own s3 bucket (you own the data. We don't charge for retention), so you can query years of data via the platform. No extra cost. Let me know what you think. Cheers.
4:23

We're currently rebranding APItoolkit to Monoscope (monoscope ). This is because the scope of the platform was beyond API monitoring (Even if it still can do that; 3rd party or own API monitoring). But Monoscope is now a general open telemetry vendor. So, anything that can be instrumented via OpenTelemetry can be monitored using Monoscope. You can also create all sorts of dashboards (and yes, you can use AI to make the charts). And you can have all your data be stored in your own s3 bucket (you own the data. We don't charge for retention), so you can query years of data via the platform. No extra cost. Let me know what you think. Cheers.

Anthony Alaribe {···}

17,825 Aufrufe • vor 7 Monaten

New configurator, fully built on data binding in Rive - set any values (even from external data), calculate the final price, and test image binding (early access) Currently exploring ways to export all configured data, so it can be integrated wherever you need to receive or store it. Built this with Uliana Kalashnikova, supported the data binding side. Try it:
0:28

New configurator, fully built on data binding in Rive - set any values (even from external data), calculate the final price, and test image binding (early access) Currently exploring ways to export all configured data, so it can be integrated wherever you need to receive or store it. Built this with Uliana Kalashnikova, supported the data binding side. Try it:

Daria Nevezhyna

63,813 Aufrufe • vor 11 Monaten

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!
1:00

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!

César A. Hidalgo

12,925 Aufrufe • vor 11 Monaten