Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Data engineering can be completely open source! You don’t need to rely on many platforms aside from maybe S3! - compute: Spark - data lake format: iceberg - orchestrator: Airflow, Mage or Dagster You can get far with just these three!

EcZachly's profile picture

Zach Morris Wilson

41,857 subscribers

15,628 görüntüleme • 1 yıl önce •via X (Twitter)

Bilim & TeknolojiEğitim
LIVE

Anya Rossi• Live Now

Private livecam show

10 Yorum

VJ profil fotoğrafı
VJ1 yıl önce

@mage_ai @dagster For a minute, I thought you were rapping.

The Information profil fotoğrafı
The Information1 yıl önce

OpenAI is betting on a little-known startup to stay ahead of Elon Musk in the supercomputer race.

Data Bass Engineer profil fotoğrafı
Data Bass Engineer1 yıl önce

@mage_ai @dagster I’d rather use open source tools so I can spend my money on Fat chainz

Zach Morris Wilson profil fotoğrafı
Zach Morris Wilson1 yıl önce

@mage_ai @dagster Spoken like a true Data Bass Engineer

November profil fotoğrafı
November1 yıl önce

@mage_ai @dagster Love the chain bro 😎

Alex Noonan profil fotoğrafı
Alex Noonan1 yıl önce

@mage_ai @dagster Dagster

bhavit sharma profil fotoğrafı
bhavit sharma1 yıl önce

@mage_ai @dagster nice jacket dude

DataEngDude profil fotoğrafı
DataEngDude1 yıl önce

@mage_ai @dagster death to #iceberg long live #deltalake

Tom profil fotoğrafı
Tom1 yıl önce

@mage_ai @dagster Spark is not compute!

👾 profil fotoğrafı
👾1 yıl önce

@mage_ai @dagster Do you like Dagster?

Benzer Videolar

Friendly reminder that you can run Qwen 3.5 (and other open-source models) on your phone And the app (PocketPal) is: - Completely free - 100% open-source - Available on iOS and Android You can then chat even if you're offline or just don't want to share personal data. Links below
0:40

Friendly reminder that you can run Qwen 3.5 (and other open-source models) on your phone And the app (PocketPal) is: - Completely free - 100% open-source - Available on iOS and Android You can then chat even if you're offline or just don't want to share personal data. Links below

Paul Couvert

25,997 görüntüleme • 3 ay önce

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:
1:19

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:

Zach Wilson

29,205 görüntüleme • 1 yıl önce

🥁 Sync any data source. Whatever context you need can now live in Notion. Our new database sync (powered by Workers) runs on our infrastructure and lets you sync any data source with an API into a Notion database. Sync customer data from Salesforce to build detailed reports. Or connect Strava and Spotify data to curate the perfect running playlist.
1:41

🥁 Sync any data source. Whatever context you need can now live in Notion. Our new database sync (powered by Workers) runs on our infrastructure and lets you sync any data source with an API into a Notion database. Sync customer data from Salesforce to build detailed reports. Or connect Strava and Spotify data to curate the perfect running playlist.

Notion

87,476 görüntüleme • 1 ay önce

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:
1:00

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:

Karan Vaidya

40,455 görüntüleme • 6 ay önce

With you can create an unlimited number of databases using AI! You can create dummy data or import data from CSV files. You can now deploy the database you built on to Supabase with a click of a button!
1:16

With you can create an unlimited number of databases using AI! You can create dummy data or import data from CSV files. You can now deploy the database you built on to Supabase with a click of a button!

Tyler Shukert

94,941 görüntüleme • 1 yıl önce

Getting a big tech data engineer job in 2016: - do you know SQL? - yes - here’s $500k Getting a big tech data engineer job in 2024: - do you know Spark, Kafka, Iceberg? - yes - did you shake hands with Bill Inmon when he invented the data warehouse? - no - rejected and ghosted Here’s why! Follow to stay up to date on data engineering!
1:22

Getting a big tech data engineer job in 2016: - do you know SQL? - yes - here’s $500k Getting a big tech data engineer job in 2024: - do you know Spark, Kafka, Iceberg? - yes - did you shake hands with Bill Inmon when he invented the data warehouse? - no - rejected and ghosted Here’s why! Follow to stay up to date on data engineering!

Zach Wilson

43,547 görüntüleme • 1 yıl önce

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake or Google BigQuery. - update your Python script to load a table in Snowflake/BigQuery - schedule your script with CRON in the cloud with a service like Heroku. - build aggregations and visualizations on top of your ingested data Only thing this misses is data quality and complex job orchestration which you can learn later! How would you learn data engineering nowadays?
0:51

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake or Google BigQuery. - update your Python script to load a table in Snowflake/BigQuery - schedule your script with CRON in the cloud with a service like Heroku. - build aggregations and visualizations on top of your ingested data Only thing this misses is data quality and complex job orchestration which you can learn later! How would you learn data engineering nowadays?

Zach Wilson

20,368 görüntüleme • 11 ay önce

❌ Data Science ✅ Data Engineering is the sexiest job of the 21st century Top 7 skills you can master for Data Engineering /🧵/
0:27

❌ Data Science ✅ Data Engineering is the sexiest job of the 21st century Top 7 skills you can master for Data Engineering /🧵/

Darshil | Data Engineer👨🏻‍🔧

69,092 görüntüleme • 3 yıl önce

On data centers and energy usage, "Data centers don't need the grid, you can put the data center where the power source is." Khosla Ventures’ Vinod Khosla #BloombergGreen ⏯️
0:48

On data centers and energy usage, "Data centers don't need the grid, you can put the data center where the power source is." Khosla Ventures’ Vinod Khosla #BloombergGreen ⏯️

Bloomberg Live

38,466 görüntüleme • 11 ay önce

You can be old school, that's completely fine. Maybe just don’t crap on others for living differently. 🤷
2:15

You can be old school, that's completely fine. Maybe just don’t crap on others for living differently. 🤷

Larkin Love

16,794 görüntüleme • 1 yıl önce

Let’s be real: Nobody knows what you want, but Smart’s got what you need so you can LIVE FOR REAL, LIVE MORE TODAY 💚 Enjoy the 5G network with Smart! Plus, get all-access data + Unli TikTok with Power All, or experience no expiry data with Magic Data! Get your Smart eSIM now! #LiveForRealWithSmart Belle Mariano Donny
1:37

Let’s be real: Nobody knows what you want, but Smart’s got what you need so you can LIVE FOR REAL, LIVE MORE TODAY 💚 Enjoy the 5G network with Smart! Plus, get all-access data + Unli TikTok with Power All, or experience no expiry data with Magic Data! Get your Smart eSIM now! #LiveForRealWithSmart Belle Mariano Donny

SMART

1,679,127 görüntüleme • 2 yıl önce

YC W25's superglue (YC W25) translates data into the exact format you need. It's an open source proxy that automatically extracts, maps & transforms data so devs don't have to deal with complex integration code. Congrats on the launch, adina goerres + Stefan Faistenauer!
1:06

YC W25's superglue (YC W25) translates data into the exact format you need. It's an open source proxy that automatically extracts, maps & transforms data so devs don't have to deal with complex integration code. Congrats on the launch, adina goerres + Stefan Faistenauer!

Y Combinator

103,140 görüntüleme • 1 yıl önce

You don't need to be an expert to help light up the world. 💡 Missing electrical grid data hinders everything from building renewables to disaster response. The MapYourGrid open source project lets anyone contribute to an open map of the world's energy grid. This is just one way developers can take climate action. Get the full plan.👇
0:40

You don't need to be an expert to help light up the world. 💡 Missing electrical grid data hinders everything from building renewables to disaster response. The MapYourGrid open source project lets anyone contribute to an open map of the world's energy grid. This is just one way developers can take climate action. Get the full plan.👇

GitHub

25,745 görüntüleme • 8 ay önce

Announcing Fire Enrich V2 - our open-source Clay-like data enrichment platform you can run with any LLM! Upload a CSV, spawn sub-agents using @Firecrawl_dev's /search and /scrape to find real-time company data, funding, or any data point. Fork the example today 👇
0:30

Announcing Fire Enrich V2 - our open-source Clay-like data enrichment platform you can run with any LLM! Upload a CSV, spawn sub-agents using @Firecrawl_dev's /search and /scrape to find real-time company data, funding, or any data point. Fork the example today 👇

Nicolas Camara

43,766 görüntüleme • 8 ay önce

This is the first example I've seen where subscribing to an AI agent gets you access to proprietary, paid data. SimilarWeb offers just 1 month of data on a free plan. On a Manus plan (free or paid), you can now get 12 months of SimilarWeb data.
0:22

This is the first example I've seen where subscribing to an AI agent gets you access to proprietary, paid data. SimilarWeb offers just 1 month of data on a free plan. On a Manus plan (free or paid), you can now get 12 months of SimilarWeb data.

Olivia Moore

287,885 görüntüleme • 5 ay önce

Lilac (Lilac) is an open-source tool that ensures your data scientists always have enough GPUs for their work. They seamlessly connect compute from any source, on-prem or cloud.
1:38

Lilac (Lilac) is an open-source tool that ensures your data scientists always have enough GPUs for their work. They seamlessly connect compute from any source, on-prem or cloud.

Y Combinator

38,348 görüntüleme • 10 ay önce

We're currently rebranding APItoolkit to Monoscope (monoscope ). This is because the scope of the platform was beyond API monitoring (Even if it still can do that; 3rd party or own API monitoring). But Monoscope is now a general open telemetry vendor. So, anything that can be instrumented via OpenTelemetry can be monitored using Monoscope. You can also create all sorts of dashboards (and yes, you can use AI to make the charts). And you can have all your data be stored in your own s3 bucket (you own the data. We don't charge for retention), so you can query years of data via the platform. No extra cost. Let me know what you think. Cheers.
4:23

We're currently rebranding APItoolkit to Monoscope (monoscope ). This is because the scope of the platform was beyond API monitoring (Even if it still can do that; 3rd party or own API monitoring). But Monoscope is now a general open telemetry vendor. So, anything that can be instrumented via OpenTelemetry can be monitored using Monoscope. You can also create all sorts of dashboards (and yes, you can use AI to make the charts). And you can have all your data be stored in your own s3 bucket (you own the data. We don't charge for retention), so you can query years of data via the platform. No extra cost. Let me know what you think. Cheers.

Anthony Alaribe {···}

17,825 görüntüleme • 8 ay önce

New configurator, fully built on data binding in Rive - set any values (even from external data), calculate the final price, and test image binding (early access) Currently exploring ways to export all configured data, so it can be integrated wherever you need to receive or store it. Built this with Uliana Kalashnikova, supported the data binding side. Try it:
0:28

New configurator, fully built on data binding in Rive - set any values (even from external data), calculate the final price, and test image binding (early access) Currently exploring ways to export all configured data, so it can be integrated wherever you need to receive or store it. Built this with Uliana Kalashnikova, supported the data binding side. Try it:

Daria Nevezhyna

63,813 görüntüleme • 11 ay önce

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!
1:00

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!

César A. Hidalgo

12,925 görüntüleme • 1 yıl önce