Загрузка видео...

Не удалось загрузить видео

На главную

Data engineering can be completely open source! You don’t need to rely on many platforms aside from maybe S3! - compute: Spark - data lake format: iceberg - orchestrator: Airflow, Mage or Dagster You can get far with just these three!

EcZachly's profile picture

Zach Morris Wilson

41,857 subscribers

15,628 просмотров • 1 год назад •via X (Twitter)

Наука и технологииОбразование
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 10

Фото профиля VJ
VJ1 год назад

@mage_ai @dagster For a minute, I thought you were rapping.

Фото профиля The Information
The Information1 год назад

OpenAI is betting on a little-known startup to stay ahead of Elon Musk in the supercomputer race.

Фото профиля Data Bass Engineer
Data Bass Engineer1 год назад

@mage_ai @dagster I’d rather use open source tools so I can spend my money on Fat chainz

Фото профиля Zach Morris Wilson
Zach Morris Wilson1 год назад

@mage_ai @dagster Spoken like a true Data Bass Engineer

Фото профиля November
November1 год назад

@mage_ai @dagster Love the chain bro 😎

Фото профиля Alex Noonan
Alex Noonan1 год назад

@mage_ai @dagster Dagster

Фото профиля bhavit sharma
bhavit sharma1 год назад

@mage_ai @dagster nice jacket dude

Фото профиля DataEngDude
DataEngDude1 год назад

@mage_ai @dagster death to #iceberg long live #deltalake

Фото профиля Tom
Tom1 год назад

@mage_ai @dagster Spark is not compute!

Фото профиля 👾
👾1 год назад

@mage_ai @dagster Do you like Dagster?

Похожие видео

Friendly reminder that you can run Qwen 3.5 (and other open-source models) on your phone And the app (PocketPal) is: - Completely free - 100% open-source - Available on iOS and Android You can then chat even if you're offline or just don't want to share personal data. Links below
0:40

Friendly reminder that you can run Qwen 3.5 (and other open-source models) on your phone And the app (PocketPal) is: - Completely free - 100% open-source - Available on iOS and Android You can then chat even if you're offline or just don't want to share personal data. Links below

Paul Couvert

25,997 просмотров • 3 месяцев назад

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:
1:19

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:

Zach Wilson

29,205 просмотров • 1 год назад

🥁 Sync any data source. Whatever context you need can now live in Notion. Our new database sync (powered by Workers) runs on our infrastructure and lets you sync any data source with an API into a Notion database. Sync customer data from Salesforce to build detailed reports. Or connect Strava and Spotify data to curate the perfect running playlist.
1:41

🥁 Sync any data source. Whatever context you need can now live in Notion. Our new database sync (powered by Workers) runs on our infrastructure and lets you sync any data source with an API into a Notion database. Sync customer data from Salesforce to build detailed reports. Or connect Strava and Spotify data to curate the perfect running playlist.

Notion

87,591 просмотров • 1 месяц назад

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:
1:00

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:

Karan Vaidya

40,455 просмотров • 6 месяцев назад

With you can create an unlimited number of databases using AI! You can create dummy data or import data from CSV files. You can now deploy the database you built on to Supabase with a click of a button!
1:16

With you can create an unlimited number of databases using AI! You can create dummy data or import data from CSV files. You can now deploy the database you built on to Supabase with a click of a button!

Tyler Shukert

94,941 просмотров • 1 год назад

Getting a big tech data engineer job in 2016: - do you know SQL? - yes - here’s $500k Getting a big tech data engineer job in 2024: - do you know Spark, Kafka, Iceberg? - yes - did you shake hands with Bill Inmon when he invented the data warehouse? - no - rejected and ghosted Here’s why! Follow to stay up to date on data engineering!
1:22

Getting a big tech data engineer job in 2016: - do you know SQL? - yes - here’s $500k Getting a big tech data engineer job in 2024: - do you know Spark, Kafka, Iceberg? - yes - did you shake hands with Bill Inmon when he invented the data warehouse? - no - rejected and ghosted Here’s why! Follow to stay up to date on data engineering!

Zach Wilson

43,547 просмотров • 1 год назад

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake or Google BigQuery. - update your Python script to load a table in Snowflake/BigQuery - schedule your script with CRON in the cloud with a service like Heroku. - build aggregations and visualizations on top of your ingested data Only thing this misses is data quality and complex job orchestration which you can learn later! How would you learn data engineering nowadays?
0:51

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake or Google BigQuery. - update your Python script to load a table in Snowflake/BigQuery - schedule your script with CRON in the cloud with a service like Heroku. - build aggregations and visualizations on top of your ingested data Only thing this misses is data quality and complex job orchestration which you can learn later! How would you learn data engineering nowadays?

Zach Wilson

20,368 просмотров • 11 месяцев назад

❌ Data Science ✅ Data Engineering is the sexiest job of the 21st century Top 7 skills you can master for Data Engineering /🧵/
0:27

❌ Data Science ✅ Data Engineering is the sexiest job of the 21st century Top 7 skills you can master for Data Engineering /🧵/

Darshil | Data Engineer👨🏻‍🔧

69,092 просмотров • 3 лет назад

You can be old school, that's completely fine. Maybe just don’t crap on others for living differently. 🤷
2:15

You can be old school, that's completely fine. Maybe just don’t crap on others for living differently. 🤷

Larkin Love

16,794 просмотров • 1 год назад

On data centers and energy usage, "Data centers don't need the grid, you can put the data center where the power source is." Khosla Ventures’ Vinod Khosla #BloombergGreen ⏯️
0:48

On data centers and energy usage, "Data centers don't need the grid, you can put the data center where the power source is." Khosla Ventures’ Vinod Khosla #BloombergGreen ⏯️

Bloomberg Live

38,466 просмотров • 11 месяцев назад

Let’s be real: Nobody knows what you want, but Smart’s got what you need so you can LIVE FOR REAL, LIVE MORE TODAY 💚 Enjoy the 5G network with Smart! Plus, get all-access data + Unli TikTok with Power All, or experience no expiry data with Magic Data! Get your Smart eSIM now! #LiveForRealWithSmart Belle Mariano Donny
1:37

Let’s be real: Nobody knows what you want, but Smart’s got what you need so you can LIVE FOR REAL, LIVE MORE TODAY 💚 Enjoy the 5G network with Smart! Plus, get all-access data + Unli TikTok with Power All, or experience no expiry data with Magic Data! Get your Smart eSIM now! #LiveForRealWithSmart Belle Mariano Donny

SMART

1,679,127 просмотров • 2 лет назад

YC W25's superglue (YC W25) translates data into the exact format you need. It's an open source proxy that automatically extracts, maps & transforms data so devs don't have to deal with complex integration code. Congrats on the launch, adina goerres + Stefan Faistenauer!
1:06

YC W25's superglue (YC W25) translates data into the exact format you need. It's an open source proxy that automatically extracts, maps & transforms data so devs don't have to deal with complex integration code. Congrats on the launch, adina goerres + Stefan Faistenauer!

Y Combinator

103,140 просмотров • 1 год назад

You don't need to be an expert to help light up the world. 💡 Missing electrical grid data hinders everything from building renewables to disaster response. The MapYourGrid open source project lets anyone contribute to an open map of the world's energy grid. This is just one way developers can take climate action. Get the full plan.👇
0:40

You don't need to be an expert to help light up the world. 💡 Missing electrical grid data hinders everything from building renewables to disaster response. The MapYourGrid open source project lets anyone contribute to an open map of the world's energy grid. This is just one way developers can take climate action. Get the full plan.👇

GitHub

25,745 просмотров • 8 месяцев назад

Announcing Fire Enrich V2 - our open-source Clay-like data enrichment platform you can run with any LLM! Upload a CSV, spawn sub-agents using @Firecrawl_dev's /search and /scrape to find real-time company data, funding, or any data point. Fork the example today 👇
0:30

Announcing Fire Enrich V2 - our open-source Clay-like data enrichment platform you can run with any LLM! Upload a CSV, spawn sub-agents using @Firecrawl_dev's /search and /scrape to find real-time company data, funding, or any data point. Fork the example today 👇

Nicolas Camara

43,766 просмотров • 8 месяцев назад

This is the first example I've seen where subscribing to an AI agent gets you access to proprietary, paid data. SimilarWeb offers just 1 month of data on a free plan. On a Manus plan (free or paid), you can now get 12 months of SimilarWeb data.
0:22

This is the first example I've seen where subscribing to an AI agent gets you access to proprietary, paid data. SimilarWeb offers just 1 month of data on a free plan. On a Manus plan (free or paid), you can now get 12 months of SimilarWeb data.

Olivia Moore

287,885 просмотров • 5 месяцев назад

Lilac (Lilac) is an open-source tool that ensures your data scientists always have enough GPUs for their work. They seamlessly connect compute from any source, on-prem or cloud.
1:38

Lilac (Lilac) is an open-source tool that ensures your data scientists always have enough GPUs for their work. They seamlessly connect compute from any source, on-prem or cloud.

Y Combinator

38,348 просмотров • 10 месяцев назад

We're currently rebranding APItoolkit to Monoscope (monoscope ). This is because the scope of the platform was beyond API monitoring (Even if it still can do that; 3rd party or own API monitoring). But Monoscope is now a general open telemetry vendor. So, anything that can be instrumented via OpenTelemetry can be monitored using Monoscope. You can also create all sorts of dashboards (and yes, you can use AI to make the charts). And you can have all your data be stored in your own s3 bucket (you own the data. We don't charge for retention), so you can query years of data via the platform. No extra cost. Let me know what you think. Cheers.
4:23

We're currently rebranding APItoolkit to Monoscope (monoscope ). This is because the scope of the platform was beyond API monitoring (Even if it still can do that; 3rd party or own API monitoring). But Monoscope is now a general open telemetry vendor. So, anything that can be instrumented via OpenTelemetry can be monitored using Monoscope. You can also create all sorts of dashboards (and yes, you can use AI to make the charts). And you can have all your data be stored in your own s3 bucket (you own the data. We don't charge for retention), so you can query years of data via the platform. No extra cost. Let me know what you think. Cheers.

Anthony Alaribe {···}

17,825 просмотров • 8 месяцев назад

New configurator, fully built on data binding in Rive - set any values (even from external data), calculate the final price, and test image binding (early access) Currently exploring ways to export all configured data, so it can be integrated wherever you need to receive or store it. Built this with Uliana Kalashnikova, supported the data binding side. Try it:
0:28

New configurator, fully built on data binding in Rive - set any values (even from external data), calculate the final price, and test image binding (early access) Currently exploring ways to export all configured data, so it can be integrated wherever you need to receive or store it. Built this with Uliana Kalashnikova, supported the data binding side. Try it:

Daria Nevezhyna

63,813 просмотров • 11 месяцев назад

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!
1:00

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!

César A. Hidalgo

12,925 просмотров • 1 год назад