正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Data engineering can be completely open source! You don’t need to rely on many platforms aside from maybe S3! - compute: Spark - data lake format: iceberg - orchestrator: Airflow, Mage or Dagster You can get far with just these three!

Zach Morris Wilson

41,857 subscribers

15,628 次观看 • 1 年前 •via X (Twitter)

科学技术教育

Anya Rossi• Live Now

Private livecam show

10 条评论

VJ 的头像

VJ1 年前

@mage_ai @dagster For a minute, I thought you were rapping.

The Information 的头像

The Information1 年前

OpenAI is betting on a little-known startup to stay ahead of Elon Musk in the supercomputer race.

Data Bass Engineer 的头像

Data Bass Engineer1 年前

@mage_ai @dagster I’d rather use open source tools so I can spend my money on Fat chainz

Zach Morris Wilson 的头像

Zach Morris Wilson1 年前

@mage_ai @dagster Spoken like a true Data Bass Engineer

November 的头像

November1 年前

@mage_ai @dagster Love the chain bro 😎

Alex Noonan 的头像

Alex Noonan1 年前

@mage_ai @dagster Dagster

bhavit sharma 的头像

bhavit sharma1 年前

@mage_ai @dagster nice jacket dude

DataEngDude 的头像

DataEngDude1 年前

@mage_ai @dagster death to #iceberg long live #deltalake

Tom 的头像

Tom1 年前

@mage_ai @dagster Spark is not compute!

👾 的头像

👾1 年前

@mage_ai @dagster Do you like Dagster?

相关视频

Friendly reminder that you can run Qwen 3.5 (and other open-source models) on your phone And the app (PocketPal) is: - Completely free - 100% open-source - Available on iOS and Android You can then chat even if you're offline or just don't want to share personal data. Links below

Friendly reminder that you can run Qwen 3.5 (and other open-source models) on your phone And the app (PocketPal) is: - Completely free - 100% open-source - Available on iOS and Android You can then chat even if you're offline or just don't want to share personal data. Links below

Paul Couvert

26,037 次观看 • 3 个月前

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:

Zach Wilson

29,205 次观看 • 1 年前

🥁 Sync any data source. Whatever context you need can now live in Notion. Our new database sync (powered by Workers) runs on our infrastructure and lets you sync any data source with an API into a Notion database. Sync customer data from Salesforce to build detailed reports. Or connect Strava and Spotify data to curate the perfect running playlist.

🥁 Sync any data source. Whatever context you need can now live in Notion. Our new database sync (powered by Workers) runs on our infrastructure and lets you sync any data source with an API into a Notion database. Sync customer data from Salesforce to build detailed reports. Or connect Strava and Spotify data to curate the perfect running playlist.

Notion

87,608 次观看 • 1 个月前

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:

Karan Vaidya

40,455 次观看 • 6 个月前

With you can create an unlimited number of databases using AI! You can create dummy data or import data from CSV files. You can now deploy the database you built on to Supabase with a click of a button!

With you can create an unlimited number of databases using AI! You can create dummy data or import data from CSV files. You can now deploy the database you built on to Supabase with a click of a button!

Tyler Shukert

94,941 次观看 • 1 年前

Getting a big tech data engineer job in 2016: - do you know SQL? - yes - here’s $500k Getting a big tech data engineer job in 2024: - do you know Spark, Kafka, Iceberg? - yes - did you shake hands with Bill Inmon when he invented the data warehouse? - no - rejected and ghosted Here’s why! Follow to stay up to date on data engineering!

Getting a big tech data engineer job in 2016: - do you know SQL? - yes - here’s $500k Getting a big tech data engineer job in 2024: - do you know Spark, Kafka, Iceberg? - yes - did you shake hands with Bill Inmon when he invented the data warehouse? - no - rejected and ghosted Here’s why! Follow to stay up to date on data engineering!

Zach Wilson

43,547 次观看 • 1 年前

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake or Google BigQuery. - update your Python script to load a table in Snowflake/BigQuery - schedule your script with CRON in the cloud with a service like Heroku. - build aggregations and visualizations on top of your ingested data Only thing this misses is data quality and complex job orchestration which you can learn later! How would you learn data engineering nowadays?

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake or Google BigQuery. - update your Python script to load a table in Snowflake/BigQuery - schedule your script with CRON in the cloud with a service like Heroku. - build aggregations and visualizations on top of your ingested data Only thing this misses is data quality and complex job orchestration which you can learn later! How would you learn data engineering nowadays?

Zach Wilson

20,368 次观看 • 1 年前

❌ Data Science ✅ Data Engineering is the sexiest job of the 21st century Top 7 skills you can master for Data Engineering /🧵/

❌ Data Science ✅ Data Engineering is the sexiest job of the 21st century Top 7 skills you can master for Data Engineering /🧵/

Darshil | Data Engineer👨🏻‍🔧

69,092 次观看 • 3 年前

You can be old school, that's completely fine. Maybe just don’t crap on others for living differently. 🤷

You can be old school, that's completely fine. Maybe just don’t crap on others for living differently. 🤷

Larkin Love

16,794 次观看 • 1 年前

On data centers and energy usage, "Data centers don't need the grid, you can put the data center where the power source is." Khosla Ventures’ Vinod Khosla #BloombergGreen ⏯️

On data centers and energy usage, "Data centers don't need the grid, you can put the data center where the power source is." Khosla Ventures’ Vinod Khosla #BloombergGreen ⏯️

Bloomberg Live

38,466 次观看 • 11 个月前

Let’s be real: Nobody knows what you want, but Smart’s got what you need so you can LIVE FOR REAL, LIVE MORE TODAY 💚 Enjoy the 5G network with Smart! Plus, get all-access data + Unli TikTok with Power All, or experience no expiry data with Magic Data! Get your Smart eSIM now! #LiveForRealWithSmart Belle Mariano Donny

Let’s be real: Nobody knows what you want, but Smart’s got what you need so you can LIVE FOR REAL, LIVE MORE TODAY 💚 Enjoy the 5G network with Smart! Plus, get all-access data + Unli TikTok with Power All, or experience no expiry data with Magic Data! Get your Smart eSIM now! #LiveForRealWithSmart Belle Mariano Donny

SMART

1,679,324 次观看 • 2 年前

YC W25's superglue (YC W25) translates data into the exact format you need. It's an open source proxy that automatically extracts, maps & transforms data so devs don't have to deal with complex integration code. Congrats on the launch, adina goerres + Stefan Faistenauer!

YC W25's superglue (YC W25) translates data into the exact format you need. It's an open source proxy that automatically extracts, maps & transforms data so devs don't have to deal with complex integration code. Congrats on the launch, adina goerres + Stefan Faistenauer!

Y Combinator

103,140 次观看 • 1 年前

You don't need to be an expert to help light up the world. 💡 Missing electrical grid data hinders everything from building renewables to disaster response. The MapYourGrid open source project lets anyone contribute to an open map of the world's energy grid. This is just one way developers can take climate action. Get the full plan.👇

You don't need to be an expert to help light up the world. 💡 Missing electrical grid data hinders everything from building renewables to disaster response. The MapYourGrid open source project lets anyone contribute to an open map of the world's energy grid. This is just one way developers can take climate action. Get the full plan.👇

GitHub

25,745 次观看 • 9 个月前

Announcing Fire Enrich V2 - our open-source Clay-like data enrichment platform you can run with any LLM! Upload a CSV, spawn sub-agents using @Firecrawl_dev's /search and /scrape to find real-time company data, funding, or any data point. Fork the example today 👇

Announcing Fire Enrich V2 - our open-source Clay-like data enrichment platform you can run with any LLM! Upload a CSV, spawn sub-agents using @Firecrawl_dev's /search and /scrape to find real-time company data, funding, or any data point. Fork the example today 👇

Nicolas Camara

43,766 次观看 • 8 个月前

Lilac (Lilac) is an open-source tool that ensures your data scientists always have enough GPUs for their work. They seamlessly connect compute from any source, on-prem or cloud.

Lilac (Lilac) is an open-source tool that ensures your data scientists always have enough GPUs for their work. They seamlessly connect compute from any source, on-prem or cloud.

Y Combinator

38,348 次观看 • 10 个月前

This is the first example I've seen where subscribing to an AI agent gets you access to proprietary, paid data. SimilarWeb offers just 1 month of data on a free plan. On a Manus plan (free or paid), you can now get 12 months of SimilarWeb data.

This is the first example I've seen where subscribing to an AI agent gets you access to proprietary, paid data. SimilarWeb offers just 1 month of data on a free plan. On a Manus plan (free or paid), you can now get 12 months of SimilarWeb data.

Olivia Moore

287,901 次观看 • 5 个月前

We're currently rebranding APItoolkit to Monoscope (monoscope ). This is because the scope of the platform was beyond API monitoring (Even if it still can do that; 3rd party or own API monitoring). But Monoscope is now a general open telemetry vendor. So, anything that can be instrumented via OpenTelemetry can be monitored using Monoscope. You can also create all sorts of dashboards (and yes, you can use AI to make the charts). And you can have all your data be stored in your own s3 bucket (you own the data. We don't charge for retention), so you can query years of data via the platform. No extra cost. Let me know what you think. Cheers.

We're currently rebranding APItoolkit to Monoscope (monoscope ). This is because the scope of the platform was beyond API monitoring (Even if it still can do that; 3rd party or own API monitoring). But Monoscope is now a general open telemetry vendor. So, anything that can be instrumented via OpenTelemetry can be monitored using Monoscope. You can also create all sorts of dashboards (and yes, you can use AI to make the charts). And you can have all your data be stored in your own s3 bucket (you own the data. We don't charge for retention), so you can query years of data via the platform. No extra cost. Let me know what you think. Cheers.

Anthony Alaribe {···}

17,825 次观看 • 8 个月前

New configurator, fully built on data binding in Rive - set any values (even from external data), calculate the final price, and test image binding (early access) Currently exploring ways to export all configured data, so it can be integrated wherever you need to receive or store it. Built this with Uliana Kalashnikova, supported the data binding side. Try it:

New configurator, fully built on data binding in Rive - set any values (even from external data), calculate the final price, and test image binding (early access) Currently exploring ways to export all configured data, so it can be integrated wherever you need to receive or store it. Built this with Uliana Kalashnikova, supported the data binding side. Try it:

Daria Nevezhyna

63,813 次观看 • 11 个月前

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!

César A. Hidalgo

12,925 次观看 • 1 年前