Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

What if you could instantly access & analyze all the dYdX data in a reliable way? KYVE makes this happen ⚡ Recently, KYVE hosted a developer workshop providing an overview of how dYdX users can leverage KYVE’s tooling for an enhanced data & overall building experience. KYVE’s Data Pipeline... show more

KYVE 💫

69,644 subscribers

16,289 просмотров • 1 год назад •via X (Twitter)

Наука и технологии Образование Финансы

Anya Rossi• Live Now

Private livecam show

Комментарии: 2

Фото профиля KYVE 💫

KYVE 💫1 год назад

ICYMI, here’s the full replay of KYVE’s latest Dev Workshop for @dYdX nodes, devs, data analysts, & other data users ⤵️

Фото профиля denbecha

denbecha1 год назад

@dYdX Get ready for $ISLM to smash through all-time highs!

Похожие видео

⚡️ What if you could sync your node to any dYdX block height in seconds? Look no further than KYVE’s KSYNC! 🤝 No more unstable peer-to-peer connections or missing archive nodes, KSYNC enables seamless access to historical blocks, state snapshots, or full blockchain history of the dYdX chain through KYVE. Here’s how dYdX users can use KSYNC ⤵️

⚡️ What if you could sync your node to any dYdX block height in seconds? Look no further than KYVE’s KSYNC! 🤝 No more unstable peer-to-peer connections or missing archive nodes, KSYNC enables seamless access to historical blocks, state snapshots, or full blockchain history of the dYdX chain through KYVE. Here’s how dYdX users can use KSYNC ⤵️

KYVE 💫

39,035 просмотров • 1 год назад

"If you're an analytical SaaS company, it's over. It's over." - Palo Alto Networks CEO Nikesh Arora "What is an analytical SaaS company? It is a company that says, “I'm going to collect a lot of data for you and analyze it for you.” I don't need you to analyze it for me. I can run models against the data and analyze it myself. Think about it: almost every SaaS company has a marketplace. You can buy applications from the Salesforce Marketplace. What do those applications say? “You have Salesforce data. I'm a marketplace app. Use me, and I'll help you analyze that data.” But I don't need to do that. You don't need those applications anymore. I can simply run an LLM against the data. Therefore, all the additional functionality that has traditionally been sold to us as incremental software modules may no longer need to be sold separately. I would much rather have an LLM run against the data." ---- From "All-In Podcast" YouTube channel, (full video link in comment)

"If you're an analytical SaaS company, it's over. It's over." - Palo Alto Networks CEO Nikesh Arora "What is an analytical SaaS company? It is a company that says, “I'm going to collect a lot of data for you and analyze it for you.” I don't need you to analyze it for me. I can run models against the data and analyze it myself. Think about it: almost every SaaS company has a marketplace. You can buy applications from the Salesforce Marketplace. What do those applications say? “You have Salesforce data. I'm a marketplace app. Use me, and I'll help you analyze that data.” But I don't need to do that. You don't need those applications anymore. I can simply run an LLM against the data. Therefore, all the additional functionality that has traditionally been sold to us as incremental software modules may no longer need to be sold separately. I would much rather have an LLM run against the data." ---- From "All-In Podcast" YouTube channel, (full video link in comment)

Rohan Paul

40,959 просмотров • 12 дней назад

#WATCH | Delhi: On the #CasteCensus, author and Advaita teacher Acharya Prashant says, "Any action depends on the intention. Caste census means collecting more data, authentic data and data in itself are neither forward nor backwards, good nor bad. It depends on what you want to what you intend to do with the data. If you want to identify the marginalised groups in a better way, so that more accurate and more efficient policy making can happen, then it's a good step, a forward step... On the other hand, if you use this data to cement cost identification further, then obviously it's a step backwards. If you use this data for electoral gains by targeting particular caste groups concentrated in particular electorates or geographies, then obviously it's a backwards thing..."

#WATCH | Delhi: On the #CasteCensus, author and Advaita teacher Acharya Prashant says, "Any action depends on the intention. Caste census means collecting more data, authentic data and data in itself are neither forward nor backwards, good nor bad. It depends on what you want to what you intend to do with the data. If you want to identify the marginalised groups in a better way, so that more accurate and more efficient policy making can happen, then it's a good step, a forward step... On the other hand, if you use this data to cement cost identification further, then obviously it's a step backwards. If you use this data for electoral gains by targeting particular caste groups concentrated in particular electorates or geographies, then obviously it's a backwards thing..."

ANI

82,457 просмотров • 1 год назад

Major program launch: Data Analytics Professional Certificate! This large, five-course sequence takes you all the way to being job-ready as a data analyst, and shows how to use Generative AI as a thought partner to enhance your work in this role. Offered by on Coursera, this is taught by Sean Barnes, Ph.D., a Data Science & Engineering Leader at Netflix. Analyzing data remains one of the most important skills in where the world is going with AI. This comprehensive certificate takes you all the way to being job-ready. Each course comes with practical projects demonstrated in real-world contexts, such as analyzing sales data for a Korean bakery, video game sales trends across different regions, or identifying factors impacting customer retention for a communications company. You'll also work on estimating fire distribution for forest fire prevention, analyzing how a diamond's properties affect its market value, and developing predictive models for retail sales analysis, carbon emissions, and coral reef conservation. Here's some of what you'll learn: - How to define data and categorize it into its many types such as discrete & continuous numerical, structured & unstructured, time series, categorical, and know what insights can be derived from the different types of data categories. - How to differentiate between data-related job roles and their responsibilities, and how data flows through an organization from the moment of capture to decision-making. - How to perform data processing functions and apply conditional formatting in spreadsheets to extract business value from your data using statistical calculations and best practices for visualizing and interpreting data. - How to use LLMs for stakeholder analysis, data exploration, and data visualization. - Best practices for using LLMs for as a thought partner to data analysis work By the end of this professional certificate program, you will have learned core statistical concepts, analysis techniques, and visualization methodologies that will serve as the foundation for working as a data analyst. The world needs more data analysts, especially ones who know how to use modern generative AI. With data science roles projected to grow 36% by 2033, the skills taught in this program create new professional opportunities in data. Sign up here!

Major program launch: Data Analytics Professional Certificate! This large, five-course sequence takes you all the way to being job-ready as a data analyst, and shows how to use Generative AI as a thought partner to enhance your work in this role. Offered by on Coursera, this is taught by Sean Barnes, Ph.D., a Data Science & Engineering Leader at Netflix. Analyzing data remains one of the most important skills in where the world is going with AI. This comprehensive certificate takes you all the way to being job-ready. Each course comes with practical projects demonstrated in real-world contexts, such as analyzing sales data for a Korean bakery, video game sales trends across different regions, or identifying factors impacting customer retention for a communications company. You'll also work on estimating fire distribution for forest fire prevention, analyzing how a diamond's properties affect its market value, and developing predictive models for retail sales analysis, carbon emissions, and coral reef conservation. Here's some of what you'll learn: - How to define data and categorize it into its many types such as discrete & continuous numerical, structured & unstructured, time series, categorical, and know what insights can be derived from the different types of data categories. - How to differentiate between data-related job roles and their responsibilities, and how data flows through an organization from the moment of capture to decision-making. - How to perform data processing functions and apply conditional formatting in spreadsheets to extract business value from your data using statistical calculations and best practices for visualizing and interpreting data. - How to use LLMs for stakeholder analysis, data exploration, and data visualization. - Best practices for using LLMs for as a thought partner to data analysis work By the end of this professional certificate program, you will have learned core statistical concepts, analysis techniques, and visualization methodologies that will serve as the foundation for working as a data analyst. The world needs more data analysts, especially ones who know how to use modern generative AI. With data science roles projected to grow 36% by 2033, the skills taught in this program create new professional opportunities in data. Sign up here!

Andrew Ng

85,012 просмотров • 1 год назад

This is, by far, one of the best uses of modern AI. If you don't use embeddings when querying your database, you are definitely leaving a lot on the table. In this video, I'll show you how to run semantic searches using OpenAI and PostgreSQL. It's all thanks to Pgai, an open-source PostgreSQL extension: Here's what will happen: 1. We'll create a simple table with news articles 2. We'll generate embeddings for those articles 3. We'll run queries on top of those embeddings For this video, I generated the embeddings using a simple query, but pgai Vectorizer would do the same automatically as new information makes it into the database. This is awesome! If you have a PostgreSQL database with data you are searching over, you should start experimenting with semantic searches immediately. For most use cases, a combination of full-text search + semantic search is the best approach. If you don't have a PostgreSQL database around, you can try free for 30 days using Timescale: Thanks to the TimescaleDB (by Tiger Data) team for partnering with me on this post!

This is, by far, one of the best uses of modern AI. If you don't use embeddings when querying your database, you are definitely leaving a lot on the table. In this video, I'll show you how to run semantic searches using OpenAI and PostgreSQL. It's all thanks to Pgai, an open-source PostgreSQL extension: Here's what will happen: 1. We'll create a simple table with news articles 2. We'll generate embeddings for those articles 3. We'll run queries on top of those embeddings For this video, I generated the embeddings using a simple query, but pgai Vectorizer would do the same automatically as new information makes it into the database. This is awesome! If you have a PostgreSQL database with data you are searching over, you should start experimenting with semantic searches immediately. For most use cases, a combination of full-text search + semantic search is the best approach. If you don't have a PostgreSQL database around, you can try free for 30 days using Timescale: Thanks to the TimescaleDB (by Tiger Data) team for partnering with me on this post!

Santiago

110,133 просмотров • 1 год назад

New JavaScript short course: Build a full-stack web application that uses RAG in JavaScript RAG Web Apps with LlamaIndex, taught by Laurie Voss, VP of Developer Relations at LlamaIndex 🦙 and npm co-founder. - Build a RAG application for querying your own data - Develop tools to interact with multiple data sources using an agent that intelligently selects the right tool for your queries - Create a full-stack web app that can chat with your data - Dig further into production-ready techniques, like how to persist your data so you aren’t constantly reindexing, and try the create-llama command line tool from LlamaIndex You can sign up here:

New JavaScript short course: Build a full-stack web application that uses RAG in JavaScript RAG Web Apps with LlamaIndex, taught by Laurie Voss, VP of Developer Relations at LlamaIndex 🦙 and npm co-founder. - Build a RAG application for querying your own data - Develop tools to interact with multiple data sources using an agent that intelligently selects the right tool for your queries - Create a full-stack web app that can chat with your data - Dig further into production-ready techniques, like how to persist your data so you aren’t constantly reindexing, and try the create-llama command line tool from LlamaIndex You can sign up here:

Andrew Ng

218,284 просмотров • 2 лет назад

We just launched a major new Data Engineering Professional Certificate on Coursera! Data underlies all modern AI systems, and engineers who know how to build systems to store and serve it are in high demand. If you're interested in learning this skill, please check out this 4-course sequence, which is designed to make you job-ready to be a Data Engineer. This is a new specialization taught by Joe Reis, the co-author of the best-selling book “Fundamentals of Data Engineering," in collaboration with AWS. (Disclosure, I serve on Amazon's board.) For many AI systems, data engineering is 80% of the work, and modeling is 20%. But people’s attention on these two topics is often flipped. This makes the job of the data engineer particularly important. In this professional certificate, you'll learn foundational data engineering skills while implementing modern data architectures using open-source tools: - Learn the key steps of the data lifecycle, to generate, ingest, store, transform, and serve data. - Learn to align with organizational goals to design the data pipeline right for your business' needs. - Understand how to make necessary trade-offs between speed, scalability, security, and cost. Joe has distilled into this specialization decades of experience helping startups and large companies with data infrastructure. He is also joined by 17 other industry leaders in the data field, who will help you learn in-demand skills for the growing field of data engineering. Please sign up here:

We just launched a major new Data Engineering Professional Certificate on Coursera! Data underlies all modern AI systems, and engineers who know how to build systems to store and serve it are in high demand. If you're interested in learning this skill, please check out this 4-course sequence, which is designed to make you job-ready to be a Data Engineer. This is a new specialization taught by Joe Reis, the co-author of the best-selling book “Fundamentals of Data Engineering," in collaboration with AWS. (Disclosure, I serve on Amazon's board.) For many AI systems, data engineering is 80% of the work, and modeling is 20%. But people’s attention on these two topics is often flipped. This makes the job of the data engineer particularly important. In this professional certificate, you'll learn foundational data engineering skills while implementing modern data architectures using open-source tools: - Learn the key steps of the data lifecycle, to generate, ingest, store, transform, and serve data. - Learn to align with organizational goals to design the data pipeline right for your business' needs. - Understand how to make necessary trade-offs between speed, scalability, security, and cost. Joe has distilled into this specialization decades of experience helping startups and large companies with data infrastructure. He is also joined by 17 other industry leaders in the data field, who will help you learn in-demand skills for the growing field of data engineering. Please sign up here:

Andrew Ng

118,937 просмотров • 1 год назад

Data teams spend weeks on simple requests. (This AI answers them in minutes.) Most data analysis is repetitive manual tasks. Data teams spend more time on setup than actual analysis. The workflow usually looks like this: → Run some exploratory data analysis in a local Jupyter notebook or environment → Pull data from multiple disconnected sources → Write code from scratch for every analysis → Export static charts that stakeholders can't explore (or wrestle with legacy BI to create a dashboard) → Manually send updates via email or Slack when data changes → Start over for each new request Most teams accept this as "how data analysis works." While business decisions wait for insights. That's where Fabi changes the entire approach. It's a powerful, AI-native platform built for teams that want to boost productivity and supercharge their data workflows. Instead of working on separate tools and manual processes, you collaborate on analysis that automatically delivers insights where teams work. Here's what makes Fabi different: AI-Native Analysis Environment ↳ SQL and Python work together with AI assistance that handles coding and debugging automatically. Smart Automation Workflows ↳ Automatically send AI-powered reports and summaries right where business works in Slack, email, and spreadsheets. Universal Data Integration ↳ Analyze data from files, Google Sheets, Airtable, plus your data warehouse and databases in one place. Collaborative Data Apps ↳ Create interactive dashboards that stakeholders can explore and ask follow-up questions directly. What you can do with Fabi that legacy BI can't: ➟ Send AI-generated insights directly to Slack channels ➟ Automatically email data summaries to stakeholders ➟ Analyze uploaded files without complex ETL processes ➟ Collaborate on analysis like Google Docs for data ➟ Build workflows that push insights to spreadsheets Perfect for teams that want to move beyond the constraints of legacy and increase their impact. Teams using Fabi see immediate results: ✓ Insights delivered in minutes instead of days ✓ Reduced context switching between tools ✓ Stakeholders explore data independently ✓ Workflows automated to save hours of manual work From analysis to automated delivery - all in one AI-native environment. 📌 Try Fabi today: 👉 Follow Fabi.ai and marc for Fabi updates. 🔄 Repost to help other teams streamline data analysis #DataAnalysis #ModernBI #DataOps #InteractiveDashboards #FabiPartnership #SponsoredByFabi

Data teams spend weeks on simple requests. (This AI answers them in minutes.) Most data analysis is repetitive manual tasks. Data teams spend more time on setup than actual analysis. The workflow usually looks like this: → Run some exploratory data analysis in a local Jupyter notebook or environment → Pull data from multiple disconnected sources → Write code from scratch for every analysis → Export static charts that stakeholders can't explore (or wrestle with legacy BI to create a dashboard) → Manually send updates via email or Slack when data changes → Start over for each new request Most teams accept this as "how data analysis works." While business decisions wait for insights. That's where Fabi changes the entire approach. It's a powerful, AI-native platform built for teams that want to boost productivity and supercharge their data workflows. Instead of working on separate tools and manual processes, you collaborate on analysis that automatically delivers insights where teams work. Here's what makes Fabi different: AI-Native Analysis Environment ↳ SQL and Python work together with AI assistance that handles coding and debugging automatically. Smart Automation Workflows ↳ Automatically send AI-powered reports and summaries right where business works in Slack, email, and spreadsheets. Universal Data Integration ↳ Analyze data from files, Google Sheets, Airtable, plus your data warehouse and databases in one place. Collaborative Data Apps ↳ Create interactive dashboards that stakeholders can explore and ask follow-up questions directly. What you can do with Fabi that legacy BI can't: ➟ Send AI-generated insights directly to Slack channels ➟ Automatically email data summaries to stakeholders ➟ Analyze uploaded files without complex ETL processes ➟ Collaborate on analysis like Google Docs for data ➟ Build workflows that push insights to spreadsheets Perfect for teams that want to move beyond the constraints of legacy and increase their impact. Teams using Fabi see immediate results: ✓ Insights delivered in minutes instead of days ✓ Reduced context switching between tools ✓ Stakeholders explore data independently ✓ Workflows automated to save hours of manual work From analysis to automated delivery - all in one AI-native environment. 📌 Try Fabi today: 👉 Follow Fabi.ai and marc for Fabi updates. 🔄 Repost to help other teams streamline data analysis #DataAnalysis #ModernBI #DataOps #InteractiveDashboards #FabiPartnership #SponsoredByFabi

Andrew Bolis

36,504 просмотров • 11 месяцев назад

MCP is mind-blowing. Here is a genius setup that will earn you a promotion with your manager. I have multiple data sources: data in S3, MySQL, and local files. Using Spice (an open-source project), I can co-locate all of that data behind a single SQL endpoint and augment it using MCP. Before, accessing data was slow, and I had to make multiple requests to get anything. After using Spice, I can query the data much faster (since it's mirrored locally), and I don't have to worry about where the data is coming from. Spice acts as an MCP server, so I can connect to it from my IDE or the Claude Desktop app and have access to all the data. But even more cool than that: Spice also acts as an MCP client, so it can connect to other MCP servers. This is where things get really interesting! Watch the video. I'm querying multiple data sources from my IDE and using the Google Maps MCP server for driving directions.

MCP is mind-blowing. Here is a genius setup that will earn you a promotion with your manager. I have multiple data sources: data in S3, MySQL, and local files. Using Spice (an open-source project), I can co-locate all of that data behind a single SQL endpoint and augment it using MCP. Before, accessing data was slow, and I had to make multiple requests to get anything. After using Spice, I can query the data much faster (since it's mirrored locally), and I don't have to worry about where the data is coming from. Spice acts as an MCP server, so I can connect to it from my IDE or the Claude Desktop app and have access to all the data. But even more cool than that: Spice also acts as an MCP client, so it can connect to other MCP servers. This is where things get really interesting! Watch the video. I'm querying multiple data sources from my IDE and using the Google Maps MCP server for driving directions.

Santiago

142,696 просмотров • 1 год назад

The Financial Revolution You Didn't Know You Needed: Account Aggregator You may have caught Amitabh Bachchan discussing Account Aggregator (AA) on Kaun Banega Crorepati recently. It wasn't just talk; it is a financial revolution in the making. The Problem: Meet Priya, an entrepreneur with a vision but no credit history. Traditional banks shut their doors on her, labeling her a 'risk.' The Solution: Enter Account Aggregator. With AA, Priya could consolidate her financial data, from bank accounts to tax returns, into one secure platform. She could then share this data with lenders on her terms. The Outcome: Priya secured a loan within days, not weeks. She took control of her financial destiny, all thanks to AA. Why Choose Agya for Your AA Journey: Security: Your data is encrypted and secure. Efficiency: One platform for all your financial data. Inclusivity: Whether you're a student, a homemaker, or an entrepreneur, Agya is for you. Amitabh Bachchan's Take: "Account Aggregator is more than a service; it's a paradigm shift in how we handle our finances." Your Next Step: Priya's story is not an outlier; it's a testament to what Agya can do for you. Take control of your financial data. Shape your financial future. Credit for the video: Kaun Banega Crorepati on Sony Pictures Entertainment Narendra Modi Nirmala Sitharaman Sahamati Nandan Nilekani ReserveBankOfIndia Ministry of Finance BG Mahesh Siddharth Shetty Pramod Varma Dilip Asbe 🇮🇳 Amrish Rau Sahil Kini Vinay Kesari irfn Vikram Bhat

The Financial Revolution You Didn't Know You Needed: Account Aggregator You may have caught Amitabh Bachchan discussing Account Aggregator (AA) on Kaun Banega Crorepati recently. It wasn't just talk; it is a financial revolution in the making. The Problem: Meet Priya, an entrepreneur with a vision but no credit history. Traditional banks shut their doors on her, labeling her a 'risk.' The Solution: Enter Account Aggregator. With AA, Priya could consolidate her financial data, from bank accounts to tax returns, into one secure platform. She could then share this data with lenders on her terms. The Outcome: Priya secured a loan within days, not weeks. She took control of her financial destiny, all thanks to AA. Why Choose Agya for Your AA Journey: Security: Your data is encrypted and secure. Efficiency: One platform for all your financial data. Inclusivity: Whether you're a student, a homemaker, or an entrepreneur, Agya is for you. Amitabh Bachchan's Take: "Account Aggregator is more than a service; it's a paradigm shift in how we handle our finances." Your Next Step: Priya's story is not an outlier; it's a testament to what Agya can do for you. Take control of your financial data. Shape your financial future. Credit for the video: Kaun Banega Crorepati on Sony Pictures Entertainment Narendra Modi Nirmala Sitharaman Sahamati Nandan Nilekani ReserveBankOfIndia Ministry of Finance BG Mahesh Siddharth Shetty Pramod Varma Dilip Asbe 🇮🇳 Amrish Rau Sahil Kini Vinay Kesari irfn Vikram Bhat

Nikhil Kumar

35,314 просмотров • 2 лет назад

10,000 valid trajectories collected — and we did it ahead of schedule, in just 5 days. We’ve reached our data milestone for "Little Prince's Rose." This community-driven dataset is now entering the pipeline for cleaning, augmentation, and model training, advancing the intelligence of Franka at this very moment. While the Discord role claiming is closing, the experiment will remain open as a welcoming portal for everyone to experience the Axis and how we tackle the data bottleneck for robotics. Robot intelligence is not built by a few; it's built by all.🌹

10,000 valid trajectories collected — and we did it ahead of schedule, in just 5 days. We’ve reached our data milestone for "Little Prince's Rose." This community-driven dataset is now entering the pipeline for cleaning, augmentation, and model training, advancing the intelligence of Franka at this very moment. While the Discord role claiming is closing, the experiment will remain open as a welcoming portal for everyone to experience the Axis and how we tackle the data bottleneck for robotics. Robot intelligence is not built by a few; it's built by all.🌹

Axis Robotics

23,424 просмотров • 6 месяцев назад

🔎🤖LangSmith Insights Agent Really excited to launch our first in-product agent This agent lives inside LangSmith and combs through traces, giving you insights into: 🧑‍🤝‍🧑how users are using your agent ⁉️how your agent may be messing up 🛃{your custom insight here} The problem we saw was that people were launching agents... and didn't know how their users were actually using them! You put a chat box in front of people, and they may ask it anything - the surface area for agents is often super wide In addition - agents would fail silently. They could give a bad response - this wouldn't show up in error logs, but its good to know. If you know what look for, you can set up LLM as a judge evaluators. But what if you don't? (most people don't initially) The best way to figure this out - as Hamel Husain says - "look at your data". But LLMs are really good at looking at your data! So can they do it for you? This is exactly what insights agent attempts to do. It's live in LangSmith today. You can read more about it here:

🔎🤖LangSmith Insights Agent Really excited to launch our first in-product agent This agent lives inside LangSmith and combs through traces, giving you insights into: 🧑‍🤝‍🧑how users are using your agent ⁉️how your agent may be messing up 🛃{your custom insight here} The problem we saw was that people were launching agents... and didn't know how their users were actually using them! You put a chat box in front of people, and they may ask it anything - the surface area for agents is often super wide In addition - agents would fail silently. They could give a bad response - this wouldn't show up in error logs, but its good to know. If you know what look for, you can set up LLM as a judge evaluators. But what if you don't? (most people don't initially) The best way to figure this out - as Hamel Husain says - "look at your data". But LLMs are really good at looking at your data! So can they do it for you? This is exactly what insights agent attempts to do. It's live in LangSmith today. You can read more about it here:

Harrison Chase

98,520 просмотров • 9 месяцев назад

Building Data Pipelines has levels to it: - level 0 Understand the basic flow: Extract → Transform → Load (ETL) or ELT This is the foundation. - Extract: Pull data from sources (APIs, DBs, files) - Transform: Clean, filter, join, or enrich the data - Load: Store into a warehouse or lake for analysis You’re not a data engineer until you’ve scheduled a job to pull CSVs off an SFTP server at 3AM! level 1 Master the tools: - Airflow for orchestration - dbt for transformations - Spark or PySpark for big data - Snowflake, BigQuery, Redshift for warehouses - Kafka or Kinesis for streaming Understand when to batch vs stream. Most companies think they need real-time data. They usually don’t. level 2 Handle complexity with modular design: - DAGs should be atomic, idempotent, and parameterized - Use task dependencies and sensors wisely - Break transformations into layers (staging → clean → marts) - Design for failure recovery. If a step fails, how do you re-run it? From scratch or just that part? Learn how to backfill without breaking the world. level 3 Data quality and observability: - Add tests for nulls, duplicates, and business logic - Use tools like Great Expectations, Monte Carlo, or built-in dbt tests - Track lineage so you know what downstream will break if upstream changes Know the difference between: - a late-arriving dimension - a broken SCD2 - and a pipeline silently dropping rows At this level, you understand that reliability > cleverness. level 4 Build for scale and maintainability: - Version control your pipeline configs - Use feature flags to toggle behavior in prod - Push vs pull architecture - Decouple compute and storage (e.g. Iceberg and Delta Lake) - Data mesh, data contracts, streaming joins, and CDC are words you throw around because you know how and when to use them. What else belongs in the journey to mastering data pipelines?

Building Data Pipelines has levels to it: - level 0 Understand the basic flow: Extract → Transform → Load (ETL) or ELT This is the foundation. - Extract: Pull data from sources (APIs, DBs, files) - Transform: Clean, filter, join, or enrich the data - Load: Store into a warehouse or lake for analysis You’re not a data engineer until you’ve scheduled a job to pull CSVs off an SFTP server at 3AM! level 1 Master the tools: - Airflow for orchestration - dbt for transformations - Spark or PySpark for big data - Snowflake, BigQuery, Redshift for warehouses - Kafka or Kinesis for streaming Understand when to batch vs stream. Most companies think they need real-time data. They usually don’t. level 2 Handle complexity with modular design: - DAGs should be atomic, idempotent, and parameterized - Use task dependencies and sensors wisely - Break transformations into layers (staging → clean → marts) - Design for failure recovery. If a step fails, how do you re-run it? From scratch or just that part? Learn how to backfill without breaking the world. level 3 Data quality and observability: - Add tests for nulls, duplicates, and business logic - Use tools like Great Expectations, Monte Carlo, or built-in dbt tests - Track lineage so you know what downstream will break if upstream changes Know the difference between: - a late-arriving dimension - a broken SCD2 - and a pipeline silently dropping rows At this level, you understand that reliability > cleverness. level 4 Build for scale and maintainability: - Version control your pipeline configs - Use feature flags to toggle behavior in prod - Push vs pull architecture - Decouple compute and storage (e.g. Iceberg and Delta Lake) - Data mesh, data contracts, streaming joins, and CDC are words you throw around because you know how and when to use them. What else belongs in the journey to mastering data pipelines?

Zach Wilson

16,688 просмотров • 1 год назад

We can only trust X 🚨 META/ Facebook planning to build data centers in China, store their data in China, and then GIVE THE CHINESE GOVERNMENT ACCESS to the data “In this document, Facebook is talking about making Chinese user data available to the Chinese government because they're going to store that data in China. Is that correct?” “Correct” “When you store that data in China, Americans who exchange messages or other information with Chinese Facebook users, that would mean the Chinese government could get access to the American data as well, is that correct? “Through the POP servers? Potentially.” “And Facebook was willing to take that risk?” “Yes. There was a lot of discussion about this and ultimately yes.” “I mean, this is, this is extraordinary. This is exactly contrary to what Facebook has represented for years here. They're willing to build data centers, store data in China. They are willing explicitly to give the Chinese government access to it”

We can only trust X 🚨 META/ Facebook planning to build data centers in China, store their data in China, and then GIVE THE CHINESE GOVERNMENT ACCESS to the data “In this document, Facebook is talking about making Chinese user data available to the Chinese government because they're going to store that data in China. Is that correct?” “Correct” “When you store that data in China, Americans who exchange messages or other information with Chinese Facebook users, that would mean the Chinese government could get access to the American data as well, is that correct? “Through the POP servers? Potentially.” “And Facebook was willing to take that risk?” “Yes. There was a lot of discussion about this and ultimately yes.” “I mean, this is, this is extraordinary. This is exactly contrary to what Facebook has represented for years here. They're willing to build data centers, store data in China. They are willing explicitly to give the Chinese government access to it”

Wall Street Apes

80,854 просмотров • 1 год назад

Your AI Agents can only analyze 1GB of data Enterprises have 1,000,000x as much data We fixed that... It took 3 years and building an entirely new execution environment to pull this off. TextQL lets AI agents analyze your ENTIRE enterprise data - not just the 0.0001% they can handle today. Here's what makes it so different: 1. ALL your data, not samples. AI today: 1GB demos. Your reality: 350TB across 47 systems. TextQL: Everything. At once. 2. Real enterprise questions.ChatGPT writes poems. We answer why renewal rates dropped 3% after ERP upgrades. 3. No more sandbox limitations.Everyone else: Jupyter notebook with 10GB RAM. Us: A data center. ------------------------------ In 2021, I watched Fortune 500s spend millions on AI that could only analyze data that fits on a USB stick. Today: - We process 10+ petabytes daily - Our biggest customer analyzes 100TB in real-time - Average query spans 50+ data sources And we just proved what everyone said was impossible - AI that can actually build a digital twin of your enterprise ------------------------------ To celebrate, we're giving away free data architecture assessments. We'll show you exactly how much of your data is invisible to AI right now (spoiler: it's probably 99.9%). Comment "TextQL" and we'll send you the assessment link.

Your AI Agents can only analyze 1GB of data Enterprises have 1,000,000x as much data We fixed that... It took 3 years and building an entirely new execution environment to pull this off. TextQL lets AI agents analyze your ENTIRE enterprise data - not just the 0.0001% they can handle today. Here's what makes it so different: 1. ALL your data, not samples. AI today: 1GB demos. Your reality: 350TB across 47 systems. TextQL: Everything. At once. 2. Real enterprise questions.ChatGPT writes poems. We answer why renewal rates dropped 3% after ERP upgrades. 3. No more sandbox limitations.Everyone else: Jupyter notebook with 10GB RAM. Us: A data center. ------------------------------ In 2021, I watched Fortune 500s spend millions on AI that could only analyze data that fits on a USB stick. Today: - We process 10+ petabytes daily - Our biggest customer analyzes 100TB in real-time - Average query spans 50+ data sources And we just proved what everyone said was impossible - AI that can actually build a digital twin of your enterprise ------------------------------ To celebrate, we're giving away free data architecture assessments. We'll show you exactly how much of your data is invisible to AI right now (spoiler: it's probably 99.9%). Comment "TextQL" and we'll send you the assessment link.

ethan ding 📊

162,101 просмотров • 1 год назад

The $KLED Enterprise Portal Full Demo – Search and License Any Data in Seconds: For the last 6 weeks, we've had thousands of users sign up for the first data marketplace. We've paid out over $200,000 to them for uploading their camera rolls, homework, dashcam footage, etc. We've now amassed petabytes and petabytes of data from these users and large enterprises, and tagged it all with rich metadata. Now OpenAI, xAI, Google AI, or any other large lab can easily search and license that data using our newly completed enterprise portal. If you're a large lab, send me a DM and get access to the largest, never-before-seen consumer dataset on the planet. This is the biggest update for Kled since launch.

The $KLED Enterprise Portal Full Demo – Search and License Any Data in Seconds: For the last 6 weeks, we've had thousands of users sign up for the first data marketplace. We've paid out over $200,000 to them for uploading their camera rolls, homework, dashcam footage, etc. We've now amassed petabytes and petabytes of data from these users and large enterprises, and tagged it all with rich metadata. Now OpenAI, xAI, Google AI, or any other large lab can easily search and license that data using our newly completed enterprise portal. If you're a large lab, send me a DM and get access to the largest, never-before-seen consumer dataset on the planet. This is the biggest update for Kled since launch.

Avi Patel

91,697 просмотров • 1 год назад

After 4 months of legal groundwork, engineering, and partnership development, we’re excited to introduce Kled Life. A fully new experience inside the Kled app that reimagines how user data is shared, and how users are compensated. Kled users can now link their Netflix, Uber, Gmail, Spotify, and over 100 more apps directly to their Kled profile. Every time you use those linked accounts, Kled securely captures your behavioral data, how you listen, watch, shop, move, and pays you for it. How many songs you played today. How many hours you streamed. Which games you love most. It’s data that’s usually extracted and sold behind your back. Now, you get a share of the game. All in under 60 seconds of setup. We’re excited to be rolling this out with our wonderful partners and collaborators who have made this possible. We’ve tasked over 300 select users to test out these features while we simultaneously organize this new data pipeline for enterprises. We expect Kled Life to triple the size of our user payouts and enterprise contract demand once rolled out to the public. Get paid for your data, get paid for using your phone, get paid for labeling data. One app. The ultimate data side hustle.

Kled AI

52,199 просмотров • 10 месяцев назад