Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

Building Data Pipelines has levels to it: - level 0 Understand the basic flow: Extract → Transform → Load (ETL) or ELT This is the foundation. - Extract: Pull data from sources (APIs, DBs, files) - Transform: Clean, filter, join, or enrich the data - Load: Store into a... warehouse or lake for analysis You’re not a data engineer until you’ve scheduled a job to pull CSVs off an SFTP server at 3AM! level 1 Master the tools: - Airflow for orchestration - dbt for transformations - Spark or PySpark for big data - Snowflake, BigQuery, Redshift for warehouses - Kafka or Kinesis for streaming Understand when to batch vs stream. Most companies think they need real-time data. They usually don’t. level 2 Handle complexity with modular design: - DAGs should be atomic, idempotent, and parameterized - Use task dependencies and sensors wisely - Break transformations into layers (staging → clean → marts) - Design for failure recovery. If a step fails, how do you re-run it? From scratch or just that part? Learn how to backfill without breaking the world. level 3 Data quality and observability: - Add tests for nulls, duplicates, and business logic - Use tools like Great Expectations, Monte Carlo, or built-in dbt tests - Track lineage so you know what downstream will break if upstream changes Know the difference between: - a late-arriving dimension - a broken SCD2 - and a pipeline silently dropping rows At this level, you understand that reliability > cleverness. level 4 Build for scale and maintainability: - Version control your pipeline configs - Use feature flags to toggle behavior in prod - Push vs pull architecture - Decouple compute and storage (e.g. Iceberg and Delta Lake) - Data mesh, data contracts, streaming joins, and CDC are words you throw around because you know how and when to use them. What else belongs in the journey to mastering data pipelines?show more

Zach Wilson

51,663 subscribers

16,851 görüntüleme • 1 yıl önce •via X (Twitter)

Bilim & Teknoloji Eğitim

Anya Rossi• Live Now

Private livecam show

9 Yorum

Zach Morris Wilson profil fotoğrafı

Zach Morris Wilson1 yıl önce

Subscribe to for more insights!

HUDI profil fotoğrafı

HUDI2 yıl önce

🚀 Introducing DATA DATA DATA: The Official HUDI Podcast! 🔥 💡 The podcast where data becomes power. We dive into Web3, crypto, data ownership, and the future. Engage, learn, and be part of the revolution. 🌍🔗 Knowledge is power—join the journey. #DataDataData #HUDI #Web3 #Podcast #DataOwnership #BigTech #Privacy #Crypto #AI #DigitalRevolution

A. Ibrahim, PhD profil fotoğrafı

A. Ibrahim, PhD1 yıl önce

Great insights Zach. Do you cover all these in any of your bootcamps? If yes link please. Met you last year in San Francisco btw. I am a fan. Thanks.

Mitch Call M profil fotoğrafı

Mitch Call M1 yıl önce

Classic, these are big value posts Zach

Ulises FV profil fotoğrafı

Ulises FV1 yıl önce

Data

N profil fotoğrafı

N1 yıl önce

Lovely summary!

Mamadou Tafsir Diallo profil fotoğrafı

Mamadou Tafsir Diallo1 yıl önce

Data

Quophi Amakye profil fotoğrafı

Quophi Amakye1 yıl önce

Data

The South Indian profil fotoğrafı

The South Indian1 yıl önce

Data

Benzer Videolar

In the next 2-3 years, the skills you need to build an ETL process will fundamentally differ from anything we've done in the last decade. The EXTRACT and LOAD steps of an ETL pipeline have been solved forever. They are boring, and I suspect they will continue to be boring. But the TRANSFORM step is where the money is at (and where innovation will continue happening!) An ETL pipeline that can use Large Language Models effectively to transform data is a whole new ballgame. I recorded a video to talk about this. And more importantly, I talk about how you can build ETL pipelines to transform UNSTRUCTURED data (the kind of data that's really hard to handle).

In the next 2-3 years, the skills you need to build an ETL process will fundamentally differ from anything we've done in the last decade. The EXTRACT and LOAD steps of an ETL pipeline have been solved forever. They are boring, and I suspect they will continue to be boring. But the TRANSFORM step is where the money is at (and where innovation will continue happening!) An ETL pipeline that can use Large Language Models effectively to transform data is a whole new ballgame. I recorded a video to talk about this. And more importantly, I talk about how you can build ETL pipelines to transform UNSTRUCTURED data (the kind of data that's really hard to handle).

Santiago

56,412 görüntüleme • 1 yıl önce

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:

Here’s how I would learn data engineering in 2025: 1. The basics: - learn SQL — SELECT, FROM, WHERE, GROUP BY, JOIN, HAVING, etc - learn Python — data structures: objects, arrays, tuples, namedtuples — algorithms: recursion, loops 2. Intermediate - learn distributed compute — pick up PySpark or Snowflake or BigQuery - learn data make architecture — pick up iceberg or delta lake - learn job orchestration — pick up Airflow or Mage - learn data quality — pick up Great expectations 3. Advanced - learn the data modeling techniques — one big table vs kimball vs Inmon vs data vault techniques - learn machine learning features and vector databases — pick up pinecone and how to fine tune LLMs with high quality data My newsletter has a deeper roadmap here:

Zach Wilson

29,968 görüntüleme • 1 yıl önce

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake or Google BigQuery. - update your Python script to load a table in Snowflake/BigQuery - schedule your script with CRON in the cloud with a service like Heroku. - build aggregations and visualizations on top of your ingested data Only thing this misses is data quality and complex job orchestration which you can learn later! How would you learn data engineering nowadays?

Here's how I would learn data engineering basics in 2025: - Find a data source you care about (examples: gaming APIs, stock market, web scraping, etc) - Use Python to interact and ingest your source. Initially just write the data to a CSV. - Setup an account with Snowflake or Google BigQuery. - update your Python script to load a table in Snowflake/BigQuery - schedule your script with CRON in the cloud with a service like Heroku. - build aggregations and visualizations on top of your ingested data Only thing this misses is data quality and complex job orchestration which you can learn later! How would you learn data engineering nowadays?

Zach Wilson

20,412 görüntüleme • 1 yıl önce

Traditional data pipelines don't work for RAG applications. There are 3 issues with them: 1. Traditional data engineering solutions are optimized to handle structured data. RAG applications rely primarily on unstructured data. 2. The connector ecosystem to load data from unstructured data sources is very immature. 3. Traditional solutions do not offer any way to transform unstructured data into an optimized vector search index. The goal of a RAG Pipeline is to solve these problems. The number one objective is to create a reliable vector search index using factual knowledge and relevant context. This sounds easy, but it's one of the biggest challenges we face when building RAG applications. At a high level, there are four different stages in the architecture of a RAG pipeline: 1. Ingestion: Here is where the pipeline loads the information from the data source. 2. Extraction: Where the pipeline processes the input data and decides how to retrieve the text contained inside them. 3. Transform: Where the pipeline chunks the data and generates document embeddings. 4. Load: Where the pipeline creates a search index in a vector database and loads the document embeddings. There are different rabbit holes at each one of these stages. Here are three of them: 1. Ingesting data once is simple. The hard part is refreshing the vector database whenever the original data source changes. 2. Extracting the content of a plain text document is simple. The hard part is to extract content from complex documents containing tables, images, or cross-references. 3. A simple continual chunking strategy with an overlap is simple. The hard part is to find the optimal strategy for your specific knowledge base and the way you are planning to query it. In the attached video, I'll show you how you can build an enterprise-grade RAG Pipeline that solves every one of the above problems. I'll use Vectorize. They partnered with me on this post. You can use them to build RAG pipelines optimized for accurate context retrieval. If you have a few documents lying around, set up a free account and give it a try.

Traditional data pipelines don't work for RAG applications. There are 3 issues with them: 1. Traditional data engineering solutions are optimized to handle structured data. RAG applications rely primarily on unstructured data. 2. The connector ecosystem to load data from unstructured data sources is very immature. 3. Traditional solutions do not offer any way to transform unstructured data into an optimized vector search index. The goal of a RAG Pipeline is to solve these problems. The number one objective is to create a reliable vector search index using factual knowledge and relevant context. This sounds easy, but it's one of the biggest challenges we face when building RAG applications. At a high level, there are four different stages in the architecture of a RAG pipeline: 1. Ingestion: Here is where the pipeline loads the information from the data source. 2. Extraction: Where the pipeline processes the input data and decides how to retrieve the text contained inside them. 3. Transform: Where the pipeline chunks the data and generates document embeddings. 4. Load: Where the pipeline creates a search index in a vector database and loads the document embeddings. There are different rabbit holes at each one of these stages. Here are three of them: 1. Ingesting data once is simple. The hard part is refreshing the vector database whenever the original data source changes. 2. Extracting the content of a plain text document is simple. The hard part is to extract content from complex documents containing tables, images, or cross-references. 3. A simple continual chunking strategy with an overlap is simple. The hard part is to find the optimal strategy for your specific knowledge base and the way you are planning to query it. In the attached video, I'll show you how you can build an enterprise-grade RAG Pipeline that solves every one of the above problems. I'll use Vectorize. They partnered with me on this post. You can use them to build RAG pipelines optimized for accurate context retrieval. If you have a few documents lying around, set up a free account and give it a try.

Santiago

40,646 görüntüleme • 2 yıl önce

Major program launch: Data Analytics Professional Certificate! This large, five-course sequence takes you all the way to being job-ready as a data analyst, and shows how to use Generative AI as a thought partner to enhance your work in this role. Offered by on Coursera, this is taught by Sean Barnes, Ph.D., a Data Science & Engineering Leader at Netflix. Analyzing data remains one of the most important skills in where the world is going with AI. This comprehensive certificate takes you all the way to being job-ready. Each course comes with practical projects demonstrated in real-world contexts, such as analyzing sales data for a Korean bakery, video game sales trends across different regions, or identifying factors impacting customer retention for a communications company. You'll also work on estimating fire distribution for forest fire prevention, analyzing how a diamond's properties affect its market value, and developing predictive models for retail sales analysis, carbon emissions, and coral reef conservation. Here's some of what you'll learn: - How to define data and categorize it into its many types such as discrete & continuous numerical, structured & unstructured, time series, categorical, and know what insights can be derived from the different types of data categories. - How to differentiate between data-related job roles and their responsibilities, and how data flows through an organization from the moment of capture to decision-making. - How to perform data processing functions and apply conditional formatting in spreadsheets to extract business value from your data using statistical calculations and best practices for visualizing and interpreting data. - How to use LLMs for stakeholder analysis, data exploration, and data visualization. - Best practices for using LLMs for as a thought partner to data analysis work By the end of this professional certificate program, you will have learned core statistical concepts, analysis techniques, and visualization methodologies that will serve as the foundation for working as a data analyst. The world needs more data analysts, especially ones who know how to use modern generative AI. With data science roles projected to grow 36% by 2033, the skills taught in this program create new professional opportunities in data. Sign up here!

Major program launch: Data Analytics Professional Certificate! This large, five-course sequence takes you all the way to being job-ready as a data analyst, and shows how to use Generative AI as a thought partner to enhance your work in this role. Offered by on Coursera, this is taught by Sean Barnes, Ph.D., a Data Science & Engineering Leader at Netflix. Analyzing data remains one of the most important skills in where the world is going with AI. This comprehensive certificate takes you all the way to being job-ready. Each course comes with practical projects demonstrated in real-world contexts, such as analyzing sales data for a Korean bakery, video game sales trends across different regions, or identifying factors impacting customer retention for a communications company. You'll also work on estimating fire distribution for forest fire prevention, analyzing how a diamond's properties affect its market value, and developing predictive models for retail sales analysis, carbon emissions, and coral reef conservation. Here's some of what you'll learn: - How to define data and categorize it into its many types such as discrete & continuous numerical, structured & unstructured, time series, categorical, and know what insights can be derived from the different types of data categories. - How to differentiate between data-related job roles and their responsibilities, and how data flows through an organization from the moment of capture to decision-making. - How to perform data processing functions and apply conditional formatting in spreadsheets to extract business value from your data using statistical calculations and best practices for visualizing and interpreting data. - How to use LLMs for stakeholder analysis, data exploration, and data visualization. - Best practices for using LLMs for as a thought partner to data analysis work By the end of this professional certificate program, you will have learned core statistical concepts, analysis techniques, and visualization methodologies that will serve as the foundation for working as a data analyst. The world needs more data analysts, especially ones who know how to use modern generative AI. With data science roles projected to grow 36% by 2033, the skills taught in this program create new professional opportunities in data. Sign up here!

Andrew Ng

85,182 görüntüleme • 1 yıl önce

Is the future of AI $NEAR … ? OpenAI is racing toward a $1T IPO, quickly becoming a centralized beast of epic proportions. But what happens to your data in that world?? As chronear explains, “We can't let centralized actors determine how they store or how they use user data. It should be the users who actually control the custody or use of their data." NEAR AI is building a truly open AI, open source, where users control their data and store their data locally. What AI do you use daily? And would you switch if a better decentralized option existed?

Is the future of AI $NEAR … ? OpenAI is racing toward a $1T IPO, quickly becoming a centralized beast of epic proportions. But what happens to your data in that world?? As chronear explains, “We can't let centralized actors determine how they store or how they use user data. It should be the users who actually control the custody or use of their data." NEAR AI is building a truly open AI, open source, where users control their data and store their data locally. What AI do you use daily? And would you switch if a better decentralized option existed?

Bri Teresi

25,101 görüntüleme • 6 ay önce

We just launched a major new Data Engineering Professional Certificate on Coursera! Data underlies all modern AI systems, and engineers who know how to build systems to store and serve it are in high demand. If you're interested in learning this skill, please check out this 4-course sequence, which is designed to make you job-ready to be a Data Engineer. This is a new specialization taught by Joe Reis, the co-author of the best-selling book “Fundamentals of Data Engineering," in collaboration with AWS. (Disclosure, I serve on Amazon's board.) For many AI systems, data engineering is 80% of the work, and modeling is 20%. But people’s attention on these two topics is often flipped. This makes the job of the data engineer particularly important. In this professional certificate, you'll learn foundational data engineering skills while implementing modern data architectures using open-source tools: - Learn the key steps of the data lifecycle, to generate, ingest, store, transform, and serve data. - Learn to align with organizational goals to design the data pipeline right for your business' needs. - Understand how to make necessary trade-offs between speed, scalability, security, and cost. Joe has distilled into this specialization decades of experience helping startups and large companies with data infrastructure. He is also joined by 17 other industry leaders in the data field, who will help you learn in-demand skills for the growing field of data engineering. Please sign up here:

We just launched a major new Data Engineering Professional Certificate on Coursera! Data underlies all modern AI systems, and engineers who know how to build systems to store and serve it are in high demand. If you're interested in learning this skill, please check out this 4-course sequence, which is designed to make you job-ready to be a Data Engineer. This is a new specialization taught by Joe Reis, the co-author of the best-selling book “Fundamentals of Data Engineering," in collaboration with AWS. (Disclosure, I serve on Amazon's board.) For many AI systems, data engineering is 80% of the work, and modeling is 20%. But people’s attention on these two topics is often flipped. This makes the job of the data engineer particularly important. In this professional certificate, you'll learn foundational data engineering skills while implementing modern data architectures using open-source tools: - Learn the key steps of the data lifecycle, to generate, ingest, store, transform, and serve data. - Learn to align with organizational goals to design the data pipeline right for your business' needs. - Understand how to make necessary trade-offs between speed, scalability, security, and cost. Joe has distilled into this specialization decades of experience helping startups and large companies with data infrastructure. He is also joined by 17 other industry leaders in the data field, who will help you learn in-demand skills for the growing field of data engineering. Please sign up here:

Andrew Ng

120,581 görüntüleme • 2 yıl önce

DataTransfer iOS app wip. Ver 1.0.0 Perfect for Anyone that wants to quickly, easily, and reliably pull out all media data from device and can use to restore on another iOS device. Made for iOS only. Mainly started this project because there’s no app I can reliably use without it crashing or too much $ for a simple data backup, plus I need something like this for iOS data recovery situations when customers come in with iOS device for data transfer but FMI on and they cannot remember the login. So this way I can backup data FAST and have multiple ways to transfer the backup files, (Air drop wired or wireless, iCloud, etc) Also, main focus for this app will be elegant UI and high reliability and speed for backups.

DataTransfer iOS app wip. Ver 1.0.0 Perfect for Anyone that wants to quickly, easily, and reliably pull out all media data from device and can use to restore on another iOS device. Made for iOS only. Mainly started this project because there’s no app I can reliably use without it crashing or too much $ for a simple data backup, plus I need something like this for iOS data recovery situations when customers come in with iOS device for data transfer but FMI on and they cannot remember the login. So this way I can backup data FAST and have multiple ways to transfer the backup files, (Air drop wired or wireless, iCloud, etc) Also, main focus for this app will be elegant UI and high reliability and speed for backups.

Mr. Creator - EuphoriaTools.com

20,727 görüntüleme • 1 yıl önce

Instagram launches forced Data Collection to use their app If you login and get this prompt, you must accept data all this data collection or you will be promoted to log out and not allowed to use the App Here is a full down of what’s mandatory to give up to use Instagram: Collection and use of personal data: They collect our activity on Instagram (posts, stories, likes, comments, searches, time spent), device info, IP address, contacts (if synced), messages, location (when enabled), demographics, and inferred interests Sharing of data with other Meta companies. They take all the personal data they collected and share across Facebook, WhatsApp, Threads, Oculus, etc. Cross-border transfers of personal data: Your data is sent and stored in the US and other countries where Meta has offices/data centers or partners. Location information: Precise location (GPS when you use features like Stories or check-ins), approximate location from IP/Wi-Fi, and location-based activity history. Stop using META products

Instagram launches forced Data Collection to use their app If you login and get this prompt, you must accept data all this data collection or you will be promoted to log out and not allowed to use the App Here is a full down of what’s mandatory to give up to use Instagram: Collection and use of personal data: They collect our activity on Instagram (posts, stories, likes, comments, searches, time spent), device info, IP address, contacts (if synced), messages, location (when enabled), demographics, and inferred interests Sharing of data with other Meta companies. They take all the personal data they collected and share across Facebook, WhatsApp, Threads, Oculus, etc. Cross-border transfers of personal data: Your data is sent and stored in the US and other countries where Meta has offices/data centers or partners. Location information: Precise location (GPS when you use features like Stories or check-ins), approximate location from IP/Wi-Fi, and location-based activity history. Stop using META products

Wall Street Apes

266,148 görüntüleme • 3 ay önce

How does an FDE map the processes to deploy AI in a real business? We use 3 sources. Source 1 is people. - Companies tell us: "this person's been at the firm for 20 years and they just handle it." - We hear this from Fortune 50 companies and from SMBs. - So in finance, you talk to the head of AP, AR, reconciliations, billing, banking, and FP&A. You make a "human API." Source 2 is the systems of record. - With real-time access to Salesforce for 3 to 4 weeks, you see what data comes in and how often people correct it. Source 3 is the documents in SharePoint, Drive, Notion, Slack, Teams, and Gmail. If you use the system data without the interviews, you miss half of the picture. The split is different for each company. A large company can have 10 years of Salesforce data. An SMB has most of it in people's heads. Choose your angle for each company before you build.

How does an FDE map the processes to deploy AI in a real business? We use 3 sources. Source 1 is people. - Companies tell us: "this person's been at the firm for 20 years and they just handle it." - We hear this from Fortune 50 companies and from SMBs. - So in finance, you talk to the head of AP, AR, reconciliations, billing, banking, and FP&A. You make a "human API." Source 2 is the systems of record. - With real-time access to Salesforce for 3 to 4 weeks, you see what data comes in and how often people correct it. Source 3 is the documents in SharePoint, Drive, Notion, Slack, Teams, and Gmail. If you use the system data without the interviews, you miss half of the picture. The split is different for each company. A large company can have 10 years of Salesforce data. An SMB has most of it in people's heads. Choose your angle for each company before you build.

The Startup Ideas Podcast (SIP) 🧃

35,097 görüntüleme • 24 gün önce

$BTC Statistical Study using Claude - A Beginner's Workflow Here's an example of a z-score study on $BTC - still tinkering so don't take this as overtly useful information but the creation of a dashboard for the visualization of statistical data is phenomenal. my current workflow: > import $BTC time data - .csv file (can get this information from multiple venues - Binance is where I got mine) > creating bins - if you have a larger data set - you can use recent data to prevent overt bias in the long direction or filter consolidation and trending regime data into separate bins for statistical analysis - however, you will have to define thresholds and determine what that entails. > defining metrics in Claude that you want to use for statistical analysis e.g. for z-score what is it based on and what type of calculation? make sure you understand the calculations being performed for any metrics that you are doing a study for and modify them accordingly. > prompting Claude to do a statistical analysis with specific instructions and then tell it to create visualization for this. I've been messing around with this and I'm seriously impressed by the output.

$BTC Statistical Study using Claude - A Beginner's Workflow Here's an example of a z-score study on $BTC - still tinkering so don't take this as overtly useful information but the creation of a dashboard for the visualization of statistical data is phenomenal. my current workflow: > import $BTC time data - .csv file (can get this information from multiple venues - Binance is where I got mine) > creating bins - if you have a larger data set - you can use recent data to prevent overt bias in the long direction or filter consolidation and trending regime data into separate bins for statistical analysis - however, you will have to define thresholds and determine what that entails. > defining metrics in Claude that you want to use for statistical analysis e.g. for z-score what is it based on and what type of calculation? make sure you understand the calculations being performed for any metrics that you are doing a study for and modify them accordingly. > prompting Claude to do a statistical analysis with specific instructions and then tell it to create visualization for this. I've been messing around with this and I'm seriously impressed by the output.

Stoic

19,816 görüntüleme • 1 yıl önce

Sharing our latest short course: Building and Evaluating Data Agents, created in collaboration with Snowflake and taught by Anupam Datta (Anupam Datta) and Josh Reini (Josh Reini). A data agent extracts data from sources such as files or databases, analyzes it, and provides insights and visualizes its findings. But most data agents struggle with reliability or can't handle multi-step reasoning. In this course, you'll learn to build, trace, and evaluate a multi-agent workflow that plans tasks, pulls context from structured and unstructured data, performs web search, and summarizes or visualizes the final results. Learn more and enroll for free!

Sharing our latest short course: Building and Evaluating Data Agents, created in collaboration with Snowflake and taught by Anupam Datta (Anupam Datta) and Josh Reini (Josh Reini). A data agent extracts data from sources such as files or databases, analyzes it, and provides insights and visualizes its findings. But most data agents struggle with reliability or can't handle multi-step reasoning. In this course, you'll learn to build, trace, and evaluate a multi-agent workflow that plans tasks, pulls context from structured and unstructured data, performs web search, and summarizes or visualizes the final results. Learn more and enroll for free!

DeepLearning.AI

40,878 görüntüleme • 1 yıl önce

ChatGPT connecting to external data sources offers a glimpse into what AI interoperability looks like. Ask any question in ChatGPT and it can now pull data from Box. The future of AI is AI Agents that can access your systems and data to complete any task for you.

ChatGPT connecting to external data sources offers a glimpse into what AI interoperability looks like. Ask any question in ChatGPT and it can now pull data from Box. The future of AI is AI Agents that can access your systems and data to complete any task for you.

Aaron Levie

57,745 görüntüleme • 1 yıl önce

PhD Students - How to analyze data in seconds? 1. Go to 2. Upload your data 3. Describe how you want the data to be analyzed 4. Novix will handle the entire analysis It will perform the following data analysis tasks. - Load and validate the dataset - Generate graphs for exploratory analysis - Create histograms for numeric variables - Build a correlation heatmap - Create bar charts for categorical features - Generate box plots for score distributions - Create scatter plots for relationships - Generate a final report Download the report in PDF or Word format.

PhD Students - How to analyze data in seconds? 1. Go to 2. Upload your data 3. Describe how you want the data to be analyzed 4. Novix will handle the entire analysis It will perform the following data analysis tasks. - Load and validate the dataset - Generate graphs for exploratory analysis - Create histograms for numeric variables - Build a correlation heatmap - Create bar charts for categorical features - Generate box plots for score distributions - Create scatter plots for relationships - Generate a final report Download the report in PDF or Word format.

Faheem Ullah

16,138 görüntüleme • 5 ay önce

Did you know you can use Promise.allSettled() for async error handling with a single promise? Collect errors and returned data at the same level!

Did you know you can use Promise.allSettled() for async error handling with a single promise? Collect errors and returned data at the same level!

Wes Bos

112,724 görüntüleme • 1 yıl önce

Did you know that Facebook made $134.9 billion in 2023 from selling your data? Our society has come to the point where we accepted an unfair model in which privacy is being exploited and data gifted to big tech that is selling it for billions. We intend to break the norm of how data is shared on the internet because it is outdated and unfair. You should be empowered with more control and monetization models for your data!

Did you know that Facebook made $134.9 billion in 2023 from selling your data? Our society has come to the point where we accepted an unfair model in which privacy is being exploited and data gifted to big tech that is selling it for billions. We intend to break the norm of how data is shared on the internet because it is outdated and unfair. You should be empowered with more control and monetization models for your data!

Solana ID 🪷

30,466 görüntüleme • 2 yıl önce

Use this FREE tool to analyze data in 10 sec. This tool is used by more than 500,000 researchers. 1. Go to and log in 2. Click on Data Analysis and upload your data file. 3. Write your prompt for data analysis. 4. E.g., generate graphs for exploratory data analysis. 5. Liner will generate a variety of graphs for you. 6. It also generates an insightful data analysis report. ✦ You can edit the report. For example, you can: ✦ Insert citations in the report ✦ Make changes to the text ✦ Add/remove any graphs After making changes, you can download the report. Try Liner today. It’s FREE. Liner link:

Use this FREE tool to analyze data in 10 sec. This tool is used by more than 500,000 researchers. 1. Go to and log in 2. Click on Data Analysis and upload your data file. 3. Write your prompt for data analysis. 4. E.g., generate graphs for exploratory data analysis. 5. Liner will generate a variety of graphs for you. 6. It also generates an insightful data analysis report. ✦ You can edit the report. For example, you can: ✦ Insert citations in the report ✦ Make changes to the text ✦ Add/remove any graphs After making changes, you can download the report. Try Liner today. It’s FREE. Liner link:

Faheem Ullah

11,984 görüntüleme • 6 ay önce

A local utility company in Illinois has plans to help lower residents electrical bills They will be requiring new data centers and planned data centers to sign 10 year contracts saying they will be responsible for paying for upgrade costs and electricity usage costs “That means any new data centers would be on the hook for power upgrades and any energy they plan to use, even if the data center ends up not getting built or if they don't use as much power as planned or if they close. So if they use the power that they're applying for, then they'll pay their fair share of the cost of maintaining and building our grid through their rates”

A local utility company in Illinois has plans to help lower residents electrical bills They will be requiring new data centers and planned data centers to sign 10 year contracts saying they will be responsible for paying for upgrade costs and electricity usage costs “That means any new data centers would be on the hook for power upgrades and any energy they plan to use, even if the data center ends up not getting built or if they don't use as much power as planned or if they close. So if they use the power that they're applying for, then they'll pay their fair share of the cost of maintaining and building our grid through their rates”

Wall Street Apes

29,410 görüntüleme • 7 ay önce

SQL has levels to it: - level 1 SELECT, FROM, WHERE, GROUP BY, HAVING, LIMIT Master these basic keywords and you’ll be well on your way to mastering SQL. - level 2 Mastering JOINs: Most common JOINs: INNER and LEFT Less common JOINs: FULL OUTER Joins you should avoid almost always: RIGHT and CROSS JOIN Mastering common table expressions (CTEs). The WITH keyword defines a CTE which you can imagine as a “variable” that you can query later. Using variables like this you can master algorithm techniques like recursion, breadth first search and more! CTEs also make your SQL much more readable and make your coworkers hate you less compared to nested sub queries. - level 3 Mastering window functions Window functions have 3 pieces: The function (i.e. SUM, RANK, AVG) The over clause to start the window The window definition which has 3 pieces: - how to split the window up with PARTITION BY - how to order the window with ORDER BY - how to restrict the window size with ROWS clause (useful for rolling monthly averages) Understand RANK vs DENSE_RANK vs ROW_NUMBER, I have been asked this in interviews a million times. - level 4 You understand table scans, b-tree indexes, and partitioning schemes to increase performance. Doing something like COUNT(CASE WHEN) is much better than doing multiple queries with a UNION ALL. UNION ALL is terrible for all sorts of reasons that I don’t want to get into in this post. B-trees indexes allow for efficient scanning of data in the WHERE clause. Use explain plans to understand if an index is actually being used or not! Partitioning is similar to indexes except it’s a “poor mans” index. It just keeps data in specific folders and skips the folders that don’t include the data I question. What else did I miss for mastering SQL?

SQL has levels to it: - level 1 SELECT, FROM, WHERE, GROUP BY, HAVING, LIMIT Master these basic keywords and you’ll be well on your way to mastering SQL. - level 2 Mastering JOINs: Most common JOINs: INNER and LEFT Less common JOINs: FULL OUTER Joins you should avoid almost always: RIGHT and CROSS JOIN Mastering common table expressions (CTEs). The WITH keyword defines a CTE which you can imagine as a “variable” that you can query later. Using variables like this you can master algorithm techniques like recursion, breadth first search and more! CTEs also make your SQL much more readable and make your coworkers hate you less compared to nested sub queries. - level 3 Mastering window functions Window functions have 3 pieces: The function (i.e. SUM, RANK, AVG) The over clause to start the window The window definition which has 3 pieces: - how to split the window up with PARTITION BY - how to order the window with ORDER BY - how to restrict the window size with ROWS clause (useful for rolling monthly averages) Understand RANK vs DENSE_RANK vs ROW_NUMBER, I have been asked this in interviews a million times. - level 4 You understand table scans, b-tree indexes, and partitioning schemes to increase performance. Doing something like COUNT(CASE WHEN) is much better than doing multiple queries with a UNION ALL. UNION ALL is terrible for all sorts of reasons that I don’t want to get into in this post. B-trees indexes allow for efficient scanning of data in the WHERE clause. Use explain plans to understand if an index is actually being used or not! Partitioning is similar to indexes except it’s a “poor mans” index. It just keeps data in specific folders and skips the folders that don’t include the data I question. What else did I miss for mastering SQL?

Zach Wilson

79,773 görüntüleme • 1 yıl önce