正在加载视频...

视频加载失败

Your ETL pipeline has more orchestration code than transformation logic. AWS Glue 6.0: Delivers 30% lower price, Iceberg v3 with VARIANT shredding, Spark 4.1 and Python 3.13 runtimes, Spark Declarative Pipelines and Real-Time Mode

27,908 次观看 • 7 天前 •via X (Twitter)

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Building Data Pipelines has levels to it: - level 0 Understand the basic flow: Extract → Transform → Load (ETL) or ELT This is the foundation. - Extract: Pull data from sources (APIs, DBs, files) - Transform: Clean, filter, join, or enrich the data - Load: Store into a warehouse or lake for analysis You’re not a data engineer until you’ve scheduled a job to pull CSVs off an SFTP server at 3AM! level 1 Master the tools: - Airflow for orchestration - dbt for transformations - Spark or PySpark for big data - Snowflake, BigQuery, Redshift for warehouses - Kafka or Kinesis for streaming Understand when to batch vs stream. Most companies think they need real-time data. They usually don’t. level 2 Handle complexity with modular design: - DAGs should be atomic, idempotent, and parameterized - Use task dependencies and sensors wisely - Break transformations into layers (staging → clean → marts) - Design for failure recovery. If a step fails, how do you re-run it? From scratch or just that part? Learn how to backfill without breaking the world. level 3 Data quality and observability: - Add tests for nulls, duplicates, and business logic - Use tools like Great Expectations, Monte Carlo, or built-in dbt tests - Track lineage so you know what downstream will break if upstream changes Know the difference between: - a late-arriving dimension - a broken SCD2 - and a pipeline silently dropping rows At this level, you understand that reliability > cleverness. level 4 Build for scale and maintainability: - Version control your pipeline configs - Use feature flags to toggle behavior in prod - Push vs pull architecture - Decouple compute and storage (e.g. Iceberg and Delta Lake) - Data mesh, data contracts, streaming joins, and CDC are words you throw around because you know how and when to use them. What else belongs in the journey to mastering data pipelines?

Zach Wilson

16,688 次观看 • 1 年前

Your agents can't keep up with real-time data. Especially when it's scattered across dozens of sources. Most teams waste weeks building custom connectors for every database, API, and data warehouse. Then they build ETL pipelines to sync everything. By the time your agent retrieves the data, it's already outdated. Picture this: Your Postgres database updated 5 minutes ago. Your MongoDB collection changed 2 minutes ago. Your agent is still pulling from yesterday's snapshot. This is why most production RAG systems fail. There's a better approach: MindsDB is an open-source AI platform with a federated data engine that lets you query multiple data sources in real-time using SQL - without moving any data. Here's what makes it different: ↳ Your data stays in place. No ETL pipelines or data duplication ↳ Query Postgres, MongoDB, REST APIs, and more using consistent SQL ↳ JOIN across different sources in real-time with a unified interface ↳ Works with both structured and un-structured data And here's the best part: You don't even need to write SQL. Just describe what you want in plain English, and MindsDB converts it to SQL automatically. The system does all the heavy lifting. The breakthrough for AI agents is simple: When data updates at the source, your agent gets fresh results immediately. No sync delays. No stale embeddings. No custom code for each integration. You can literally write a SQL query that joins a Postgres table with a MongoDB collection and gets live results. This is what production AI applications need but rarely get. In this video, I give you a complete walkthrough of what we just discussed and how to actually do it. Make sure you watch this till the end. I've shared the link to MindsDB's GitHub repo in the next tweet!

Akshay 🚀

65,672 次观看 • 9 个月前

Jupiter prides itself as the consumer face of Defi lend, perps, pro, prediction markets,jup has it all! now it gets even better with the ULTRA V3! the ultra V3 is Crypto’s ultimate trading engine! It introduces;👇 •Iris, our new router which uses Brent’s method for route splitting and capable of more granular splitting, up to 0.01%. •ShadowLane - Jupiter’s in-house transaction landing engine which leverages on its own validator stake and dedicated R&D efforts to consistently process transactions within sub-second latency It operates entirely on Jupiter’s infrastructure instead of relying on external providers, this eliminates the risk of artificial delays and front-running ensuring faster and more secure execution for jupiter users. With shadowlane, Jupiter minimizes your exposure to susceptible MEV risks by ensuring trades are never handed off to any external providers for onchain execution. The Ultra V3 introduces Predictive Execution. The Ultra V3 introduces JupiterZ which facilitates about $100M daily volume all with ZERO SLIPPAGE! Ultra V3 also introduces Ultra Signaling, which allows Prop AMMs to distinguish between different types of user flow when submitting quotes to Ultra. The ultra signaling differentiates Ls between “non-toxic” and “toxic” order flow when quoting. With Ultra V3, Jupiter accounts for slippage when deciding which AMM to use. This ensures that the provided quotes reflect actual onchain execution, not theoretical or inflated “best prices”. The ultra v3 also has gasless support coverage! With the Gasless Support acting as a just-in-time fee payer, you can make trades on ULTRA V3 without holding $SOL to cover gas fees, Ultra automatically calculates and covers the gas fee, deducting it from the swap amount. Ultra v3 fundamentally transforms swap execution with top tier improvements at every level resulting in •the best price, •the best execution (3x more trades landed) •the best price protection (34x better sandwich protection) note: meta aggregation with the new iris router means you literally get the best price across dflow, hashflow, okx, and more Ultra V3’s execution is insane too - 34x better sandwich protection, lower slippage, and 8-10x cheaper fees. The predictive routing + gasless support combo also make your trading experience seamless! The ULTRA v3 runs across the web, Jupiter mobile, desktop wallet and jup pro tools. It is majorly a spot construct meaning you can swap studio tokens with v3 using the jup mobile (every swap in the jup mobile is powered by the ultra V3). Try the ULTRA V3 here - or through the swap ui or pro ui in the Jupiter Mobile!

DUKE 🇲🇾

85,423 次观看 • 10 个月前

gemini 3.7 flash vs deepseek v4 pro 0813 vs muse spark 1.2 – on voxel city dioramas three models each built three crossy road-style 3d scenes – a construction site, a nyc intersection, a river with a drawbridge – as single self-contained html files the setup: Nous Research's hermes agent cli on OpenRouter, three.js skills preloaded, identical prompts per scene tasks: 1. construction site – tower crane on a working lift loop, paver laying fresh road, roller compacting it behind 2. nyc crossing – four-way intersection with a traffic light state machine, queuing cars, pedestrians crossing on the walk signal 3. river drawbridge – double-leaf bascule that lifts for tall boats, cars queuing at the barriers, animated water every scene: Three.js r185, box geometry only, a locked 20-color palette, four camera presets, and a day/night mode with bloom. one file, no build step, no assets models: Google DeepMind gemini 3.7 flash, DeepSeek v4 pro 0813, AI at Meta muse spark 1.2 muse and gemini finished every scene in two to three minutes. deepseek took 15 to 41 minutes per scene - build time, all three scenes #1 gemini 3.7 flash – 6m 43s #2 muse spark 1.2 – 7m 20s #3 deepseek v4 pro – 91m 25s - total tokens #1 muse spark 1.2 – 440,279 #2 gemini 3.7 flash – 713,855 #3 deepseek v4 pro – 20,957,568 - total price #1 muse spark 1.2 – $0.53 #2 gemini 3.7 flash – $0.56 #3 deepseek v4 pro – $4.57 - agent calls across the three builds #1 muse spark 1.2 – 12 #2 gemini 3.7 flash – 18 #3 deepseek v4 pro – 143 observations: • muse won two of the three scenes on looks with the smallest files in the test – 887 to 1,042 lines against gemini's 1,934 to 2,377. cheapest, fastest to a good frame, and shortest turned out to be the same column • deepseek burned 20.96m tokens – 29x gemini, 48x muse – across 143 agent calls. prompt caching is the only reason that cost $4.57: the cache discount absorbed roughly $30 of resent context • gemini was the only model whose files needed zero fixes to render – and the only one whose night mode is cosmetic. the sky never darkens and one camera button does nothing. clean code for a scene it never looked at follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

28,407 次观看 • 10 天前

New short course: Serverless Agentic Workflows with Amazon Bedrock. Learn to build and deploy serverless agents in this course created with Amazon Web Services and taught by Mike G Chambers, a Senior Developer Advocate at AWS specializing in GenAI. (Disclosure: I serve on Amazon's board.) Generative AI applications are becoming more complex, sophisticated, and agentic. Agentic applications have workloads that can be hard to predict in advance -- for example, what tools will it decide to call? -- and a serverless architecture helps you efficiently providing on-demand resources. This course teaches you to build and deploy a serverless agentic application. You’ll learn to create agents with tools, code execution, and guardrails, and build responsible agents for business use cases: - Build a customer service bot for a fictional tea mug business that can answering questions, retrieve information, and process orders. - Connect your customer service agent to a CRM to get customer info and log support tickets in real-time. - Explore how you invoke the agent, and see the trace to review the agent’s thought process and observation loop until it reaches its final output. - Attach a code interpreter to your agent, giving it the ability to perform accurate calculations by writing and running its own Python code. - Implement guardrails to prevent your agent from revealing sensitive information or using inappropriate language. By the end, you will have built a sophisticated AI agent capable of handling real-world customer support scenarios. Please sign up here!

Andrew Ng

81,048 次观看 • 1 年前

I'm teaching a new course! AI Python for Beginners is a series of four short courses that teach anyone to code, regardless of current technical skill. We are offering these courses free for a limited time. Generative AI is transforming coding. This course teaches coding in a way that’s aligned with where the field is going, rather than where it has been: (1) AI as a Coding Companion. Experienced coders are using AI to help write snippets of code, debug code, and the like. We embrace this approach and describe best-practices for coding with a chatbot. Throughout the course, you'll have access to an AI chatbot that will be your own coding companion that can assist you every step of the way as you code. (2) Learning by Building AI Applications. You'll write code that interacts with large language models to quickly create fun applications to customize poems, write recipes, and manage a to-do list. This hands-on approach helps you see how writing code that calls on powerful AI models will make you more effective in your work and personal projects. With this approach, beginning programmers can learn to do useful things with code far faster than they could have even a year ago. Knowing a little bit of coding is increasingly helping people in job roles other than software engineers. For example, I've seen a marketing professional write code to download web pages and use generative AI to derive insights; a reporter write code to flag important stories; and an investor automate the initial drafts of contracts. With this course you’ll be equipped to automate repetitive tasks, analyze data more efficiently, and leverage AI to enhance your productivity. If you are already an experienced developer, please help me spread the word and encourage your non-developer friends to learn a little bit of coding. I hope you'll check out the first two short courses here!

Andrew Ng

1,225,302 次观看 • 2 年前

I just compared Claude Code vs Codex vs Cursor CLI The task was to build a Next.js app with Tailwind 4 and shadcn components to collect customer feedback and showcase it with a widget. I gave all three the same prompt and let them go for 30 minutes to see what they came up with. Claude Code with Opus 4.1 Even though I told it to set up the app in the existing project folder, it tried to create a directory for it. After I interrupted and told it not to do that, it built a demo form and landing page with no errors. I had to ask it to make the demo interactive so users could submit a testimonial and preview it. The landing page looked like AI and was pretty basic, but it worked and it was done in a fraction of the time of the others. Total tokens used: 33k Codex with GPT-5 At the end of the 30 minutes I just could not get Codex to produce a working app. It got stuck in a loop of not being able to set up Tailwind 4 and despite many, MANY, attempts, I ended up with a "failed to compile" error. Total tokens used: 102k Cursor Agent with GPT-5 This was the slowest agent by far and a couple of times I actually thought it got stuck in a loop and was close to Ctrl+C'ing to cancel it. The TUI is really nice though, especially how it shows diffs and it did eventually build a working app (after one or two slight errors that needed fixing) The demo was interactive and it had a very minimal design that looked bare but also a lot less like an "AI generated" app than the Opus 4.1 design. It also wasn't too chatty and just did what it needed to do! Code quality was on a par with Opus 4.1, but it did use 5.5x as many tokens to get there. Still cheaper than Opus on a direct comparison but not when you factor in a Claude Code Max subscription. Total tokens: 188k I'll be able to do a proper comparison and record some videos when I'm back from holiday but for now, Opus is still the more capable model out of the box and Claude Code is the more complete CLI product. It will be interesting to see how Cursor evolve their CLI though with commands and subagents because I think with GPT-5 they have a real shot at providing competition for Claude Code if they can optimise output to get similar quality with less tokens. Jump to 0:40 in the video to see the two apps. Which do you think is which? ;)

Ian Nuttall

195,173 次观看 • 1 年前