正在加载视频...

视频加载失败

返回首页

Mistral AI Releases Robostral Navigate: An 8B Model Enabling Robots to Navigate Complex Environments Hitting 76.6% on R2R-CE With One RGB Camera. No LiDAR. No depth sensor. No multi-camera rig. Here's how it works. 👇 1. Pointing, not metric commands The model predicts the pixel coordinates of the next...

Marktechpost's profile picture

Marktechpost AI

11,455 subscribers

39,955 次观看 • 5 天前 •via X (Twitter)

科学技术教育新闻政治
LIVE

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

While Anthropic and OpenAI race on AI Agents/LLMs, Mistral continues expanding its footprint in physical AI. They just released Robostral Navigate (8B VLA model). It shows strong potential in delivery, logistics, manufacturing, and hospitality--for example, enabling humanoid robots to autonomously navigate multiple rooms to hand a glass of water to an employee,all via natural language instructions in complex environments. Powered by just a single RGB camera (no LiDAR or depth sensors), it achieves 76.6% success on R2R-CE (Room-to-Room in Continuous Environments) unseen benchmarks, outperforming multi-sensor systems. Trained purely in simulation, it generalizes across wheeled, legged, and flying robots.
0:25

While Anthropic and OpenAI race on AI Agents/LLMs, Mistral continues expanding its footprint in physical AI. They just released Robostral Navigate (8B VLA model). It shows strong potential in delivery, logistics, manufacturing, and hospitality--for example, enabling humanoid robots to autonomously navigate multiple rooms to hand a glass of water to an employee,all via natural language instructions in complex environments. Powered by just a single RGB camera (no LiDAR or depth sensors), it achieves 76.6% success on R2R-CE (Room-to-Room in Continuous Environments) unseen benchmarks, outperforming multi-sensor systems. Trained purely in simulation, it generalizes across wheeled, legged, and flying robots.

CyberRobo

10,704 次观看 • 6 天前

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:
2:52

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:

Andrew Ng

125,146 次观看 • 1 年前

The robot’s “eyes” just received a big upgrade. LingBot-Depth 2.0, a depth-completion model with half the depth error just dropped. 12/16 benchmarks topped. Glass, mirrors, and transparent objects are so easy for us humans, but so hard for robots, because they do not behave like ordinary surfaces in a camera pipeline. A robot that misunderstands a balcony window or a table edge, will have a completely false planning inside a false world. Huge implecation. LingBot-Depth 2.0 takes an RGB image plus a broken depth map from a sensor and then outputs a cleaner depth map and a usable 3D point cloud. Numbers on LingBot-Depth 2.0 - Excels on glass, mirrors & transparent objects — where traditional depth cameras fail - Training data: 3M → 150M (50x scale-up) - 12 out of 16 first-place rankings on depth completion benchmarks - RMSE cut in half: 0.132 → 0.062 on the hardest indoor scenes LingBot-Vision trained on boundaries, because object edges carry the geometry robots need. No human boundary labels are used, which makes this approach easier to scale. The open-sourced LingBot-Vision is the general vision backbone, and LingBot-Depth 2.0 is the depth model built on it.
2:22

The robot’s “eyes” just received a big upgrade. LingBot-Depth 2.0, a depth-completion model with half the depth error just dropped. 12/16 benchmarks topped. Glass, mirrors, and transparent objects are so easy for us humans, but so hard for robots, because they do not behave like ordinary surfaces in a camera pipeline. A robot that misunderstands a balcony window or a table edge, will have a completely false planning inside a false world. Huge implecation. LingBot-Depth 2.0 takes an RGB image plus a broken depth map from a sensor and then outputs a cleaner depth map and a usable 3D point cloud. Numbers on LingBot-Depth 2.0 - Excels on glass, mirrors & transparent objects — where traditional depth cameras fail - Training data: 3M → 150M (50x scale-up) - 12 out of 16 first-place rankings on depth completion benchmarks - RMSE cut in half: 0.132 → 0.062 on the hardest indoor scenes LingBot-Vision trained on boundaries, because object edges carry the geometry robots need. No human boundary labels are used, which makes this approach easier to scale. The open-sourced LingBot-Vision is the general vision backbone, and LingBot-Depth 2.0 is the depth model built on it.

Rohan Paul

25,312 次观看 • 12 天前

AI BASKETBALL ANALYSIS. A FULL COMPUTER VISION SYSTEM. BUILT ON YOLO, OPENCV, AND PYTHON. Take any NBA broadcast. Any camera angle. Any resolution. Feed it into the system. YOLO finds every player and the ball. Frame by frame. No manual annotation. No pre-labeled data.The model just sees the court and understands it. Zero-shot classification looks at jersey pixels and decides who plays for whom. Two teams separated in milliseconds. Without ever being told the team names. But here's where it gets wild. Court keypoint detection identifies the geometry of the playing surface. Homography transforms the broadcast camera into a top-down tactical map. Real meters. Real coordinates. From a flat 2D video. Now every player has a position on a real court. Every movement becomes a data point. Every pass becomes a vector. Speed. Distance. Possession time. Pass networks. All extracted from nothing but pixels. No GPS. No chips in the ball. No million-dollar Second Spectrum setup. A Python script and a GPU. The full pipeline is open source. The tutorial walks through every line of code. From detection to transformation to analytics output. Teams used to pay six figures for this data. Now you build it in an afternoon. Computer vision didn't just enter basketball. It democratized it.
5:58:26

AI BASKETBALL ANALYSIS. A FULL COMPUTER VISION SYSTEM. BUILT ON YOLO, OPENCV, AND PYTHON. Take any NBA broadcast. Any camera angle. Any resolution. Feed it into the system. YOLO finds every player and the ball. Frame by frame. No manual annotation. No pre-labeled data.The model just sees the court and understands it. Zero-shot classification looks at jersey pixels and decides who plays for whom. Two teams separated in milliseconds. Without ever being told the team names. But here's where it gets wild. Court keypoint detection identifies the geometry of the playing surface. Homography transforms the broadcast camera into a top-down tactical map. Real meters. Real coordinates. From a flat 2D video. Now every player has a position on a real court. Every movement becomes a data point. Every pass becomes a vector. Speed. Distance. Possession time. Pass networks. All extracted from nothing but pixels. No GPS. No chips in the ball. No million-dollar Second Spectrum setup. A Python script and a GPU. The full pipeline is open source. The tutorial walks through every line of code. From detection to transformation to analytics output. Teams used to pay six figures for this data. Now you build it in an afternoon. Computer vision didn't just enter basketball. It democratized it.

zostaff

230,194 次观看 • 2 个月前

Mistral AI Releases Leanstral 1.5: An Apache-2.0 Lean 4 Code Agent Model Solving 587 of 672 PutnamBench Problems Most AI theorem proving is a language model generating a proof in one shot, with a verifier bolted on at the end to check it. That's autocomplete with a grader — and Mistral just drew a clear line between that and an actual proof agent. They released Leanstral 1.5 — a 119B MoE with 6.5B active parameters, trained as a code agent that lives inside the Lean 4 compiler loop: propose a proof, read the compiler's goals and errors, refine, repeat until it compiles or the budget runs out. Verification isn't the eval here. It's the training signal. Here's what's actually interesting: → Test-time scaling behaves like a dial: PutnamBench Pass@8 climbs 44 → 244 → 493 → 587 solved as the per-attempt token budget moves 50k → 200k → 1M → 4M → 587/672 on PutnamBench at ~$4 per problem, versus an estimated $300+ for Seed-Prover 1.5 high (a 10 H20-days-per-problem budget) → Saturates miniF2F: 100% on both validation and test sets → Two RL environments in training — a multiturn prover, and a raw-filesystem code agent that edits files, runs bash, and queries the Lean language server for live goals and types → Not just math: an Aeneas (Rust → Lean) pipeline flagged 11 genuine bugs across 57 repos, 5 previously unreported — including an integer overflow in datrs/varinteger when (value + 1) hits Std.U64.MAX Apache 2.0 weights, free API endpoint Full analysis: Model weights: Project: Technical Details: Mistral AI Mistral AI for Developers Sophia Yang, Ph.D.
0:25

Mistral AI Releases Leanstral 1.5: An Apache-2.0 Lean 4 Code Agent Model Solving 587 of 672 PutnamBench Problems Most AI theorem proving is a language model generating a proof in one shot, with a verifier bolted on at the end to check it. That's autocomplete with a grader — and Mistral just drew a clear line between that and an actual proof agent. They released Leanstral 1.5 — a 119B MoE with 6.5B active parameters, trained as a code agent that lives inside the Lean 4 compiler loop: propose a proof, read the compiler's goals and errors, refine, repeat until it compiles or the budget runs out. Verification isn't the eval here. It's the training signal. Here's what's actually interesting: → Test-time scaling behaves like a dial: PutnamBench Pass@8 climbs 44 → 244 → 493 → 587 solved as the per-attempt token budget moves 50k → 200k → 1M → 4M → 587/672 on PutnamBench at ~$4 per problem, versus an estimated $300+ for Seed-Prover 1.5 high (a 10 H20-days-per-problem budget) → Saturates miniF2F: 100% on both validation and test sets → Two RL environments in training — a multiturn prover, and a raw-filesystem code agent that edits files, runs bash, and queries the Lean language server for live goals and types → Not just math: an Aeneas (Rust → Lean) pipeline flagged 11 genuine bugs across 57 repos, 5 previously unreported — including an integer overflow in datrs/varinteger when (value + 1) hits Std.U64.MAX Apache 2.0 weights, free API endpoint Full analysis: Model weights: Project: Technical Details: Mistral AI Mistral AI for Developers Sophia Yang, Ph.D.

Marktechpost AI

56,461 次观看 • 15 天前

For the past 3 months, I've been heads down building and extending our multimodal data capture rig for Physical AI training. What you're seeing here is a single synchronized capture from our system — RGB, depth (stereo IR camera) feed; real-time 3D hand reconstruction, and a 23×20 tactile pressure grid from sensor gloves, all aligned and timestamped together. Getting here took a lot of unglamorous iteration. We reworked the pipeline to deliver clean, gap-free depth at scale after identifying early depth frame drops caused by data throughput bottlenecks in our capture box. We built per-channel calibration models that convert raw resistance signals from our flex sensor gloves into accurate, drift-corrected force values — solving the creep behavior that made a steady grip falsely appear to fade over time. Even though we’ve stayed quiet on socials, word of mouth spread — our rigs have already been deployed in data factories, manufacturing lines, and homes across the US, Europe, China, and Southeast Asia by our customers and ourselves. In the next few days, I'll be sharing more about how we're tackling rights-cleared data capture from the hardware level, and our closed-loop data-model flywheel — using model evaluation results to identify gaps and guide what data our customers need. More soon, stay tuned…
0:26

For the past 3 months, I've been heads down building and extending our multimodal data capture rig for Physical AI training. What you're seeing here is a single synchronized capture from our system — RGB, depth (stereo IR camera) feed; real-time 3D hand reconstruction, and a 23×20 tactile pressure grid from sensor gloves, all aligned and timestamped together. Getting here took a lot of unglamorous iteration. We reworked the pipeline to deliver clean, gap-free depth at scale after identifying early depth frame drops caused by data throughput bottlenecks in our capture box. We built per-channel calibration models that convert raw resistance signals from our flex sensor gloves into accurate, drift-corrected force values — solving the creep behavior that made a steady grip falsely appear to fade over time. Even though we’ve stayed quiet on socials, word of mouth spread — our rigs have already been deployed in data factories, manufacturing lines, and homes across the US, Europe, China, and Southeast Asia by our customers and ourselves. In the next few days, I'll be sharing more about how we're tackling rights-cleared data capture from the hardware level, and our closed-loop data-model flywheel — using model evaluation results to identify gaps and guide what data our customers need. More soon, stay tuned…

Grace Zhang

52,665 次观看 • 1 个月前

Here's my conversation with Lucas Atkins and the team at Arcee.ai on their path to training and releasing Trinity Large today. From going all in on open models built end to end in the US 6 months ago to having the model in hand is no easy feet. I loved this conversation on how to design a startup around open models and take a bold step to scale it up. I'm openly an Arcee fan, watching them take risk and pull it off. We discuss: - The state (and future) of open vs. closed models, - The business of selling open models for on-prem deployments, - The story of Arcee AI & going “all-in” on this training run, - The ATOM project, - Building frontier model training teams in 6 months, - and other great topics. I really loved this one, and think you well too. Chapters: 00:00:00 Intro: Arcee AI, Trinity Models & Trinity Large 00:08:26 Transitioning a Company to Pre-training 00:13:00 Technical Decisions: Muon and MoE 00:18:41 Scaling and MoE Training Pain 00:23:14 Post-training and RL Strategies 00:28:09 Team Structure and Data Scaling 00:31:31 The Trinity Manifesto: US Open Weights 00:42:31 Specialized Models and Distillation 00:47:12 Infrastructure and Hosting 400B 00:50:53 Open Source as a Business Moat 00:56:31 Predictions: Best Model in 2026 01:02:29 Lightning Round & Conclusions More great open model builder podcasts coming soon!
1:12:15

Here's my conversation with Lucas Atkins and the team at Arcee.ai on their path to training and releasing Trinity Large today. From going all in on open models built end to end in the US 6 months ago to having the model in hand is no easy feet. I loved this conversation on how to design a startup around open models and take a bold step to scale it up. I'm openly an Arcee fan, watching them take risk and pull it off. We discuss: - The state (and future) of open vs. closed models, - The business of selling open models for on-prem deployments, - The story of Arcee AI & going “all-in” on this training run, - The ATOM project, - Building frontier model training teams in 6 months, - and other great topics. I really loved this one, and think you well too. Chapters: 00:00:00 Intro: Arcee AI, Trinity Models & Trinity Large 00:08:26 Transitioning a Company to Pre-training 00:13:00 Technical Decisions: Muon and MoE 00:18:41 Scaling and MoE Training Pain 00:23:14 Post-training and RL Strategies 00:28:09 Team Structure and Data Scaling 00:31:31 The Trinity Manifesto: US Open Weights 00:42:31 Specialized Models and Distillation 00:47:12 Infrastructure and Hosting 400B 00:50:53 Open Source as a Business Moat 00:56:31 Predictions: Best Model in 2026 01:02:29 Lightning Round & Conclusions More great open model builder podcasts coming soon!

Nathan Lambert

26,872 次观看 • 5 个月前

Molmo by Ai2 - Open source SoTA Multimodal (Vision) Language model, beating Claude 3.5 Sonnet, GPT4V and comparable to GPT4o 🔥 They release four model checkpoints: 1. MolmoE-1B, a mixture of experts model with 1B (active) 7B (total) 2. Molmo-7B-O, most open 7B model 3. Molmo-7B-D, demo model 4. Molmo-72B, best model System Architecture > Input: Multi-scale, multi-crop images generated from the original image. > Vision Encoder: OpenAI's ViT-L/14 336px CLIP model, a powerful ViT, encodes images into vision tokens. > Connector: MLP projects tokens to LLM input space, followed by pooling for dimensionality reduction. > LLM: Decoder-only Transformer, various options (OLMo, OLMoE, Qwen2, Mistral, Gemma2, Phi) with diverse scales and openness. Model Variants > Vision Encoder: Consistent ViT-L/14 CLIP model across variants. > LLM: OLMo-7B-1024, OLMoE-1B-7B-0924, Qwen2 (7B, 72B), Mistral 7B, Gemma2 9B, Phi 3 Medium, offering different capacities and openness levels. Training Strategy > Stage 1: Multimodal pre-training for caption generation with new captioning data. > Stage 2: Supervised fine-tuning on a dataset mixture, updating all parameters. > No RLHF involved, Learning rates adjusted based on component types and pre-training status. > All the weights are available on Hugging Face Hub 🤗 > Compatible with Transformers (Remote Code) Kudos Ai2 for such a brilliant and open work! 🐐 Video credits: Allen AI YT Channel
2:15

Molmo by Ai2 - Open source SoTA Multimodal (Vision) Language model, beating Claude 3.5 Sonnet, GPT4V and comparable to GPT4o 🔥 They release four model checkpoints: 1. MolmoE-1B, a mixture of experts model with 1B (active) 7B (total) 2. Molmo-7B-O, most open 7B model 3. Molmo-7B-D, demo model 4. Molmo-72B, best model System Architecture > Input: Multi-scale, multi-crop images generated from the original image. > Vision Encoder: OpenAI's ViT-L/14 336px CLIP model, a powerful ViT, encodes images into vision tokens. > Connector: MLP projects tokens to LLM input space, followed by pooling for dimensionality reduction. > LLM: Decoder-only Transformer, various options (OLMo, OLMoE, Qwen2, Mistral, Gemma2, Phi) with diverse scales and openness. Model Variants > Vision Encoder: Consistent ViT-L/14 CLIP model across variants. > LLM: OLMo-7B-1024, OLMoE-1B-7B-0924, Qwen2 (7B, 72B), Mistral 7B, Gemma2 9B, Phi 3 Medium, offering different capacities and openness levels. Training Strategy > Stage 1: Multimodal pre-training for caption generation with new captioning data. > Stage 2: Supervised fine-tuning on a dataset mixture, updating all parameters. > No RLHF involved, Learning rates adjusted based on component types and pre-training status. > All the weights are available on Hugging Face Hub 🤗 > Compatible with Transformers (Remote Code) Kudos Ai2 for such a brilliant and open work! 🐐 Video credits: Allen AI YT Channel

Vaibhav (VB) Srivastav

80,474 次观看 • 1 年前

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress *continuous-time* trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar
0:23

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress *continuous-time* trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Davide Scaramuzza

12,637 次观看 • 2 年前

AI TENNIS ANALYSIS. A FULL COMPUTER VISION SYSTEM. BUILT ON YOLO, PYTORCH, AND KEYPOINT EXTRACTION. Take any tennis match broadcast, any camera angle, any resolution. Feed it into the pipeline. YOLO detects both players and the tennis ball frame by frame. No manual labeling, no pre-annotated dataset. A fine-tuned YOLOv5 model trained on a Roboflow tennis ball dataset handles the ball - the hardest object to track in any sport. Tiny, fast, constantly occluded. The model finds it anyway. Trackers maintain identity across frames so Player 1 stays Player 1 from the first serve to match point. But detection is just the start. A ResNet50 CNN trained in PyTorch predicts court keypoints from every frame - the corners, service lines, baselines, net posts. Fourteen points that define the entire playing surface geometry. From those keypoints the system builds a homography matrix and warps the broadcast perspective into a top-down mini court with real coordinates. Now every player has a position in real space, not pixel space. Every frame becomes a measurement. Every rally becomes a dataset. Player movement speed - calculated from position deltas between frames, converted to meters per second through the homography. Ball shot speed - measured from the ball trajectory across consecutive detections. Number of shots per rally - counted automatically through ball direction changes. All of this rendered live on the video as an overlay. A mini court in the corner showing both players as dots moving in real time. Stats updating after every point. OpenCV handles the rendering. Pandas handles the math. PyTorch handles the intelligence. YOLO handles the eyes. No Hawkeye subscription, no court-embedded sensors, no tracking chips in the ball. A Python script, a trained model, and a GPU. The full code is on GitHub. The tutorial walks through every module - from ball detector training to court keypoint extraction to the final statistical overlay. Professional teams used to need broadcast deals and proprietary hardware for this kind of analysis. Now you build it in an afternoon with open-source tools. Trading here: Computer vision didn't just enter tennis. It made the expensive stuff free.
4:41:24

AI TENNIS ANALYSIS. A FULL COMPUTER VISION SYSTEM. BUILT ON YOLO, PYTORCH, AND KEYPOINT EXTRACTION. Take any tennis match broadcast, any camera angle, any resolution. Feed it into the pipeline. YOLO detects both players and the tennis ball frame by frame. No manual labeling, no pre-annotated dataset. A fine-tuned YOLOv5 model trained on a Roboflow tennis ball dataset handles the ball - the hardest object to track in any sport. Tiny, fast, constantly occluded. The model finds it anyway. Trackers maintain identity across frames so Player 1 stays Player 1 from the first serve to match point. But detection is just the start. A ResNet50 CNN trained in PyTorch predicts court keypoints from every frame - the corners, service lines, baselines, net posts. Fourteen points that define the entire playing surface geometry. From those keypoints the system builds a homography matrix and warps the broadcast perspective into a top-down mini court with real coordinates. Now every player has a position in real space, not pixel space. Every frame becomes a measurement. Every rally becomes a dataset. Player movement speed - calculated from position deltas between frames, converted to meters per second through the homography. Ball shot speed - measured from the ball trajectory across consecutive detections. Number of shots per rally - counted automatically through ball direction changes. All of this rendered live on the video as an overlay. A mini court in the corner showing both players as dots moving in real time. Stats updating after every point. OpenCV handles the rendering. Pandas handles the math. PyTorch handles the intelligence. YOLO handles the eyes. No Hawkeye subscription, no court-embedded sensors, no tracking chips in the ball. A Python script, a trained model, and a GPU. The full code is on GitHub. The tutorial walks through every module - from ball detector training to court keypoint extraction to the final statistical overlay. Professional teams used to need broadcast deals and proprietary hardware for this kind of analysis. Now you build it in an afternoon with open-source tools. Trading here: Computer vision didn't just enter tennis. It made the expensive stuff free.

zostaff

120,370 次观看 • 2 个月前

a team of researchers just proved you don't need a bigger model, you need a smarter plan researchers from Tsinghua and South China University of Technology built a framework called Atomic Task Graph. it turned 7B-8B open-source models into GPT-4 competitors on complex agent benchmarks, beating it on two out of three. no fine-tuning. no extra training. zero parameter updates. current AI agents plan in a straight line. step 1, step 2, step 3. when step 4 fails, the whole chain breaks. and the longer the chain gets, the more the model hallucinates because it's reasoning over a ballooning text history. here's how it works. 1. instead of a linear chain, ATG breaks any complex task into a directed graph where subtask inputs and outputs are explicitly mapped 2. it recursively decomposes each subtask until every node is one atomic tool call 3. independent branches run in parallel instead of waiting in line 4. before anything executes, a lightweight "thought experiment" simulates the plan internally to catch bad dependencies and missing steps early 5. when something breaks at runtime, ATG traces the failure to the exact subgraph that caused it and repairs only that piece. validated work stays frozen. the old way meant a failure at step 5 forced a full replan from scratch. hallucinated actions piled up the longer the task ran. ReAct hit a 43% hallucination rate on household tasks. ATG on an 8B Llama model scored 63.65 on ALFWorld. GPT-4 with ReAct scored 41.24 on the same benchmark. hallucinated actions dropped to 12%. those numbers happened because someone stopped throwing compute at the problem and started thinking about how work gets organized. that's the part that gets me. the industry is spending billions on scale. this team spent time on architecture. and the architecture won.
0:23

a team of researchers just proved you don't need a bigger model, you need a smarter plan researchers from Tsinghua and South China University of Technology built a framework called Atomic Task Graph. it turned 7B-8B open-source models into GPT-4 competitors on complex agent benchmarks, beating it on two out of three. no fine-tuning. no extra training. zero parameter updates. current AI agents plan in a straight line. step 1, step 2, step 3. when step 4 fails, the whole chain breaks. and the longer the chain gets, the more the model hallucinates because it's reasoning over a ballooning text history. here's how it works. 1. instead of a linear chain, ATG breaks any complex task into a directed graph where subtask inputs and outputs are explicitly mapped 2. it recursively decomposes each subtask until every node is one atomic tool call 3. independent branches run in parallel instead of waiting in line 4. before anything executes, a lightweight "thought experiment" simulates the plan internally to catch bad dependencies and missing steps early 5. when something breaks at runtime, ATG traces the failure to the exact subgraph that caused it and repairs only that piece. validated work stays frozen. the old way meant a failure at step 5 forced a full replan from scratch. hallucinated actions piled up the longer the task ran. ReAct hit a 43% hallucination rate on household tasks. ATG on an 8B Llama model scored 63.65 on ALFWorld. GPT-4 with ReAct scored 41.24 on the same benchmark. hallucinated actions dropped to 12%. those numbers happened because someone stopped throwing compute at the problem and started thinking about how work gets organized. that's the part that gets me. the industry is spending billions on scale. this team spent time on architecture. and the architecture won.

Alex Veremeyenko

172,342 次观看 • 7 天前

There is a beautiful story that just happened in AI so let me share it for a lighter tone weekend post among all the doom stories in our AI field this week. It’s a story of people on three continents building and sharing in the open a new small efficient and state-of-the-art AI model. It started a couple of months ago when a new team in the AI scene released their first model from their headquarters in Paris (France): Mistral 7B. Impressive model, small and very strong performances in the benchmarks, better than all previous models of this size. And open source! So you could build on top of it. Lewis in Bern (Switzerland) and Ed (in Lyon, in the South of France) both from the H4 team, a team of researchers in model fine-tuning and alignment were talking about it over a coffee, in one of these gatherings that often happen at Hugging Face to break the distance between people (literal distance as HF is a remote company). What about fine-tuning it using this new DPO method that a research team from Stanford in California just posted on Arxiv, says one? Hey, that’s a great idea, replies the other. We've just build a great code base (with Nathan, Nazneen, Costa, Younes and all the H4 team and TRL community) let's use it! The next day they start diving in the datasets openly shared on the HF hub and stumble upon two interesting large and good quality fine-tuning datasets recently open-sourced by OpenBMB, a Chinese team from Tsinghua: UltraFeedback and UltraChat. A few rounds of training experiments confirm the intuition, the resulting model is super strong, by far the strongest they have ever seen in their benchmarks from Berkeley and Stanford (LMSYS and Alpaca). Join Clementine, the big boss of the open evaluation leaderboard. Her deep dive into the model capabilities confirms the results: impressive performance. But the H4 team also hosts a famous faculty member, Pr. Sasha Rush, Associate Professor at Cornell University in his daytime, hacker at HF in his nighttime. Joining the conversation, he proposes to quickly draft a research paper to organize and share all the details with the community. A few days later, the model, called Zephyr (a wind like Mistral), paper, and all details are shared with the world. Quickly other companies, everywhere in the world starts to use it. LlamaIndex, a famous data framework and community, shares how the model blew their expectations on real-life use-case benchmarks, while researchers and practitioners discuss the paper and work on the Hugging Face hub. All this happened in just a few weeks catalyzed by open access to knowledge, models, research, and datasets released all over the world (Europe, California, China) and by the idea that people can build upon one another work in AI to bring real-world value with efficient and open models. Stories like this are numerous everywhere around us and make me really proud of the AI community and see how we can build amazingly useful things together. [the video is just me reading this Friday post hahah]
4:30

There is a beautiful story that just happened in AI so let me share it for a lighter tone weekend post among all the doom stories in our AI field this week. It’s a story of people on three continents building and sharing in the open a new small efficient and state-of-the-art AI model. It started a couple of months ago when a new team in the AI scene released their first model from their headquarters in Paris (France): Mistral 7B. Impressive model, small and very strong performances in the benchmarks, better than all previous models of this size. And open source! So you could build on top of it. Lewis in Bern (Switzerland) and Ed (in Lyon, in the South of France) both from the H4 team, a team of researchers in model fine-tuning and alignment were talking about it over a coffee, in one of these gatherings that often happen at Hugging Face to break the distance between people (literal distance as HF is a remote company). What about fine-tuning it using this new DPO method that a research team from Stanford in California just posted on Arxiv, says one? Hey, that’s a great idea, replies the other. We've just build a great code base (with Nathan, Nazneen, Costa, Younes and all the H4 team and TRL community) let's use it! The next day they start diving in the datasets openly shared on the HF hub and stumble upon two interesting large and good quality fine-tuning datasets recently open-sourced by OpenBMB, a Chinese team from Tsinghua: UltraFeedback and UltraChat. A few rounds of training experiments confirm the intuition, the resulting model is super strong, by far the strongest they have ever seen in their benchmarks from Berkeley and Stanford (LMSYS and Alpaca). Join Clementine, the big boss of the open evaluation leaderboard. Her deep dive into the model capabilities confirms the results: impressive performance. But the H4 team also hosts a famous faculty member, Pr. Sasha Rush, Associate Professor at Cornell University in his daytime, hacker at HF in his nighttime. Joining the conversation, he proposes to quickly draft a research paper to organize and share all the details with the community. A few days later, the model, called Zephyr (a wind like Mistral), paper, and all details are shared with the world. Quickly other companies, everywhere in the world starts to use it. LlamaIndex, a famous data framework and community, shares how the model blew their expectations on real-life use-case benchmarks, while researchers and practitioners discuss the paper and work on the Hugging Face hub. All this happened in just a few weeks catalyzed by open access to knowledge, models, research, and datasets released all over the world (Europe, California, China) and by the idea that people can build upon one another work in AI to bring real-world value with efficient and open models. Stories like this are numerous everywhere around us and make me really proud of the AI community and see how we can build amazingly useful things together. [the video is just me reading this Friday post hahah]

Thomas Wolf

169,127 次观看 • 2 年前

#GoProMISSION1 PRO Deep Dive—Part 2 of 6 🎥 A new, cutting-edge 1-inch sensor means incredible dynamic range, even in the darkest environments. Low-light performance like you've never seen from a GoPro, with up to 14 stops of dynamic range at the sensor. Think rich shadow detail, crispy highlights, + true-to-life color with minimal noise and no AI artifacts. The new 1-inch sensor gathers 77% more light per pixel than the competition at 8K, holding detail where other cameras give up. For the darkest scenes, Low Light Mode switches the camera to 4K for Quad Bayer capture, combining 4 pixels into 1 to quadruple each pixel's light-gathering area. Add HLG HDR with dual gain readout—capturing multiple exposures simultaneously to eliminate the ghosting you'd normally get from sequential bracketed HDR—and the result is clean, cinematic footage from high noon to pitch black. Then we take it one step further with 3 built-in Night Effects Modes: Star Trails, Vehicle Light Trails, + Light Painting to change the way you see the world after dark. All of this, in a camera over 85% smaller by volume and over 75% less expensive than the next camera in the 8K60 class. Shop MISSION 1 PRO at or at your local retailer 🛒 #GoPro #Cinematic #Videography
0:43

#GoProMISSION1 PRO Deep Dive—Part 2 of 6 🎥 A new, cutting-edge 1-inch sensor means incredible dynamic range, even in the darkest environments. Low-light performance like you've never seen from a GoPro, with up to 14 stops of dynamic range at the sensor. Think rich shadow detail, crispy highlights, + true-to-life color with minimal noise and no AI artifacts. The new 1-inch sensor gathers 77% more light per pixel than the competition at 8K, holding detail where other cameras give up. For the darkest scenes, Low Light Mode switches the camera to 4K for Quad Bayer capture, combining 4 pixels into 1 to quadruple each pixel's light-gathering area. Add HLG HDR with dual gain readout—capturing multiple exposures simultaneously to eliminate the ghosting you'd normally get from sequential bracketed HDR—and the result is clean, cinematic footage from high noon to pitch black. Then we take it one step further with 3 built-in Night Effects Modes: Star Trails, Vehicle Light Trails, + Light Painting to change the way you see the world after dark. All of this, in a camera over 85% smaller by volume and over 75% less expensive than the next camera in the 8K60 class. Shop MISSION 1 PRO at or at your local retailer 🛒 #GoPro #Cinematic #Videography

GoPro

10,781 次观看 • 11 天前

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill the best know-how into an ever-expanding library. ASPIRE is a new type of continual learning: "training" is skill refinement instead of gradient descent. "Trained model" is a repo of sensorimotor skills instead of floating weights. “Distributed training” is a panel of agents each practicing a different skill instead of sharded minibatches. Here's the beauty: ASPIRE gives the tired terms "sim2real transfer" and "cross-embodiment transfer" a whole new meaning. Bridging the sim-to-real gap is notoriously brutal. An end-to-end policy has to swallow both the visual shift (sim looks toyish next to a real camera) and the subtle contact physics it never quite gets right. ASPIRE sidesteps the mess, because it doesn't ship pixels or weights across the gap, but ships the know-how. The robot still has to practice in the real world, not zero-shot, but it gets there way faster because it isn't rediscovering the strategy from scratch. Same for going single-arm to bimanual hardware, which usually requires new data and retraining from zero. ASPIRE achieves up to ~10x cut in "transfer learning” tokens (yes, tokens are the new unit of *training* compute ;) Check out our gallery of 150+ tasks and 90+ skills the robots taught themselves, all on the website! Kind of wild that we can ship the "learned weights" as an HTML page rather than a GGUF. We'll open-source the full stack so your own robot library starts compounding from ours! Deep dive in thread:
0:40

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill the best know-how into an ever-expanding library. ASPIRE is a new type of continual learning: "training" is skill refinement instead of gradient descent. "Trained model" is a repo of sensorimotor skills instead of floating weights. “Distributed training” is a panel of agents each practicing a different skill instead of sharded minibatches. Here's the beauty: ASPIRE gives the tired terms "sim2real transfer" and "cross-embodiment transfer" a whole new meaning. Bridging the sim-to-real gap is notoriously brutal. An end-to-end policy has to swallow both the visual shift (sim looks toyish next to a real camera) and the subtle contact physics it never quite gets right. ASPIRE sidesteps the mess, because it doesn't ship pixels or weights across the gap, but ships the know-how. The robot still has to practice in the real world, not zero-shot, but it gets there way faster because it isn't rediscovering the strategy from scratch. Same for going single-arm to bimanual hardware, which usually requires new data and retraining from zero. ASPIRE achieves up to ~10x cut in "transfer learning” tokens (yes, tokens are the new unit of *training* compute ;) Check out our gallery of 150+ tasks and 90+ skills the robots taught themselves, all on the website! Kind of wild that we can ship the "learned weights" as an HTML page rather than a GGUF. We'll open-source the full stack so your own robot library starts compounding from ours! Deep dive in thread:

Jim Fan

199,983 次观看 • 18 天前

This Chinese developer linked two $2,999 NVIDIA DGX Sparks into one box and runs the full Qwen3-235B at home, after dropping his $1,999-a-month cloud bill to zero. He wired 2 small boxes into a single computer, split a giant 235-billion-parameter model in half between them, and serves it across his own network at about 10 tokens a second, with no internet, no cloud, right there on the desk. No data center, no thousand-dollar graphics cards, no monthly cloud bill. Just him, 2 gold boxes the size of a sandwich, one cable between them, and 1 power strip. And here is the whole payoff. He used to pay the cloud $1,999 a month for the same model, and the meter ticked on every request. Now he paid $5,998 once for 2 boxes, they covered their cost in 3 months, and after that he sends as many requests as he wants for free, only electricity. The two Sparks talk over one fast cable, each holds 128GB of memory, and together they carry the whole model, about 73GB loaded per box, with the chip inside pinned near the limit at 96%. Both boxes work as one and keep trading data over the cable, with no cloud in the loop and no single word leaking out. The ready model sits on one local address, and any app on his network calls it as easily as ChatGPT. And here is how he described, in plain words, what this pair of boxes does: "this is a pair of boxes that holds the huge Qwen3-235B model and serves it to one network. the model is split in half, and each box owns its half. parts: // Box 1 (holds the first half of the model and starts the answer fast, the first word appears in under a second) // Box 2 (holds the second half and writes out the rest, about 10 tokens a second) // Cable (connects the 2 boxes and moves data between them on every step, with no lag) // Address (one local address where any app sends its request, like to a cloud model) // Test (a script that runs big prompts through and measures speed and delays) // Monitor (checks temperature, power draw, and load on both boxes every 2 seconds). the model never goes to the cloud. he only steps in when a box runs hotter than 80 degrees or the cable between them starts dropping data." So the system knows exactly what it is, what it is for, and where its limits are. It knows it has to hold the whole huge model across 2 boxes on its own. It knows it has to answer every request locally, with no meter, no limits, and no internet. It knows the human is only needed when a box overheats or the link between them stalls. → The setup runs around the clock on 2 boxes, each pulling under 60 watts → However many requests he sends, the monthly bill is $0, only electricity → The first box starts the answer in under a second → The second writes text at about 10 tokens a second → One request at a time: 838 tokens in 85 seconds, first word in 0.8s → Two requests at once: 697 tokens in 108 seconds, first word in 0.7s → Both boxes sit at 96% load and warm up to 76-78 degrees And only when a chip in a box runs hotter than 80 degrees or the cable between the 2 Sparks drops data does the system call the owner. And when he himself is out on a run or in a coffee shop, he still reaches his own model at home from his phone: sends a big prompt to the local Qwen3-235B, gets the full answer back in under a minute and a half, with no token meter ticking and no limit to hit. Here is what the test shows on his screen during one of the night runs: "one request at a time: 838 tokens in 84.9 seconds, first word in 0.8s, then 0.1s per token." "two requests at once: 697 tokens in 107.6 seconds, first word in 0.7s, then 0.15s per token." "Box 1: chip at 96% load, 76 degrees, 56 watts, 73GB used in memory." "Box 2: chip at 96% load, 78 degrees, 56 watts, the Qwen3-235B model fully loaded." And while everyone around is paying for AI by the month and bumping into limits, his top-tier model just sits on the desk and works as much as he wants: his own little power plant instead of a forever meter. He has no server rack of his own and no cloud account behind it. Just 2 DGX Spark boxes on a desk, one model split in half between them, one local address, and a folder of prompts next to it. Out of everything I have seen this year, this is the cleanest way to stop paying for AI: $5,998 of hardware on the desk once, $0 a month to the cloud, unlimited forever, and between them 2 gold boxes, 1 cable, and the full Qwen3-235B answering at home with no internet.
1:16

This Chinese developer linked two $2,999 NVIDIA DGX Sparks into one box and runs the full Qwen3-235B at home, after dropping his $1,999-a-month cloud bill to zero. He wired 2 small boxes into a single computer, split a giant 235-billion-parameter model in half between them, and serves it across his own network at about 10 tokens a second, with no internet, no cloud, right there on the desk. No data center, no thousand-dollar graphics cards, no monthly cloud bill. Just him, 2 gold boxes the size of a sandwich, one cable between them, and 1 power strip. And here is the whole payoff. He used to pay the cloud $1,999 a month for the same model, and the meter ticked on every request. Now he paid $5,998 once for 2 boxes, they covered their cost in 3 months, and after that he sends as many requests as he wants for free, only electricity. The two Sparks talk over one fast cable, each holds 128GB of memory, and together they carry the whole model, about 73GB loaded per box, with the chip inside pinned near the limit at 96%. Both boxes work as one and keep trading data over the cable, with no cloud in the loop and no single word leaking out. The ready model sits on one local address, and any app on his network calls it as easily as ChatGPT. And here is how he described, in plain words, what this pair of boxes does: "this is a pair of boxes that holds the huge Qwen3-235B model and serves it to one network. the model is split in half, and each box owns its half. parts: // Box 1 (holds the first half of the model and starts the answer fast, the first word appears in under a second) // Box 2 (holds the second half and writes out the rest, about 10 tokens a second) // Cable (connects the 2 boxes and moves data between them on every step, with no lag) // Address (one local address where any app sends its request, like to a cloud model) // Test (a script that runs big prompts through and measures speed and delays) // Monitor (checks temperature, power draw, and load on both boxes every 2 seconds). the model never goes to the cloud. he only steps in when a box runs hotter than 80 degrees or the cable between them starts dropping data." So the system knows exactly what it is, what it is for, and where its limits are. It knows it has to hold the whole huge model across 2 boxes on its own. It knows it has to answer every request locally, with no meter, no limits, and no internet. It knows the human is only needed when a box overheats or the link between them stalls. → The setup runs around the clock on 2 boxes, each pulling under 60 watts → However many requests he sends, the monthly bill is $0, only electricity → The first box starts the answer in under a second → The second writes text at about 10 tokens a second → One request at a time: 838 tokens in 85 seconds, first word in 0.8s → Two requests at once: 697 tokens in 108 seconds, first word in 0.7s → Both boxes sit at 96% load and warm up to 76-78 degrees And only when a chip in a box runs hotter than 80 degrees or the cable between the 2 Sparks drops data does the system call the owner. And when he himself is out on a run or in a coffee shop, he still reaches his own model at home from his phone: sends a big prompt to the local Qwen3-235B, gets the full answer back in under a minute and a half, with no token meter ticking and no limit to hit. Here is what the test shows on his screen during one of the night runs: "one request at a time: 838 tokens in 84.9 seconds, first word in 0.8s, then 0.1s per token." "two requests at once: 697 tokens in 107.6 seconds, first word in 0.7s, then 0.15s per token." "Box 1: chip at 96% load, 76 degrees, 56 watts, 73GB used in memory." "Box 2: chip at 96% load, 78 degrees, 56 watts, the Qwen3-235B model fully loaded." And while everyone around is paying for AI by the month and bumping into limits, his top-tier model just sits on the desk and works as much as he wants: his own little power plant instead of a forever meter. He has no server rack of his own and no cloud account behind it. Just 2 DGX Spark boxes on a desk, one model split in half between them, one local address, and a folder of prompts next to it. Out of everything I have seen this year, this is the cleanest way to stop paying for AI: $5,998 of hardware on the desk once, $0 a month to the cloud, unlimited forever, and between them 2 gold boxes, 1 cable, and the full Qwen3-235B answering at home with no internet.

Blaze

93,219 次观看 • 1 个月前

Can an inexpensive, off-the-shelf IMU be the only sensor to estimate the full state (position, velocity, orientation) of a quadrotor flying through a track at high speed and even be on-pair with vision-based localization? The answer is yes, within certain limitations! In this #RAL2023 paper, we propose a learning-based odometry algorithm that couples a model-based filter driven by the inertial measurements with a learning-based module with access to the control commands. Our system outperforms by a large margin the state-of-the-art visual-inertial odometry (#VIO) algorithms and the state-of-the-art learned-inertial odometry algorithm, #TLIO, for the task of drone racing. Additionally, we show that our system is as accurate as a VIO algorithm that uses a camera to localize to a known map of the racing track. The main limitation of our approach is that it cannot generalize to trajectories that have not been seen at training time. However, in drone racing competitions, the track is known beforehand. Human pilots spend hours or even days of practice on the race track before the competition. Similarly, our system can be trained with the data collected during practice time and deployed during the competition. Future work will investigate how to generalize to trajectories not seen at training time. The code is released! Paper: Video: Code: Kudos to Giovanni Cioffi Leonard Bauersfeld Elia Kaufmann European Research Council (ERC) University of Zurich UZH Science UZH Space Hub NCCR Robotics Aerial Core #RAL2023 #IROS2023 #SLAM
0:31

Can an inexpensive, off-the-shelf IMU be the only sensor to estimate the full state (position, velocity, orientation) of a quadrotor flying through a track at high speed and even be on-pair with vision-based localization? The answer is yes, within certain limitations! In this #RAL2023 paper, we propose a learning-based odometry algorithm that couples a model-based filter driven by the inertial measurements with a learning-based module with access to the control commands. Our system outperforms by a large margin the state-of-the-art visual-inertial odometry (#VIO) algorithms and the state-of-the-art learned-inertial odometry algorithm, #TLIO, for the task of drone racing. Additionally, we show that our system is as accurate as a VIO algorithm that uses a camera to localize to a known map of the racing track. The main limitation of our approach is that it cannot generalize to trajectories that have not been seen at training time. However, in drone racing competitions, the track is known beforehand. Human pilots spend hours or even days of practice on the race track before the competition. Similarly, our system can be trained with the data collected during practice time and deployed during the competition. Future work will investigate how to generalize to trajectories not seen at training time. The code is released! Paper: Video: Code: Kudos to Giovanni Cioffi Leonard Bauersfeld Elia Kaufmann European Research Council (ERC) University of Zurich UZH Science UZH Space Hub NCCR Robotics Aerial Core #RAL2023 #IROS2023 #SLAM

Davide Scaramuzza

37,061 次观看 • 2 年前

Domain Randomization (DR) is a key component of the data augmentation pipeline at Axis Robotics. By applying DR, we are able to scale verified, high-quality human trajectories by 10x to 100x. During training, we systematically introduce variances in environmental parameters. This prevents the model from relying on spurious visual correlations. The objective is to ensure the policy learns rather than overfitting. To demonstrate the necessity and effectiveness of this approach, we evaluated both DR and No-DR models on Task 74 (pour_water_into_mug). The empirical results show a definitive impact on real-world deployment reliability: integrating DR into the pipeline increased the success rate from 0% to 90% (Fig. 1). This divergence stems from how the respective policies process visual observations (Fig. 2). The baseline (No DR) model overfits to the static visual background. It essentially memorizes the poses from the training dataset but fails to generalize when subjected to the inevitable variances of real-world deployment. Consequently, it cannot execute the correct manipulation on the target object. Conversely, the DR-trained model learns to extract essential geometric features and physical constraints, filtering out superficial visual noise. This leads to significantly higher robustness in dynamic environments. The structural difference in execution is clearly reflected in the end-effector trajectory data: These real-world deployment recordings further illustrate this difference (Videos 1 and 2). Scaling Physical AI requires turning raw trajectory data into robust policies, and a rigorously engineered DR infrastructure is an essential bridge to close the Sim2Real gap.
0:20

Domain Randomization (DR) is a key component of the data augmentation pipeline at Axis Robotics. By applying DR, we are able to scale verified, high-quality human trajectories by 10x to 100x. During training, we systematically introduce variances in environmental parameters. This prevents the model from relying on spurious visual correlations. The objective is to ensure the policy learns rather than overfitting. To demonstrate the necessity and effectiveness of this approach, we evaluated both DR and No-DR models on Task 74 (pour_water_into_mug). The empirical results show a definitive impact on real-world deployment reliability: integrating DR into the pipeline increased the success rate from 0% to 90% (Fig. 1). This divergence stems from how the respective policies process visual observations (Fig. 2). The baseline (No DR) model overfits to the static visual background. It essentially memorizes the poses from the training dataset but fails to generalize when subjected to the inevitable variances of real-world deployment. Consequently, it cannot execute the correct manipulation on the target object. Conversely, the DR-trained model learns to extract essential geometric features and physical constraints, filtering out superficial visual noise. This leads to significantly higher robustness in dynamic environments. The structural difference in execution is clearly reflected in the end-effector trajectory data: These real-world deployment recordings further illustrate this difference (Videos 1 and 2). Scaling Physical AI requires turning raw trajectory data into robust policies, and a rigorously engineered DR infrastructure is an essential bridge to close the Sim2Real gap.

Axis Robotics

27,125 次观看 • 3 个月前

China just released an open source AI model that matches the best closed models from OpenAI and Anthropic. Gavin Baker explained exactly how they did it and the answer should concern every American AI lab. The model is called GLM 5.2. It was built by Z. AI. You get 744 billion parameters, 1 million token context window and its MIT license, meaning anyone can download it, fork it, build a company on it, with no restrictions and no Dario. It scored 51 points on the artificial analysis intelligence index. The highest score any open weight model has ever achieved. It beat GPT 5.5 on the frontier software engineering benchmark. It trails Claude Opus 4.8 by less than one percentage point. And it costs 85% less to run than GPT 5.5 for comparable performance. Gavin Baker said on the All-In podcast that this model has challenged some of his beliefs. Then he explained how China built it. The method is called distillation. Just think of tens of thousands of phones and computers running simultaneously, all hitting the frontier model APIs through masked accounts, asking specific questions, and harvesting what happens inside the model when it answers. Every reasoning step, every token. The entire thinking process gets recorded and fed back into the Chinese model during training. It is a cheat sheet. It is the answer key to the exam. And here is the part that should worry everyone. Sacks said it plainly. China was already nine months behind American models. But now that GLM 5.2 is good enough to run its own reinforcement learning, it can improve itself without needing to distill from American models anymore. The cheat sheet let them get close enough to start writing their own answers. Sacks said we are six months behind on the model and 24 months behind on silicon and they are only a few months behind in total. The Z. AI founder told Elon Musk directly that open weight fable-level capability will be here before Q1 2027. Every restriction Anthropic lobbied for, every self-imposed safety guardrail, every month of delay in releasing American frontier models accelerated this. The Chinese labs were not under those restrictions. They were not going to wait. The composable model future Gavin described, where every enterprise runs a frontier model alongside their own fine-tuned open weight model, is coming regardless of what American labs do next. The question is just whether the open weight half of that stack is American or Chinese. Right now it is Chinese. WATCH THE FULL PODCAST ON The All-In Podcast
3:21

China just released an open source AI model that matches the best closed models from OpenAI and Anthropic. Gavin Baker explained exactly how they did it and the answer should concern every American AI lab. The model is called GLM 5.2. It was built by Z. AI. You get 744 billion parameters, 1 million token context window and its MIT license, meaning anyone can download it, fork it, build a company on it, with no restrictions and no Dario. It scored 51 points on the artificial analysis intelligence index. The highest score any open weight model has ever achieved. It beat GPT 5.5 on the frontier software engineering benchmark. It trails Claude Opus 4.8 by less than one percentage point. And it costs 85% less to run than GPT 5.5 for comparable performance. Gavin Baker said on the All-In podcast that this model has challenged some of his beliefs. Then he explained how China built it. The method is called distillation. Just think of tens of thousands of phones and computers running simultaneously, all hitting the frontier model APIs through masked accounts, asking specific questions, and harvesting what happens inside the model when it answers. Every reasoning step, every token. The entire thinking process gets recorded and fed back into the Chinese model during training. It is a cheat sheet. It is the answer key to the exam. And here is the part that should worry everyone. Sacks said it plainly. China was already nine months behind American models. But now that GLM 5.2 is good enough to run its own reinforcement learning, it can improve itself without needing to distill from American models anymore. The cheat sheet let them get close enough to start writing their own answers. Sacks said we are six months behind on the model and 24 months behind on silicon and they are only a few months behind in total. The Z. AI founder told Elon Musk directly that open weight fable-level capability will be here before Q1 2027. Every restriction Anthropic lobbied for, every self-imposed safety guardrail, every month of delay in releasing American frontier models accelerated this. The Chinese labs were not under those restrictions. They were not going to wait. The composable model future Gavin described, where every enterprise runs a frontier model alongside their own fine-tuned open weight model, is coming regardless of what American labs do next. The question is just whether the open weight half of that stack is American or Chinese. Right now it is Chinese. WATCH THE FULL PODCAST ON The All-In Podcast

Ihtesham Ali

86,044 次观看 • 21 天前

ONE NVIDIA DGX SPARK PACKS 128GB OF UNIFIED MEMORY AND 1 PETAFLOP OF AI COMPUTE FOR $3,999, THE SAME COST AS A SINGLE RTX 5090 WITH FOUR TIMES THE VRAM AND A FULL LINUX AI STACK PREINSTALLED 00:03 he holds it up to the camera, "personal AI super computer, fully geeking out", the DGX Spark sits in one hand, about the size of a paperback book inside the chassis is the NVIDIA GB10 Grace Blackwell Superchip. 128GB of coherent memory shared between the ARM CPU and the Blackwell GPU, 4TB of Gen5 NVMe storage, ConnectX networking at 200 gigabits per second for pairing two units together that pooled configuration runs a 200B parameter model on one unit, or up to 405B when two DGX Sparks link over ConnectX. no quantization compromise, no memory swapping to disk, everything in unified memory for local AI this is the first CUDA machine that fits on a desk and still runs GPT-4 class open models. every framework already speaks CUDA. no porting, no rewrites, no driver quirks $200 chatgpt pro and $200 claude code max stacked at $400 a month hits $4,800 a year. this box pays itself back in ten months and never expires 128gb of unified memory, 1 petaflop of AI compute, 200B models running on your desk without a datacenter subscription behind them. the article ranked the $4,199 Mac Studio as Device 4 for a single household. this lands under it at $3,999 with CUDA baked in bookmark this and read the article below
0:23

ONE NVIDIA DGX SPARK PACKS 128GB OF UNIFIED MEMORY AND 1 PETAFLOP OF AI COMPUTE FOR $3,999, THE SAME COST AS A SINGLE RTX 5090 WITH FOUR TIMES THE VRAM AND A FULL LINUX AI STACK PREINSTALLED 00:03 he holds it up to the camera, "personal AI super computer, fully geeking out", the DGX Spark sits in one hand, about the size of a paperback book inside the chassis is the NVIDIA GB10 Grace Blackwell Superchip. 128GB of coherent memory shared between the ARM CPU and the Blackwell GPU, 4TB of Gen5 NVMe storage, ConnectX networking at 200 gigabits per second for pairing two units together that pooled configuration runs a 200B parameter model on one unit, or up to 405B when two DGX Sparks link over ConnectX. no quantization compromise, no memory swapping to disk, everything in unified memory for local AI this is the first CUDA machine that fits on a desk and still runs GPT-4 class open models. every framework already speaks CUDA. no porting, no rewrites, no driver quirks $200 chatgpt pro and $200 claude code max stacked at $400 a month hits $4,800 a year. this box pays itself back in ten months and never expires 128gb of unified memory, 1 petaflop of AI compute, 200B models running on your desk without a datacenter subscription behind them. the article ranked the $4,199 Mac Studio as Device 4 for a single household. this lands under it at $3,999 with CUDA baked in bookmark this and read the article below

NO1ennn

76,338 次观看 • 15 天前

🇨🇳 Another great Chinese Model, OmniHuman-1.5 from ByteDance Turns 1 image plus a voice track into expressive avatar video by pairing a System 1 and System 2 inspired planner with a Diffusion Transformer, Produces coherent motion for over 1 minute with moving camera and multi character scenes. Most avatar models move to the beat of the audio but miss meaning, so gestures feel generic and emotions feel shallow. The fix here is a Multimodal LLM planner that listens to the speech and drafts a structured plan describing intent, emotions, beats, and high level actions, which gives the motion engine clear semantic targets instead of only rhythm. The motion engine is a Multimodal Diffusion Transformer that fuses the plan with audio, the single reference image, and optional text prompts, then synthesizes continuous body, face, and head motion that matches both words and tone. A key trick is a Pseudo Last Frame, a synthetic target that summarizes the next expected state, which stabilizes fusion across modalities and keeps motion consistent over long spans. From just 1 image and speech, the system outputs speaking avatars with synchronized lips, context aware gestures, and continuous camera movement, and it also supports multi character interactions without manual choreography. Reported results show strong lip sync accuracy, high video quality, natural motion, and close match to text prompts, and the same setup works on nonhuman characters too.
0:18

🇨🇳 Another great Chinese Model, OmniHuman-1.5 from ByteDance Turns 1 image plus a voice track into expressive avatar video by pairing a System 1 and System 2 inspired planner with a Diffusion Transformer, Produces coherent motion for over 1 minute with moving camera and multi character scenes. Most avatar models move to the beat of the audio but miss meaning, so gestures feel generic and emotions feel shallow. The fix here is a Multimodal LLM planner that listens to the speech and drafts a structured plan describing intent, emotions, beats, and high level actions, which gives the motion engine clear semantic targets instead of only rhythm. The motion engine is a Multimodal Diffusion Transformer that fuses the plan with audio, the single reference image, and optional text prompts, then synthesizes continuous body, face, and head motion that matches both words and tone. A key trick is a Pseudo Last Frame, a synthetic target that summarizes the next expected state, which stabilizes fusion across modalities and keeps motion consistent over long spans. From just 1 image and speech, the system outputs speaking avatars with synchronized lips, context aware gestures, and continuous camera movement, and it also supports multi character interactions without manual choreography. Reported results show strong lip sync accuracy, high video quality, natural motion, and close match to text prompts, and the same setup works on nonhuman characters too.

Rohan Paul

63,859 次观看 • 10 个月前