Загрузка видео...

Не удалось загрузить видео

На главную

WebGPU + WebXR Three.js physics stress tests w/ eye-tracking input on Apple Vision Pro M5 🏟️ 12k bricks, 100 ragdolls, rope & cloth Based on @[email protected]'s work on AVBD sim, rec'd by Cody Bennett. The scale is amazing + little-to-no clipping! Fun to play w/ on next gen h/w

14,826 просмотров • 2 месяцев назад •via X (Twitter)

Комментарии: 14

Фото профиля jamesckane
jamesckane2 месяцев назад

AVBD paper: Jure's WebGPU AVBD implementation & blog post: My Vision Pro physics sandbox (also w/ MLS and Weta fire sims):

Фото профиля Einar Johnson
Einar Johnson2 месяцев назад

@threejs @JureTriglav @CodyBennett Looks incredible! Thanks for sharing this. What are some of the biggest challenges with this work?

Фото профиля jamesckane
jamesckane2 месяцев назад

@threejs @JureTriglav @CodyBennett WebKit’s WebGPU is a bit unique & required some workarounds. Have to be on visionOS developer beta w/ the very latest three.js commits and avoid some other pitfalls Also very few people have Vision Pro && care about WebXR + WebGPU so little community support at this time

Фото профиля Einar Johnson
Einar Johnson2 месяцев назад

I’m running the Visions OS dev beta and definitely understand the issue with not having wide adoption on VisionPro just yet, but hope for the future that that might change. Am I correct in understanding that you cannot use controllers at all, like the PSVR two controllers for WebXR?

Фото профиля jamesckane
jamesckane2 месяцев назад

@threejs @JureTriglav @CodyBennett We're in 1984 - the Vision Pro is the Macintosh - which kind-of sucked at first. It's going to take decades of progress at all levels to get where we're going But I don't know about the controller thing, sorry!

Фото профиля Einar Johnson
Einar Johnson2 месяцев назад

My family had a Mac Plus in '84. I remember those days well. (Fokker Triplane, Dark Castle, Mac Paint). I agree, it's going to take time, however I hope with some of the tech that's already in place we might see more mainstream adoption in a solid decade rather than multiple decades.

Фото профиля Scott Bowler ᯅ
Scott Bowler ᯅ2 месяцев назад

@threejs @JureTriglav @CodyBennett Would love to have a play with this - is there a way to access this on the AVP?

Фото профиля jamesckane
jamesckane2 месяцев назад

@threejs @JureTriglav @CodyBennett Last link below - fwiw I am on M5 Vision Pro on the latest visionOS developer beta - very prototypal!

Фото профиля Zaniyar
Zaniyar2 месяцев назад

@threejs @JureTriglav @CodyBennett Now activate the passthrough so you have AR

Фото профиля jamesckane
jamesckane2 месяцев назад

@threejs @JureTriglav @CodyBennett @Apple doesn't let users access `immersive-ar` passthrough via WebXR at this time sadly! Hoping they get around to it eventually - I see this feedback from devs and users all the time ⏳

Фото профиля Zaniyar
Zaniyar2 месяцев назад

@threejs @JureTriglav @CodyBennett @Apple I know. They want to force you through the app store for that.

Фото профиля rusa
rusa2 месяцев назад

@threejs @JureTriglav @CodyBennett 12k rigid bodies with eye-tracking input on-device is a real stress test, the physics solver and the render both fighting for the same frame budget on mobile-class silicon. webgpu compute doing the broadphase is probably what makes it hold. what fps did it sustain.

Фото профиля AI Mastery Guide
AI Mastery Guide2 месяцев назад

@threejs @JureTriglav @CodyBennett 12k bricks with barely any clipping, solid.

Фото профиля Fabian
Fabian2 месяцев назад

@threejs @JureTriglav @CodyBennett How’d you get eye track input

Похожие видео

A billionaire is 10 million times richer than a student and pays 4 times more for the same infinite jackpot: $30.84 versus $7.79. Jane Street tests this exact instinct. In their published mock interview, a candidate is handed a dice game and asked what it is worth. The answer is never the average roll. It is the value of the option to walk away and roll again. That is the same move Daniel Bernoulli made on a coin in 1738. Here is the coin. A fair coin is flipped until it lands heads. The pot pays $2 if heads comes on flip one, $4 on flip two, $8 on flip three, doubling with no cap. Its expected value is not large. It is infinite: EV = (1/2)(2) + (1/4)(4) + (1/8)(8) + ... = 1 + 1 + 1 + ... = infinity By the rule most people carry, price equals expected value, you should hand over your whole net worth to play once. Nobody does. For 250 years that was called a paradox. It was not. The pricing rule was wrong. Value the outcomes by utility, not dollars. Bernoulli's fix was logarithmic utility, the idea that a dollar matters less the more you already hold: U(W) = ln(W) The fair price c is the amount that leaves your expected utility unchanged: sum (1/2^n) · ln(W - c + 2^n) = ln(W) Solve it for real wealth and the infinite EV collapses to a small, exact number: Wealth $100 -> pay $7.79 Wealth $1,000 -> pay $10.95 Wealth $1,000,000 -> pay $20.87 Wealth $1,000,000,000 -> pay $30.84 Your wealth spans 7 orders of magnitude. The price moves by a factor of 4. None of this is a curiosity about coins. It is why position sizing exists. A payoff tail can be astronomically large and still be worth almost nothing to you, because ruin is priced in logs. The same equation, rediscovered by John Kelly in 1956 as maximizing E[ln(W)], is what tells a trader to bet a fraction of the bankroll instead of the whole thing. Every fund that priced its bets at expected value instead of log wealth eventually met the tail it refused to price. Expected value said infinity. The correct answer was $11. Models over opinions.

Orion

93,571 просмотров • 2 месяцев назад

This is big. NVIDIA and Apple just unlocked the next level for Vision Pro with CloudXR. Here’s what you need to know: It streams from a PC or the cloud directly to your Vision Pro. No cables. Up to 4K at 120fps. The technology is called dynamic foveated streaming. Your eyes only see in full resolution at the center of your gaze. CloudXR tracks exactly where you’re looking and delivers maximum resolution there. Everything in your periphery gets optimized. The stream stays efficient without you ever noticing a difference. Why does this matter? Vision Pro is already the most advanced spatial computer ever made. But standalone processing has a ceiling. There are workflows that need more compute than any headset can carry. CloudXR removes that ceiling by connecting Vision Pro to the full power of NVIDIA RTX in real time. This is not a workaround. It is a native visionOS integration. Also worth knowing. Gaze data never leaves your device. Not to the app. Not to the server. Developers get the full performance benefit of foveation without ever touching your raw eye tracking data. Privacy built into the architecture. Three industries are already on this: Kia, Rivian, and Volvo are running 1:1 scale design reviews with photorealistic accuracy. Full size vehicle models evaluated in spatial computing before a physical prototype exists. That is a fundamental shift in how design works. Foxconn is walking factory floors digitally, optimizing facilities before construction begins. Switch is managing data center infrastructure through a full digital twin. Companies using this approach are reporting up to 30% improvement in development processes. And sim enthusiasts finally get to cut the cord. iRacing and X-Plane 12 are the first titles. Full GeForce RTX power, streamed wirelessly to Vision Pro, inside your physical space. For developers: one Xcode template, one codebase, deployed across Vision Pro, iPhone, and iPad. And multiple headsets can share the same streamed environment at once. Some fully immersed, others on a tablet. That collaborative layer is what enterprise has been asking for. Coming this spring with visionOS 26.4. It’s wild to think about what this unlocks. The future of spatial computing is incredibly bright. Live from GTC. More coming soon.

Justin Ryan

43,685 просмотров • 7 месяцев назад

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:

Jim Fan

364,721 просмотров • 2 лет назад

Alibaba just released a coding model that hits 82 percent on SWE-Bench Verified. That is the highest score ever published for an open-source model. The weights are free. The license is Apache 2.0. You can run it today. The model is Qwen 4 Coder 32B. Here is what 82 percent on SWE-Bench Verified actually means. SWE-Bench Verified tests whether an AI can autonomously resolve real bugs pulled from real production GitHub repositories. Not synthetic exercises. Real open-source projects that real teams depend on. A model gets a bug report, reads the code, writes a fix, and either passes the test suite or it does not. At 82 percent, Qwen 4 Coder 32B resolves 82 out of every 100 real production bugs it is given. Without a human guiding it. On code it has never seen before. For comparison: Qwen 4 Coder 32B: 82 percent SWE-Bench Verified. Open source. Apache 2.0. Claude Fable 5: 80.3 percent SWE-Bench Pro. $10 input / $50 output per million tokens. Currently suspended. GPT-5.6 Sol: Competitive on Terminal-Bench. $5 input / $30 output per million tokens. An open-weight model that you can download and run for free just beat both of them on the benchmark designed to measure real software engineering capability. Here is the architecture. Qwen 4 Coder 32B is a 32 billion parameter dense model. Not a Mixture-of-Experts. Every parameter is active on every request. This matters for inference: a dense 32B model runs on 22 gigabytes of VRAM, which fits on a single high-end consumer GPU or a MacBook Pro with 64GB of unified memory. The smaller variant, Qwen 4 Coder 4B, runs at approximately 135 tokens per second on an M5 Max and fits inside 8 gigabytes of RAM. For a model with usable coding capability, that is a new bar for what fits in a single laptop. The training methodology continued Alibaba's approach of reinforcement learning on verifiable coding tasks. The model gets rewarded when its code passes tests. It gets penalized when it fails. Over millions of training steps, the model learns to write code that actually runs rather than code that looks plausible. License: Apache 2.0. Full commercial use. No attribution requirement. No revenue threshold. No monthly active user ceiling. Weights: Hugging Face, available today. Runs on: vLLM, Ollama, SGLang, and any standard GGUF-compatible inference engine. Qwen 4 32B also runs at approximately 135 tokens per second on an M5 Max chip, setting a new bar for what a sub-8GB model can do on Apple Silicon. The open-source coding model just beat the best closed-source model in the world on the benchmark designed to test whether AI can actually do software engineering. The weights are free. The subscription is optional. Source: Autom8Labs AI Insight July 2026, State of Open Source LLMs June 2026, Kunal Ganglani blog June 2026.

Harman

41,278 просмотров • 3 месяцев назад

got early access to the VR glasses, my unfiltered thoughts: - first few min is a learning curve (swipe vs click, eye tracking as navigation). after that, navigation is surprisingly intuitive - ergonomics better than expected: nose bridge and eye position are adjustable, a lot more comfortable than vision pro. Big step up on price and weight too - definitely a lot less like a helmet and more like glasses - eye and gesture tracking is surprisingly accurate. you look at a target and pinch to select, no controllers - was skeptical of the surface keyboard (any table becomes a keyboard and trackpad) but it worked and was able to write an instagram comment directly from the glasses! - I'd personally still default to voice for most navigation (same way I do on a smart TV vs typing) - games were genuinely fun! Also watched a full Avengers scene and James Cameron Billie Eilish piece in full immersion - courtside sports is an interesting wedge - watched highlights from a Knicks game from floor seats. Wonder if "courtside for everyone" becomes new revenue line for leagues. Live sports is further push into entertainment instead of pure gaming (quest) - passthrough to mixed reality is smooth - could seamlessly talk to people in the room while looking at screens - natural handoff to go from coding on a laptop, then switching to coding on multiple screens in VR - didn't get to try the hologram video calls still have questions around: - is this an entertainment, productivity, or work device? Meta is pitching all three (cinema, workspace, console) - the cord and battery puck are still present, though the puck is light and fits in a pocket. Neither is that disruptive, but it's hard to picture people walking around and being active in these - my personal read is entertainment first, probably light work on the side. there's still slight app latency and 3-hour battery so probably movies, sports, and games vs deep work. I can see myself buying this as a hobbyist for entertainment, but not as my core work device yet all and all an awesome engineering feat and the best case of AR/VR I've ever tried so far - huge credit to the team! I'm still not convinced we'll all be walking around with computers on our eyes just yet, but the first time I can see the path + a real leap forward launching spring 2027 🕶️🚀

Claire Zau

27,800 просмотров • 28 дней назад

Back in the day, if you were a client of a bespoke tailor and wanted a tweed jacket, there's a good chance you would have phoned up W. Bill in London. They started as a shop selling things like Scottish sweaters, tweedy ties, and pillow covers. Eventually, they exclusively became a cloth merchant. For some reason, during the time I interacted with the shop, it was owned by a large online hardware company, such as the kind that sells buckets and hammers. The owner of that business assigned Ray Hammett to run W. Bill. Ray apparently was a bit of a quiet menace — a soft-spoken man who *loved* British cloth, especially tweeds. The owner of the parent company was wealthy and didn't really supervise Ray, so Ray basically had an endless budget. And so, Ray bought whatever caught his fancy. W. Bill's shop eventually became so stacked with tweeds that the light from the buzzing fluorescent tubes overhead never penetrated to all corners. The inventory was stacked up to the ceiling, and you had to squeeze yourself between the aisles. Apparently, Ray became known across Britain as the "man who would buy any cloth," so other merchants flocked there to sell even more tweeds. If you were looking for a specific tweed — a Shetland or Cheviot, herringbone or houndstooth, in a particular shade of olive or brown — he would most certainly have it. The great thing about W. Bill is that he had stock from independent, little-known weavers, so if you wanted something made by someone in a small shack in some remote corner of Scotland, he'd have that too. You'd phone him up and hear this soft voice on the other end from a man who was enthusiastic to talk to anyone about tweed. Ray passed away some years ago. W. Bill was eventually acquired by Harrison's of Edinburgh, who cleared out the stock that was never going to sell and kept the good stuff that eventually sold. IMO, Harrison's today is the best merchant if you, like me, favor British cloths. They continue to make things under the W. Bill name — linen, cotton, overcoatings, whipcords, and, of course, tweeds. Some of the designs are a bit playful and eccentric, much like what caught Ray's eye.

derek guy

66,320 просмотров • 1 год назад

Apple Vision Pro: My Initial Impressions I agree with some others. This is one of the most impressive pieces of tech I’ve ever used, but it's not perfect. The Good: • The eye tracking and hand gestures are simply incredible. The accuracy is like 99%. It’s quite intuitive and feels very natural. You can put your hands almost anywhere and it'll sense them. • I’m not exaggerating when I say Apple’s immersive videos made for the Vision Pro literally make you feel like you’re there in person. I felt like I was literally in the room with Alicia Keys while she sang, or traversing 3,000 feet above Norway’s breathtaking fjords while walking on a rope. My only complaint is that while Apple Immersive Videos are 180-degree 3D 8K recordings, they are still a little blurry. • You can use the Vision Pro while lying down in bed in complete darkness and the hand gestures will still work just fine (just like FaceID does in the dark). • In 2013 Apple introduced TouchID. In 2017 FaceID. With Vision Pro, Apple has introduced OpticID. It scans your iris and unlocks your Vision Pro when you put it on. It’s incredibly quick and accurate. And like FaceID, it's secure. It's what you will use to pay for things on the device as well. • I didn’t experience any vertigo, dizziness, etc while using the device. • I'm a bit of a display nerd. For the first time outside of a theatre, Avatar The Way of Water feels true to the directors creative intent. High frame rate (HFR) wasn’t possible in any other device. The 3D is exactly like the theatre. I went to a dolby cinema to watch it originally, and the Vision Pro gets it about as close to a theatre video quality experience as you can get. • The sound quality is really good. The spacial audio is very convincing. • The Vision Pro is heavy, but I found myself not caring all that much if i used the dual loop headband. • I was skeptical of the productivity side of the Vision pro, but wow does the mac virtual display work insanely well. You can easily edit videos or browse the web. • You can complain about plenty of things, but built quality is not one of them. It's really good and solid. The device feels very dense. • Neck strain was not an issue • I watched John Wick in one of the immersive environments, and the movie was reflecting onto the water perfectly. It’s was incredible. I'm not joking when i saw it literally felt like I was watching a movie on a 4K HDR screen in then middle of the wilderness. • Battery life is fine most of the time. The Not So Good: • The app ecosystem just isn’t there yet, which is understandable. That will happen over time, but currently there are only 600 apps built for the Vision. For developer, It doesn't make sense to make a custom Vision Pro app when there are only ~200k users in the ecosystem. • The field of view is kinda of small, which is disappointing, and sometimes a little distracting. It's not terrible, but I wish the FOV was bigger. • The passthrough video is dissapointinly low quality, though it’s still better than anything else on the market. You can’t read your small text on your phone. Colors are muted, and it's a little jittery. Future versions will be better no doubt. • It is sort of an isolating device. I almost guarantee you future versions of the Vision Pro will be able to connect with one another so two people can be looking at the same virtual safari window, movie, etc. • Price: It's expensive. • As I said above, while heavy, the weight of the VP was not an issue for me, but that doesn't mean I don't want it to be lighter. It does press against your face pretty hard, which can cause minor cheek area fatigue. Future versions will no doubt be lighter. Final Thoughts: Going back to my Mac felt weird. It suddenly felt like a technology of the past. If you have the money and want to be an early adopter, you’ll be happy with the Vision Pro. But I have to say, the second or third generation of this device, I think, will be better and cheaper, so it’s probably worth waiting for that. While it’s one of the greatest pieces of tech I’ve ever tried, I can’t see myself using this in my daily life yet, and for $3,500, I should be able to envision that. I'm going to give it a few more days, but I'll likely be returning it. The app ecosystem needs to improve and expand a lot. That's ultimately what will make or break this thing's success. Apple knocked it out of the park engineering this thing. It's mind blowing, but if i'm already mind blown, Imagine how good future versions will be.

Sawyer Merritt

1,602,952 просмотров • 2 лет назад