正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Ultrasound-derived 3D reconstruction with zero manual processing (CARTO v8) 1. Rotate ICE probe 2. Acquire frames 3. Receive annotated, segmented 3D LA model (with LAA cut away already) compliments of deep learning algorithm Moussa Mansour

Tom De Potter

3,225 subscribers

11,791 次观看 • 2 年前 •via X (Twitter)

教育健康养生科学技术

Anya Rossi• Live Now

Private livecam show

3 条评论

ALBERTO ALFIE 的头像

ALBERTO ALFIE2 年前

@M__Mosca @MoussaMansour10 It is real !!! Machines have taken control of the world.

Jakub Baran 的头像

Jakub Baran2 年前

@MoussaMansour10 It is nice that after so many year’s ICE will have proper position in EP lab. Being ICE user for over 15 years i can tell that it has much more to offer. Lok for

Balaram Krishna Hanumanthu 的头像

Balaram Krishna Hanumanthu2 年前

@MoussaMansour10 Good but not perfect

相关视频

totally blown away by Google Gemini 3 + three.js.. just generated a 3D product interactive app with zero coding.. use hand gesture to rotate, scale, control xyz and lighting of 3D model AI is taking over.. accept or not tutorial with prompts & code below:

Sensitive content

totally blown away by Google Gemini 3 + three.js.. just generated a 3D product interactive app with zero coding.. use hand gesture to rotate, scale, control xyz and lighting of 3D model AI is taking over.. accept or not tutorial with prompts & code below:

el.cine

416,631 次观看 • 7 个月前

ULS FAM! New feature of CARTO 3 V8! Look at the video of the autogenerated machine-learning 3D map using CARTOSOUND FAM with only ICE catheter. Check our paper #Europace👉 Luigi Di Biase Fengwei Jasper Zou Domenico Giovanni Della Rocca, MD Vito Grupposo Nicola Tarantino Andrea Natale MD

ULS FAM! New feature of CARTO 3 V8! Look at the video of the autogenerated machine-learning 3D map using CARTOSOUND FAM with only ICE catheter. Check our paper #Europace👉 Luigi Di Biase Fengwei Jasper Zou Domenico Giovanni Della Rocca, MD Vito Grupposo Nicola Tarantino Andrea Natale MD

Xiaodong Zhang (Andy), MD, PhD, FHRS, FACC

30,615 次观看 • 2 年前

Reconstructing the world in 3D from random internet photos is a deep rabbit hole. In 2008, Photosynth proved that 3D reconstruction of smaller monuments was possible. The next year, "Building Rome in a Day" proved it could be done for entire cities. Since then, there’s been a wild lineage of research from early point clouds to modern NeRFs and 3DGS --with a similar cast of characters appearing along the way. In this deep dive, we'll break it all down, and explore why IARPA is funding what comes next. 0:00 The Internet's 3D Model of the World 0:40 Building Rome in a Day (2009) 2:09 The Long Tail Problem 3:18 MegaDepth and the Mannequin Challenge 4:44 NeRFs and 3D Gaussian Splatting 7:19 MegaScenes, Doppelgangers, and Foundation Models 10:00 MegaDepth-X Punches Through the Wall 11:26 Why IARPA Is Funding What's Next 13:51 The 4D Frontier and the Sensorium

Reconstructing the world in 3D from random internet photos is a deep rabbit hole. In 2008, Photosynth proved that 3D reconstruction of smaller monuments was possible. The next year, "Building Rome in a Day" proved it could be done for entire cities. Since then, there’s been a wild lineage of research from early point clouds to modern NeRFs and 3DGS --with a similar cast of characters appearing along the way. In this deep dive, we'll break it all down, and explore why IARPA is funding what comes next. 0:00 The Internet's 3D Model of the World 0:40 Building Rome in a Day (2009) 2:09 The Long Tail Problem 3:18 MegaDepth and the Mannequin Challenge 4:44 NeRFs and 3D Gaussian Splatting 7:19 MegaScenes, Doppelgangers, and Foundation Models 10:00 MegaDepth-X Punches Through the Wall 11:26 Why IARPA Is Funding What's Next 13:51 The 4D Frontier and the Sensorium

Bilawal Sidhu

51,997 次观看 • 22 天前

全网疯传神老师这个提示词太好玩了我改了一版裸眼3D版本然后又花了1美金做了一个视频产品展示历史3D图提示词： Role & Subject: A museum-grade, hyper-realistic 16:9 3D infographic masterpiece titled "THE EVOLUTION OF [Product Name]". The visual style combines cinematic product photography with advanced technical blueprint aesthetics, emphasizing extreme depth and tangible 3D presence. The Hero Lineup (3D Volumetric Core): • A complete chronological sequence of 8-12 historical versions of [Product Name], from earliest prototype to futuristic concept • Arranged on a precision-engineered floating ruler/timeline base with depth shadows and reflection mapping • 3D Rendering Enhancement: - Octane Render + Unreal Engine 5 volumetric lighting - PBR (Physically Based Rendering) materials with: * Sub-surface scattering for organic materials * Metallic/roughness maps for mechanical parts * Micro-displacement for surface texture depth - Each object casts realistic shadows on the background plane - Edge rim lighting (0.3 intensity) to separate objects from background - Depth of field: Sharp focus on center objects, subtle bokeh on edges • Material Evolution Story: - Early versions: Weathered textures (rust, wear marks, patina) - Mid-era: Industrial matte finishes with visible manufacturing seams - Modern: Pristine glossy surfaces with screen-space reflections - Future: Holographic elements with volumetric fog effects Brand Atmosphere (The 3D Canvas): • Background: Deep [Brand Color] with physical texture layers: - Embossed vintage patent drawings (normal-mapped) - Floating translucent glass panels with engraved engineering notes - Volumetric god rays piercing through layered newspaper clippings - Particle system: Subtle floating dust motes with motion blur • Header Zone: - Brand logo: Extruded 3D metal badge with bevel and emboss - Title typography: Carved letterpress effect with depth shadows - Metallic foil stamp simulation with anisotropic highlights The "Hyper-Dense" 3D Information Layer: A complex, organized data-maximalist layout with spatial depth: 1. Dense Annotation Network: - Ultra-fine white hairlines (0.5px) connecting components to data blocks - Lines have slight 3D curvature following perspective rules - Connection nodes: Small metallic spheres with Fresnel reflections 2. Contextual Era Modules: - Floating glass panels hovering above the timeline - Each decade module has: * Frosted glass material with transparency gradient * Iconographic markers in raised relief * Subtle animated pulse effects (optional) 3. Magnifying Lens Inserts: - Circular/hexagonal cutouts with beveled glass edges - Extreme macro close-ups showing: * Material grain (wood, metal, plastic crystals) * Internal mechanisms with exploded-view depth - Lens distortion shader for realism 4. Technical Specs Strip (Bottom): - Structured data bar with 3D segmented panels - Each cell: Slightly recessed with inner shadow - Typography: Laser-etched effect on brushed metal texture - Micro LED indicator lights between data points 3D Enhancement Technical Specs: • Rendering Engine: Octane Render / Unreal Engine 5.3 • Global Illumination: HDRI environment map (studio lighting setup) • Camera Settings: - Focal Length: 35mm (slight wide-angle for information density) - F-Stop: f/5.6 (selective focus) - Chromatic Aberration: 0.2 (photographic realism) • Post-Processing Stack: - Ambient Occlusion (AO) for micro-shadows - Screen Space Reflections (SSR) - Volumetric lighting with light shaft scattering - Color Grading: Cinematic LUT with lifted blacks - Sharpness: 130% (crisp data legibility) • Material Library: - Metals: Anisotropic brushed aluminum, polished chrome - Glass: IOR 1.52 with caustics - Plastics: Translucent with thickness variation - Paper: Subsurface scattering for vintage documents Output Parameters: --ar 16:9 --v 6.1 --style raw --stylize 400 --quality 2 --chaos 10 Additional Flags (for AI generators): --3d-rendering --volumetric-lighting --octane-quality --pbr-materials --depth-of-field Metadata Variables to Replace: [Product Name] → Your subject (e.g., "POP MART Designer Toys") [Brand Color] → Primary brand color (e.g., "Deep Navy Blue #1A2332") [Material Vibe] → Core material story (e.g., "Vinyl to Resin to Smart Materials") [Key Components] → Specific product features (e.g., "Joints, Paint Finish, Packaging") Pro Tips for Maximum 3D Impact: 1. Use "isometric 3D layout" for uniform depth perception 2. Add "floating holographic UI elements" for futuristic tech feel 3. Include "cross-section cutaway views" to show internal structure 4. Apply "chromatic aberration" around high-contrast edges for lens realism 5. Insert "measurement calipers" as 3D props for scale reference

serva huang

12,281 次观看 • 9 个月前

Love seeing Silico (Goodfire ) used to probe our EchoJEPA's representations! this is exactly the kind of interpretability work that's been missing for JEPA-style models. One thing that makes EchoJEPA particularly interesting to interpret: unlike MAE-based approaches, it never reconstructs pixels. The model learns entirely in latent space through masked prediction, so you can't just look at decoder outputs to understand what it captured. Attribution onto a temporally aligned 3D mesh is a much more honest probe of what the representations actually encode. What we found in building EchoJEPA: training on 18M echo videos across 300K patients, the model learns to disentangle cardiac anatomy from ultrasound noise (speckle, reverberation artifacts) almost entirely through self-supervision. With 1% labeled data it already outperforms supervised baselines trained on 100%. The latent space is doing real anatomical work, but until you can visualize it like this, "real anatomical work" is mostly a claim. Paper + code: |

Love seeing Silico (Goodfire ) used to probe our EchoJEPA's representations! this is exactly the kind of interpretability work that's been missing for JEPA-style models. One thing that makes EchoJEPA particularly interesting to interpret: unlike MAE-based approaches, it never reconstructs pixels. The model learns entirely in latent space through masked prediction, so you can't just look at decoder outputs to understand what it captured. Attribution onto a temporally aligned 3D mesh is a much more honest probe of what the representations actually encode. What we found in building EchoJEPA: training on 18M echo videos across 300K patients, the model learns to disentangle cardiac anatomy from ultrasound noise (speckle, reverberation artifacts) almost entirely through self-supervision. With 1% labeled data it already outperforms supervised baselines trained on 100%. The latent space is doing real anatomical work, but until you can visualize it like this, "real anatomical work" is mostly a claim. Paper + code: |

Bo Wang

29,611 次观看 • 4 个月前

3D scanning and rendering is moving so fast - got my splats up and running and I'm mind blown getting ~100fps for this complex 3D scene ⬇️ 🤯 1. WAY faster than NeRF: For comparison, NeRFs would takes around 10 seconds per frame (!) Instead I'm zipping around with FPV controls without breaking a sweat - though I do crash a few times towards the end of the video lol 2. Old Meets New: Gaussian Splatting is cool in that it fuses classical graphics and deep learning techniques. Like NeRFs, this is still a radiance field - just without the slower (ne)ural rendering part. 3. Explicit Representation: Instead you represent a 3D scene as a collection of ellipsoidal "splats" called gaussians. Each gaussian has a position, size, and color. Rendering in real-time is done by projecting into the image plane and alpha blending. 4. Photorealistic Effects: Gaussian splatting use spherical harmonics to represent the view-dependent effects and lighting - allowing surfaces to change color when viewed from different angles, enabling greater photorealism. It doesn't use a neural network, but the training loop is similar to deep learning. 5. Enables Direct Editing: But it's not just speed - with Gaussian Splatting you also get 3D editing support! So you can select, move, and delete stuff, even relight stuff. This type of editing has been more tedious to do with NeRFs and their implicit black box representations. 📲 More tests cooking! Much more to unpack here including simpler explanations. If you enjoyed this post, you might enjoy my feed: Bilawal Sidhu

3D scanning and rendering is moving so fast - got my splats up and running and I'm mind blown getting ~100fps for this complex 3D scene ⬇️ 🤯 1. WAY faster than NeRF: For comparison, NeRFs would takes around 10 seconds per frame (!) Instead I'm zipping around with FPV controls without breaking a sweat - though I do crash a few times towards the end of the video lol 2. Old Meets New: Gaussian Splatting is cool in that it fuses classical graphics and deep learning techniques. Like NeRFs, this is still a radiance field - just without the slower (ne)ural rendering part. 3. Explicit Representation: Instead you represent a 3D scene as a collection of ellipsoidal "splats" called gaussians. Each gaussian has a position, size, and color. Rendering in real-time is done by projecting into the image plane and alpha blending. 4. Photorealistic Effects: Gaussian splatting use spherical harmonics to represent the view-dependent effects and lighting - allowing surfaces to change color when viewed from different angles, enabling greater photorealism. It doesn't use a neural network, but the training loop is similar to deep learning. 5. Enables Direct Editing: But it's not just speed - with Gaussian Splatting you also get 3D editing support! So you can select, move, and delete stuff, even relight stuff. This type of editing has been more tedious to do with NeRFs and their implicit black box representations. 📲 More tests cooking! Much more to unpack here including simpler explanations. If you enjoyed this post, you might enjoy my feed: Bilawal Sidhu

Bilawal Sidhu

337,090 次观看 • 3 年前

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: **Highlights:** 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: Highlights: 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch

Bo Wang

178,579 次观看 • 2 年前

Cause and direction of the Nepal disaster in a 3D map. A massive wave of ice, rocks, mud and floodwater travelled from the high mountains deep into Nepal, destroying roads, bridges, buildings and other infrastructure along the way. 1. Glacier / High Mountain - A lower part of a Himalayan glacier collapsed, sending huge amounts of ice and rock downhill. The exact cause is still being investigated. 2. Lhende Khola - The ice and rocks crashed into the Lhende Khola, triggering a massive flood filled with mud, rocks and debris. 3. Rasuwagadhi - The powerful surge reached Rasuwagadhi, around 20 km from the suspected collapse site, and entered Nepal. 4. Timure - The flood rushed toward Timure, sweeping away roads, bridges, buildings and vehicles. 5. Syapru Besi - The flood continued toward Syapru Besi, causing major damage to homes, roads, hydropower projects and other infrastructure. 6. Trishuli River - Carrying the flood wave deeper into the valley. 7. Further Downstream - The flood then affected areas including Nuwakot, Dhading, Chitwan, Gorkha, Tanahu and Nawalparasi East.

Cause and direction of the Nepal disaster in a 3D map. A massive wave of ice, rocks, mud and floodwater travelled from the high mountains deep into Nepal, destroying roads, bridges, buildings and other infrastructure along the way. 1. Glacier / High Mountain - A lower part of a Himalayan glacier collapsed, sending huge amounts of ice and rock downhill. The exact cause is still being investigated. 2. Lhende Khola - The ice and rocks crashed into the Lhende Khola, triggering a massive flood filled with mud, rocks and debris. 3. Rasuwagadhi - The powerful surge reached Rasuwagadhi, around 20 km from the suspected collapse site, and entered Nepal. 4. Timure - The flood rushed toward Timure, sweeping away roads, bridges, buildings and vehicles. 5. Syapru Besi - The flood continued toward Syapru Besi, causing major damage to homes, roads, hydropower projects and other infrastructure. 6. Trishuli River - Carrying the flood wave deeper into the valley. 7. Further Downstream - The flood then affected areas including Nuwakot, Dhading, Chitwan, Gorkha, Tanahu and Nawalparasi East.

Nalini Unagar

442,136 次观看 • 8 天前

1. Essence of the Problem: Algorithm “Hesitation” and System “Jitter” The “clear → blurry → clear → blurry” cycle you see on the preview screen is essentially the AI algorithm dynamically switching between multiple image-processing paths. During 10× telephoto preview, Samsung’s multimodal imaging system makes decisions based on several concurrent signals: Scene Classifier (scene recognition) AI Detail Enhancer (texture-enhancement algorithm) Motion Estimation (motion detection) HDR Weight Selection (highlight suppression or shadow lift) The issue is that these modules lack a unified arbitration layer. When multiple modules give conflicting judgments about the same frame (for example, “static subject” vs. “slightly moving object”), the algorithm repeatedly enables and cancels enhancement strategies. The result is a visual oscillation of “pre-load → cancel → pre-load → cancel.” This reflects architectural uncertainty within Samsung’s image-processing framework. 2. Deeper Systemic Issue: Unstable Coordination Between ISP and AI In recent Galaxy generations, Samsung’s imaging stack consists of three main components: Exynos/Snapdragon ISP layer (hardware-level processing) Samsung Multi-Frame Engine (multi-frame fusion) Galaxy AI Pipeline (deep-learning post-processing) The core problem is that these modules do not operate within the same clock domain. The AI processing unit runs asynchronously on the NPU, while the ISP and multi-frame fusion run synchronously on the main SoC. In certain scenarios, when the AI result hasn’t returned yet, the ISP outputs the preview frame first—causing frame-to-frame style fluctuations. This isn’t a performance issue; it’s a scheduling bug in the system architecture. Apple avoids this by implementing a unified “Image Core” framework within the A17 Pro. All AI decisions, HDR merges, and white-balance calculations occur within one synchronized pipeline. As a result, the preview image already matches the final shot almost perfectly. 3. User-Level Impact: Inconsistent Output and Experience Fragmentation This “algorithm hesitation” leads to three direct consequences: Preview and final image mismatch — what users see is not what they get. Large variations between shots — even under identical conditions, different AI branches produce completely different looks. Loss of operational trust — users cannot predict results and hesitate to press the shutter. In imaging experience terms, this is actually more serious than sharpness or noise issues, because it breaks the user’s sense of stability and reliability with the device. 4. My View: Samsung’s AI Imaging Needs a “Referee System” The root cause isn’t insufficient power or hardware; it’s the absence of an orchestration layer. Samsung has too many independent sub-modules (super-resolution, noise reduction, detail enhancement, color reconstruction, depth recognition, AI HDR, etc.) but no master controller to decide when to activate them, how to prioritize, or how to manage latency. The ideal solution would be to: Establish a Central Scene Controller Manage all AI sub-modules with unified priority scheduling and decision memory Maintain temporal consistency of algorithmic states across consecutive frames Only then can Samsung truly fix its “algorithm instability” problem and move its Galaxy imaging pipeline toward maturity.

1. Essence of the Problem: Algorithm “Hesitation” and System “Jitter” The “clear → blurry → clear → blurry” cycle you see on the preview screen is essentially the AI algorithm dynamically switching between multiple image-processing paths. During 10× telephoto preview, Samsung’s multimodal imaging system makes decisions based on several concurrent signals: Scene Classifier (scene recognition) AI Detail Enhancer (texture-enhancement algorithm) Motion Estimation (motion detection) HDR Weight Selection (highlight suppression or shadow lift) The issue is that these modules lack a unified arbitration layer. When multiple modules give conflicting judgments about the same frame (for example, “static subject” vs. “slightly moving object”), the algorithm repeatedly enables and cancels enhancement strategies. The result is a visual oscillation of “pre-load → cancel → pre-load → cancel.” This reflects architectural uncertainty within Samsung’s image-processing framework. 2. Deeper Systemic Issue: Unstable Coordination Between ISP and AI In recent Galaxy generations, Samsung’s imaging stack consists of three main components: Exynos/Snapdragon ISP layer (hardware-level processing) Samsung Multi-Frame Engine (multi-frame fusion) Galaxy AI Pipeline (deep-learning post-processing) The core problem is that these modules do not operate within the same clock domain. The AI processing unit runs asynchronously on the NPU, while the ISP and multi-frame fusion run synchronously on the main SoC. In certain scenarios, when the AI result hasn’t returned yet, the ISP outputs the preview frame first—causing frame-to-frame style fluctuations. This isn’t a performance issue; it’s a scheduling bug in the system architecture. Apple avoids this by implementing a unified “Image Core” framework within the A17 Pro. All AI decisions, HDR merges, and white-balance calculations occur within one synchronized pipeline. As a result, the preview image already matches the final shot almost perfectly. 3. User-Level Impact: Inconsistent Output and Experience Fragmentation This “algorithm hesitation” leads to three direct consequences: Preview and final image mismatch — what users see is not what they get. Large variations between shots — even under identical conditions, different AI branches produce completely different looks. Loss of operational trust — users cannot predict results and hesitate to press the shutter. In imaging experience terms, this is actually more serious than sharpness or noise issues, because it breaks the user’s sense of stability and reliability with the device. 4. My View: Samsung’s AI Imaging Needs a “Referee System” The root cause isn’t insufficient power or hardware; it’s the absence of an orchestration layer. Samsung has too many independent sub-modules (super-resolution, noise reduction, detail enhancement, color reconstruction, depth recognition, AI HDR, etc.) but no master controller to decide when to activate them, how to prioritize, or how to manage latency. The ideal solution would be to: Establish a Central Scene Controller Manage all AI sub-modules with unified priority scheduling and decision memory Maintain temporal consistency of algorithmic states across consecutive frames Only then can Samsung truly fix its “algorithm instability” problem and move its Galaxy imaging pipeline toward maturity.

PhoneArt

28,588 次观看 • 10 个月前

Here are 10 AI video editor GitHub repos worth bookmarking: 1. Shotcut Most actively maintained open source video editor in 2026. 14K stars. Cross-platform with AI-assisted features. Just shipped a new release April 30, 2026. 2. Kdenlive The closest open source alternative to Adobe Premiere Pro. Multi-track editing, proxy editing, VST audio, and customizable workspace. Best for professional workflows. 3. OpenShot The easiest entry point for beginners. Drag and drop, 400+ transitions, 3D titles, and AI-assisted trimming. 5,700 stars. 4. Blender Not just 3D. Blender's video sequence editor and compositing pipeline is used in professional film production. 18,300 stars. Unmatched for VFX. 5. Recordly Screen recorder with auto-zoom, cursor polish, webcam overlays, and styled frames built in. Built for demo videos and walkthroughs. 6. Wan2.1 Alibaba's open source text-to-video model. Cinema-grade 1080p generation. Apache 2.0. The gold standard for open source video generation in 2026. 7. HunyuanVideo Tencent's 13B parameter open source video model. 11.9K stars. Handles 720p and 1080p with high temporal coherence. 8. CogVideoX Apache 2.0 licensed. Loads natively via Hugging Face Diffusers. Strong prompt following and smooth frame transitions. Needs 16GB VRAM minimum. 12.5K stars. 9. Open-Sora Most starred open source video generation project at 24K stars. Full training pipeline for $200K. Production-level output quality. 10. Mochi 1 Focused entirely on motion quality. The most natural-looking physics of any open source video model. Water, fabric, and human gestures without AI jitter. Apache 2.0.

Here are 10 AI video editor GitHub repos worth bookmarking: 1. Shotcut Most actively maintained open source video editor in 2026. 14K stars. Cross-platform with AI-assisted features. Just shipped a new release April 30, 2026. 2. Kdenlive The closest open source alternative to Adobe Premiere Pro. Multi-track editing, proxy editing, VST audio, and customizable workspace. Best for professional workflows. 3. OpenShot The easiest entry point for beginners. Drag and drop, 400+ transitions, 3D titles, and AI-assisted trimming. 5,700 stars. 4. Blender Not just 3D. Blender's video sequence editor and compositing pipeline is used in professional film production. 18,300 stars. Unmatched for VFX. 5. Recordly Screen recorder with auto-zoom, cursor polish, webcam overlays, and styled frames built in. Built for demo videos and walkthroughs. 6. Wan2.1 Alibaba's open source text-to-video model. Cinema-grade 1080p generation. Apache 2.0. The gold standard for open source video generation in 2026. 7. HunyuanVideo Tencent's 13B parameter open source video model. 11.9K stars. Handles 720p and 1080p with high temporal coherence. 8. CogVideoX Apache 2.0 licensed. Loads natively via Hugging Face Diffusers. Strong prompt following and smooth frame transitions. Needs 16GB VRAM minimum. 12.5K stars. 9. Open-Sora Most starred open source video generation project at 24K stars. Full training pipeline for $200K. Production-level output quality. 10. Mochi 1 Focused entirely on motion quality. The most natural-looking physics of any open source video model. Water, fabric, and human gestures without AI jitter. Apache 2.0.

Kanika

17,726 次观看 • 2 个月前

Everyone is sleeping on Meta's SAM 3 release. But it's actually a big deal. Here's why: Companies spend millions paying humans to label images and videos frame by frame. A single autonomous driving dataset? Months of work, hundreds of annotators, millions in cost. Without labeled data, you can't train custom models. Without custom models, you're stuck with generic solutions. This is why most companies never move past pilots. SAM 3 breaks this cycle. First let's look at the evolution: SAM 1 segmented objects when you clicked on them. Revolutionary, but one object at a time. SAM 2 added video tracking with memory. Game-changing, but you still manually prompted every object. SAM 3 changes everything with text prompts. Type "yellow school bus" and it finds ALL of them in your image or video. Not just one. Every instance across thousands of frames. Now here's where people get confused: "Can't I just use GPT-5 or Gemini for this?" No, and here's why that's a terrible approach. Large multimodal LLMs are great for reasoning, but they're slow and expensive for production visual tasks. You're paying API costs per image, waiting seconds for responses, getting inconsistent results. SAM 3 runs in 30 milliseconds on a single GPU for 100+ objects. That's 100x faster, and you own the infrastructure. More importantly, SAM 3 gives you precise pixel-level masks, not descriptions. Try asking an LLM to segment every defective part on a manufacturing line in real-time. It won't work. SAM 3 does this effortlessly. The real breakthrough is their data engine. Meta built an AI-human hybrid system that's 5x faster for complex annotations. They trained SAM 3 on 4 million unique visual concepts - 50x more than existing benchmarks like LVIS. SAM 3 is trained on 4 million unique visual concepts, it handles everything: - Text-based concept search - Interactive refinement with clicks - Video tracking across frames - Zero-shot detection of new concepts The model is open source. Weights, code, and benchmarks are on GitHub. If you're building computer vision applications, this is the foundation model to evaluate. The annotation time savings alone will pay for integration costs within weeks. Find the relevant links in the next tweet!

Everyone is sleeping on Meta's SAM 3 release. But it's actually a big deal. Here's why: Companies spend millions paying humans to label images and videos frame by frame. A single autonomous driving dataset? Months of work, hundreds of annotators, millions in cost. Without labeled data, you can't train custom models. Without custom models, you're stuck with generic solutions. This is why most companies never move past pilots. SAM 3 breaks this cycle. First let's look at the evolution: SAM 1 segmented objects when you clicked on them. Revolutionary, but one object at a time. SAM 2 added video tracking with memory. Game-changing, but you still manually prompted every object. SAM 3 changes everything with text prompts. Type "yellow school bus" and it finds ALL of them in your image or video. Not just one. Every instance across thousands of frames. Now here's where people get confused: "Can't I just use GPT-5 or Gemini for this?" No, and here's why that's a terrible approach. Large multimodal LLMs are great for reasoning, but they're slow and expensive for production visual tasks. You're paying API costs per image, waiting seconds for responses, getting inconsistent results. SAM 3 runs in 30 milliseconds on a single GPU for 100+ objects. That's 100x faster, and you own the infrastructure. More importantly, SAM 3 gives you precise pixel-level masks, not descriptions. Try asking an LLM to segment every defective part on a manufacturing line in real-time. It won't work. SAM 3 does this effortlessly. The real breakthrough is their data engine. Meta built an AI-human hybrid system that's 5x faster for complex annotations. They trained SAM 3 on 4 million unique visual concepts - 50x more than existing benchmarks like LVIS. SAM 3 is trained on 4 million unique visual concepts, it handles everything: - Text-based concept search - Interactive refinement with clicks - Video tracking across frames - Zero-shot detection of new concepts The model is open source. Weights, code, and benchmarks are on GitHub. If you're building computer vision applications, this is the foundation model to evaluate. The annotation time savings alone will pay for integration costs within weeks. Find the relevant links in the next tweet!

Akshay 🚀

46,438 次观看 • 9 个月前

hy3 vs fable 5 vs opus 4.8 vs sonnet 5 Tencent Hy just dropped hy3 – their new open-weight model under apache 2.0. following the april preview they scaled up post-training, and it now rivals flagship open models with 2-5x the params. api pricing: ~$0.15 in / ~$0.59 out per 1m tokens. built for coding, office work, frontend, agentic tasks so we ran a test: hy3 vs fable 5 vs opus 4.8 vs sonnet 5 three prompts, one-shot each: • ocean wave crumbling a sand castle (canvas) • looping factory assembly line (html/css/js) • interactive 3d city with three.js + orbitcontrols self-contained files, no libraries beyond the cdn where asked totals across all three prompts: 1. hy3 – 1231 loc / 14m34s 2. sonnet 5 – 1373 loc / 18m55s 3. fable 5 – 1546 loc / 18m32s 4. opus 4.8 – 1904 loc / 27m21s hy3 is the fastest and the leanest by a wide margin we had opus 4.8 analyze hy3's code. the read: - sand castle: checklist-complete but the crumble is parametric, not physical. it shrinks and slumps the towers and fades alpha instead of dissolving into grains. the cheap-but-plausible interpretation. the tell of a smaller model - factory line: the arm-to-part sync is actually causal, not faked. it triggers each robot early by exactly the arm's descent time, so the tap lands right as the part arrives. it also pre-seeds the belt so it never cold-starts empty. clean state machine. one latent bug – a part gets marked processed before checking if the robot is free, so at a faster spawn rate a "laptop" could ship missing a part. never fires at current timing, but the invariant isn't enforced - 3d city: genuinely frontier-adjacent. correct modern setup (pcfsoft shadows, srgb, aces tone mapping, damped orbit + auto-rotate pause). clones the window texture per building and scales the uv repeat to each building's dimensions so windows don't stretch. downside: no instancing – ~800 texture clones across 200 buildings. runs fine, not optimized. roads are implicit gaps, not explicit planes our observations: • hy3 is quite fast • its animations are simple but you can see it trying – it adds detail, and the 3d render sits at the same level as the frontier models • sonnet 5 is weak here. hy3 beats it on the sand castle and the 3d render, level on the conveyor • opus 4.8 is anthropic's best model after the fable 5 nerf – it beats fable on the conveyor and the 3d render net: hy3 runs clean and well-formed across all three with zero syntax errors, even version-matching the three.js core and examples build. it's economical rather than ambitious – it does the minimum viable version of each hard requirement well, and only reaches for the expensive interpretation on the 3d task a very coherent profile for a cost-optimized open-weight model follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

34,961 次观看 • 1 个月前

I stole this startup idea from a video with 453,000 views and 71,000 comments from parents. A dad built a paper YouTube feed for his daughter. She scans a QR code, the video downloads to a local server, plays with zero ads, algorithm, or surprise "just 5 more minutes daddy!" He can set it to power off after a certain number of scans. Screen time with guardrails. Parents lost their minds in the comments. Let's turn that demand into a business you can build this weekend. IRL YouTube for Kids. Simple offers + an add on upsell: 1. Done-for-you kit. $99. 3D printed housing, clicky button, wired up nice. Fun little brand. You open the box and assemble it in 20 minutes. This is the move for parents who want it to just work. 2. Do-it-yourself kit. $49. Shopping list, assembly instructions, 3D print files. You source the parts yourself. Embed Amazon affiliate links in every component so you make money whether they buy the kit or not. + Recurring: layer on a content subscription as the upsell. Curated video picks by age and interest. Nicely designed binder full of scan cards that shows up at your door every month. Nostalgic and tactile. The kind of thing millennial parents didn't know they needed. Give your kids the 90s childhood you want them to have. You already know the content works. You can make this same style video with different hooks and ride the wave. The audience is hungry. Package this puppy up and print some profit. As always, I'm rooting for you.

Idea Browser

34,266 次观看 • 5 个月前

🚀AI NEXUS 2.0 - A NEW ERA BEGINS Happy One Year Anniversary! During the bear… we proved the model: • Over 350K installs of the AI Nexus Mobile App • 150K wallets onboarded to Multiversᕽ • Turning people’s socials → playable 3D avatars in minutes • 60+ partner projects onboarded to our ecosystem But it's not enough. Now, we upgrade... Welcome to AI Nexus 2.0 AI Nexus 2.0 = AI + Gaming Infrastructure. Powered by $A1X. Cross-chain. Creator-driven. Identity-first. Built for everyone: from everyday mobile users to full game studios. AI Nexus becomes a superlayer for digital identity, gaming and AI-powered creation. And over the coming weeks (before end of the year), we’ll reveal every part of it in clear, bite-sized drops. What’s Coming in AI Nexus 2.0? 1. Two brand-new apps (launching sooner than you think!). 2. Deeper strategic, revenue-driven integrations with Ice Open Network, Multiversᕽ, , Linera ⛓️ & more. 3. Marketplace for Avatars / Assets; powered by NFTs. 4. SDK for effortless 3D avatar creation. 5. $A1X total supply reduction. 6. More AI tools coming to AI Nexus Mobile App & games to the Clone Machine. 7. New staking + developer incentive programs. AI Nexus 2.0 is bigger, broader, and built to scale. And we’re just getting started.

AI Nexus

172,609 次观看 • 8 个月前

HERMES AGENT: 5 USEFUL COMMUNITY PLUGINS RIGHT NOW Hermes already ships with strong core features — Skills, MCP, Memory, Curator, and a plugin system. These extensions don’t replace anything in the core. They fill practical gaps. Here are the ones worth installing: 1. hermes-plugins by 42-evey Solid collection. Adds goal management, inter-agent communication, cost tracking, and easier model switching. Good for running Hermes autonomously. 2. web-search-plus Much stronger web search than the default. Supports multiple providers, smart routing, quality scoring, and a proper research mode. 3. super-hermes by Cranot Adds “Prisms” — structured analytical frameworks. Makes the agent noticeably better at deep analysis and catching its own blind spots. 4. Hermes Agent Self-Evolution (by Nous Research) Official companion that automatically improves skills and prompts using DSPy + GEPA. Useful if you want the agent to get better over time with minimal manual work. 5. Advanced Memory Providers When built-in memory isn’t enough for long projects, external providers give Hermes stronger long-term recall and context handling. All of these work cleanly on top of Hermes v0.16.0 without conflicting with core systems. Which plugins or extensions are you actually using with Hermes right now? Github links 👇

HERMES AGENT: 5 USEFUL COMMUNITY PLUGINS RIGHT NOW Hermes already ships with strong core features — Skills, MCP, Memory, Curator, and a plugin system. These extensions don’t replace anything in the core. They fill practical gaps. Here are the ones worth installing: 1. hermes-plugins by 42-evey Solid collection. Adds goal management, inter-agent communication, cost tracking, and easier model switching. Good for running Hermes autonomously. 2. web-search-plus Much stronger web search than the default. Supports multiple providers, smart routing, quality scoring, and a proper research mode. 3. super-hermes by Cranot Adds “Prisms” — structured analytical frameworks. Makes the agent noticeably better at deep analysis and catching its own blind spots. 4. Hermes Agent Self-Evolution (by Nous Research) Official companion that automatically improves skills and prompts using DSPy + GEPA. Useful if you want the agent to get better over time with minimal manual work. 5. Advanced Memory Providers When built-in memory isn’t enough for long projects, external providers give Hermes stronger long-term recall and context handling. All of these work cleanly on top of Hermes v0.16.0 without conflicting with core systems. Which plugins or extensions are you actually using with Hermes right now? Github links 👇

YanXbt

34,564 次观看 • 3 个月前

U-Net by hand ✍️ ~ 17 steps walkthrough below I consider U-Net as a key milestone in deep learning, the first image-to-image model that really worked! It came out of medical imaging, an unusual place, not from NeurIPS or CVPR or ACL. Now it is the backbone of diffusion models, which you see in almost all modern image generation models. I drew the network as a C so the matrix multiplication flows naturally down. Tilt your head to the right and it is a U again. 🤣 Goal: push a 3 x 16 image down to a 2 x 4 bottleneck and back out again, filling in every cell yourself. = 1. Given = An image of three channels, R, G and B, sixteen pixels wide, and every kernel the network will use. = 2. Convolution 1 = Let us slide the first kernel over the image. Each output is one multiply-and-add over a 2 x 3 window, and the result is the green feature map. = 3. Find the maxima = We circle the largest value in each 1 x 2 window. Circling first is worth the extra step: it is the pooling decision, made before anything is written down. = 4. Max pool 1 = Let us copy those maxima down. Sixteen columns become eight, and half the detail is gone for good. = 5. Convolution 2 = We convolve again with the second kernel, deeper into the contracting path. The feature map is blue now. = 6. Find the maxima again = Same move as step 3, on the blue map. = 7. Max pool 2 = Eight columns become four. = 8. The bottleneck = Let us convolve once more. This is the bottom of the U, a 2 x 4 block that is everything the network kept. = 9. Spread it out = We start back up. The transposed convolution writes each bottleneck value into a wider grid, leaving gaps between them. = 10. Transposed convolution 1 = Let us fill those gaps by convolving over the spread-out grid. Four columns become eight. = 11. The first skip = We copy the encoder's matching row straight across. This is the skip connection, and it is the whole reason a U-Net can recover detail that pooling threw away. = 12. Convolution with the skip = Let us convolve the upsampled features together with the copied ones. = 13. Spread it out again = Same as step 9, one level up. = 14. Transposed convolution 2 = Eight columns become sixteen, back to the width we started at. = 15. The second skip = The encoder's first feature map comes across, the one made before any pooling happened. = 16. Convolution and ReLU = We convolve, then cross out every negative and set it to zero. = 17. Output convolution = Let us apply the last kernel. Out comes R', G' and B', an image the same size as the one we started with. The outputs: R' = [3, 0, 7, 0, 7, 0, 17, 0, 3, 0, 9, 0, 2, 0, 6, 0] G' = [1, 20, 1, 10, 1, 12, 1, 19, 2, 5, 1, 11, 1, 3, 1, 7] B' = [4, 20, 8, 10, 8, 12, 18, 19, 5, 5, 10, 11, 3, 3, 7, 7] Congrats! You just calculated a U-Net by hand. 💾 Save this post!

U-Net by hand ✍️ ~ 17 steps walkthrough below I consider U-Net as a key milestone in deep learning, the first image-to-image model that really worked! It came out of medical imaging, an unusual place, not from NeurIPS or CVPR or ACL. Now it is the backbone of diffusion models, which you see in almost all modern image generation models. I drew the network as a C so the matrix multiplication flows naturally down. Tilt your head to the right and it is a U again. 🤣 Goal: push a 3 x 16 image down to a 2 x 4 bottleneck and back out again, filling in every cell yourself. = 1. Given = An image of three channels, R, G and B, sixteen pixels wide, and every kernel the network will use. = 2. Convolution 1 = Let us slide the first kernel over the image. Each output is one multiply-and-add over a 2 x 3 window, and the result is the green feature map. = 3. Find the maxima = We circle the largest value in each 1 x 2 window. Circling first is worth the extra step: it is the pooling decision, made before anything is written down. = 4. Max pool 1 = Let us copy those maxima down. Sixteen columns become eight, and half the detail is gone for good. = 5. Convolution 2 = We convolve again with the second kernel, deeper into the contracting path. The feature map is blue now. = 6. Find the maxima again = Same move as step 3, on the blue map. = 7. Max pool 2 = Eight columns become four. = 8. The bottleneck = Let us convolve once more. This is the bottom of the U, a 2 x 4 block that is everything the network kept. = 9. Spread it out = We start back up. The transposed convolution writes each bottleneck value into a wider grid, leaving gaps between them. = 10. Transposed convolution 1 = Let us fill those gaps by convolving over the spread-out grid. Four columns become eight. = 11. The first skip = We copy the encoder's matching row straight across. This is the skip connection, and it is the whole reason a U-Net can recover detail that pooling threw away. = 12. Convolution with the skip = Let us convolve the upsampled features together with the copied ones. = 13. Spread it out again = Same as step 9, one level up. = 14. Transposed convolution 2 = Eight columns become sixteen, back to the width we started at. = 15. The second skip = The encoder's first feature map comes across, the one made before any pooling happened. = 16. Convolution and ReLU = We convolve, then cross out every negative and set it to zero. = 17. Output convolution = Let us apply the last kernel. Out comes R', G' and B', an image the same size as the one we started with. The outputs: R' = [3, 0, 7, 0, 7, 0, 17, 0, 3, 0, 9, 0, 2, 0, 6, 0] G' = [1, 20, 1, 10, 1, 12, 1, 19, 2, 5, 1, 11, 1, 3, 1, 7] B' = [4, 20, 8, 10, 8, 12, 18, 19, 5, 5, 10, 11, 3, 3, 7, 7] Congrats! You just calculated a U-Net by hand. 💾 Save this post!

Tom Yeh

17,510 次观看 • 1 个月前

Gemini-1.5 Pro has its spotlight stolen today, and people are poking fun at Sora vs Google memes. Well, I think it's the biggest boost in LLM capability so far in 2024. v1.5's 10M token context (1) excels at retrieval; (2) generalizes zero-shot to extremely long instructions like full tutorials and codebases; and (3) works across modalities such as text, audio, and video. Here's a stunning example: v1.5 learns to translate from English to Kalamang purely in context, following a full linguistic manual at inference time. Kalamang is a language spoken by fewer than 200 speakers in western New Guinea. Gemini has never seen this language during training and is only provided with 500 pages of linguistic documentation, a dictionary, and ~400 parallel sentences in context. It basically acquires a sophisticated new skill in the neural activations, instead of gradient finetuning. I talked about the Myth of Context Length many times before: don't get too excited by claims of 1M or even 1B context tokens. LSTMs already achieved literally infinite context length 25 yrs ago! What truly matters is how well the model actually uses the context to solve real-world problems, and Gemini-1.5 has surpassed the SOTA with flying colors. The paper is also well-written with lots of solid quantitative analysis on in-context memorization and generalization. Paper: “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context” Congrats to Jeff Dean Oriol Vinyals Sundar Pichai and team!

Gemini-1.5 Pro has its spotlight stolen today, and people are poking fun at Sora vs Google memes. Well, I think it's the biggest boost in LLM capability so far in 2024. v1.5's 10M token context (1) excels at retrieval; (2) generalizes zero-shot to extremely long instructions like full tutorials and codebases; and (3) works across modalities such as text, audio, and video. Here's a stunning example: v1.5 learns to translate from English to Kalamang purely in context, following a full linguistic manual at inference time. Kalamang is a language spoken by fewer than 200 speakers in western New Guinea. Gemini has never seen this language during training and is only provided with 500 pages of linguistic documentation, a dictionary, and ~400 parallel sentences in context. It basically acquires a sophisticated new skill in the neural activations, instead of gradient finetuning. I talked about the Myth of Context Length many times before: don't get too excited by claims of 1M or even 1B context tokens. LSTMs already achieved literally infinite context length 25 yrs ago! What truly matters is how well the model actually uses the context to solve real-world problems, and Gemini-1.5 has surpassed the SOTA with flying colors. The paper is also well-written with lots of solid quantitative analysis on in-context memorization and generalization. Paper: “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context” Congrats to Jeff Dean Oriol Vinyals Sundar Pichai and team!

Jim Fan

278,517 次观看 • 2 年前

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

Tom Yeh

67,883 次观看 • 2 年前

ANTHROPIC LEAKED A FILE WHERE 8 ENGINEERS COST $2.40 A DAY INSTEAD OF $120,000 A MONTH Microsoft and Google already run this - the same 8 seats cost them $73 a month. the ring turns once a day, 3 shifts hand off the work, zero meetings. the scout has tomorrow's agenda ready before you sit down. the architect splits the work, 2 builders run parallel branches. contracts get written before the workers - skip that and every worker guesses what the last one meant. with contracts all three read one shape: what goes in, what comes out. 3 branches at once means you pay the slowest lane, not the sum of three. nothing reaches you unrejected: the cheap model asks does it run, the expensive one asks should it merge. a gate that never rejects anything is a status badge, not a gate. the night crew takes the long jobs that never fit a working day. 1 role out of 8 writes back into the file - cut that edge and the rest repeat monday for a year. save this and paste it into Claude Code - you write the file, the file runs the company ↓

ANTHROPIC LEAKED A FILE WHERE 8 ENGINEERS COST $2.40 A DAY INSTEAD OF $120,000 A MONTH Microsoft and Google already run this - the same 8 seats cost them $73 a month. the ring turns once a day, 3 shifts hand off the work, zero meetings. the scout has tomorrow's agenda ready before you sit down. the architect splits the work, 2 builders run parallel branches. contracts get written before the workers - skip that and every worker guesses what the last one meant. with contracts all three read one shape: what goes in, what comes out. 3 branches at once means you pay the slowest lane, not the sum of three. nothing reaches you unrejected: the cheap model asks does it run, the expensive one asks should it merge. a gate that never rejects anything is a status badge, not a gate. the night crew takes the long jobs that never fit a working day. 1 role out of 8 writes back into the file - cut that edge and the rest repeat monday for a year. save this and paste it into Claude Code - you write the file, the file runs the company ↓

Sprytix

698,581 次观看 • 15 天前