Loading video...

Video Failed to Load

Go Home

sam.cpp 👀 Inference of Meta's Segment Anything Model on the CPU Project by Yavor Ivanov - powered by

ggerganov's profile picture

Georgi Gerganov

53,881 subscribers

264,542 views • 2 years ago •via X (Twitter)

EducationGamingScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

10 Comments

Georgi Gerganov's profile picture
Georgi Gerganov2 years ago

repo: The project is still in it's early stages, but @YavorGI has already done a great job with the implementation. Please check it out and let's help him bring SOTA image segmentation to our devices!

Georgi Gerganov's profile picture
Georgi Gerganov2 years ago

Is this what the Terminator used 🤔

kache's profile picture
kache2 years ago

@YavorGI amazing!

Kenneth Cassel's profile picture
Kenneth Cassel2 years ago

@YavorGI whoa looks pretty fast. we've been using FastSAM for faster SAM inference. but it has slightly worse perf than SAM excited to check this out 👀

Georgi Gerganov's profile picture
Georgi Gerganov2 years ago

@YavorGI Speed is not there yet, but lots of room for improvement. Should by OK an Apple Silicon I think

bryan pratte's profile picture
bryan pratte2 years ago

@YavorGI What is the inference time per frame end to end?

Mevlut Cafer Arslan's profile picture
Mevlut Cafer Arslan2 years ago

@YavorGI Vision Pro SDK + this + a model that can extract information from segmentations 🥹

𝕏 NiNo's profile picture
𝕏 NiNo2 years ago

@yacineMTB @YavorGI @Scobleizer 👀 with AppleVisionPro and then SegmentationsInfos…. 🔥

Nikhila Ravi's profile picture
Nikhila Ravi2 years ago

@YavorGI Very cool! What's the breakdown in the time for 1) embedding extraction per image and 2) mask prediction per prompt?

Mayowa Daniel's profile picture
Mayowa Daniel2 years ago

@YavorGI Exciting! Just whipped this up and it got merged so quickly too! Trying to get my feet wet with the whole ai thing Been a minute since I touched c++ so I’m looking forward to good first issues and open to being a fly on the wall/pair 🤲🏽

Related Videos

Detecting airplanes at DFW Airport from a single text prompt. Meta's Segment Anything Model 3 and the geosam R package make powerful image detection tools accessible.
0:51

Detecting airplanes at DFW Airport from a single text prompt. Meta's Segment Anything Model 3 and the geosam R package make powerful image detection tools accessible.

Kyle Walker

31,475 views • 5 months ago

📣 MLPerf Inference v6.0 results are in. Learn how systems powered by NVIDIA Blackwell set the pace on inference, delivering the highest AI factory throughput. 🔗
1:02

📣 MLPerf Inference v6.0 results are in. Learn how systems powered by NVIDIA Blackwell set the pace on inference, delivering the highest AI factory throughput. 🔗

NVIDIA Data Center

33,408 views • 2 months ago

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!
0:56

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,583 views • 2 years ago

Introducing the NVIDIA Project DIGITS personal AI supercomputer, powered by the GB10 Grace Blackwell Superchip: Project DIGITS enables developers to prototype, fine-tune and inference models locally and seamlessly deploy at scale to the data center or cloud. #CES2025 #NVIDIAGraceBlackwell
0:23

Introducing the NVIDIA Project DIGITS personal AI supercomputer, powered by the GB10 Grace Blackwell Superchip: Project DIGITS enables developers to prototype, fine-tune and inference models locally and seamlessly deploy at scale to the data center or cloud. #CES2025 #NVIDIAGraceBlackwell

NVIDIA

51,983 views • 1 year ago

#M5StackNew 🎊 The LLM630 Compute Kit is an #AI large language model (#LLM) inference development kit, powered by the #Axera #AX630C SoC with a 3.2 TOPs NPU, it delivers efficient AI inference for tasks like computer vision (CV) and LLM processing.
0:45

#M5StackNew 🎊 The LLM630 Compute Kit is an #AI large language model (#LLM) inference development kit, powered by the #Axera #AX630C SoC with a 3.2 TOPs NPU, it delivers efficient AI inference for tasks like computer vision (CV) and LLM processing.

M5Stack

16,383 views • 1 year ago

Unbiased AI -- at Scale Truly Open, inference at scale, running on Subnet 4. Powered by the Bittensor ecosystem. Full episode on Youtube and Spotify:
0:57

Unbiased AI -- at Scale Truly Open, inference at scale, running on Subnet 4. Powered by the Bittensor ecosystem. Full episode on Youtube and Spotify:

Openτensor Foundaτion

45,393 views • 1 year ago

Introducing AI Click powered by Segment Anything (SAM) for biological images, on Biodock. We’ve worked hard to make SAM much more useful for bio - read below to find out how!
0:44

Introducing AI Click powered by Segment Anything (SAM) for biological images, on Biodock. We’ve worked hard to make SAM much more useful for bio - read below to find out how!

Michael

57,576 views • 3 years ago

Introducing FOMO - FLock Open Model Offering The full-cycle decentralized AI model economy is now on Base, powered by FLock.
1:37

Introducing FOMO - FLock Open Model Offering The full-cycle decentralized AI model economy is now on Base, powered by FLock.

FLock.io

65,838 views • 5 months ago

Super stoked that Jordão Bragantini 's napari plugin for Meta's Segment Anything Model 2: napari-sam2, as well as our cell tracking software ultrack were showcased during Chris Cox's Keynote at Meta's Connect 2024. Jordao and I were in the audience; it was great fun! Chan Zuckerberg Biohub Network
0:56

Super stoked that Jordão Bragantini 's napari plugin for Meta's Segment Anything Model 2: napari-sam2, as well as our cell tracking software ultrack were showcased during Chris Cox's Keynote at Meta's Connect 2024. Jordao and I were in the audience; it was great fun! Chan Zuckerberg Biohub Network

Loïc A. Royer 💻🔬⚗️

41,045 views • 1 year ago

The Segment Anything Model (SAM) by Meta AI is a step toward the first foundation model for image segmentation. SAM is capable of one-click segmentation of any object from photos or videos + zero-shot transfer to other segmentation tasks. Try the demo ➡️
0:51

The Segment Anything Model (SAM) by Meta AI is a step toward the first foundation model for image segmentation. SAM is capable of one-click segmentation of any object from photos or videos + zero-shot transfer to other segmentation tasks. Try the demo ➡️

AI at Meta

186,324 views • 3 years ago

Fund your Kalshi account with Bitcoin, trade on anything. Powered by @ZeroHashX
0:39

Fund your Kalshi account with Bitcoin, trade on anything. Powered by @ZeroHashX

Kalshi

86,126 views • 1 year ago

THE TAKEDOWN: AOC losing to facts in today's Takedown segment powered by new search partner Perplexity
4:23

THE TAKEDOWN: AOC losing to facts in today's Takedown segment powered by new search partner Perplexity

The Ben Shapiro Show

169,095 views • 1 year ago

Private Uniswap is fully functional with The Anon Project. Powered by RAILGUN.
0:59

Private Uniswap is fully functional with The Anon Project. Powered by RAILGUN.

RAILGUN - Private Ethereum DeFi

22,050 views • 27 days ago

Releasing moondream-zig! It is a fast, implementation of moondream2 inference on the CPU written from-scratch in Zig :) moondream-zig provides 1.5-2x faster inference compared to huggingface on the same device. moondream vik
1:21

Releasing moondream-zig! It is a fast, implementation of moondream2 inference on the CPU written from-scratch in Zig :) moondream-zig provides 1.5-2x faster inference compared to huggingface on the same device. moondream vik

snow

29,245 views • 1 year ago

Done. Long pending demand of people of #Koraput Parliament Segment: Gunupur - Therubali New Railway Line Project approved by the Central Government. 🙏
1:01

Done. Long pending demand of people of #Koraput Parliament Segment: Gunupur - Therubali New Railway Line Project approved by the Central Government. 🙏

Saptagiri Ulaka

14,151 views • 1 year ago

Glimpse of the model houses for resettlement of people who are affected by Parandur Greenfield Airport Project. A township just near the project site!!
1:30

Glimpse of the model houses for resettlement of people who are affected by Parandur Greenfield Airport Project. A township just near the project site!!

Chennai Updates

34,074 views • 5 months ago

RESIDENT MOUTHFUL Ashley Graham/POV Vore Animation. Apparently, she's more prepared to do anything to survive than we thought 🫦👀 Animation Collab with CEO of EatFun (Appdattta) Ashley Model by MokujinH🔞 Esophagus Model by meknes Stomach Model by GarrisenGameArt🔞🎮🎨
1:38

Sensitive content

RESIDENT MOUTHFUL Ashley Graham/POV Vore Animation. Apparently, she's more prepared to do anything to survive than we thought 🫦👀 Animation Collab with CEO of EatFun (Appdattta) Ashley Model by MokujinH🔞 Esophagus Model by meknes Stomach Model by GarrisenGameArt🔞🎮🎨

Paramouth Pictures

72,978 views • 3 years ago

Deterministic inference: getting the exact same output every time you run an LLM with identical inputs. Try it yourself: Powered by EigenAI
0:31

Deterministic inference: getting the exact same output every time you run an LLM with identical inputs. Try it yourself: Powered by EigenAI

nader dabit

29,335 views • 7 months ago