Marc Sun's banner
Marc Sun's profile picture

Marc Sun

@_marcsun • 1,697 subscribers

Machine Learning Engineer @huggingface Open Source team

Shorts

`transformers` + `torchao` quantization + `torch.compile` for faster inference speed and less memory usage 🔥 Demo of "meta-llama/Meta-Llama-3.1-8B-Instruct" quantized in 4-bit weight-only :

`transformers` + `torchao` quantization + `torch.compile` for faster inference speed and less memory usage 🔥 Demo of "meta-llama/Meta-Llama-3.1-8B-Instruct" quantized in 4-bit weight-only :

24,571 views

Videos

LIVE
1.2k

Anya Rossi

sweetdream.ai

SweetDream.aiSponsored•Livecam
Streaming Now

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

HD live stream
Exclusive private shows
1.2k viewers online

Free preview available • Premium content

Millions of GGUF downloads later, those same llama.cpp checkpoints can now run in 🤗 transformers. Same models, more ways to use them, and fast local inference on Mac powered by ggml kernels! Blog: ggml kernels:
0:37
_marcsun's profile picture

Millions of GGUF downloads later, those same llama.cpp checkpoints can now run in 🤗 transformers. Same models, more ways to use them, and fast local inference on Mac powered by ggml kernels! Blog: ggml kernels:

Marc Sun

66,530 views • 1 month ago

Your LLM knows the answer before it writes a single token. So we stopped making it write. transformers serve now turns any LLM or VLM into a decision model with `/v1/systemone` endpoint. Decision model standardization coming soon in transformers by Arthur Zucker Demos below👇
0:43
_marcsun's profile picture

Your LLM knows the answer before it writes a single token. So we stopped making it write. transformers serve now turns any LLM or VLM into a decision model with `/v1/systemone` endpoint. Decision model standardization coming soon in transformers by Arthur Zucker Demos below👇

Marc Sun

13,808 views • 17 days ago

No more content to load