Tristan Hume's banner

Tristan Hume

@trishume • 8,988 subscribers

Performance optimization lead @AnthropicAI. Profiling, distributed systems, dev tools, interpretability. [email protected]

Shorts

Here's Claude 3 Haiku running at >200 tokens/s (>2x as fast as prod)! We've been working on capacity optimizations but we can have fun testing those as speed optimizations via overly-costly low batch size. Come work with me at Anthropic on things like this, more info in thread 🧵

Here's Claude 3 Haiku running at >200 tokens/s (>2x as fast as prod)! We've been working on capacity optimizations but we can have fun testing those as speed optimizations via overly-costly low batch size. Come work with me at Anthropic on things like this, more info in thread 🧵

99,886 views