Melvin Vivas's banner
Melvin Vivas's profile picture

Melvin Vivas

@melvindvivas • 2,079 subscribers

Local models, Agents, AI Engineering AI Coworkers https://t.co/UCRp7F3gZ9 Pylib Local AI https://t.co/MtiGBNIqKS Book a call https://t.co/2YVZbo5saw

Shorts

This is awesome. Thanks Mia for this Was able to run with the RTX 3090 use this .env config DRAFT=mtp CONTEXT_SIZE=262144 CACHE_QUANT=8,4 GPU_MEM_GB=22 262K context: - Run 1: 65.98 tok/s - Run 2: 63.23 tok/s - Run 3: 64.41 tok/s Average: ~64.5 tok/s.

This is awesome. Thanks Mia for this Was able to run with the RTX 3090 use this .env config DRAFT=mtp CONTEXT_SIZE=262144 CACHE_QUANT=8,4 GPU_MEM_GB=22 262K context: - Run 1: 65.98 tok/s - Run 2: 63.23 tok/s - Run 3: 64.41 tok/s Average: ~64.5 tok/s.

87,084 Aufrufe