13 Comments
Comments from the original post on X (Twitter)

@steveruizok
Currently using gemini flash lite for this. I tried groq's models and they're faster but too dumb.
7 likes

@theopomies
Unlucky, best answer was “seeing as I'm not six, I don't really have a favourite color.”
2 likes

@benhylak
use logprobs + openai (if you're not already) does flash-lite do top n tokens? didn't realize
2 likes

@steveruizok
logprobs only works on davinci models from forever ago. Doesn't seem like the right game to play anyway, though maybe I can keep working on it. None of the gemini models do logprobs (though they say they do)
2 likes

@TelepathicPug
This looks like fun.
2 likes

@patricktrack
It’s like a killer choose your own adventure
1 like

@FarberKostya
What library are you using for the TUI?

@aryanbaburajan
How are you getting Gemini to respond only a few tokens? Are you perhaps just running the same prompt 3 times and picking the first n tokens from each?

@steveruizok
Im just asking for three options of the next most likely three words, with a probability assigned

@cheatyyyy
asking an llm what the three next options will likely be is like asking water where it's going to flow next if you want to keep doing it this way, turn reasoning off on flash lite, that's probably where the latency is coming from i personally would use logprobs locally

@BuildingGunn
We still talk about you, classic Lego dragon.
12,591 likes

@jack
my weekend project to learn about bluetooth mesh networks, relays and store and forward models, message encryption models, and a few other things. bitchat: bluetooth mesh chat...IRC vibes. TestFlight: GitHub:
7,248 likes

@aidenybai
recently seeing the rise of "plastic software" while this software is fast to create and consume, it's next-to-impossible to refactor and reuse
1,158 likes
