ok, building this

Currently using gemini flash lite for this. I tried groq's models and they're faster but too dumb.

Unlucky, best answer was “seeing as I'm not six, I don't really have a favourite color.”

use logprobs + openai (if you're not already) does flash-lite do top n tokens? didn't realize

logprobs only works on davinci models from forever ago. Doesn't seem like the right game to play anyway, though maybe I can keep working on it. None of the gemini models do logprobs (though they say they do)

This looks like fun.

It’s like a killer choose your own adventure

What library are you using for the TUI?

How are you getting Gemini to respond only a few tokens? Are you perhaps just running the same prompt 3 times and picking the first n tokens from each?

Im just asking for three options of the next most likely three words, with a probability assigned

asking an llm what the three next options will likely be is like asking water where it's going to flow next if you want to keep doing it this way, turn reasoning off on flash lite, that's probably where the latency is coming from i personally would use logprobs locally

We still talk about you, classic Lego dragon.

my weekend project to learn about bluetooth mesh networks, relays and store and forward models, message encryption models, and a few other things. bitchat: bluetooth mesh chat...IRC vibes. TestFlight: GitHub:

recently seeing the rise of "plastic software" while this software is fast to create and consume, it's next-to-impossible to refactor and reuse