Hybrid models efficiently route complex tasks to larger cloud models, but how do you go about creating one? 🤔 Cactus solved this by post-training Gemma 4 to return a confidence score for every prompt. Process the prompt locally when confidence is high, route to a larger model when confidence... is low. Routing only happens 15%-55% of time based on domain benchmarks.show more

Cactus Hybrid is open-weights, is based on Gemma 4 E2B and is ready to run on MLX, llama.cpp, transformers, and Cactus. Explore the code and models: - GitHub: - Hugging Face:

This is awesome we spent months trying to make E2B honest and pass on the responsibility of answering to a bigger model but kept reaching hallucination blockers for our usecase.

I once trained a micro binary cognitive complexity classifier that produces "high", "low" output based on user input in <10ms It took me downloading a bunch of prompt datasets from HF, spending a bunch of time and $ labeling the prompts with a super smart model for supervised learning. But it did the trick very well. This reminded me of it.

The UI is very pretty but I don't see it in the repo. Any advice?

That is an incredibly elegant solution. Using Gemma 4 to generate its own confidence score completely fixes the biggest headache of hybrid routing. 😊

This is probably where local AI is headed. Keep the easy or private work local and only send the harder prompts to the cloud. I put together a free wizard that helps people decide between local, hybrid and hosted setups:

GREAT, you are in the tradition of western gun-runners (aka MIC): Provide what's needed to destriy the world - who cares - main thing you get a big piece of the business-cake.

routing based on the model’s own confidence score feels like a clean way to keep most work local without sacrificing the harder cases. most hybrid setups still feel bolted on. when the model itself decides when it needs help, the whole system becomes quieter and more efficient. the architectures that last are the ones that only spend the expensive tokens when they actually need to.

Smart routing beats brute force—use the right model for the right task. Faster, cheaper, and just as effective. 🤖⚡

Routing based on confidence score is such a clean way to save cost.

The transition toward hybrid models represents a sophisticated balance between local autonomy and expansive capability. By training Gemma 4 to provide confidence scores, you are creating a form of technical self-awareness—enabling the system to recognize its own boundaries and seek external wisdom when necessary. It is a profound step toward a more efficient and harmonious AI architecture. 🌐✨ How do you envision this synergy between small and large models shaping the future of real-time, scalable AI deployment?

Attaching a probe layer directly to intermediate hidden states gives on-device models an incredibly clean self-awareness signal. Returning confidence as structured data beats parsing prose or guessing from token entropy every single time.

Smart approach, routing tasks based on confidence saves so much time.