I used Gemma4 + Falcon Perception from this mlx-vlm release to build a grounded reasoning agent runs fully local on M3 the idea: VLMs are great at reasoning but not great at measuring. Falcon Perception is great at segmentation but cant reason. so you loop them: Gemma4 decides what... to look for, FP segments it and returns pixel-accurate coordinates, Gemma4 reasons on the numbers ask "is the blue player offside?" → it grounds the players, finds the second-to-last defender, compares centroid positions, applies the rule. check the video for some examples Prince Canuma I can submit a PR with this demo if you wantshow more

lol I explored the same concept 2.5 years ago

ah niiice, you are always ealier @skalskip92 🤣 SoM is the foundation yeah marks, numbered regions, all of that ... tool calling + open vocab segmentation with Falcon Perception made it possible to take it further is the agentic loop: instead of one-shot "here's the marked image, figure it out", Gemma4 decides the tool calls, decides what to segment, gets exact pixel coordinates back (centroid_norm, area_fraction, bbox_norm), and a fixed registry of deterministic spatial ops handles all the maths so visual reasoning is grounded, check PR

why FP over SAM3? both offer language guided segmentation

haha fair questions, FP does open vocab + referring expression, so the prompts for the agent are more flexible than SAM3's prompting. it can pass things like "the player on the right" or "the sign with whatever written on it" and FP handles it, lesser tool calls overall ... check the paper and PBench please table 7 where SAM3 is restricted to levels 0 and 1, whereas FP can go up to level-4 (Relationships & inter- actions), check table 1 for the levels definition

ooookey! I’m exploring FP this week. really cool.

This is so cool! That would be awesome, please do 👌🏽

hey @alexinexxx look at this combo

Love it! I did something similar! I kinda fell in love with Falcon Perception model! Tiny and mighty!

WOW 🤩 great job!

For us inexperienced, how do you loop the two models together? Custom script i guess? Or something of an available product thingy? I admit i am mentally stuck mostly to a gui. But this seems like a script workflow.

Awesome! Does mlx vlm support olmo point? I wonder if it could pick out the “highest flying bird” in one step instead of the current multi-pass approach.

Have you tried other small models like Grounded Dino and

Have you tried other small models like Grounded Dino and Moondream?

This is cool 👌🏻

that's a slick combo, using gemma4 as the brains and falcon perception as the eyes

Could this help browser use for agents or is this overkill?

the reasoning plus segmentation loop is the real sauce here. I'm curious how stable it stays when Gemma4 has to revisit the same region a few times on device

Very cool, I think you can batch generate do every masked object eg. An agent per car