Here is a huge trick to cut down on hallucinations: I have an agent that processes phone receipts using GPT-4o. Whenever I go to a restaurant or pay for something, I take a photo of the receipt and upload it to Google Drive. The agent goes through the folder... and extracts the following information from each receipt: • Name of the place • Category of the place • Amount • Tax amount • Tip amount Each field requires an individual prompt. The result is a JSON object that I export to a separate folder. This is awesome, but there's a problem: The agent occasionally makes mistakes when interpreting a receipt; unfortunately, I couldn't detect them as they happened. Here is the trick: I'm now using a challenger model (o3-mini) to check the work of GPT-4o. This increases the cost of processing receipts but reduces the mistakes to almost nothing. The best part: 1. Whenever the challenger disagrees with the main model, I produce detailed information about the problem to improve the prompts and make the extraction more robust. 2. It's very easy to turn this process into a Human-in-the-Loop workflow where I get notifications of every disagreement and an opportunity to settle them. I recorded the attached video to show you how this works using Unstract.show more

Unstract is open-source, and it takes 3 minutes to run locally. They partnered with me on this post. It's pretty popular: they process over 7 million pages monthly and turn them into actionable structured data. Here is their GitHub repository:

Here is the video on YouTube, in case you want to watch it there:

Discover Automate your document signing process in minutes. Try now!

Does the cost balance against taking 10 seconds to manually punch these values into a basic form ?

A company processing the receipts from its 1000 employees would like a word with you.

What’s the advantage over using o3-mini alone with no 4o?

Better solution is to abolish taxation this is 💯 a waste of human and AI potential... and they blow the money on shit that helps nobody anyways

Add dspy to the loop. Train it on good receipts dspy will iteratively rewrite the prompt to optimise towards a metric

Alternatively you could try my way to get an accurate output - for free. Text only. But I might look at images when I can get v0-dev to do it right.

Yeah I internally call this an “observer.” As Humans we make mistakes but can self-correct "hallucinations" on the fly. An on-the-fly observer in an Agent is slow/expensive, so it’s more efficient to run it at the end, using even a smaller model can be effective