We built Talos - a full CNN inference engine running directly on silicon. Every multiply, buffer, and data path lives as real digital logic on the FPGA. This is what deep learning looks like when the model becomes hardware👇

this was one hell of a ride. building it alongside @krish_chhajer made it even more special. if you want the full story, check out: full hardware repo is open too. systemverilog, docs, .mif weights, testbenches, all of it:

the funny part is that the simulation worked in a couple weeks. getting it onto real silicon took months. that gap is the whole story. in simulation you fight logic which on hardware you fight physics. routing, timing closure, memory limits, clock budgets. things you don’t see until the chip says no.

the model is tiny on paper. 28×28 input - 4 conv kernels - pool - flatten - fully connected to 10 classes but on an fpga every multiply, every buffer, every bus has to exist physically. you don’t “allocate tensors”. you build them.

one moment that really felt like we started thinking in hardware: we reordered the layers. software: conv → relu → maxpool talos: conv → maxpool → relu initialize maxpool at zero and relu comes for free. same math, thousands of cycles saved.

our first real architecture was honestly naive. parallel everything. multiple cnn blocks, wide neuron buses, all firing at once. and the worst part? it simulated perfectly. then we tried to put it on the chip, but Quartus refused to fit it. not slow, not buggy, it was just physically impossible to place and route.

so we stepped back and rethought the whole thing. instead of brute forcing parallelism, we leaned into time. one cnn, one maxpool, four passes, all driven by a tiny finite state machine which lets you do the same math, way less silicon. that’s the real fpga tradeoff you don’t appreciate until you live it: you’re always trading time against area.

the next breakthrough was realizing we were still thinking too much like software. we were storing pooled activations, planning to feed them into the fc layer later. but on hardware, buffers are expensive. so we just… stopped storing them. each pooled value streamed straight into 10 parallel mac accumulators. no intermediate memory, no giant buses, just data flowing forward and getting absorbed immediately. that single decision shrunk the design more than anything else.

after that, memory became the final boss. 676 weights per neuron doesn’t sound like much until routing blows up. keeping them in logic was creating absolute chaos for the fitter. moving everything into m10k rom blocks with a shared address bus changed everything. utilization dropped, routing cleaned up, and for the first time the design actually fit.

this project completely changed how we see deep learning. in software, models look like graphs. in hardware, they become pipelines. latency stops being a metric and starts being geometry. you’re not writing code anymore, you’re shaping math into silicon. and once you build it this way, it’s obvious: under all the abstraction, modern ai is still macs and memory movement. talos just made that impossible to ignore.

working demo example:

@taalas_inc

what happens when the goats of ece come together. absolute heat🔥

🤝🤝

double the goats on this proj🐐🐐

ur the goat twin

sick project and the animations are fire

tysm bro

@suryasure05 This is 🔥🔥🔥

🔥🔥🔥 qualcomm stock is going to go up in may

you already know 📈

THE hardware guy

the talker

so goated

🪿

let’s goo!!

@xgawtham very cool

@xgawtham appreciate it bro

This is so fricking cool!!

thank uuu

That looks really cool 😎 FPGA implementations are next level.

🙏

oh shit this is sick

thanks evan!

sick!!!

🙏

insane!!!!!

🙌

always fun to team up with the one and only @luthiraabeykoon many more to come 🚀

take it to the moon

Oh wow, so you are using FPGAs... Amazing

Epic stuff!

very fun work! I worked at a startup in Sweden and we did very similar research!

thanks brooke! always nice to run into someone who’s been in the same trenches

How much different from talaas

hmmmmm sounds a bit too familiar hahhaha

extremely based, love seeing uoft winning

you already know!

Talos: goat-level work redefining on-chip inference

🙌🙏

Nova is very similar to this and took me a few days... Things are accelerating quickly lol

super cool!

🙌

Awesome project, DM if you’re interested in an internship working on even realer hardware (ASIC)

this is very interesting

So cool!! Love the use of FPGA!

that is pretty cool!

if you build software with physics and maths as a first class citizen using design by contract and hypothesis testing to enforce i shouldn't be that hard to just output whatever your coding in python or research world straight to SYCL/ intel one API and then you can target any fabric you want.

How do you manufacture the chips? It must be a lot of iteration

CNNs are so 2015

Does it also runs transformer networks?