I’m thrilled to present the KAN Convolutional Layer, a promising neural architecture for image processing. Last week KANs came out, as an alternative to the MLP. We extended this idea to Convolutional Layers, creating the KAN Convolution. Join me in this thread to more about it🧵

how about KANvolution?😂

HAHAHAHAHAH

All the code is available in the following repository: Co Authored with @JackSpolski @SantiPourteau @antotepsich Lets dive in what we've created 👇

What is a KAN? A KAN connection applies a function defined as a learned B Spline, plus a Residual activation function b(x), and all this times a learnable parameter w. The whole node value is the sum of the different ’s that enter the node.

What is a KAN Convolution? KAN Convolutions are very similar to convolutions, we apply a Learnable Non Linear function in each edge, and then add them up. The kernel of the KAN Convolution is equivalent to a KAN Linear Layer of 4 inputs and 1 output neuron.

Results In our first evaluations with MNIST Dataset, we got very tight results between KAN vs Classic convolutions with a MLP at the end, with the last one getting 0.06 more accuracy. But the KAN Convs with a KAN got only 0.04 less accuracy with almost half the parameters!

Work in progress In the coming days and weeks we will be thoroughly tuning the hyperparameters of the models we use in the comparison. In addition, we will evaluate on different and more complex datasets to extract definitive conclussions.

And that's all for now, if you liked it give the repo a star⭐, help spreading the work with a RT and follow me for more AI content!

@Ethan_smith_20 a bit of what was discussed last time, but I think the jump needs to be better directed, i.e., I don't think splines are best suited here (but I am of course willing to shut up and let them cook)

Amazing!

Muy buen trabajo!

Muchas gracias Pato!

Felicitaciones Alex y co!

Muchas gracias!

based af

training speed? is it parallelizable ?

@LodestoneE621 We will update in the following days with training speed evaluations, but if you checkout the notebook in the repo you should get a clue. The current problem with KANs is that they depend on non linear functions that you cant parallelize in GPU ATM

Amazing!!

@TechRadar9

Nice :)

It sounds like a prophecy but I'm looking forward to KAN proving it will replace MLP in all Vision encoder architectures and Language encoders (or it proves itself to be just an illusion from expectations :)))) I wonder if you want to experiment on transformer architectures?

@DeeperThrill Code in next tweet

@readwise save

Nice job! What about the training time of the KAN convolutions vs the classic ones?

@threadreaderapp unroll

!!! this is SO COOL to already see KANs getting put to work Cant wait to see what else comes 🍿

This KAN Convolution idea is fascinating! Given KANs' attention to function complexity, I'm curious about how you're quantifying the effective receptive field or spatial feature hierarchy compared to traditional CNNs. 🤔

Inference speed?

Is it possible to adapt it to the following ccn model .

Very interesting work! Does some version of the convolution theorem still apply to KAN convolutions? As in, is there a dual of KAN convolution in the Fourier domain?

Not sure about it, my gut tells me that the theorem as is doesnt stand, but is work to do!

Yeah I'm thinking the non-linearity means there is no Fourier transform dual but maybe there is a Volterra/Wiener series transform

Your thread is gaining traction! #TopUnroll 🙏🏼@QTMDots for 🥇unroll

Have you thought about the possibility to do equivariant convolutions as well? Efficiency is especially important in that field.

Sorry, but, what do you mean when you say Equivariant Convolutions?

I mean as in equivariant and geometric deep learning. Standard convolutions are equivariant to translations, but in some domains (eg medical image analysis) the objective is also for example rotation invariant.

Here’s an example of such work:

What's the point ? Novel architectures should be easier to compute on current machines This is just another alternate formulation of a parameterized network, it can't bear any advantage in and of itself.

That is so freakin awesome. Thanks for working on this and in the open

Really cool, think this could scale reasonably to more layers?

honestly i’m struggling to see how KANs are in any way superior to classical MLPs. Just the non-parallelized architecture should indicate that it isn’t likely to overtake MLPs for ML anytime soon.

Great stuff! I wonder if the spline enables better performance in generative tasks, due to the introduction of non linearity in the kernel.

Why do you use a SiLU activation on the input x before passing it through the residual connection? Referring to

Awsome! What about explainability? Is there any added value on this regard?

Thanks, one could visualize the function that each pixel of the kernel learns, but IMO it is very hard to conclude something out of that. But we are working on it. In one of my figures you can see the outputs of conv layers and it Seems that they learn the shape.

Lovely! All the best