ag2718
Joined 4 months ago
Karma
178
Submissions
15
Member since
Jun 9
Recent submissions
Comment · 4 months ago
This is a simplification in the blog post: each activation doesn't map one-to-one onto a physical FPGA LUT primitive, but is instead represented as a "logical LUT" (L-LUT) that Vivado synthesizes into distributed RAM. These L-LUTs map very naturally to FPGA fabric (as compared to multiply-accumulate operations), making them a very useful implementation-level abstraction.
Comment · 4 months ago
Our end-to-end implementation can be found here! https://github.com/Duchstf/KANELE
Comment · 4 months ago
Some very cool applications of small models! It seems that this scale of models tends to be sufficient when doing simpler classification, anomaly detection, signal processing, etc. as compared to generative modeling (where larger models are usually necessary).
Comment · 4 months ago
That is a really cool application of FPGA-based machine learning that I would not have thought of :)
Comment · 4 months ago
This is definitely true: one could imagine a model with a mix of the two layers or a simple linear / MLP-like kernel doing "preprocessing" before KAN layers. Other work that explores task performances for KANs and MLPs generally finds KANs are worse at non-symbolic tasks, but it would be interesting to see if hybrid architectures could improve on this failure mode.
Comment · 4 months ago
Thank you :)
Comment · 4 months ago
Ah I see, that's an interesting point about higher depth potentially having other benefits. For our work on smaller models (e.g. generally <5 layers), this might not have been as relevant but I would definitely be interested to see implications for much deeper networks. As to your point about KANs performing better or worse depending on the specific task, we definitely did notice this to some extent (symbolic tasks were the best, non-symbolic tasks such as image recognition were the worst).
Comment · 4 months ago
A key benefit of KANs is expressivity, as each layer is significantly more expressive than an MLP layer. This can be seen in our benchmarks: KAN networks need fewer layers than MLPs to match or beat their performance, even in software.
However, on GPUs, KAN implementations are far less efficient than MLPs: since B-spline locality is hard to exploit and lookup operations aren't as efficient. This is your original point about MLPs training and running faster on modern architectures: each KAN layer is more expressive, but its poor hardware efficiency makes it a net negative (at least for current approaches).
On FPGAs, LUT lookups are cheap, so KANs' expressive layers map to very hardware-efficient implementations, and the resulting networks are thus much more compact and efficient than equivalent MLPs.
On your second point: low precision is certainly viable for both inference and learning (as shown in our work), and quantization can even have a mild regularizing effect. However, task performance generally worsens with lower precision (here and across the literature): the use of low precision is fundamentally a result of the efficiency-performance tradeoff.
Comment · 4 months ago
There is definitely a precision-performance tradeoff to consider. We explored this through ablation studies on bitwidth precision / resource usage in our work (Figure 6a in https://arxiv.org/pdf/2512.12850, Figure 4 in https://arxiv.org/pdf/2602.02056). Further exploration into the mechanics here would definitely be useful.
Regarding your point that "90% of the benefit of KANs can be gained from a small variety of function shapes": even within the B-spline basis, the shapes are quite uniform. Much of the actual benefit of scaling up the basis size comes from learning more complex, piecewise-polynomial activation functions. Scaling up the number of basis functions (i.e. more granular intervals) also increases locality and allows the activation function's value across different parts of the domain to be learned semi-independently. (There obviously is a tradeoff here with overfitting.)
The number of basis functions (G+S) is largely what determines how expressive the activation is, as it relates to your point: "you could have a representation that scales from a standard relu perceptron though KANs to something with weighted inputs and fancy weighted activation functions."
Comment · 4 months ago
You're correct that this work is not very applicable for LLMs and that the focus here is primarily on latency.
Comment · 4 months ago
Yes, definitely: this type of work is applicable in domains where software run on general-purpose processors cannot meet latency or power requirements.
Comment · 4 months ago
Hmm the post is still up for me?
Comment · 4 months ago
Yes, this work is focused on accelerating very small models, typically for real-time systems that require extremely low power or low latency.
One primary application of this work is in high-energy physics (https://home.cern/smarter-decisions-at-the-speed-of-collisio...). Ultrafast and real-time learning is also very applicable for problems in quantum computing, plasma control, etc. (https://arxiv.org/pdf/2602.02005).