323454

So far we've been focused on edge devices like the Zynq, Artix and Zynq Ultrascale+ families. Tensil certainly works on larger devices but it's not as optimized there as we'd like it. If that's interesting to you, I'd love to talk and understand your use case in more depth.

The Intel FPGA side is interesting, as you say there are fewer projects targeting their technologies for ML use cases. We haven't tested support for their boards yet, but there is nothing in our generated RTL that is exclusive to Xilinx. The only thing we'd need to add is new drivers for their platforms.

Sorry if this was unclear - in a datacenter use case you are right, but for an edge deployment, you will usually need to quantize, prune or compress your ML model to get it working as fast as you'd like on a sufficiently small CPU/GPU. Compared with running your ML model unchanged on those platforms, Tensil can run with the performance ranges listed above. You can also quantize and use Tensil too!

FINN is a very cool project, but usually requires big changes to your model in order to work, e.g. quantizing down to 1 or 2 bit weights. It also works best on large FPGAs which are unsuitable for edge deployments. Tensil works out of the box with any model (no need to quantize / compress) and on small edge FPGAs.

Cheers, and great question! FPGAs are pretty amazing devices, but one thing that's been holding them back is how difficult they have been to work with. Typically to actually make use of an FPGA you'd need to have an FPGA expert and an embedded software engineer on your team, along with all the requisite tools and materials.

That has started to change dramatically in the last decade, with open source FPGA toolchains like yosys, runtimes like the PYNQ framework and RTL generator tools like Tensil being developed. When you put these things together, working with FPGAs starts to become as easy as using any other compute platform. For that reason, I think there are lots of applications involving FPGAs that will soon be invented to take advantage of this trend. One could speculate that the reason Intel and AMD are buying up FPGA vendors is because they see the potential there.

As far as head-to-head comparisons go, as long as you're running the workload it was designed for in the environment it was designed for, an ASIC will always be the best possible perf per watt. The question is what happens when you go outside those bounds. Can you take your model, swap out a layer, and have it run just as fast on your Coral or NPU? Probably not, at least right now. But with Tensil, you can re-run your architecture search to find the best accelerator, and take advantage of it right away.

I don't have hard numbers at hand, but I'd estimate something like an order of magnitude improvement for using DSP for multiplication vs not. If they're available on the fabric, you'll definitely want to use them! If this is an experiment you want to run, I'd be very happy to help you figure out how to do it.

Thanks for pointing this out! The UP5K does look promising.

Thanks! The general answer is that it depends on your model and on which FPGA platform we're talking about, but in a head-to-head benchmark test you'll find results in the ballpark of 2-10x CPU and 0.5-2x GPU. As you point out, the power and cost are big differentiators. The other thing to consider is (as another commenter mentioned) that usually inference on CPU or GPU will require you to do some model quantization or compression, which can degrade model accuracy. Tensil can give you a way around that dilemma, so that you can have great performance without sacrificing accuracy.

Just saw your edit re: model compression. One thing that Tensil can do is help you avoid the need to quantize or compress your model entirely! For example, we've found that using a 16-bit fixed point numeric data type preserves almost all the model accuracy while not sacrificing performance thanks to the huge amount of parallelism available on FPGA.

The broader point is that Tensil is extremely flexible, so you can try out lots of different accelerator configurations to find the one that works best for your ML model. Think of it as optimizing the hardware first, then the software if needed.

We're actually working on a tool to manage and automate this hardware architecture search - watch this space!

Thank you for the kind words! Just to clarify, the core technology here is free and open source, anyone can use it right now for free. We do have commercialization plans in addition - we may explore things like additional paid features for enterprise use or paid tiers of extra support.

Regarding LSTMs, yes. We're aiming to support all machine learning model architectures: do you have any particular models you're interested in that we should be prototyping with?

For CGRAs, we don't have any immediate plans to explicitly support them. What kind of use case do you have in mind? Generally, any platform that can implement a blob of generated RTL should be something we can work with quite easily.

You absolutely can use it in a data centre. You can even tape out an ASIC using these designs! Currently we've done most of our prototyping with edge FPGA platforms but if you want to try other platforms we'd love to help you get started. You can email me at [email protected] or use the contact methods on the website.

Great question - TVM / OctoML are a great option if you have an off-the-shelf ML model and off-the-shelf hardware. Tensil is different in that you can actually customize the accelerator hardware itself, allowing you to get the best trade-off of performance / accuracy / power usage / cost given your particular ML workload. This is especially useful if you want to avoid degrading the accuracy of your models (e.g. through quantization) to achieve performance targets.