> Although the GPU performs inference faster than the FPGA,
one of the primary bottlenecks of deep learning inference is
memory bandwidth and the availability of on-chip memory
(Balasubramanian et al., 2021). A RTX 3090 has 24GB
VRAM running at 1219 MHz with a base core clock of 1395
MHz (TechPowerUp, 2024). In comparison, a VU9P FPGA
has 345.9 MB of combined on-chip BRAM and URAM,
running at a much slower clock speed of around 200-300
MHz depending on the module; however, with much lower
clock speeds, the FPGA is able to achieve better efficiency
on power and energy consumption, as shown below.
So as far as I can understand, the biggest "bottleneck"/limiting factor with using FPGAs for LLMs is the available memory -- with current large models exceeding 40 GiB in parameter size, GPUs and TPUs with DRAM look like the only way to go forward for the months to come ... Thoughts?
So as far as I can understand, the biggest "bottleneck"/limiting factor with using FPGAs for LLMs is the available memory -- with current large models exceeding 40 GiB in parameter size, GPUs and TPUs with DRAM look like the only way to go forward for the months to come ... Thoughts?