IMO the next big shift in AI hardware is when some of the upcoming logic-compatible memory types mature. Inference inherently likes many small local pools of memory, with their own interface each. This is impractical with currently available memory other than sram.
IMO the next big shift in AI hardware is when some of the upcoming logic-compatible memory types mature. Inference inherently likes many small local pools of memory, with their own interface each. This is impractical with currently available memory other than sram.