Another benefit of specialized hardware, besides dispatch costs, is the cost of moving data around. As your chip gets wider you want more physical (rather than architectural) registers with more ports meaning super linear growth. And your bypass network also grows quadratically in transistor terms. And as your core gets bigger physically you lose more power moving data longer distances.
But in dedicated hardware you can just gang your operations into dataflows where the output of one stage feeds into the physically adjacent next stage with no need to make a trip through the register file or bypass network.
A lot of the benefit of hardware vector operations over scalar operations is in the dispatch cost, but most of the benefit from hardware matrix operations over hardware vector operations is from reduced data movement.
EDIT: Of course, the post is from 2012 back when nobody was doing hardware matrix multiplication so it's understandable.
But in dedicated hardware you can just gang your operations into dataflows where the output of one stage feeds into the physically adjacent next stage with no need to make a trip through the register file or bypass network.
A lot of the benefit of hardware vector operations over scalar operations is in the dispatch cost, but most of the benefit from hardware matrix operations over hardware vector operations is from reduced data movement.
EDIT: Of course, the post is from 2012 back when nobody was doing hardware matrix multiplication so it's understandable.