Comment by famouswaffles

Sure there's a better way. https://google-research.github.io/seanet/audiopalm/examples/

There's no reason autoregressive LMs can't be used to model audio data.