While that most definitely is a consideration, AVIF lacks many features and much of the functionality offered by JPEG XL. An investment in JPEG XL, while only being beneficial for one task like you say, would have huge benefits given just how many images we see and process on the web.
AVIF certainly gets the hardware support advantage, but it fails in other regards where JPEG XL shines. Looking at it either way, JPEG XL is far from slow, and I'd argue that the other benefits outweigh that single shortcoming. Bandwidth should also be treated as a consideration, as you mentioned, and JPEG XL generally leads to smaller file sizes.
Realistically, this boils down to AVIF being largely worse than JPEG XL, with the exception of performance. Performance that could be improved for JPEG XL should hardware vendors choose to provide better support.
AVIF certainly gets the hardware support advantage, but it fails in other regards where JPEG XL shines. Looking at it either way, JPEG XL is far from slow, and I'd argue that the other benefits outweigh that single shortcoming. Bandwidth should also be treated as a consideration, as you mentioned, and JPEG XL generally leads to smaller file sizes.
Realistically, this boils down to AVIF being largely worse than JPEG XL, with the exception of performance. Performance that could be improved for JPEG XL should hardware vendors choose to provide better support.