Try running `firefox MOZ_LOG=PlatformDecoderModule:5`, navigating to a video and checking that the decode is being done though VA-API. Depending on your GPU, you might have to use something like enhanced-264ify to force Youtube to use formats that are supported by hardware decode.
More info here:
https://wiki.archlinux.org/index.php/firefox#Hardware_video_...