If author is around: amazing work!!! Multimodal from scratch :)
I'm curious if you have the test clip you use, I got to the end and was like "wait....is that a good result! The words are completely different!"
Then I re-read a couple times scanning carefully for references to what the audio is.
This quote[^1] makes me think the sample is music, as that would explain why the end result is good -- it's trying to describe a sound file of just music, not a sound file that is a spoken word version of the "ground truth":
[^1] "For dataset, I chose MusicCaps. I did not see any convenient links to download processed/segmented audio files, so I wrote a small script to download the Youtube videos."
I'm curious if you have the test clip you use, I got to the end and was like "wait....is that a good result! The words are completely different!"
Then I re-read a couple times scanning carefully for references to what the audio is.
This quote[^1] makes me think the sample is music, as that would explain why the end result is good -- it's trying to describe a sound file of just music, not a sound file that is a spoken word version of the "ground truth":
[^1] "For dataset, I chose MusicCaps. I did not see any convenient links to download processed/segmented audio files, so I wrote a small script to download the Youtube videos."