I think this is a nice approach. You may even be able to take it further; if you're training end to end based on users' queries, you can probably have the query and image representations in the same space and use a simple similarity measure in place of the tiny neural net (something like OpenAI's CLIP model).
The tricky part will be scaling it -- not just for speed, but keeping the index size down. Also, you'll need to already have some version of image search to collect the training data.
The tricky part will be scaling it -- not just for speed, but keeping the index size down. Also, you'll need to already have some version of image search to collect the training data.