Comment by PaulHoule

Cosine similarity works for this but the right way to think about it is as a classical ML classification problem with all the tools from

https://scikit-learn.org/stable/supervised_learning.html

For instance you will probably get better results with SVM or a not-so-deep perceptron or maybe random forest model than you will with cosine similarity. You can also probability calibrate such a model

https://scikit-learn.org/stable/modules/calibration.html

which is quite useful.