Do you worry that you risk introducing bias into your interview process with this sort of unstructured questioning? There is quite a bit of research [1] demonstrating that structured and standardized interviews across candidates are one of the most crucial ways of preventing various types of bias, conscious or not.
[1] Here's a useful summary article: https://hbr.org/2016/04/how-to-take-the-bias-out-of-intervie...