Comment by chsasank

One reason to use beautiful soup is that selenium is slow. You need to open the whole webpage including images, css etc. With requests/beautiful soup you can just parse the collected URLs very fast.