tehwhale

You sound like Elon. LIDAR is far superior to radar in a lot of cases. The best situation would be for Tesla's to use a mix of both radar and lidar.

Our company has extremely deeply tinted and reflective exterior windows that even at night (outside) with lights inside, you can barely see in if your face is 1" away from the glass.

While this looks good, I don't think it's feasible for a web crawler in most cases. Crawlers want to crawl a ton of URLs and it would have to make a request to your service for each and every URL.

What's the plan here? Check for a sitemap.xml (which generally only contains crawlable URLs anyway) or crawl the index and look for all links and send a request to your service for every URL before crawling it?

I personally think it would be better suited as a library where you can pass it a robots.txt and it'll let you know if you can crawl a URL based on that.