Comment by msgodel

It would be nice if this could get standardized http headers so bots could still use sites but they effectively pay for use. That seems like the best of all possible worlds to me, the whole point of HTML is that robots can read it, otherwise we'd just be emailing eachother pdfs.