mozillas
Joined 12 years ago
Joined 12 years ago
Yahoo handles it well, I think. They will send an email to your recovery email address to inform you that unless you log into your account in the next 30 days or so, it will be closed.
That's what I did, but I might have had different requirements. If you don't have a lot to crawl and you don't have to do it very often(once a week or less), you can probably space out the requests enough so that the server doesn't feel it. It helps a lot if you use some caching as well for the website itself in this case. I think it depends a lot on the requirements of the project. But using two machines is safer I think, although it might complicate things a bit.
Keep in mind that there's probably better technical advice out there than mine. I'm a hobbyist developer.
I also remember about a side project that involves by crawling HN and displaying the most mentioned books in the comments. I think the author was making a bit of money by having a link to Amazon for each book(and using an affiliate code). You could do something similar but for popular Wikipedia articles for example. Or you could use Reddit as a source and instead of searching for popular books to search for sneakers or music.
Of course, I don't know if these ideas are too simple or too complicated, interesting or not for you and the person who will give you a grade.
If things go well, you can keep the website or sell it for more than you bought it. If the revenue doesn't improve or becomes smaller you can sell the website and get some of the money back. Either way, you'll learn some new things.
Worked OK for me with the default context size. 2048, like you see in most examples was too slow for my taste.