Golang or Python for the large scale distributed crawler?
To deal better with concurrency which one would you opt for, as for the task queues, I have checked golang has "Machinery" library and for python its celery (we are currently using this). The bottleneck we are seeing is celery is very hard to debug for the issues in case of the gevent.
Sometimes it works fine with 100 threads, and sometimes it completely stalls and become idle without giving any errors etc.
I was thinking about this approach:
- Golang to take care of the requests/HTTP work and store them in the DB.
- Python for the NLP tasks etc.
Sometimes it works fine with 100 threads, and sometimes it completely stalls and become idle without giving any errors etc.
I was thinking about this approach:
- Golang to take care of the requests/HTTP work and store them in the DB. - Python for the NLP tasks etc.
Any insights/suggestion would be helpful.