I'm not well versed enough in Elixir to give a good answer.
The hard problems in crawling is mostly dealing with a very large and potentially highly mutable state. It's a concurrent problem, but a fairly easy one and a good use case for a traditional thread pool since there is latency is mostly a non-issue.
The hard problems in crawling is mostly dealing with a very large and potentially highly mutable state. It's a concurrent problem, but a fairly easy one and a good use case for a traditional thread pool since there is latency is mostly a non-issue.