If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing.
I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the chance, but I’m slowly outsourcing critical pieces to C++ and Rust libraries. Multiprocessing has been a source of subtle portability errors as it has changed over different minor versions.
I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the chance, but I’m slowly outsourcing critical pieces to C++ and Rust libraries. Multiprocessing has been a source of subtle portability errors as it has changed over different minor versions.