There are lots of efforts to make pandas faster. Dask, pandarallel, and this.
They all stay at your computer and in your RAM though.
I wonder, what is the benefit of using this compared to a truly distributed engine like spark? If your job barely scales maybe that is what you need rather than maximizing your cores or memory usage?
(Note, I have not worked with spark, so it is an honest question.)
They all stay at your computer and in your RAM though.
I wonder, what is the benefit of using this compared to a truly distributed engine like spark? If your job barely scales maybe that is what you need rather than maximizing your cores or memory usage?
(Note, I have not worked with spark, so it is an honest question.)