Comment by wdroz

I switched from pandas to polars[0] . I'm happy with the performance gain. You still need to use Spark/dask on "big" data, but polars give you more room before reaching to these solutions.

[0] -- https://www.pola.rs/