I guess I'm wondering what the point of pandas is here. And maybe in general? We hired a data engineer awhile ago and he kept complaining that the "python kernel" was crashing and he needed bigger instances to run his pandas scripts. I took a look and realized he was trying to pull the entire database of a nearly a billion rows into python to do some trivial joins, groupings, and filtering. I'm thinking, what are you doing man.
I think this is pretty common in the pandas world right
I think this is pretty common in the pandas world right