One of the things that most pisses me off about numpy, pandas, and the whole scipy ecosystem is that everything is "immediate mode": all operations evaluate _instantly_ to new arrays or dataframes. There's no opportunity for any component to evaluate a whole expression tree and optimize it, e.g., by loop hoisting.
The right way to design a data analysis DSL is to do it the way Python's dask does it: build an operation graph and execute it as the last step. The trouble is that Dask doesn't get it right either because, as part of its graph formation, it computes sizes of operands, and computing operand sizes can involve huge amounts of computation, and so dask, in effect, is also an "immediate mode" system.
What I really want to see is something lazy that can do sane query planning and that can work within limited system resources. Maybe one day I'll open source the work I've done in this space. Query languages are infinitely nicer for analytics than data processing libraries.
The right way to design a data analysis DSL is to do it the way Python's dask does it: build an operation graph and execute it as the last step. The trouble is that Dask doesn't get it right either because, as part of its graph formation, it computes sizes of operands, and computing operand sizes can involve huge amounts of computation, and so dask, in effect, is also an "immediate mode" system.
What I really want to see is something lazy that can do sane query planning and that can work within limited system resources. Maybe one day I'll open source the work I've done in this space. Query languages are infinitely nicer for analytics than data processing libraries.