KDB[1] is about 1000x faster than Spark[2], and is only about 600kb (and most of that is shared library dynamic linker stuff that makes interfacing with the rest of the OS easier). A big part of why it's fast is because it's small -- once you're inside cache memory everything gets faster.
That's the real cost of memory in 2017. So what did we gain for paying it?
KDB[1] is about 1000x faster than Spark[2], and is only about 600kb (and most of that is shared library dynamic linker stuff that makes interfacing with the rest of the OS easier). A big part of why it's fast is because it's small -- once you're inside cache memory everything gets faster.
That's the real cost of memory in 2017. So what did we gain for paying it?
[1]: https://news.ycombinator.com/item?id=13481824
[2]: http://tech.marksblogg.com/billion-nyc-taxi-rides-spark-2-1-...