marsupialtail_2
Joined 4 years ago
Joined 4 years ago
Incidentally the most effectively way to stall a project according to the CIA is to have a huge guiding committee with clearly diverging interests.
Redis will win because it's focused on its users. It's competitors will lose. Like OpenSearch, like OpenCL etc.
I may sound like a rich apologist, but please believe me when I say it is harder to spend 10 hrs a day cranking on a risky startup if you know you can be clubbing with daddy's money
You don't have to use the system I am building, but it's worth thinking about that design.
I work on Quokka (https://github.com/marsupialtail/quokka). I support Iceberg reads. Recently we are adding SQL support from just parsing the DuckDB logical plan, though that is very challenging as well.
The Python world lacks a standard for a plug and play SQL query optimizer. Apache Calcite is good for the JVM world, but not great if you are trying to cut out the JVM.
I have users demanding Iceberg writes and Hudi reads/writes. I don't know what to tell them, since I don't have the resources to add a reader/writer myself for those projects.
Hopefully as DuckDB becomes more popular we will see Python bindings for these popular data lake formats this year.
I am not focused on complex data types though.