carlineng
Joined 11 years ago
Karma
656
Submissions
143
Member since
Mar 12
Recent submissions
Comment · 1 year ago
This is really a question of economics. The biggest organizations with the most ability to hire engineers have need for technologies that can solve their existing problems in incremental ways, and thus we end up with horrible technologies like Hadoop and Iceberg. They end up hiring talented engineers to work on niche problems, and a lot of the technical discourse ends up revolving around technologies that don't apply to the majority of organizations, but still cause FOMO amongst them. I, for one, am extremely happy to see technologies like DuckDB come along to serve the long tail.
Comment · 1 year ago
Not yet, but I believe the DataCouncil staff recorded it and will post it to their YouTube channel sometime in the next few weeks: https://www.youtube.com/@DataCouncil/videos
Comment · 1 year ago
I just watched the author of this feature and blog post give a talk at the DataCouncil conference in Oakland, and it is obvious what a huge amount of craft, ingenuity, and care went into building it. Congratulations to Hamilton and the MotherDuck team for an awesome launch!
Comment · 1 year ago
I agree 100% that this needs to be more of a thing. For data engineers building data pipelines, queries are like functions, and table schemas are like types. There needs to be a way to write a query that runs on an abstract interface, rather than an actual table. To do this, most folks rely on string templating in Python or Jinja, which makes the development process really cumbersome. As a result, most teams end up in scenarios where data pipelines are always a big mess of spaghetti SQL, or they are stuck maintaining complex frameworks that abstract away common logic, but are inscrutable to the average user.
I wrote a longer blog post about this recently: https://carlineng.com/?postid=holy-grail-data-engineering#bl...
Comment · 2 years ago
The argument is more than that -- namely that in addition to having sophisticated AI, Google also controls the OS (Android) and hardware (Pixel). Being able to integrate best-in-class AI at every level of the stack is a tremendous advantage. OpenAI can't do this because they don't control the OS, and will always need to go through an app. Apple can play since they control OS and hardware, but at the moment they appear pretty far behind in the AI aspect.
[ my public key: https://keybase.io/carlineng; my proof: https://keybase.io/carlineng/sigs/SxKhzboFFXEbC-x-9-NGEtke2DKLBS2wJCzYxL2vPV8 ]