aduffy
Joined 11 years ago
Karma
1,918
Submissions
242
Member since
May 13
do or do not, there is no try
View on Hacker NewsRecent submissions
Comment · 1 month ago
Guy who still writes Quora answers
Comment · 1 month ago
Maybe I'm getting old but man, motel rooms run $80 now?
Comment · 2 months ago
“…and the naming convention is load bearing”
close tab
Comment · 2 months ago
This was true six months ago but today models are indeed excellent at coding especially if you give them guidance.
Comment · 2 months ago
It’s been wild watching smart people have their brain cooked on a constant diet of X and LLMs in the past two years.
Models are excellent at code but they are not people managers. My guess this will ultimately lead a bunch of companies to revert to average returns or fold.
Cal Newport was right.
Comment · 2 months ago
idk why this is getting downvoted, I also got this sense, plugged it into Pangram and indeed, 80% AI-written score.
I guess that's fine, but after awhile I get a spidey-sense reading something that feels like a Claude session.
Comment · 4 months ago
That is fair.
FWIW I think if you are just doing pure analytics and nothing else, Parquet will probably continue to do the job for you just fine, and you don't need to touch your workloads at all.
These new formats I think will find a niche where people aren't just running Spark jobs, but doing lots of systems building over large tables. If you're building a PB-scale data warehouse, you care a lot about the file format b/c it is a big factor in your performance curve, and you're willing to ship new experimental codecs in response to new datatypes you want to support that the system wasn't originally designed for, or you want to use a newly invented compressor.
Comment · 4 months ago
> Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up.
This is really more of an expectation that has been put on file formats by the query engines. Spark/Datafusion/DuckDB wouldn't really know what to do with a multi-table file.
> Parquet is unfortunately very good just by virtue of being first, and so widely supported
IMO that is not how technology works. It is great that Parquet is so good at a lot of things, but that does not mean just because it came first that it deserves to be the only analytic file format forever.
> Its main result seems to be improved random access which, although certainly welcome, is not the point of columnar storage, as columnar storage was invented to exchange random access for something else: fast analytics
Fast analytics, as well as newer ML-shaped workloads, are inherently mix of batch scans and random access.
Some of the authors of F3 previously authored another paper that goes into the details of the shortcomings of Parquet
https://www.vldb.org/pvldb/vol17/p148-zeng.pdf
All of the newer formats that popped up recently (Vortex, Lance, F3 now) have been working on solving the problems outlined in that paper.
Lance has some interesting ideas, Vortex focuses on extensibility and performance by replacing all of Parquet's black-box encoders with fully transparent encodings. This solves the tradeoff between bulk and element decoding, allowing you to have efficient full scans and really fast random access.
E.g. Langchain recently rebuilt a system that used to be all Parquet files to use Vortex and saw a massive speedup, which they talk about more here: https://www.langchain.com/blog/introducing-smithdb
Disclaimer: I work on Vortex, so a lot of these questions about "what is the point of building a new format" are things that I have grappled with myself.
Comment · 4 months ago
> “I never got my driver's license, and I rely on Waymo to commute to an office every day," said Sarah Paige Roland, a Waymo rider in Phoenix. "I get privacy, time back, a safe ride, and I'm not obligated to talk to someone that I don't want to talk to."
I recognize that this is a luxury product but I kind of laughed out loud at this testimonial. The amount of privilege you need to have to grow up and live in *Arizona* without ever learning how to drive is insane.
Comment · 5 months ago
Move fast and meltdown
Comment · 5 months ago
I think I'm missing the excitement. This is an artist's rendering of a supposed massive orb in the sky? I am more impressed by the actual UAV footage that has been released previously.
Comment · 6 months ago
On top of this whole thing just being ridiculous, $50mm is also just not a very impressive amount of money to build out an AI data center.