yathaid
Joined 7 years ago
Joined 7 years ago
You mean the ones encoded in the regression suite? I think your argument is valid in many medium-to-faang firms where the application is going to have encoded business logic that isn't explicitly tested for. If anything, projects like PG are the exact opposite: no business context and regression suites that test every possible scenario due to the accumulation of bug fixes and context over many years.
>> Run it in production for a year in a sufficiently large system and you might be somewhere.
How do you know every release of PG doesn't break in the setting of "a year in a sufficiently large system"?
Evidence: see the amount of nonsense in the postgres rewritten in Rust story - https://news.ycombinator.com/item?id=48841676 where there are two contradictory claims made:
"postgres is so stable I will never trust a rewrite."
"covering 100% of postgres regression suite doesn't guarantee you have replicated every behavior."
Folks who are okay with LLMs think the regression test suite is the spec and is the guarantee of stability. How else can it be? If you are depending on some behavior not covered in the regression tests, how do you know the next minor release won't break you?
Folks who are against seem to imagine a platonic ideal of PG which conveniently is the original PG implementation by tautological definition. So no rewrite can ever meet their bar.
You can already imagine Anthropic working with a bunch of shady brokers to "remedy" this situation.
Please do read the full PR, it is informative and funny!
>> My workflows involve opening and closing up to hundreds of terminal or tmux tabs a day.
What?!?
Without an explanation of what they author is calling out as flaws, it is hard to take this article seriously.
I know engineers I respect a ton who have gotten a bunch of productivity upgrades using "AI". My own learning curve has been to see Claude say "okay, these integration tests aren't working. Let me write unit tests instead" and go on when it wasn't able to fix a jest issue.
__need__ is doing a lot of work here. There is no forcing function to get OEMs to do this ASAP: 1) the market doesn't really care that much 2) there are no regulations around this (and even if they were, can you immediately recall a tech exec going to jail for breaking the law ... )
good is doing a lot of heavy lifting here. The point of TLA+/Pluscal is to have a proof of the soundness of the design.
This highlights one of my personal bugbears. People have a mental barrier when it comes to recurring, low-cost payments; even though the net sum is small in comparison to other things that they wouldn't think twice to pay for.
A $5 latte every workday comes to (260 * 5) $1300 annually. Obsidian sync is $96. Why would you not pay this amount for a tool you use everyday?
They are not informal AFAIK - https://github.com/PGPerfFarm. A lot of these heavy duty OS projects have explicit Perf testing/nighlites (see Lucene's at https://benchmarks.mikemccandless.com/)
If you are saying the PG regression suite doesn't cover these perf tests - that is fair. I consider perf to be part of the "regression framework" generally.
>> How do you know your new tests are right? That the new behavior is right?
This is a more generic question in the LLM world. Reliable verifiers is what drives LLM loops. If you don't have these, you have no idea if you are actually making progress or you just generated code that returns 42 for every question. You needs something to actually ground the LLM output against. For existing features, the reliable verification is the existing regression/perf suites. For new features, the regression/perf suites should expand to fit. There are ways to go at this that range from ad-hoc (line/branch coverage) to fully formalized (verification-aware languages like Dafny).