bugfix-66
Joined 4 years ago
Karma
501
Submissions
264
Member since
Sep 24
A debugging game for hackers: https://BUGFIX-66.com
View on Hacker NewsRecent submissions
Comment · 3 years ago
Comment · 3 years ago
You don't test on the training set.
Everybody in ML (e.g., me) knows that.
But that's what we're seeing.
I'm saying: "Here's a test outside the training set."
Comment · 3 years ago
That's right. There was no snark intended.
Comment · 3 years ago
I seriously doubt this appeals to the modern Hacker News crowd, but it might appeal to a couple of you:
Published in October 2022, The Art of Computer Programming: Volume 4B is dazzling. It's like a guide to expressing (in non-obvious ways) all kinds of problems as some variation on Exact Cover (or Boolean Satisfiability) and using wicked tight little general-purpose backtracking solvers to solve them.
If you want a peek, watch Knuth's 2018 lecture on Dancing Links Exact Cover:
Comment · 3 years ago
The training set is full of code for these problems.
Put your query into Google and see how many thousands of answers appear.
Your tests are in the training set.
Comment · 3 years ago
Look at how many times that program appears when you paste your question into Google.
The language model was trained on it.
Comment · 3 years ago
Named after the two companies that promoted it.
Comment · 3 years ago
It was trained on numerous Leetcode solutions (question/answer), which you can find on GitHub. People are proud of their Leetcode efforts: they post their solutions, as if it is an achievement.
Now tell me, how many of the BUGFIX-66 problems can it solve?
Copilot can solve a few of the simple ones at the beginning (simple matrix multiplication, simple radix sort, etc., that appear often in the training data, and some of the harder ones whose solution appears on GitHub, e.g., the uncorrected prediction/correction compressor/decompressor whose solutions were front-page on Hacker News).
But as you go down the list, into material outside the training set, the language model won't be able to solve it.
Try it. Paste the tile, the introductory text, and the unfixed code, and see whether the omnipotent AI can solve these (quite straightforward) problems. For how many does it need the hint?
Comment · 3 years ago
Radix sort is also very simple, e.g., https://bugfix-66.com/834f0677c85b23c0bf1047d3654ab7c27ff054...
And djb's vectorized sorting networks are pretty great: https://sorting.cr.yp.to/
Comment · 3 years ago
Now do it for an algorithm+explanation the training set doesn't contain a million copies of, please.
For example, substitute "The Burrows-Wheeler Inverse Transform" for "bubble sort".
Comment · 3 years ago
Look at Section 8 and Section 9 here:
http://tug.ctan.org/info/tex-nutshell/tex-nutshell.pdf
How is ChatGPT's answer wrong? The regex is matching square brackets instead of the correct OPTIONAL curly braces. Unseparated arguments aren't handled. The regex for the id is also wrong. The leading slash is correct, but other than that, the whole fucking thing is wrong!
And of course it can't handle macro calls within the macro arguments (nesting).
It's just an absolute clusterfuck, from top to bottom. The chatbot gave its user a useless piece of garbage, and Hacker News doesn't know enough to throw it away. Instead, we parade it around like some great success!
Fooled by a language model.