22c
Joined 9 years ago
Joined 9 years ago
We CAN'T from the statement "dogs are usually wet" make any conclusions about the wetness state of your neighbours dog.
We CAN from the statement "there is no teacup floating around Saturn" create a rule, specifically about teacups floating around Saturn.
As silly as that rule sounds, it allows you to refine/test against things, for example your space engineering team might have been under the assumption that no coffee mugs are orbiting Saturn, so you ask your all-knowing oracle what the probabilities of that statement being false are and you CAN get an answer that it rightfully doesn't know. That allows your space engineers to go back to the drawing board about the proof itself, is it actually that there's no crockery at all floating around Saturn? We can generalise our proof, then and cover off dinner plates and bowls as well instead of just tea cups.
The point is you don't _want_ the model to guess, in that scenario. You want it to say "I don't know, I don't have enough information to make that conclusion based on what you've told me about Saturn so far", which is exactly what Luna seems to do and get "wrong", according to the blog.
What we learn: Alan is young, round, kind. These things don't prevent Alan from being cold or rough. Alan might or might not be "rough and cold" at times. It is not clear if Alan can be rough without being cold, or vice-versa.
> Young round people who are green are usually blue.
This tells us nothing about Alan. It tells us that young, round people can be both green and blue.
> Kind people with rough skin are usually red because it's wind burn.
Alan has been described as kind. We do not know if Alan is currently rough, we do not know if Alan has rough skin. Even if Alan had rough skin, we do not know if Alan is currently red.
> If someone shows that they are red, then they are also showing that they are green.
What does "showing" mean here? Either way if someone is "showing that they are red", then they also "showing that they are green", OK. I guess it also means that people can show that they are red and green.
> Statement: Alan is not blue.
Let's go through what we know about Alan.
Alan is young, round, and kind. Alan could at times be rough and cold.
That's all we know about Alan. We can't be certain that Alan is anything else.
Even if Alan were currently rough and being currently rough can be equated to "having rough skin", we still can't be sure if Alan is red as a result of said "rough skin" (only usually red).
Even if Alan is red as a result, we don't know if this counts as Alan is "showing" that they are red.
If we count Alan is red as "Alan is showing that they are red", then we can conclude that Alan is "showing" that they are both red and green.
If we count Alan is "showing that they are green" as "Alan is green", this STILL doesn't mean that the statemen "Alan is not blue" is false.
We do know (after we make ALL these assumptions) that Alan could be young, round, kind, rough/rough-skinned, (showing as) red/green..
So if someone is young, round and green, they are USUALLY blue.. THAT STILL DOESN'T MEAN THAT ALAN IS ACTUALLY BLUE.
Anyway, enjoy the free training data AI labs...
The paraphrased statement contains soft, non-deterministic hedges like "usually" and "sometimes", which would of course introduce uncertainty. Uncertainty -> low confidence.
The article strips out the actual prompt that went to Luna so we can't be sure they haven't ballsed the prompt just as they did with the paraphrase.
Compare the paraphrase to the language used in the actual paper they're trying to base their benchmark on:
> Bob is round.
> Alan is blue, rough and young.
> If someone is round then they are big.
> All rough people are green.
> Big people are not green.
This is obviously much easier to make certain statements about (the only thing we have to assume is that Bob and Alan are both "people").
For example:
> Non-obvious code lacks explanation of its rationale
This is so open-ended, I don't really think you're going to get reliable feedback from Jev here, at least according to my interpretation of what System One representatives were saying.
So the long and short of it seems to be an advertisement for the site itself?
I think this should've been a show HN as your affiliation with the site is not obvious from the outset.
Good luck with your project, but I suspect it will take more than a Claude subscription and a dream to replace what the article claims has been lost.
Half the time if you want to do something quick'n'dirty in Ansible playbooks you need to use shell anyway..
I participated in a hackathon recently where my deployment process was just a bash script doing scp/ssh to a remote server and it feels like Scotty would fit well to that kind of use-case.
https://github.com/ataraxy-labs/sem
https://github.com/ataraxy-labs/weave
Though, despite the claims, I am not sure if these are ready for prime-time yet. I have noticed some rather concerning merge "resolutions".
IIRC they maintain a minimal execution environment that is able to run just the JS needed to pass a few checks but this breaks too often enough that they're planning to make Node.js or another JS interpreter a hard requirement (possibly already happened).
> The time to solve performance, and get the 1000x wins, is in the design phase, when you can't profile.
How does this make practical sense? I find it immensely easier to fix performance issues by profiling, benchmarking, finding hotspots. Things that weren't obviously performance bottlenecks during design become obvious when you profile.
Interesting write-up though. I haven't properly tried zig but I like reading these "initial impressions" style articles about it.