Benjammer

Why would you not just build your own harness in October 2026? Gateway, memory, tool calls, LLM api - the basics are pretty simple to get started and get to a point where your harness can be self-improving to add any new features that you need.

Recently? Blind (teamblind.com) culture has been taking over tech for like a decade at this point.

I usually just say “make sure this code is professional and ready to deliver as a senior engineer” and it usually infers all that stuff you said plus more things as well. I try to give it the goal and let it decide what to do.

One thing I usually keep having to point out directly is to remove all “progress tracking” code comments and make sure all comments are appropriate for long term maintenance in the code base. Claude tends to leave comments like “button click causes save now, no longer uses onBlur” when the code really never used onBlur, that was just a thing Claude wanted to do earlier in the same task/branch and I redirected it at some point.

Now do Long Island City in Queens, where 44th Ave, 44th rd, and 44th st are all in a row of blocks parallel to each other.

> the quality really does matter.

If this level of quality/rigor does matter for something like a game, do you think the market will enforce this? If low rigor leads to a poor product, won't it sell less than a good product in this market? Shouldn't the market just naturally weed out the AI slop over time, assuming it's true that "quality really does matter"?

Or were you thinking about "matter" in some other sense than business/product success?

>Since COVID in CA, it feels like driving has become far more dangerous with much more lawlessness regarding excessive speeding and running red lights, going into the left lane to turn right in front of stopped cars, all sorts of weird things

NYC has had the same effect since COVID, and over the last year or two it's gotten to the point where every single light at every busy intersection in Manhattan you get 2-3 cars speeding through the red light right after it turns. I bike ride a lot so I'm looking around at drivers a lot, and for the most part the crazy drivers seem to be private citizens in personal cars, not Uber or commercial/industrial drivers.

>It is the loose equivalent of asking why are you getting hung up on the type of a variable in a programming language? A float or a string? Who cares if it works?

No, it's not. This is like me saying "string and float are two types of variables" and you going "what is a 'type' even??? Bertrand Russell said some bullshit and that means I'm right and you suck!"

Wind and sunshine are both types of weather, what are you talking about?

>They belong in different categories

Categories of _what_, exactly? What word would you use to describe this "kind" of which LLMs and humans are two very different "categories"? I simply chose the word "cognition". I think you're getting hung up on semantics here a bit more than is reasonable.

So the idea is what? What's the successful outcome look like for this test, in your mind? What should good software do? Respond and say there are 5 legs? Or question what kind of dog this even is? Or get confused by a nonsensical picture that doesn't quite match the prompt in a confusing way? Should it understand the concept of a dog and be able to tell you that this isn't a real dog?

It always feels to me like these types of tests are being somewhat intentionally ignorant of how LLM cognition differs from human cognition. To me, they don't really "prove" or "show" anything other than simply - LLMs thinking works different than human thinking.

I'm always curious if these tests have comprehensive prompts that inform the model about what's going on properly, or if they're designed to "trick" the LLM in a very human-cognition-centric flavor of "trick".

Does the test instruction prompt tell it that it should be interpreting the image very, very literally, and that it should attempt to discard all previous knowledge of the subject before making its assessment of the question, etc.? Does it tell the model that some inputs may be designed to "trick" its reasoning, and to watch out for that specifically?

More specifically, what is a successful outcome here to you? Simply returning the answer "5" with no other info, or back-and-forth, or anything else in the output context? What is your idea of the LLMs internal world-model in this case? Do you want it to successfully infer that you are being deceitful? Should it respond directly to the deceit? Should it take the deceit in "good faith" and operate as if that's the new reality? Something in between? To me, all of this is very unclear in terms of LLM prompting, it feels like there's tons of very human-like subtext involved and you're trying to show that LLMs can't handle subtext/deceit and then generalizing that to say LLMs have low cognitive abilities in a general sense? This doesn't seem like particularly useful or productive analysis to me, so I'm curious what the goal of these "tests" are for the people who write/perform/post them?

amusement park --> park amusement... Is that the joke?