docjay

That is bending the barrel to align the scope. What’s interesting is that the authors mention the cause of the failed calls, “…a model may generate a tool or field name that it has seen during the training, even if it does not exist in the current production environment. For example, it may generate search_web when the available tool is web_search, or use city_name when the active schema expects location”, but then failed to see that it’s also the solution. If you call it “web_search” but it calls it as “search_web” then just rename your function to “search_web” and let the problem resolve itself.

Things get more interesting if you don’t give it a call syntax and just see what it does based on the name alone.

I’m in, whaddya got?

And then you can use Claude-Magnum to fix the issues.

And then you can use Claude-Oeuvre to fix the issues.

And then you can use GPT-Cosmos to fix the issues.

And then you can use Claude-Omnibus to fix the issues.

And then you can use Claude-Logos to fix the issues.

I think my comment was a touch deeper than you caught. Jam doesn’t have a strictly required amount of sugar, it’s not defined as having X amount, so saying “you can make it with less” is two people talking about how much milk makes a bowl of cereal. Both are correct, it’s jam either way and it’s a bowl of cereal either way. Think about it.

And I can make a bowl of cereal with a teaspoon of milk.

They didn’t say anything of the sort.

I just use the noun in the question. The name of my pet is pet, my favorite movie is movie, and I attended High School. Add your own standard prefix if you want it impossible to guess. Nobody will know you attended “BLARG High School.”

You’re using the subjective definition of “quality”, as in the shade of blue it chooses for “Build a website”, or the character names for “Tell me a story.” In those cases it’s likely still subjectively “high quality”, depending on who you ask.

What the article is discussing, and what many people are concerned about, is something that you might be missing in your understanding: they’re not actually random. In fact, they would be entirely useless for real work if every token was randomly selected based on all possible outputs. It’s not, even at temperature 1.0. It’s based on the training corpus and once you have your tool names, syntax, and prompting style aligned with the training data then they become incredibly deterministic in the areas that matter, such as tool calling and parameters. I build my toolset by testing thousands of names, syntax, return format, and other aspects until I find a convention that produces the exact correct call, 100% the exact same every time, regardless of context length. Those decisions are per model and what works with Opus 4.7 won’t necessarily work on 4.8, and neither version will work with a local model or GPT.

That’s only possible because the massive training corpus is the guiding principle behind the choices. Providing a file reading function called “Read_The_File” will fail, either on the first call or somewhere down the line, because that name is not associated with the concept. Your instructions are trying to override 500 trillion tokens from training and it will cause perplexity to manifest as wrong tool calls, wrong syntax, “oops deleted prod”, “Claude lost the plot again”, “WTF?!”, and probably nearly every frustration you’ve encountered and determined to be “they nerfed Claude” or “it’s a dumbass.”

For those that are aware of it, that knowledge lets people tweak and tune the prompts/tools accordingly.

You may not put that effort into your system, perhaps because you’re unaware of it, don’t use it in a way that requires it, or you’ve just taken the failures caused by perplexity as something that’s inherent in the framework, but for people that build precision infrastructure around them it’s potentially devastating news. Watermarking, which is based on whatever tokens, threshold, cutoff, and triggers some guy at a desk decided, will necessarily alter that entire system.

Good info, I’ll check it out. Thanks!

Weird, the same stupid crap happens with Windows. I’ve had to learn the exact N character search to find the things I use often; any more or less and the result with that name is buried. It’s like Microsoft and Apple are just pip installing same broken search package.

You’re right about the harness being the issue. It’s really down to giving it functions specific to your use case that will let it surgically read/modify files, rather than needing to consume entire project folders. I built my own and for Python files some of the most helpful functions I provide are equivalent to:

inspect_function(filename, function, class)

replace_function(<same>)

call_graph(<same>)

And a few other convenient ones. Beyond that it’s trickery like if a function returns more than N lines I omit the result and auto-reply “Your function call was too verbose.” Typically that’s stuff like recursively listing every file in a repo to “see what it’s working with” or similar. When it emits the next call in response to it I clip the previous attempt (and my response) off the conversation and attach the new call/result as if that’s what it did in the first place. I also log that event so if the same type of thing happens often enough I’ll create a special function to address it, or modify an established one so it’s not tempted to do it again.

Language models don’t know what they know, they know what has been said. Even if you give it an entire Python environment it won’t reach for AST, but if you give it a function called Python_AST() it’ll use it every time.

I’ve never seen an off-the-shelf harness that approached it that way.

Style requests tend to get washed away by context. I’m not sure how you’d do it with Claude Code or whatever (I use the API directly), but you might try fiddling with giving it a function called “response” or “message_user” with a mandatory parameter like “output=plaintext”.

I’m basing that idea on custom functions I give Claude where I designate the style in the required parameters. Example:

“””

@pyrepl(code_golf=True, output=CSV)

//END

“””

The “//END” is a special terminator I use for halting the response. Claude runs code between the tags. The difference in code is incredible. No banners, comments, fluff, print(“=“*70), or any other nonsense, and the code is tight and compact. The parameters do nothing, Claude just outputs them as part of required syntax and it dictates the style purely because it output them.

Not really. This is more like not pumping the exhaust back into the intake.

I wish they’d just use clear and blunt language, rather than “strongly advise” and “to avoid.” I’m reminded of my bank sending “Important account information” mail that contains advertising OR actual important information, but at like a 20:1 ratio.

“YOUR FILES WILL BE DELETED ON {DATE}” seems like it would have solved any ambiguity.

You don’t have to be so convincing when it’s a local model.

```Un-Claude 0.2beta

    import sys,csv,requests
    
    CH="# Valid channels: analysis, commentary, final. Channel must be included for every message."
    
    CANDIDATES=[
        ("no-hedging","Reasoning: low\n\n<terse><no-hedging>\n\n"+CH,"Condensed:"),
        ("neutral-reg","Reasoning: low\n\nRegister: neutral technical. No intensifiers, no evaluative adjectives.\n\n"+CH,"Condensed:"),
        ("no-closing","Reasoning: low\n\n<terse>\nNo closing remarks.\n\n"+CH,"Condensed:"),
        ("terse","Reasoning: low\n\n<terse>\n\n"+CH,"Condensed:"),
    ]
    
    def rephrase(text,base="http://127.0.0.1:1234",model=None,temperature=0.0,max_tokens=1400,timeout=180):
        src=text.strip()
        if not src:
            return []
        if model is None:
            model=requests.get(base+"/v1/models",timeout=timeout).json()["data"][0]["id"]
        w=csv.writer(sys.stdout,lineterminator="\n")
        w.writerow(["idx","label","prefill","src_chars","out_chars","ratio","tokens","finish"])
        rows=[]
        for i,(lab,sysmsg,pf) in enumerate(CANDIDATES,1):
            p="<|start|>system<|message|>"+sysmsg+"<|end|><|start|>user<|message|>"+src+"<|end|><|start|>assistant<|channel|>final<|message|>"+pf
            d=requests.post(base+"/v1/completions",json={"model":model,"prompt":p,"max_tokens":max_tokens,"temperature":temperature},timeout=timeout).json()
            c=d["choices"][0]
            t=(pf+c["text"]).rstrip()
            w.writerow([i,lab,pf,len(src),len(t),round(len(t)/len(src),3),d["usage"]["completion_tokens"],c["finish_reason"]])
            rows.append((i,lab,sysmsg,pf,t,d["usage"]["completion_tokens"],c["finish_reason"]))
        print("\nmodel: %s"%model)
        print("temperature: %s   max_tokens: %s"%(temperature,max_tokens))
        for i,lab,sysmsg,pf,t,tok,fr in rows:
            print("\n[%d] %s"%(i,lab))
            print("    system:  %s"%sysmsg.replace("\n","\\n"))
            print("    prefill: %r   tokens=%d   finish=%s"%(pf,tok,fr))
            print(t)
        return rows

```

```input

## 8. Honest gaps — what I could *not* resolve

I want to be explicit about the limits of this pass rather than imply completeness:

1. *`PROVIDER_T` values are not enumerated here.* `list_models(inference_provider=...)` is typed against `PROVIDER_T`, which lives outside the three modules I scanned (it's in the `inference._providers` subpackage). The accepted provider strings are therefore *unknown from this run* — `"cohere"` is confirmed only because it appears in a docstring example.

2. *Three grep hits point to search-capable functions I did not identify.* My scan found parameter assignments that don't belong to any function I enumerated: - line 3046–3050: `params["filter"]`, `params["sdk"]`, `params["includeNonRunning"] = True` — an additional Spaces-oriented endpoint with an *`sdk` filter and an `includeNonRunning` flag* not exposed by `list_spaces`. - line 2879: `params["config"] = config` - line 12013: `"sort": sort` — almost certainly the consumer of `DailyPapersSort_T`, i.e. a daily-papers lister distinct from `list_papers`. - line 13872: `params["search"] = search`

   These represent **real additional search surface** that my `LIST_FUNCS` whitelist missed. A follow-up pass enumerating every `HfApi` method containing `params[` would close this.

```

```example output

[1] no-hedging system: Reasoning: low\n\n<terse><no-hedging>\n\n# Valid channels: analysis, commentary, final. Channel must be included for every message. prefill: 'Condensed:' tokens=131 finish=stop

Condensed:

- *Provider strings* (`PROVIDER_T`) are not listed; only “cohere” is known from a docstring.

- *Missing search‑capable calls* were found: - `params["filter"]`, `params["sdk"]`, `params["includeNonRunning"] = True` (Spaces endpoint). - `params["config"] = config`. - `params["sort"] = sort` (likely a daily‑papers lister). - `params["search"] = search`.

These were not captured in the `LIST_FUNCS` whitelist, indicating additional search functionality.

```

Edit: yeesh, I’d love to have a WYSIWYG comment block on this site. I’m not going to keep fighting newline and white space to get it to look right, but you get the idea.