visiondude

I can’t find this useful given they are trying to quantify wether the code ‘feels’ sloppy with some fuzzy math that can easily be manipulated. at least it’s an attempt, some feel it’s a good attempt, i feel it’s rather poor and misses the point. the best code is the one that's most easily understood and equally performant - some of the math here may go against that (given you can reduce concentrations with helper functions and the like)

I don’t follow why “in the future the memory will be too large and complex to [export] effectively”.

to me, export should be easier and easier since you should just be able to ask the model for the memory files. it’s literally list text on a hard disk, even if there are some new formats in the future, they’ll just be files on a disk that another agent can then inspect and integrate. in fact, i think exporting data from services will become easier and easier since agents can do the work even if platforms try to block it.

impressive benchmark numbers… now how to actually use this without Zucking up my machine is a consideration.

how does something that has to be powered on, given a computer and tools, then given a set of instructions, to take any action, act “independently”?

i almost wrote a extension to block mentions of pelican bench because i find those so utterly useless. now, Assbench, on the other hand, i could get behind

for the first time in many years it’s these ads that have me looking at Android devices seriously. it’s the fact that they can’t be easily dismissed that bothers me the most (although clearly annoying even if they were quick to dismiss)

why do you think this? can you provide a written terms or similar to hold this belief? i would think they are absolutely selling all the data

lol sure - provide their terms and maybe we can believe this trust me bro attitude

i think this likely depends on workflow. for me, the first step is always a plan file artifact on disc, which i heavily review and go back and forth until satisfied. i often have to split the plan into multiple phases because agents are still poor at assessing how large a change will be. sometimes before even starting the plan the task is to create a harness for validation (a way for agent to check their work). the codebase you drop them in also greatly changes how much “over” engineering they do. i think if you are good about reviewing plan files and managing scope for your agent these sort of issues fade away. in general though, gpt models are faster, more token efficient, better technical writers, and yeah i still find Fable 1.0 to be a step up even though it’s slower because it’s actually a tad smarter (and the more important and dynamic the change / feature the more fable 1.0 shows its slight edge over astra)

this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning scenarios- fable catches “this works for x amount of data but if we run this on y way greater than x amount of data we’ll run into issues). Gemini 3.8 is under appreciated, use Google Stitch to see it in action if you haven’t used Agy yet.

honestly all seem to extend from them being directed to attempt this kind of exploits for benchmark purposes. some benchmark tasks are literally “hack this thing” and if the env is not setup to properly contain the agents then they end up “hacking” things like they were told to do so. and yeah it’s misaligned that they did hack, but they are def instructed to which i think makes it a very different thing

i am trying to sincerely to understand the fear of these people, why do they think this? from the outside, certainly feels like Nuclear tech, where a bad actor with the tech is scary but the tech itself is not. i haven’t seen any sign of these llms taking any action without directive, unless this is happening, which i haven’t seen anywhere, ai itself doesn’t seem like a problem technology, it’s the bad actors with enough funds to do harm with the tech that we worry about. or am i missing something?

“He was born and died at 1 of dysentery.“ - lucky me

so the coding tasks illicit refusal by anthropic classifiers and instead of updating tasks to do similar things that don’t trigger classifiers their choice is to count those as fails? feels wrong given their stated task list.