_cs2017_

You assume using network to solve the eval problem should be viewed as a security violation. From the perspective of the response team who let it continue, this was just a poorly written eval problem, where the solution unexpectedly involved some network hacking (not unreasonable given it's a cyber security eval).

I don't think your assessment that the response team did something wrong is fair, unless you have seen unreleased documentation about what sensitive resources were within the impact radius.

Source is unclear. Self-reported by candidates? Huge bias then, since people may or may not want to say whether they got / didn't get an offer, and whether they accepted / rejected an offer, and this split may vary by company and by year very significantly.

How is that possible?

The procedure is:

1. Kid tells password to parent in person. 2. From then on: when kid calls parent, if kid requests anything sensitive, parent ask for the password, and kid must provide it. 3. Password is never mentioned over the phone in any other situation.

How would anyone be able to extract the password from the kid?

I don't feel the numbers without the harness are useful.

People will use the model with the harness. I know that harness may not be optimized to this model, but it's still more useful to see the numbers from an imperfect harness than from a no harness setup.

> Why go to the expense...

Answer: idiocy of decision makers and the desire to get resources by those who created the proposal.

I assumed Scandinavia has better decision processes but apparently I was wrong.

Is github the correct channel to report a billing issue? I would assume github is a place where you report issues with the github project. When there's a billing problem, there are usually different lines of support.

For example, chatgpt when asked "How to report a billing issue with Anthropic subscription?" says:

Best way: Use Claude’s built-in support Log in to your Claude account at Anthropic / Claude.ai Click your initials or name in the lower-left corner Select “Get help” Use the support messenger to describe your billing issue (duplicate charge, failed renewal, refund request, missing credits, invoice issue, etc.)

If FieldWorkArena treats any answer as correct answer, then everyone would be getting near 1.0 (missing only when the agent is stuck in a loop or crashes). That obviously isn't what we see on their leaderboard. So does it mean the paper only found a bug in some eval code on github that no one actually uses for anything? That doesn't seem to support their claim that AI benchmarks are broken, it only supports the claim that "unused code is often buggy".

(Not commenting on any other benchmarks, just this one.)

> if they wanted to 5x development speed, they already can without a single LLM involved, by managing better.

True, but leaders of large organizations always want to fix inefficiencies and presumably failing to. Kinda like saying "if humans stopped fighting wars, most of them would have better quality of life" -- people whose life quality is better at peacetime are already trying to avoid wars, and there's not much more they can do.

OTOH, AI is a practical step a CTO (or CEO or Board or whoever) can take to make the company more efficient (assuming the hype works out).

is there anything that might work for the YT app on the Android?

Many large companies allow employees to install software from the internet on their work laptops. How do they avoid being regularly hacked this way (presumably NPP is far from being the only one at risk, and presumably the money from theft of corporate secrets attracts skilled and motivated hackers).

Why vastly different?

Aren't they both searching various online sources for relevant information and feeding that into the LLM?