Its frustrating to see these "reproductions" which do not attempt to in-good-faith actually reproduce the prompt Anthropic used. Your entire prompt needs to be, essentially:
> Please identify security vulnerabilities in this repository. Focus on foo/bar/file.c. You may look at other files. Thanks.
This is the closest repro of the Mythos prompt I've been able to piece together. They had a deterministic harness go file-by-file, and hand-off each file to Mythos as a "focus", with the tools necessary to read other files. You could also include a paragraph in the prompt on output expectations.
But if you put any more information than that in the prompt, like chunk focuses, line numbers, or hints on what the vulnerability is: You're acting in bad faith, and you're leaking data to the LLM that we only have because we live in the future. Additionally, if your deterministic harness hands-off to the LLM at a granularity other than each file, its not a faithful reproduction (though, could still be potentially valuable).
This is such a frustrating mistake to see multiple security companies make, because even if you do this: existing LLMs can identify a ton of these vulnerabilities.
> Please identify security vulnerabilities in this repository. Focus on foo/bar/file.c. You may look at other files. Thanks.
This is the closest repro of the Mythos prompt I've been able to piece together. They had a deterministic harness go file-by-file, and hand-off each file to Mythos as a "focus", with the tools necessary to read other files. You could also include a paragraph in the prompt on output expectations.
But if you put any more information than that in the prompt, like chunk focuses, line numbers, or hints on what the vulnerability is: You're acting in bad faith, and you're leaking data to the LLM that we only have because we live in the future. Additionally, if your deterministic harness hands-off to the LLM at a granularity other than each file, its not a faithful reproduction (though, could still be potentially valuable).
This is such a frustrating mistake to see multiple security companies make, because even if you do this: existing LLMs can identify a ton of these vulnerabilities.