Leaderboards are not that useful for measuring real-life effectiveness of the models atleast in my day-today usage.
I am currently struggling to diagnose an ipv6 mis-configuration in my enormous aws cloudformation yaml code. I gave the same input to Claude Opus, Gemini and ChatGPT ( o1 and 4o).
4o was the worst. verbose and waste of my time.
Claude completely went off-tangent and began recommending fixes for ipv4 while I specifically asked for ipv6 issues
o1 made a suggestion which I tried out and it fixed it. It literally found a needle in the haystack. The solution is working well now.
Gemini made a suggestion which almost got it right but it was not a full solution.
I must clarify diagnosing network issues on AWS VPC is not my expertise and I use the LLMs to supplement my knowledge.
I am currently struggling to diagnose an ipv6 mis-configuration in my enormous aws cloudformation yaml code. I gave the same input to Claude Opus, Gemini and ChatGPT ( o1 and 4o).
4o was the worst. verbose and waste of my time.
Claude completely went off-tangent and began recommending fixes for ipv4 while I specifically asked for ipv6 issues
o1 made a suggestion which I tried out and it fixed it. It literally found a needle in the haystack. The solution is working well now.
Gemini made a suggestion which almost got it right but it was not a full solution.
I must clarify diagnosing network issues on AWS VPC is not my expertise and I use the LLMs to supplement my knowledge.