Comment Comment by steveklabnik > Current LLMsOne thing that happened here is that they aren't using current LLMs:> Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released.That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact. parent steveklabnik 1 year ago
One thing that happened here is that they aren't using current LLMs:
> Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released.
That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact.