Hey everyone, I just wanted to chime in and say that GPT-J is incredibly legit. Every aspect of GPT-J is production grade — no difference in process, quality, or results, compared to any other big name research lab.
I also want to apologize to Eleuther for giving them a hard time in the past. My earlier concerns were unjustified. To be completely honest, I was jealous they achieved everything I tried to achieve with my own open source research lab attempt. It took a long time to even recognize that jealousy in myself, let alone set it aside. Sorry.
The credit for this work goes almost entirely to kindiana, aka Ben Wang. Remember that name; you’ll be seeing a lot of it in the coming decade. It’s clear to me that whichever lab he ends up at (he’s an undergrad! Google let him slip away because he didn’t have a degree!), he’s gonna be changing the world. Don’t know what, don’t know how, know he will.
Every aspect of that codebase is immaculate. Most research code is not pretty; this looks carved out of marble and placed in a museum.
Without Eleuther’s TPU resources, this work wouldn’t have happened. Tensorfork (my lab) didn’t get access to the TPU VM alpha. And TPU VMs were an absolute necessity here. (TPU VMs are a new thing; they’ve been in alpha since December, but only recently launched. If curious see https://github.com/shawwn/website/blob/master/jaxtpu.md and https://github.com/shawwn/website/blob/master/mlmind.md for why it’s the future of ML.)
Eleuther also helped test the model thoroughly. Leo Gao (go follow him: https://twitter.com/nabla_theta?s=21) ran GPT-J through the gauntlet. He was the primary person behind The Pile, the training data that makes any of this possible. I can say with absolute certainty and no hesitation that there are no “gotchas” here.
Eleuther’s https://6b.eleuther.ai page looks wonderful too. It’s like a free OpenAI API playground that everyone can try. Keeping it running for months is no small achievement. (Set top_p to 1.0 and temp to 0.8; the defaults are pretty bad.)
Congratulations, and thank you everyone for all your hard work. The world is so much better for having access to this level of quality.
I also want to apologize to Eleuther for giving them a hard time in the past. My earlier concerns were unjustified. To be completely honest, I was jealous they achieved everything I tried to achieve with my own open source research lab attempt. It took a long time to even recognize that jealousy in myself, let alone set it aside. Sorry.
The credit for this work goes almost entirely to kindiana, aka Ben Wang. Remember that name; you’ll be seeing a lot of it in the coming decade. It’s clear to me that whichever lab he ends up at (he’s an undergrad! Google let him slip away because he didn’t have a degree!), he’s gonna be changing the world. Don’t know what, don’t know how, know he will.
Every aspect of that codebase is immaculate. Most research code is not pretty; this looks carved out of marble and placed in a museum.
Without Eleuther’s TPU resources, this work wouldn’t have happened. Tensorfork (my lab) didn’t get access to the TPU VM alpha. And TPU VMs were an absolute necessity here. (TPU VMs are a new thing; they’ve been in alpha since December, but only recently launched. If curious see https://github.com/shawwn/website/blob/master/jaxtpu.md and https://github.com/shawwn/website/blob/master/mlmind.md for why it’s the future of ML.)
Eleuther also helped test the model thoroughly. Leo Gao (go follow him: https://twitter.com/nabla_theta?s=21) ran GPT-J through the gauntlet. He was the primary person behind The Pile, the training data that makes any of this possible. I can say with absolute certainty and no hesitation that there are no “gotchas” here.
Eleuther’s https://6b.eleuther.ai page looks wonderful too. It’s like a free OpenAI API playground that everyone can try. Keeping it running for months is no small achievement. (Set top_p to 1.0 and temp to 0.8; the defaults are pretty bad.)
Congratulations, and thank you everyone for all your hard work. The world is so much better for having access to this level of quality.