Comment by ACCount37

"China does it in a cave with a box of scraps" is a myth. Chinese labs play the shell game to get their hands on a lot of compute outside China.

Tricks like distillation save compute in the RL leg of the process - where a lot of the frontier labs puts their own training run compute.