The base model already have a good grasp of language, so that man pages look like coherent sentences. It also knows python and similar language well, which means that things like 1+5 or print(["hi" for _ in range(5)]) work.
I did try to train from scratch, but it was much weaker (also added a lot of aux losses).
The base model already have a good grasp of language, so that man pages look like coherent sentences. It also knows python and similar language well, which means that things like 1+5 or print(["hi" for _ in range(5)]) work.
I did try to train from scratch, but it was much weaker (also added a lot of aux losses).