I am early in my journey but I’m stumbling on the basic structure of these models.
Is this structurally a vanilla transformer (or encoder/decoder) with tweaks to the tokenizer, the loss function, the hyper parameters, and the method of training?
Is whatever this is representative of most of the publicized releases? For instance the recent Orca 2 paper didn’t seem to have any “structural” changes. Is there a better term for these distinctions?
I don’t mean to downplay the importance of those changes, I am merely trying to understand in a very broad sense what changes have what impacts.
I am early in my journey but I’m stumbling on the basic structure of these models.
Is this structurally a vanilla transformer (or encoder/decoder) with tweaks to the tokenizer, the loss function, the hyper parameters, and the method of training?
Is whatever this is representative of most of the publicized releases? For instance the recent Orca 2 paper didn’t seem to have any “structural” changes. Is there a better term for these distinctions?
I don’t mean to downplay the importance of those changes, I am merely trying to understand in a very broad sense what changes have what impacts.