This paper feels way too abstract, to the point it makes it hard to understand what the team actually did.
For instance, the paper claims it beat GPT-4-Turbo and Gemini-Pro-1.5 on certain tasks... but it doesn't include any of the questions they asked GPT4 or Gemini, so it's hard to guess whether these results have any value at all.
It's also unclear what they even trained their custom transformer to do. It has a custom tokenizer, but they don't give a list of tokens (aside from a few examples in the diagrams like "Barrack", "Michelle", "Trump"). They talk about in-distribution and out-of-distribution tasks, but they don't give any examples of these tasks and what they look like.
This feels like accidental complexity. It wouldn't have been hard to add a few more appendices with eg a list of 20 or so in-distribution sentences they asked the model to complete and 10 out-of-distribution sentences. Instead all they include is diagrams comparing performance for different hyperparameters and stuff, but we don't even know what the models are being tested on.
For instance, the paper claims it beat GPT-4-Turbo and Gemini-Pro-1.5 on certain tasks... but it doesn't include any of the questions they asked GPT4 or Gemini, so it's hard to guess whether these results have any value at all.
It's also unclear what they even trained their custom transformer to do. It has a custom tokenizer, but they don't give a list of tokens (aside from a few examples in the diagrams like "Barrack", "Michelle", "Trump"). They talk about in-distribution and out-of-distribution tasks, but they don't give any examples of these tasks and what they look like.
This feels like accidental complexity. It wouldn't have been hard to add a few more appendices with eg a list of 20 or so in-distribution sentences they asked the model to complete and 10 out-of-distribution sentences. Instead all they include is diagrams comparing performance for different hyperparameters and stuff, but we don't even know what the models are being tested on.