Edit: ah. This paper is from May 2023. Might be worth putting that in the title.
---
> Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words
These models are 5 years old.
I have to wonder if the authors have seen RWKV 7 0.1B, because it blows away just about every other model I've seen at that size.
The capabilities it has vs the examples in the paper are night and day.
---
> Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words
These models are 5 years old.
I have to wonder if the authors have seen RWKV 7 0.1B, because it blows away just about every other model I've seen at that size.
The capabilities it has vs the examples in the paper are night and day.
https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-1