A 0.5B parameter model with a 32k context length that also makes good use of that full window?! That's very interesting.
The academic benchmarks on that particular model relative to 1.5B-2B models are what you would expect, but it would make for an excellent base for finetuning/embedding generation.
The academic benchmarks on that particular model relative to 1.5B-2B models are what you would expect, but it would make for an excellent base for finetuning/embedding generation.