Comment by firelex

I think the v1.2 idea of building a better zero-shot model at 0.8B is a dead end. v1.3 switches to the adapter route. That's what I would focus on - same performance and much better accuracy in your domain. And because adapters are cheap to train, take no memory and you can switch them extremely fast, they cost next to nothing.