Comment by mrbonner

My guess is that they use an encoder-only model as the foundation and then do RLDC. Why? 1) it doesn’t need text generation 2) limited context window (40k last time I check)

Those are telltales of a Bert model.