Comment by philovivero

Start searching SuperHOT and RoPE together. 8k-32k context length on regular old Llama models that were originally intended to only have 2k context lengths.