Comment by snippyhollow

We changed RoPE's theta from 10k to 1m and fine-tuned with 16k tokens long sequences.