It seems worth noting that the input of gte-small is limited to 512 tokens. The tokenizers for sure aren’t the same, but I imagine this is significantly less than ada-002, whose input is limited to 8191 tokens. That said, I don’t imagine that embedding huge full documents is necessarily the right approach.
I would love to see a comparison for some typical use cases using various methods of chunking input documents.
I would love to see a comparison for some typical use cases using various methods of chunking input documents.