I think it's astounding how "easy" we can get away with stuff these days. But how does this scale to larger datasets? Say, capturing all signalling traffic for a telco. It's not hard to have 100s of GB a day - that's only a couple 10s of Mbps. Suddenly 20GB/sec means 5-10s queries just for that day.
They can also leverage the bulk speed to reduce the size of posting lists. Just index each term to a chunk of, say, 1-5GB. And don't even index full terms, just hash them - a 64-bit hash gives you enough room for around 10 billion unique terms without causing much problems with collisions. New data in the last hour can be brute forced while the index is being generated.
OTOH they seem successful so I suppose it's working quite fine.
They can also leverage the bulk speed to reduce the size of posting lists. Just index each term to a chunk of, say, 1-5GB. And don't even index full terms, just hash them - a 64-bit hash gives you enough room for around 10 billion unique terms without causing much problems with collisions. New data in the last hour can be brute forced while the index is being generated.
OTOH they seem successful so I suppose it's working quite fine.