Comment by derefr

Honestly, I don't think there's any way to get around it, if you've got an indexer daemon in the mix. It's pretty much the same as trying to store billions of rows in a RDBMS table, except that file systems+metadata indexes don't have any concept of table partitioning.