That's funny, it must be. Paged Out seems really cool, and I'm probably going to write much more than one page if I do. They also have a community ads program which sounds nice. 2600 too, but like you said, I’m probably going to wait until I have a more reasonable index size.
My disk space situation is very bad. Like I said, I’m running on a Beelink EQR5 with only 500GB of storage. If my estimates are right, which they probably aren’t, it would take around 5TB for 50 million documents. So maybe 150TB or so should be enough for a billion.
Crawling speed is also a limiting factor, although not because of network speed. My internet is fine, but my crawler currently only crawls at around 1 million pages a day. It should have crawled around 30 million pages by now since it’s been running for over a month, but when I check the batches it’s produced, it’s probably closer to 2 million max, so there’s clearly a major issue somewhere.
Along with that bug, I also need to make the batch processor much faster. It processes documents and also adds embeddings using BERT, which takes up a significant amount of time. So it doesn’t index 1 million a day, maybe only 30k/day which is obviously something I really need to improve.
If the project keeps growing, I’ll probably eventually move to something much better than my current setup.
My disk space situation is very bad. Like I said, I’m running on a Beelink EQR5 with only 500GB of storage. If my estimates are right, which they probably aren’t, it would take around 5TB for 50 million documents. So maybe 150TB or so should be enough for a billion.
Crawling speed is also a limiting factor, although not because of network speed. My internet is fine, but my crawler currently only crawls at around 1 million pages a day. It should have crawled around 30 million pages by now since it’s been running for over a month, but when I check the batches it’s produced, it’s probably closer to 2 million max, so there’s clearly a major issue somewhere.
Along with that bug, I also need to make the batch processor much faster. It processes documents and also adds embeddings using BERT, which takes up a significant amount of time. So it doesn’t index 1 million a day, maybe only 30k/day which is obviously something I really need to improve.
If the project keeps growing, I’ll probably eventually move to something much better than my current setup.