nox21125
Joined 6 months ago
Karma
1
Submissions
11
Member since
Apr 26
Recent submissions
Comment · 5 months ago
Thanks again for the support!
Comment · 5 months ago
Yeah, I realized after making the comment that Paged Out articles are only one page, but that should still work. I'll probably make a page, and also use the Community Ads to promote as well.
Your storage estimates are a lot lower than what I’m seeing on my setup. I think the main reason is that Slick stores way more than just extracted text and a basic inverted index. Most of my indices contain a huge amount of metadata, structured fields, and semantic search data.
For example, nearly all of my major indices use 384-dimensional BERT embeddings with Lucene/Elasticsearch HNSW vector indexing, which adds a pretty significant amount of overhead. I’m also storing metadata, schema information, image/video fields, social tags, ranking signals, and multiple text representations.
Just my web index alone is already around 55GB for only 2.4 million documents, and the other major indices combined add another 100+ GB on top of that. The vector data alone is probably going to become enormous at larger scales.
So I think the 13TB estimate for a billion pages is probably realistic for a much leaner BM25-style setup using mostly extracted text and a simpler index, but for my current architecture it’ll probably end up quite a bit higher unless I heavily optimize storage later on.
CommonCrawl seems like a good idea, so I may try playing around with it to see how it is. If I can fix the bugs in my crawler though, and upgrade my setup, I should be able to start crawling much better and filter much better.
Comment · 5 months ago
That's funny, it must be. Paged Out seems really cool, and I'm probably going to write much more than one page if I do. They also have a community ads program which sounds nice. 2600 too, but like you said, I’m probably going to wait until I have a more reasonable index size.
My disk space situation is very bad. Like I said, I’m running on a Beelink EQR5 with only 500GB of storage. If my estimates are right, which they probably aren’t, it would take around 5TB for 50 million documents. So maybe 150TB or so should be enough for a billion.
Crawling speed is also a limiting factor, although not because of network speed. My internet is fine, but my crawler currently only crawls at around 1 million pages a day. It should have crawled around 30 million pages by now since it’s been running for over a month, but when I check the batches it’s produced, it’s probably closer to 2 million max, so there’s clearly a major issue somewhere.
Along with that bug, I also need to make the batch processor much faster. It processes documents and also adds embeddings using BERT, which takes up a significant amount of time. So it doesn’t index 1 million a day, maybe only 30k/day which is obviously something I really need to improve.
If the project keeps growing, I’ll probably eventually move to something much better than my current setup.
Comment · 5 months ago
Thanks! I really appreciate the detailed feedback and suggestions.
The idea of supplementing the index with older Common Crawl/Blekko-era data is definitely interesting, especially for preserving pages that are gone now. The metasearch + independent reranking concept is interesting too, but one of the main goals with Slick is staying completely independent long term.
I know that comes with much slower growth and a lot more work, but I think it's better than building on top of another search engine for 5 years and then suddenly having that engine massively change direction. I actually only recently learned that Google is planning to heavily rework Search around AI as well, which honestly reinforced my decision to keep Slick independent instead of relying heavily on another engine (https://san.com/cc/googles-shift-to-ai-powered-search-result...).
Right now I'm mainly focused on improving Slick's own crawl/index quality instead of relying too heavily on external sources.
I've taken a look at 4get.ca, which is Canadian apparently (I am too), it's really good. Although again, I'm not leaning too heavily into metasearch unless maintaining a fully independent index becomes unrealistic. I have already written over 15 thousand lines of code for this engine already, over a year of coding.
I've never noticed the "search engines disappearing", probably because they're disappearing. I should probably read up on that. Most likely it's because they can't afford to run the project anymore, whether it's mentally or financially. I've experienced this too. I'm actively trying to promote to get new supporters of the search engine, to no avail.
I don't think I'll feel disinclined to work on the project any time soon, but if I ever do, I'll be sure to tell you. You are my first supporter after all.
I'm currently not looking for employees right now, but I appreciate the offer. I've been able to do this much on my own, and it's just uphill from here. Improving the ranking bugs I mentioned in my blog, getting more supporters so I have an incentive to get infrastructure, improving my crawler, etc.
I really appreciate the support.
Comment · 6 months ago
>> Your boundary is at hue 173, greener than 57% of the population. For you, turquoise is blue.
very subtle changes in color after the first two. it also seems to be repeating blue -> green -> blue -> green, for me atleast.
Comment · 6 months ago
This is really cool, feels super nostalgic. Good writing too
Blog: https://blog.slicksearchhq.com