n1xis10t

What does the icon look like?

Such a weird thing. .aigc isn’t a real toplevel domain, and when I searched for it I found that apparently TikTok uses it as an abbreviation for “ai generated content”. That fits with the fake headlines. I was going to suggest checking /etc/hosts to see if some lines had been put in there by malware to make fake domain names capable of resolving, but you say the links don’t work. So that’s weird.

So is this just something wrong with Google?

Wow, so ~43 million a day, ~15 billion a year? That’s sick.

How much space do you have on the server, and how many pages do you intend to scale to?

Yeah that’s pretty weird. I thought that maybe it was websites getting hacked and then redirecting to malicious stuff, but then I did your recommended search and it all seems very benign, and the videos seem to match the page title and urls. So I thought maybe it was some service that people use to redirect to youtube but have their own link instead of a youtu.be link, but a search for "hook global" "youtube" "redirect" doesn’t appear to reveal what this is, it just shows more examples of the weirdness. Including pages that are apparently just from youtube. Maybe if you turn off redirection in your browser and then take a look at the source for the html page that redirects you, it’ll give a clue. I can’t right now (on mobile) but I can probably try later.

What will happen if you don’t use AI? Will they fire you?

Was the reason for your attempt to search in this way that you got the “internal server error” message (that I just got) when you tried to use the website’s search feature?

It seems pretty cool, but I don’t think I’m the best person to ask. Most of social media I don’t use, and I’m not terribly interested if the results are from the apis directly. If it was an independent index of these sites, or if it took a really long time to get all the available results from the apis and then reranked them, then I would be interested because there is a lot of censorship and downranking, and it would be nice to get past that.

I just tried it with the query “reckless ben”, and in the “all” tab a lot of the results are from various websites (like kotaku), where do these results come from?

How does it work? Is it like a meta search engine where your service sends the query to all the apis, or do you actually have an index of all this stuff?

I haven’t tried that, thanks for the tip. It is far from perfect of course, because there are tons of great high quality sites that use .com. What we really need is a good search engine that is the scale of google, but those don’t happen along very often. Cuil was really big, but they really messed it up.

Thank you for saving me from the misery and regret.

Gotcha, it’ll be interesting to see how it progresses.

Oh, the rules are that paged out articles are only one page, so a longer article would have to go somewhere else like 2600.

The collection of about 1.073 million pages in extracted text form that I have takes up about 4.8 GiB spread across 15 files (but compressed they’re only 2.1 GiB), so if you were just downloading them until your hard drive filled up you’d have about 107 million pages, and you’d need something like 5TiB for a billion pages. These are the WET files from CC, which are extracted text only. I know the WARC files are made so that if you know the correct offset in bytes, you can take out individual documents without decompressing the whole file, but I’m not sure if the WET ones work the same way. If they do, your pile of text could be a bit less than half the size and still usable with an index.

I don’t know how much more space an index of the data would take up, but I think it really depends on how complicated it is. If the index is super basic, like “give me a keyword and I’ll give you a list of docs that it appears in”, then I think the index should be smaller than the text collection. You use embeddings and stuff, so I don’t know how big it would be.

Marginalia search has about 1 billion pages, and when someone asked how big the index is on disk, he said this: “16 TB for the unprocessed crawl data (compressed). 7.7 TB for the files that actually constitute the index (positions data, reverse index)” I’m guessing that the “unprocessed crawl data” is raw html, and that’s why it’s significantly larger than my Blekko-era Common Crawl extracted text based estimate.

So with an uncompressed pile of extracted text and a Marginalia style index, one billion pages would be about 13 TB on disk. He says “positions data” though, so I think that means that the locations of keywords in documents is part of the index. Probably the original extracted text and the position data don’t both need to be there (and they’re probably about the same size), so you would just pick between having the original documents and needing to use compute to find the keyword positions for ranking, or having the keyword positions for ranking and needing to use compute to reconstruct the original documents (if needed). So if you pick one instead of having both, the whole thing probably just takes up about 7.7TB.

Oh also, downloading these files from the Common Crawl should go really fast. One file has about 73000 documents in it, and takes up around 141 MiB (in it’s compressed form, but that’s the form it’ll be downloaded in.)

These wouldn’t get you recent stuff of course, but they would make the index size way bigger, and so the quality would go up but it would be dated. It would be like resurrecting Blekko. For context, Greg Lindahl said that their largest index was 4 billion pages, but that their crawl frontier was much larger.

Here’s another idea: Download tons of old stuff from the Common Crawl / Blekko, but only keep and index the pages that are inaccessible today. This would make your search engine as competitive [edit: probably complementary is a better word] as possible, because it draws from resources that the other engines don’t have. I’m pretty sure the standard is to prune 404’s from search indexes, which seems very silly to me because cached page content can be served, or a link to the Wayback machine can be given. I suppose there are a couple partial exceptions, because Kagi, Brave, and either yep.com or Yandex will give some results from the wayback machine, but I imagine this is a very small part of what they have.

Would you mind writing a comment about this search engine you have found or created? I’m intrigued, but I don’t like clicking on links right away.