fullstacking

heres the rub, whos to say in the future or even now that google is really respecting the robot.txt in the first place. I know for a fact that my sites are still getting hit by AI bots but I'm "blocking" them via cloudflare. Way too many IP's originating from data center ISP. Just saying respecting the robot.txt maybe just PR lip service.

This reads like a self centered narcissist.

"How dare this person write something not pleasing me and my needs..."

Wow, first thank you. I mean you no disrespect as I do think DuckDuckGo is the better of the options. Sadly for me Microsoft is a big deal as I'm a Jew by ethnicity who takes great issue with the harm Israel has caused people in the name of my ancestors. For me anything that helps provide Microsoft any financial support is real a conflict of interest for me. That said these are my own personal issues and I don't expect other people to confirm to my own ethical views. Outside of that I'm also a programmer of 25+ years who has spent a great deal of my time watching Microsoft cause a great deal of harm to the industry. IDK about you, but I feel github has been hijacked, and don't get me started on all the tax dollars we have been spent on sharepoint type BS that can't seem do the basic requirements our governments / institutions have paid for. So yeah anything you can do to distance yourself from them would be something that makes me take a real look at duckduckgo again.

Aside my ranting about M$, if I may ask why go beyond web search, or put another way how do you not end up on the same path as google / bing going that path?

I think the issue of search being free greatly depends on who is providing the content / doing the work. If the links are provided by the public than yeah it should be free, but if the links are provided by a singular entity who had to do the work then the question is did they provide you a service and what is it worth to you? I would argue the core issue is less a monetary issue and more a public domain vs corporate ownership issue. We need a publicly owned web search so that the ethics involved with using said search can be better enforced.

I've been thinking a lot about what it would take to build a simple search engine again. I think the time for a decentralized search engine that is human vetted is now. In a way that's what sites like reddit/HN ect have become, but I think we need a real search engine not just a social network. Looking to alternatives does not leaving me feeling great right now, DuckDuckGo is just bing IE microsoft and that's not really a better trade.

I meant from a software architecture / maintenance standpoint. I assume its that much more work to backport vs just pushing the next change, I also assume this type of issue is only going to happen more thus more backporting / hotpatches ect.