anthonyhn
Joined 4 years ago
Karma
258
Submissions
55
Member since
Jul 27
Recent submissions
Comment · 2 years ago
First off, I want to thank you and the other members of the CC Foundation, the CC data set is an incredible resource to everyone.
Much of the UA data, including CCBot, is from an upstream source[0]. I was torn on whether CCBot and other archival bots should be included in the configs, since these services are not AI bot scraping services. I've added an exclusion for CCBot[1] and the archival services from the recommended configs.
[0] https://darkvisitors.com/agents/ccbot
[1] https://github.com/anthmn/ai-bot-blocker/commit/ae0c2c40fd08...
Comment · 2 years ago
For those not using cloudflare but who have access to web server config files and want to block AI bots, I put together a set of prebuilt configs[0] (for Apache, Nginx, Lighttpd, and Caddy) that will block most AI bots from scraping contents. The configs are built on top of public data sources[1] with various adjustments.
Comment · 2 years ago
> search engine such an easy thing to produce?
Yes, I run my own independent search engine[0].
> successful
Now that's the challenging part, especially since Mozilla needs to fund browser development. The initial differentiation for a Mozilla search would have been difficult, but at their peak they had 30%+ market share, and if the default search on Firefox was Mozilla Search then they might have been able to make it all work financially. DuckDuckGo makes over $100 million in revenue per year[1], if a Mozilla Search made as much money annually, then even though it's below their current $500 million search contract with Google, with some fiscal responsibility they probably would have had enough to support a search engine and browser development concurrently.
[0] https://ichi.do/
[1] https://techreport.com/statistics/software-web/duckduckgo-st...
Comment · 3 years ago
>The biggest issue is that an ancient version of Firefox is the only viable option for a web browser.
There was some work done on porting Palemoon, Otter Browser, and QtWebEngine to ArcaOS. A preview version of a QtWebEngine-based browser[0] was available for download and may work with newer websites (given QtWebEngine is based on Chromium), but I have not tested it myself.
[0] https://www.os2world.com/cms/index.php/past-news/80-news/sof...
Comment · 3 years ago
> But I always find I have nothing to speak about.
Odds are you do have something interesting to speak about. Many people are experts in very niche things and don't even realize it. You may be very proficient with a niche piece of software that is not well documented, or may have created software to solve a very specific problem. Writing blog posts about your niche knowledge can be tremendously helpful; I can't tell you how many times a single blog post about an obscure problem has saved me hours (possibly even days or weeks) of research when I've encountered the same problem.
Comment · 3 years ago
>Wish I had an invite.
They have an IRC chat room where you can request an invite[0]. They typically ask if you have a personal website, git repo, interesting projects, etc when inviting people.
Comment · 3 years ago
Lisp has very minimal syntax and so you can build a lisp interpreter fairly easily in many languages. Most languages don't have a readily available python interpreter that you can embed, and building one that is feature parity with CPython (the most popular python implementation) is not an easy task.
Comment · 3 years ago
Wikipedia has a page on single-user personal wikis that are great for organizing your notes[0].
If you're looking for something simple and unixy, Zim is a pretty good choice[1]. It's an offline GTK-based GUI application for creating personal wikis that saves all the wiki pages as Markdown files and can export your wiki as HTML using various templates. Zim has been available in many linux package repos for over a decade and is GPL-2.0 licensed.
[0] https://en.wikipedia.org/wiki/Personal_wiki?useskin=vector#S...
Comment · 3 years ago
Yes, it can be quite frustrating. On my search engine[0] I tag sites that use captcha services and browser checks so that the user has more transparency and can choose to avoid sites that use these services if they want.
[0] https://ichi.do
Comment · 3 years ago
I work on a search engine in my spare time as a side project. What I've learned from working on a search engine is that even though the recent GPT models are quite good for general purpose search, there are still ample opportunities in search, and generally not enough people looking at search to cover everything.
Comment · 3 years ago
If you don't mind me asking, what do you use Haskell for at work? Also was the company using Haskell before, or did you bring Haskell to the job? It's always fascinating hearing stories about people using less common languages like Common Lisp or Haskell on the job!
Comment · 3 years ago
GNU Parallel is pretty useful. You can use it to run command line programs in parallel. For example, if you wanted to gzip compress all individual files in a directory and limit the number of jobs equal to the number of CPU cores on your machine:
ls | parallel --dry-run --jobs `nproc` 'gzip {}'
(Note you have to remove the --dry-run switch to actually run this, --dry-run only prints out the commands parallel will run, not actually execute them)
I run an search engine at https://ichi.do. Please feel free to test it out.
If you would like to get in touch, please feel free to contact me through github (https://github.com/anthmn) or send me an email ([email protected]).