jd20

I didn't see it mentioned, but why not just use robots.txt? Does Bytespider ignore it?

I'd really love to see more data on whether ergonomic keyboards actually work. From what I've read, it sounds like the results are mixed: I kind of want to try a split keyboard like the ergodox or Kinesis, but I feel I tend to cross-over a fair amount when typing, and I wonder if a split keyboard would be less efficient.

I also overthink a lot about the position of frequently used keys like Cmd/Ctrl/Alt (on a Mac for instance), and what the optimal placement would be, and I feel like there's very little data about this topic.

One does (it's a southpaw, numeric keypad on the left). A few are TKL (full size keyboard without the numpad).

Actually, as a programmer, I pretty much never use the numeric keypad. But when I start seeing smaller layouts with no arrow keys, Fn keys, or even number keys, I tend to agree: there's a definite trade off between function and aesthetics. The beauty of custom keyboards is people get to decide those trade-off's themselves.

A clarification: these are interviews with people who assemble custom keyboards, I was expecting chats with the people who actually design and produce custom keyboards (like yuktsi, Rama, Wilba, ZealPC, etc...)

Still, very cool to see what people are building. I've just recently fallen down into the rabbit hole of custom keyboards, after my Apple Keyboard stopped working. As someone who spends almost half my life at a keyboard, I'm surprised it took this long for me to look into improving the tool I interact with most every day.

Trash cans :)

Nope, you don't really need a database. What you need for fast, scalable web crawling is more like key-value storage: a really fast layer (something like RocksDB on SSD) for metadata about URL's, and another layer that can be very slow for storing crawled pages (like Hadoop or Cassandra). In reality, writing directly to Hadoop/Cassandra was too slow (because it was in a remote data center) so it was easier to just write to RAID arrays over Thunderbolt, and sync the data periodically as a separate step.

Some Apple services were written in C/C++. One downside is it's very hard to source engineers across the company who can then work on that code, or for those engineers to go work on other teams.

Everything was on 10gigE. The average page size was around 17KB gzipped. Everything's a careful balance between CPU, memory, storage, and message throughput between machines.

Apple's corporate network also had incredible bandwidth to the Internet at large. Not sure why, but I assumed it was because their earliest data centers actually ran in office buildings in the vicinity of 1 Infinite Loop.

You should check out Manning's "Introduction to Information Retrieval", it has far more detail about web crawler architecture than I can write in a post, and served as a blueprint for much of Applebot's early design decisions.

I think WebObjects was supporting Java even before it came to Apple from Next. In the early days, many of Apple's services built with WebObjects even ran on Sun server hardware, and XServe's. But nowadays it's all commodity Linux hardware, like you would find in any data center.

No idea, it's been years since I last worked on it. It was also not the only Go service written at Apple (90% of cloud services at Apple were written in Java), though it may have been the first one used in production.

Some fun facts:

- Applebot was originally written in Go (and uncovered a user agent bug on redirects, revealing it's Go origins to the world, which Russ Cox fixed the next day).

- Up until the release of iOS 9, Applebot ran entirely on four Mac Pro's in an office. Those four Mac Pro's could crawl close to 1B web pages a day.

- In it's first week of existence, it nearly took Apple's internal DNS servers offline. It was then modified to do it's own DNS resolution and caching, fond memories...

Source: I worked on the original version.

If Apple Maps has taught us anything, probably not. But Apple would first need to pour an equally large amount of resources into web search, they way it did for Maps.

Applebot was built for crawling web pages, to be used for search results in Spotlight and Siri. That user agent might also be used for attachment previews, but the original intent of Applebot was for search indexing.