komon

Can't you just run it under Rosetta?

Effective system design comes from understanding the constraints of the problem, understanding the constraints of technologies you could use to solve the problem, and understanding what parts of your solution needs to change as those constraints change.

For instance, a web crawler. Establish your boundaries right away. You can't crawl the entire web right out the gate, so you need to decide where to stop. 1M pages? 1B? Does your crawler run continuously, or in batches, on a schedule? Are you concerned with avoiding crawling duplicate links? How often do you want to revisit links you've seen before? Are you going to limit page sizes you'll keep? Are you keeping images and other files, or only HTML content? Do you need to be able to execute JavaScript? How do you want to access the results: search, machine learning, analytics, etc? How many requests for those results do you expect in a given time frame? What's the maximum time you'd like to wait for a response?

All of these questions will affect your system design.

Take those answers and do some napkin math to figure out the scale you're working with.

If you are crawling only 1M pages and keeping only the HTML of pages 50kB in size you can easily use any database off the shelf with little tuning, for pretty much any use case (modulo response time requirements). For allowing larger files and images you'll probably want to reach for commercial object storage (unless you're a whizzkid file server guy), and if you're crawling a billion links you'll likely need to think carefully about horizontal scale and partitioning, and if you need to provide search or analytical processing you'll likely want to bring in specialized data stores, which means setting up some pipes to connect them through replication or batch processing...

It goes on and on like that. You need a broad knowledge of the capabilities and limitations of different softwares across various categories to understand what your options are and when the complication of adding them would be warranted. I find it helps to understand the historical context under which they were first developed, so that you can understand the problem they were trying to solve.

In general, I've had the misfortune of not being the dumbest guy in the room at any given time, so I've not received much in the way of mentorship. I've had to mostly learn by doing, which gives me a particular bias towards building over buying and avoiding bringing in new software dependencies where I can help it.

I encourage you to learn by doing as well, even if you stop around the first or second roadblock, or about the time you start needing to spend your own money to go further. Maybe try implementing things locally in a way that doesn't let you do more from your personal machine per se, but instructs you in the complication of setting up e.g. kafka or elastic search, and their benefits and tradeoffs.

Think hard about how you would monitor the system to make sure it's working well, think about the dashboard you'd want to see to assure you of the health of the system. Think about where the bottlenecks to performance would be. Think about what you'd change to ensure it stays up 99.999% of the time, even when you're deploying changes to it or scaling up the database. Think about what you'd need to do to ensure that you never lose data.

And then read other people's experiences, but remember to take a grain of salt with you. People like to talk a big game, you'll hear people say that tool XYZ is the ONLY choice for a serious project and... that may or may not be true. Depends on the scale.

But usually in system design interviews people want you to whip out all the keywords and show you know why you're using them and what they're for. So when prepping for those, don't be afraid to say "streaming CDC events with debezium and kafka" instead of "batch job loading new records from postgres" or w/e, as long as it's applicable and you can justify it.

System design interviews also tend to revolve around a few categories of problems (web crawling/search, Google docs, building social media feeds, etc etc) so reading other people's answers for those will start giving you the flavor.

I give the system design interviews at my company (well, now I mostly train other people how to do them) so usually I look for people to ask this constraint questions, show breadth of knowledge, reveal where they're strongest by going deep on a few topics, and the more senior they are the more I want them to lead the conversation and ask me questions rather than the other way around. I don't really expect people to come in and deliver a rote-memorized google-scale answer off the top of their head, and tbh I'd be really suspicious if they did.

I think you need FDA clearance to be able to market a test as actually being effective for treatment/prevention etc. For exactly the same reason woo-woo supplements must disclaim that they aren't intended for that purpose

These are mutually exclusive states, actually.

The "chance at making FU money" and "100% more work/stress" and "losing funding or... goes bust and can't make payroll", that's stuff that happens at startups that are paying _below_ the VC-backed market rate.

Once a company gets money to be paying employees in the third hump of the trimodal distribution of SWE pay, those "exciting" parts are long gone. You're no longer in the running to receive a hilarious amount of money should everything 10-100x, and you're no longer worried about going bust and not making payroll.

Even a pre-series A startup I worked at saw the writing on the wall with 5 months of runway left and switched from trying to get funded to finding an M&A deal. At the very least you can get the team acquired by someone in the expansion phase and get the rare chance to interview as a group and compare notes and have an exec try and negotiate your salary _up_ for once.

By the time a startup is paying VC-backed market rate, the level of stress you have just comes from the usual suspects: management, culture toxicity, etc. some of that you can absorb, some might prompt you to find a new job, just like normal.

There's still a level of "hoo-rah, we're a startup" present of course, but I just consider this a bad attempt to keep the magic of when the company and products were small alive. I let it affect me none.

$JOB-2 got sued over website accessibility. The way these cases go is someone uses the WAVE browser extension to identify accessibility issues, and anything yellow or red becomes part of the cause of action. IIRC we settled out of court.

We were pretty large, big-name online merchandise retailer. Definitely put our heels to the fire on giving the frontend a design refresh.

A reference to Postel's Law: be conservative in what you produce and liberal in what you accept.

The law references that you should strive to follow all standards in your own output, but you should make a best effort to accept content that may break a standard.

This is useful in the context of open standards and evolving ecosystems since it allows peers speaking different versions of a protocol to continue to communicate.

The assertion being made here is that the world has become too fraught with exploiting this attitude for it to continue being a useful rule

I enjoyed Schild's Ladder and Permutation City immensely

The output from an LLVM frontend compiler is LLVM IR, which is sort of a high-level architecture-agnostic assembly.

The LLVM backend compiles this IR to the target architecture's machine code.

Some optimizations can (and do?) happen in the backend compiler, but for the most part it is up to the frontend to emit IR that is well optimized within the semantics of the frontend language.

The history of Rust, for example, is filled with examples of optimizations made not to the standard library, but to how the frontend emits IR.

Beyond that, different languages and ecosystems are going to have differing levels of optimization in their stdlib and runtime that will have an outsize effect on day-to-day performance.

I wonder what your actual cache hit rate is? Sounds like it could use some tuning since 100-150ms sounds consistent with most requests hitting the origin?

Well, due process is a right co-equal to free speech, so which rights override which others in which circumstances will come down to legal precedent.

My understanding is that the FBI or other non-judicial body cannot unilaterally issue a gag order. Subpoenas and gag orders related to them are granted by judges.

(Which isn't to say that the relationship between the judicial branch and law enforcement bodies is always pure and equal)

KV storage is SQLite, and SQLite replication and syncing using an out-of-process daemon is becoming a solved problem.

For that application at least no changes to the runtime would be necessary

Here's two models:

"Open Core", where the core product is open source and you provide closed source extensions, generally high-value integrations and enteprise-y features like SSO, active directory support, etc.

"Paid Services" where you have offerings on top of the open product. You might offer trainings, consultations, support contracts, custom development work (e.g. aforementioned high-value integrations). Also very common is to offer a fully-managed SaaS experience, but I wouldn't expect that to be your key to success; many open-core businesses fail expecting that full hosting to be more attractive than it is.

It's becoming more popular to offer "on-prem"/"self-hosted" support. Cloud costs are going to get cut after 3rd party products among your customers, so getting your product counted in column B gives you a little edge. Even if they need to cut their support contract down they're still using your product and are more likely to come back when the storm is over.

What you're observing is not the rise and fall of best practice and good engineering. All you're seeing is the hype cycle.

Best practice and good engineering in web dev is the same as it is in any other software field: the best stack is the one you know cold.

Unfortunately, there are many popular stacks, and all of them have something to say about all the others. And 80% of learning materials for any given stack aren't from official sources, but are instead blog posts and videos and tweets from tech influencers. When they parrot those same criticisms, it lends the air of a grassroots shifting of the whole community.

The fact of the matter is that despite the huge levels of hype flux, large, popular, stable apps are still being made with technology "boring" tech.

Despite SPAs, rails and Django apps are still built rendering server side templates and sprinkling jQuery where necessary. Despite GraphQL, REST APIs are still the state of the art for most APIs. Despite kubernetes and docker, many apps are still deployed to heroku or VPS providers.

It can be intimidating for newcomers because it can be difficult to separate hype from fact. My advice is to pick something, the more boring the better, and know it cold and ignore all claims of its death.

To address concerns about projects dying, that is pretty legitimate. Unfortunately webdevs deploy trusted code into a highly untrustworthy and antagonistic environment. New technologies do come to play and requires adapters and plugins to be written. I wouldn't worry about overall size of community as long as security issues are getting addressed.

DynamoDB is an interesting beast. It's hard for me to recommend it for most use cases anyway.

It was designed to power experiences that look like shopping carts: relatively short-lived, mostly append-only, needing a flattish performance curve as the number of items and users go up, with a really natural partition key.

Using it as the primary storage of your app means you play the part of query planner a lot of the time.

If your data is especially graph-structured you might end up doing so-called "single table design" where all objects end up in the same table that has its keys constructed carefully (compound keys formed by string concatting), And has secondary sparse indices to power other access patterns.

And definitely plan on overfetching as an _optimization_ to prevent many small queries to dynamo, especially in a GraphQL context where we have no idea what fields are being requested at fetch time.