bigmutant

Pretty common attitude from folks who have never worked in one of the BigTech companies where Java rules (Amazon being a prime example). Since they never encounter Java in the "SF-style Startup" world, they assume that it must be dead. Meanwhile hundreds-of-thousands of Engineers deal with hundreds-of-millions (billions?) of lines of Java every day

DynamoDB is used *everywhere* in AMZN Retail, this is absolutely not surprising. Plus the vast majority of internal Services are using EC2 in the form of Apollo/ECS. So OP probably hit some parts of the site that are hosted in us-west-2. For all I know they started routing all requests for us-east-1 traffic to other DCs, figuring latency is a fine trade-off for availability

To clarify, most of CDO (Consumer Devices Other) does run on AWS in the sense that NAWS is the target state, MAWS is legacy and actively (slowly) being migrated off of. CDO (including Alexa) has been using DynamoDB/Lambda/Kinesis/SQS etc forever, its just the compute and kind-of network layers that are still MAWS. Even then, a large part of CDO has moved from Apollo to ECS/FarGate/whatever unholy Hex or DataPath thing they're pushing these days

Source: Ex-AMZN

Sure, I did the same, BS/MS with a focus on Compilers/Programming Languages. It's been personally gratifying to understand programming "end-to-end" and to solve some tricky problems, but 99% of folks aren't going to hit those problems. There are tons of people interacting with Cloud Services every day that aren't aware of the basic issues like:

- Consistency models (can I really count on data being there? What do I have to do to make sure that stale reads/write conflicts don't occur?)

- Transactions (this has really fallen off, especially in larger companies outside of BI/Analytics)

- Causality (how can I handle write conflicts at the App Layer? Are there Data Structures ie CDTs that can help in certain cases?)

Even basic things like "use system time/monotonic clocks to measure elapsed time instead of wall-clock time" aren't well known, I've personally corrected dozens of CRs for this. Yes this can be built in to libs, AI agents etc but it never seems to actually be, and I see the same issues repeated over-and-over. So something is missing at the education layer

Def agree. Most people will never touch an Abstract Syntax Tree or even Expression Trees. Almost everyone working in back-end will use Cloud Services, will make mistakes based on assumptions of what they provide

The fundamental problems are communication lag and lack of information about why issues occur (encapsulated by the Byzantine Generals problem). I like to imagine trying to build a fault-tolerant, reliable system for the Solar System. Would the techniques we use today (retries, timeouts, etc) really be adequate given that lag is upwards of hours instead of milliseconds? But that's the crux of these systems, coordination (mostly) works because systems are close together (same board, at most same DC)

As others have said, this is a solved problem in a lot of companies. Basic answers are: 1. Queuing 2. Asynchronous APIs (don't wait for the 'real' response, just submit the transaction) 3. Call-backs to the Client

A good async setup can easily handle 100k+ TPS

If you want to go the synchronous route, it's more complicated but amounts to partitioning and creating separate swim-lanes (copies of the system, both at the compute and data layers)

Absolutely not true in my experience. MySQL has its share of issues (all DBs do) but it is rock-solid when using the correct engine (InnoDB for most cases, RocksDB for high-throughput writes, Memory for caching). MySQL is very hard to beat for very high-volume OLTP workloads, both reads and writes. Its replication systems were years ahead of other systems (SQL Server, Postgres, SQLite doesn't have replication). DuckDB AFAIK is OLAP and they don't compete in the same space. Every DB system has "the things its good at" and MySQL really shines at very high-volume OLTP spread across partitions.

Agree, Data Quality in-the-wild is a huge concern. I've led efforts to establish Lineage/Quality in large orgs and doing this after-the-fact is a massive undertaking. Having this "up-front" before all the data pipelines (origination, transformation, pre-processing) calcify saves a lot of headache down the road.

Never really understood the folks pushing for RPC-over-HTTP. RPC is for systems that are close together (ie intra-DC). These simple rules work well: 1. JSON-over-HTTP for over the web 2. RPC (pick your flavor) for internal service-to-service

I will say that Amazon's flavor (Coral-RPC) works well and doesn't come with a ton of headache, its mostly "add ${ServiceName}Client to build" and incorporate into the code. Never mind its really odd config files

Related note, I've never understood why Avro didn't take off over GRPC, I've used Avro for one project and it seems much easier to use (no weird id/enumerations required for fields) while maintaining all the code-gen/byte-shaving

Northrop Grumman had a lot of folks from Crenshaw/Hawthorne/Carson when I was there, due to a partnership program with the local Cal State (Long Beach). All of the security staff was from that area too. Good folks, would 100% work with them again.

On the other hand, I've seen exactly 1 guy at the FANG I work at. What's the difference? I think it's companies like Northrop realizing that folks from under-represented communities have great value and prioritize that instead of whatever the current HackerRank-based interview process selects for

The Tech-Priests of Mars are calling

Sort of. The main issue is that AMZ/AWS doesn't differentiate (by title) between someone recently promoted to L6 (generally ~8-10 YOE) and someone deep in to L6 (~12-15 YOE). Other companies solve this with the Staff Engineer title, but AMZ doesn't have that. If you're an L6 who reports to an L7 Sr. Manager then you're functionally a Staff Engineer and supposed to be on the Principal track. Mileage varies across orgs/teams.

Gist: "junior" L6 SDE is more like Google L5, "senior" L6 SDE tracks mostly with Google L6 (or even L7 in rare cases).