hinkley

A lot of my poor opinion on developer opinions dates back to a couple projects where Product and Marketing were begging us for substantial performance improvements and the teams pushed back saying this was as fast as you could expect. I got 2x in the first six months on both projects, and delivered 25% per quarter on one for almost 2 years (that maths to around 8x) before I started to scrape the bottom of my barrel.

It’s predominantly a way to shut down thinking, which is why it’s so often misquoted. Knuth’s version is more nuanced. Theirs is not at all.

As a compromise I got really good at doing optimizations that also improved readability. Things like hoisting and better variable naming combined together can be mistaken for just code cleanup.

I also discovered that keyhole optimizations are more palatable. Which is to say, move the slow bit yo a separate function with a single responsibility. Then do the optimization in a separate commit. Then the other devs don’t have to look at your changes and if they turn out to have a bug, we can always revert the change later. They almost never do, but having the option seems to grease the wheels.

That’s a rather astounding amount of cash to have extracted per capita.

Don't we, though?

One of the things I didn't get to do at my last place before leaving was that I was putting together a proposal to beef up our CI infrastructure to be more cores per agent. I was trying to figure out the math to prove we would save money by doing so.

I had a batch process I became responsible for that ran on the same VM as an important production service and every once in a while it became a noisy neighbor. It ended up getting moved onto a much smaller cluster for the compute part of the operation after I did some pretty aggressive traffic shaping work to get the cost down by more than a factor of 10.

At one point I got the code into a shape where I thought I could just run it directly on the CI machine, and the lower CPU availability would be offset by the higher IPC overhead and cache misses. Problem was that it was still about 2.3x slower running on 1/3 as many cores. But those VMs idled for hours at a time while waiting for a 10 minute job to be started.

Near as I could figure, if we doubled the size of the instances the build agents ran on, I could shut down that cluster and just about offset the cost of the more expensive build agents exactly, and speed up quite a lot of builds in the process (many build steps do not parallelize but some do).

If one other team had a batch process they could just run directly on the CI machines, that would be a net reduction in operating costs. We had quite a few teams doing batch processing. CI servers have a lot of tricks you can use to queue up tasks rather than running them concurrently.

That was a different time and that’s a very small sample of survivor bias.

There's also a pretty big morale hit for the people who fall below the remuneration cutoff of peers they are almost as good as or feel they are better than. I've seen a few different people lose almost all of their fucks because some dweeb is buying fancy cars and they're still paying their mortgage every month.

It's also often also part of the employee retention plan. If you discover your boss has lied to you shortly after an IPO or a merger, then odds are good they were trying to hold onto you by any means necessary until their bonus was paid out.

I've had several people I trusted destroy our relationship this way over the years.

It’s not a judgement of intellect but of character. Those people were forced out of Google when the advertising/boiler room mentality took over.

Amazon kind of does say it but they were clever about it. They don’t let rhetoric win only raw numbers. That means they will always shill or scam for profit. Has been that way for decades but only now are people paying attention.

An IPO is where the investors cash out to spend money on something else. It’s where the founders and early employees get rich enough to start chasing other interests, or just retire outright.

However customers often take a while to adapt to changing conditions.

If for instance you enshittify a tool brand, first the supply chain needs to drain of old inventory and then customers need to be confronted with the new product before they react, and that may take several interactions.

Ok this case that doesn’t describe people using search but does describe a monthly or quarterly ad spend. Once the feature factory shuts down it takes a bit to notice the new trend.

It is unfortunate and only sometimes sustainable to alienate a third of your user base to extract 2x as much money from the rest for a net 30% increase in short term revenue. It’s not repeatable usually.

His voice work in Nausicaa was perhaps the first time I’d heard of him.

This graph invites some pertinent questions. For instance, it suggests that their search revenue growth went linear sometime in 2020 and might be going asymptotic as of the last year. So I want to know what happens there over the next several quarters. Is the current quarter anomalous, or do deferred payments always make it look like that?

Second, is the geometric growth of Cloud cannibalizing their search teams, leading to less effort being put into sales. And so it’s not so much that demand is tapering off as that supply is being constrained as an opportunity cost. That $1m in salary nets more profit from Cloud than from Search.

As cpburns2009 says, the benefit of inlining code (which this technique often applies to) is that you make copies at multiple call sites and so they don’t have to share the same hottest branch.

The danger of being excited about old techniques is giving the impression that you are years behind everyone else.

A coworker once said of a particular breed of manager who is an emphatic late adopter: it’s as if every time he goes to the dentist he reads an old trade journal in the waiting room and comes back to work with a “new” idea that everyone else is already busy replacing.

Are there any actual newer techniques? For instance, putting 11% of your hardware on reserve for hardware failures seems a bit generous to me for such an expensive cluster. Even if it’s down from some much larger number. One solution to this I’ve seen for handling cpu and memory variability with bin packing is to pool the spare capacity between tasks. For instance a data center with four zones might have 7 backup generators, instead of eight or twelve, counting on clusters of generator failures not to happen frequently.

And of course there’s always running low priority tasks on the spare hardware and evicting them on a fault. However those low priority tasks tend to develop their own shadow priorities when they are in house. But that’s a management problem not a technical one.