claytonjy

Suppose I have hundreds or thousands of containers from the same python image running in production, and want Maxwell to speed them up.

How many Maxwells do I need? Maxwell isn’t free; when is it worth it vs when are the gains not enough to justify? How well can we estimate that up front?

Does Maxwell only look at one container, or each of them individually? Do they all get the same patches or are they all treated independently?

Does the profiling or patching cause temporary delays? If so how much, how might it compare to a GC pause or similar?

Vaguely similar, since electrical wires emit EM fields that are disturbed by human bodies, you can use a special camera to see who's in a building based on those disturbances. I'm not sure how high-fidelity or easy to spoil it is, but this was real technology at least 15 years ago.

Betteridge’s law prevails. A fun read, but not as fun as the title suggests.

tl;dr Modern art existed before the CIA and even CIA predecessors did not originate it, but they did push it abroad to show that America is great because we put up with modern art made by socialists depite most americans disliking both the art and the artists

They do, it’s called the AMA, American Medical Association. Very tight control of who can be a doctor, how many new doctors per year, etc. which ensures salaries stay high. They also fight against other roles like nurses doing too much doctor-like work, helping doctor pay while harming nursing pay.

I haven’t worked at a Big Tech monorepo place, but I hope/assume that in the service dependency case, you would touch both in one changeset and the deploy tooling understands the dependency tree and orders the deployments correctly. Without that i agree its the polyrepo approach just with all PRs in the same repo instead of multiple.

In the library scenario, I know tooling like bazel ensures that’s one changeset, not 3+. Tests run for both the changed library and all consumers of it in the same pass. You’re right that it might loop in others for review who weren’t expecting it, but i think that’s the same in a polyrepo approach.

if service A calls service B, and service B adds a new endpoint, or a new optional argument, service A needs an update to take advantage of it

if a library is used by multiple services, and gets an important bug fix, each service using the library needs to update to get the fix

these are sequences of changes, not literally at the same time or requiring deployment coordination, but when this happens a lot people start asking about monorepos

does one not imply the other? if it can run faster in the same hardware, it should also run as fast on lower spec hardware

the last step, #7, is that fully automated or is this humans calling humans? I imagine everything before then is quite automated, and are thus happening very quickly, so I'm curious if the last piece possible being manual has the potential to blow the whole thing up by being too slow.

thanks for pointing out the linear PR stuff, hadn’t seen that. Interesting that while a dozen other companies are trying to muscle in on the hosting/versioning side of github, rather fewer are working on the PR side.

Isn’t this how most tech companies, if not most companies in general, operate? Ive only had one BYOD job in 15 years. Even if you’re a contractor this is common.

I think there’s a bit more nuance and complexity here.

I built and operated an application that used 1-2000 L4 GPUs in production for a couple of years. Long-term GCP reservations running in a GKE cluster. At that scale we had a few GPU failures per week, and once saw 3 in a day. Nearly every failure required an engineer to manually intervene to get rid of the bad node, and then file a ticket with GCP support as they requested.

NVIDIA GPUs throw an “XID” code when they fail, which can be seen from serial port logs for GCP compute nodes (not through k8s!). If you’re lucky, it fires right as your application starts to fail, but there’s often a delay of several minutes. Even when you get an XID, by default GKE only responds to one or a few of them. You can expand the list via configuration, but the reconciliation loop is so slow that might still take 10-15 minutes during which a pod is puking errors and someone might be getting paged.

They’re working on it, and we never saw a single XID after we migrated to H100s, so the situation is improving. I imagine other clouds are even worse, though iirc Azure was leading some effort to improve k8s node problem detector to include accelerator problems so maybe they have a better story.

Training workloads are rather more sensitive to a node failure given that many modern training runs (including SFT etc) need multiple nodes where topology matters, and they might not have another e.g. 8xH100 box in the right place when one fails.