whiatp

I would call out if it is a solder on TPM (or worse a removable module) MITM attacks work against the TPM. (I don't know why, but apparently new dell servers are still shipping with solder on TPMs and having the BIOS/UEFI firmware measure into that instead of just using the AMD EYPYC CPU's fTPM.) TPM isn't actually going and measuring anything on its own, it just gets PCR extend commands containing hashes sent over from the CPU that it has to trust are meaningful. If you can watch the bus (usually just SPI) it would be trivial to monitor the traffic to the TPM during a reboot and watch the hashes it is expecting to see, at least for the boot artifacts, which could be replayed into the TPM after it is disconnected, and then the TPM would happily unseal the secrets for the attacker's microcontroller, even for sealed secrets they weren't able to watch go past while monitoring. So, IMO against physical access the PCRs aren't doing a whole lot, and the whole thing is resting on a PIN. (Obviously this is a huge problem for servers, where a PIN is probably not in use.)

Yeah, if the CPU has a an embedded TPM that would be way better than the external ones. I'll admit that my experience comes from somewhat dated hardware.

The point I was making was about registering new EKs/certs into your set of trusted servers. While you can tell the TPMs apart from each other, knowing the EK/cert belongs to your hardware's TPM vs someone else's is the tricky part.

Something worth calling out is that, as far as I've seen, most server TPM implementations are not great against physical access attacks. If servers might be physically compromised (eg. you are leaving a server unsupervised in a colo) shenanigans are still possible.

The TPMs are on separate chips from the main processor. If something were to man-in-the-middle the communications with the TPM, the hash digests can be "corrected" so the TPM thinks the boot artifacts were in the intended state. At least the ones I've worked with were SPI, but I've seen I2C ones as well. Either way, these are low speed, easy to mess with buses.

Also you want to pay real close attention to how you onboard new devices. The article states

> The EK comes with a x509 cert signed by the manufacture’s PKI. So the EK proves the TPM is legit.

This lets you know the TPM you are performing remote attestation of is made by a particular manufacturer, but an attacker can go buy a TPM chip from the right manufacturer off digikey, and feed it the intended hashes in pcr extend commands. For the attacks the TPM is supposed to prevent, you have to assume they could re-direct the tpm requests your remote validation service is trying to run to their own device by compromising the boot artifacts. You still have to figure out how to make sure your workflows are onboarding the TPM from _your_ hardware, not just a TPM from the same manufacturer.

This article is about using TPMs on servers, not employee devices.

Many large distributed systems are built around pushing data through web requests, and human readable request/response formats (JSON, XML) are the most popular, and require integer to string conversions for serialization.

I remember a coworker having to fight with an old platform's build not working because our user/group IDs were bigger than 2^16. I can't remember which utility was causing the problem, I'd have to guess tar. This is when we learned to play the archive a VM game.

I'm curious what the concern is with the rust editions mechanics in place. Each crate gets to define the language edition it is compiled with. Even if dependencies up convert to later editions they can still be linked against by crates that are an older edition.

As for the broader crate ecosystem, if crates you depend on drop support for APIs you depend on, that could cause you to get stuck on older unsupported releases. Though that is no different of a problem than any other language.

Something I've noticed that I never really see called out is how easy it is to review rust code diffs. I spent a lot of my career maintaining company internal forks of large open source C programs, but recently have been working in rust. The things I spent a lot of time chasing down while reviewing C code diffs, particularly of newer team members, is if they paid attention to all the memory assumptions that were non-local to the change they made. Eg. I'd ask them "the way you called this function implies it _always_ frees the memory behind that char*. Is that the case?" If they didn't know the answer immediately I'd be worried and spend a lot more time investigating the change before approving.

With rust, what I see is generally what I get. I'm not worried about heisenbug gotchas lurking in innocent looking changes. If someone is going to be vibe coding, and truly doesn't care about the language the product ends up in, they might as well do it in a language that has rigid guardrails.

Long time back I'd play PS2 games in a chat window in EverQuest while waiting for mobs to spawn. I had a capture card that would overlay over a particular shade of purple that I discovered while trying to screen shot something from a game. I made an empty chat window in EQ that color and where it overlapped the card's application window behind the video would render. Was super jank picture in picture, but it worked.

PMTU just doesn't feel reliable to me because of poorly behaved boxes in the middle. The worst offender I've had to deal with was AWS Transit Gateway, which just doesn't bother sending ICMP too big messages. The second worst offender is, IMO (data center and ISP) routers that generate ICMP replies in their CPU, meaning large packets hit a rate limited exception punt path out of the switch ASIC over to the cheapest CPU they could find to put in the box. If too many people are hitting that path at the same time, (maybe) no reply for you.

More rare cases, but really frustrating to debug was when we had an L2 switch in the path with lower MTU than the routers it was joining together. Without an IP level stack, there is no generation of ICMP messages and that thing just ate larger packets. The even stranger case was when there was a Linux box doing forwarding that had segment offload left on. It was taking in several 1500 byte TCP packets from one side, smashing them into ~9000 byte monsters, and then tried to send those over a VPNish network interface that absolutely couldn't handle that. Even if the network in the middle bothered to generate the ICMP too big message, the source would have been thoroughly confused because it never sent anything over 1500.

This made me think of a series of "war room" meetings I had been part of early in my career. Strangely enough, also a defect revealed when the platform was low on memory. This was also the issue where I learned the value of documenting experiments and results once an investigation has taken a non-trivial amount of time. Not just to show management what you are doing, but to keep track of all the things you have already tried rather than spinning in circles.

The war room meetings were full of managers and QA engineers reporting on how many times they reproduced the bug. Their repro was related to triggering a super slow memory leak in the main user UI. I had the utmost respect for the senior QA engineer who actually listened to us when we said we could repro the issue way faster, and didn't need the twice daily reports on manual repro attempts. He took the meetings from his desk, 20 feet away, visible through the glass wall of the room we were all crammed into. I unfortunately didn't have the seniority to do the same.

Since I can't resist telling a good bug story:

The symptom we were seeing is that when the system was low on memory, a process (usually the main user UI, but not always) would get either a SIGILL at a memory location containing a valid CPU instruction, or a floating point divide by zero exception at a code location that didn't contain a floating point instruction. I built a memory pressure tool that would frequently read how much memory was free and would mmap (and dirty) or munmap pages as necessary to hold the system just short of the oom kill threshold. I could repro what the slow memory leak was doing to the system in seconds, rather than wait an hour for the memory leak to do it.

I wanted to learn more about what was going on between code being loaded into memory and then being executed, which lead me to look into the page fault path. I added some tracing that would dump out info about recent page faults after a sigill was sent out. It turns out all of the code that was having these mysterious errors was always _very_ recently loaded into memory. I realized when Linux is low on memory, one of the ways it can get some memory back is to throw out unmodified memory mapped file pages, like the executable pages of libraries and binaries. In the extreme case, the system makes almost no forward progress and spends almost all of its time loading code, briefly executing it, and then throwing it out for another process's code.

I realized there was a useful looking code path in the page fault logic we would never seem to hit. This code path would check if the page was marked as having been modified (and if I recall correctly, also if it was mapped as executable.) If it passed the check, this code would instruct the CPU to flush the data cache in the address range back to the shared L2 cache, and then clear the instruction cache for the range. (The arm processor we were using didn't have any synchronization between the L1 instruction and L1 data cache, so writing out executable content requires extra synchronization, both for the kernel loading code off disk, as well as JIT compilers.) With a little more digging around. I found the kernel's implementation of scatter gather copy would set that bit. However, our SOC vendor, in their infinite wisdom, made a copy of that function that was exactly the same, except that it didn't set the bit in the page table. Of course they used it in their SDIO driver.

bitbake -e <recipe> is super useful for that game. It dumps out a complete history of where all variables were set/changed, and their values along the way. I also use it to do what I call "variable shopping," where I roughly know what path/name content I need, but not what the variable it is in is called.

I spent a lot of time debugging an internal fork of an open source BGP implementation (really old quagga.) The confederation code always struck me as being nothing but weird exceptions to how BGP normally worked. I was happy to never hear a network engineer suggest confederations with a straight face.