Darkphibre

Joined 8 years ago

Karma
679
Submissions
223
Member since
Dec 11
Decades of engineering / architectural work. 20 years at Microsoft. 5 years SQL Server Analysis Services. 7 years at Xbox Advanced Technology Group. 10 years at 343 architecting realtime telemetry pipelines used by every game from Gears to Forza to Halo to Minecraft to Solitaire. I'm now a Principal Data Solutions Architect for a billion dollar startup, helping transition the organization to leverage AI and guide our organization on event-driven design for scale.
View on Hacker News

Recent submissions

Comment · 16 days ago
I'm with you there. I'm on expensive plans with everyone because I run out of tokens and rotate/use the models that are best in their niche...

CLI Model Pct Remaining Reset Date claude all models 15% 3d claude Fable 50% 3d grok all models 0% 4d agy Gemini 36% 5d agy Claude and GPT models 0% 5d codex codex 0% 6d

And I also just used two of my three get-another-week-free tokens for codex.

If you've got an agentic swarm, or are working on multiple big projects at once, or have agents parsing data... you burn through tokens fast. The last four days in API would have cost me well over $3k.

Comment · 2 months ago
Comment · 6 months ago
Comment · 6 months ago
Thanks! It was definitely a highlight working on such bleeding-edge technologies with crazy smart people. Pouring over custom PowerPC CPU Errata was divine.

Oh, and we only had (if I recall) 1ms per frame on one core to do all our payload packaging and dequeue messages from the circular buffer. Thats' where the 20k/s hard limit came in... we could have handled SO much more. Our entire message usually landed around 100-150 bytes if I recall, using bitpacked structures.

One thing I didn't anticipate: Memory stomping would result in everyone pointing fingers at our department, because we would inevitably be the ones that would crash (usually with our hardening asserts). We had to start flagging our memory blocks as unwritable when our thread was idle during debug mode, so that offenders would crash when they touched our memory.

Comment · 6 months ago
Great video! Love the concepts discussed, and the tigerstyle mentioned by adityaathealye. This is such a hard problem.

For Halo 4, I architected a system for our telemetry that that had to support subsecond roundtrip user lifetime aggregation. With up to 400ms one-way latencies, that left us with 200ms client + server time to manage lifetime counts. 60ms on the client for batching and network stack delays, and 80ms in the cloud for aggregation after routing delays.

On the client side we leveraged a lot of those TigerStyle ideas (static-allocation, simplified function surface area, explicit limits, strongly asserted). And for the cloud we effectively wrote Diagonal Scaling (Stage 7).

For gamedevs to use this, it had to be as lightweight as possible and introduce no risk of priority inversion or hitching. I wrote a many-reader/many-writer lockless circular queue with performance measured in the microseconds, it was a beast.

I was so proud of our tiny team for pulling off the vision. When we launched, there was a bug in the game: someone left in telemetry in a tight render loop during a cutsceen, so our 20k/s client-side throttles kicked in.

We hit an average of 700k/s on launch day, with peak ingestion throughput of 831k/s (2 weeks to hit a trillion). And we didn't break a sweat.

Partner teams that were trying to provide our reporting capabilities started slowing down, though, haha, and brought it to our attention ("We, uh, can't scale anymore, there's no more servers on the eastcoast data center we can allocate")... so we hit the killswitch on that category of event.

That was another piece I was proud of writing: each instrumentation point did a quick binary mask check for 64 categories and 64 subcategories to see if it should emit... one reason why the instrumentation times were so blazing fast, we had minimal branching based of a hotly-cached variable that would hang around L1 because it was touched so frequently.

Querying that day for insights, though... aggregation queries touching the launch day (that weren't on the per-user hotpath layer that was our primary use-case) would 30x query duration. XD

OH! And I forgot to mention. The tech was impressive enough, that it was quickly adopted as the backbone by every title that Microsoft published, from Gears of War to Solitaire, Forza to Minecraft.

And for comparison: The official, initial Xbox One telemetry stack used pipe-delimited strings, and supported 1-5 transactions/console/second (vs our 20k).

Comment · 8 months ago
I was hesitant about this article being interesting, or a good use of my limited time, but it turns out it was well written and held an interesting insight!

I find it fascinating that, even aware of the importance of the phrase, I tend to gloss over it as one conceptual unit and hardly even register its existence, like the

Comment · 8 months ago
I always loved this question when I played the 'Why' game with my kids: They ask why, and I'd ELI5. Then they'd ask why, and the process continued until I could excitedly say "We don't know for sure!! We think it might be XYZ, but we're still exploring that frontier."
Comment · 3 years ago
Oh! I'm on the east side. I've been attending Brazillian Zouk classes, but I've been interested in swing. Where do you go?

And I completely agree about event organization! It really introduces you to a wider swath of people than your initial search may have turned up.

Way to go!

Comment · 3 years ago
I hear ya! One of my hobbies is to be a somewhat known stage & exotic dancer / dance instructor in VR using 5-11pt tracking (depending on my mood and production quality). The number of "features" that get in the way, and lack of representation for VR dancers and performers has been frustrating, especially as the world moves more and more towards 3-point robot avatars.

Actually had a chance to provide my perspective to a FAANG Research Group, but I was laid off (and they were disbanded).

Comment · 3 years ago
Thanks for the warning, genuinely appreciated!! I actually opted not to read this comment, I enjoy the immersion.
Comment · 3 years ago
I just don't see how the genie is put back in the bottle. Optimizations and new techniques are coming in at a breakneck pace, allowing for models that can run on consumer hardware.
Comment · 3 years ago
Seems to me like the pathogen is just adapting to have a new transport mechanism? Rather than it decomposing or making use of microplastics.

Rather than attaching to seaweed, it can also stick to microplastics and (hopefully) be ingested by marine wildlife. At least as I read the articles.

Comment · 3 years ago
I suppose the argument would be that the key search space is reduced. Statistically speaking, you know the weights of certain binary pairs if the image is not evenly distributed. But I'm guessing that it'd only drop the average search space by... two or three powers of two for most keys?
Comment · 3 years ago
Heh... as someone with prosopagnosia, this wouldn't help too much. But I agree it'd help most people!!
Comment · 3 years ago
The metric of "hauling 40 Tesla Model S vehicles up every sixty minutes." is a strange one. A '93 Honda Accord would be divisible to the minute (60/hr). Or even 52lbs/second... Though I suppose those don't sound as flashy.

Alternatively, I'd be curious how many tesla batteries in raw materials that equated to per hour.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton