hazard

> I expect it to cost somewhere in the neighborhood of an experienced dev because that’s the alternative

But there's two ways this could be true: Frontier models costs rise, or (more likely) experience dev costs fall

Pangram check on the article: 94% of this text is AI

A key thing that's driving this is the fact that, currently, Gemini is kind of shit compared to the real frontier models from OpenAI and Anthropic. Add on to that the need for Google to serve an absolute insane number of these summaries and have low latency for each one - this means no real thinking time and a comparatively small models.

So it's partially AI, sure, but it's also because you're getting the summaries from what is an extremely stupid AI compared to the "good stuff" that's out there.

I worked very briefly at Baseten, and I can say that it was a perpetual annoyance (from an engineering perspective) that customers would complain about issues with their models but we couldn't actually see the inputs/outputs. I don't know about the other providers, but at Baseten they literally weren't stored anywhere.

In a moment of weakness, I joined a high-IQ group (Triple Nine Society). The whole mailing list was just people having the same political arguments that were on the rest of social media, just with a bigger vocabulary.

Put an airtag into literally every bag you check. Helps you figure out not just whether they made it on the flight, but also if they ended up at some baggage claim on the other end of the airport instead of the one they were supposed to be at

your LLM's temperature is set too high

I recommend reading some foundational materials on federated learning and distributed inference. The fundamental issues are (1) compute (2) latency and (3) security.

On the compute side, you can't load most individual experts into even high-end consumer-grade cards (e.g. RTX 5090) because the weights are simply too big.

On the latency side, you need to ship the activations and wait for the somewhat underpowered cards to actually do the matrix multiplications, then send the results back. This means each token takes hundreds of milliseconds or even more, which is borderline unusable.

Finally if you're running a P2P network, the only way to verify that peers aren't lying to you is to run duplicate calculations - I don't see how SHA-256 or signed model state help here unless I'm missing something (What are you calculating the hash of? What exactly is signing attesting to?).

This is one of the few places where borrowing ideas from cryptocurrencies actually makes sense - you could imagine a system where peers place bonds and forfeit them if they lie about calculations. You can look at projects like Bittensor and others for how these kind of things are currently implemented.

Didn't sfcompute launch with a vision of something like this and then cancel it because CFTC decided it was running an unregulated futures exchange? How are you getting around regulatory issues like this?

>if they’re going to also impose a slight delay between when posts show up in the API vs when they’re posted on the website. help

Yes, they say it will be a few milliseconds. This service is squarely targeted at algo trading firms.

After being in the workforce for decades, this whole issue is just so incomprehensible to me.

I went an ungrad school that was top-5 in engineering. But my experience - and in the experience of other people I've talked to - formal undergrad education was, and always has been, a farce. At best, you learn through working on projects that are meaningful to you and learn "how to be an adult" (and later, you learn how to manage the enormous financial debt you acquired). But more typically, it's pure credentialism - no one cares what your grades were, only what school you graduated from.

The amount of actual learning that goes on from classes is minimal, but somehow we can't shift the overton window away from this silly game of grades that don't measure anything meaningful.

After graduating, I've was asked about my grades exactly twice in my life -- once when I applied to a master's program, and at one job interview (the company had a policy of asking about GPA for anyone who graduated less than 10 years ago).

I'm pro-education but anti-school, and all this nonsense makes me this way even more.

This looks great, but I'd really like to see associated exercises (and solutions) to make it useful for self-study

> Some engineers, given a fixed token budget, generate exponentially more (and better) output. Other engineers waste their tokens. The variance is enormous, and unlike most performance variance, it is now directly measurable. HR has never had a clearer signal of leverage.

This doesn't make sense to me. "A clearer signal of leverage" implies an objective way to measure software engineering output, which has been the white whale of engineering management for the last 50 years.