Misdicorl

Your place of last resort with kafka is simply to replay the message back to the same kafka topic since you know it's up. In a simple single consumer setup just throw a retry count on the message and increment it to get monitoring/alerting/etc. Multi consumer? Put an enqueue source tag on it and only process the messages tagged for you. This won't scale to infinity but it scales really really far for really really cheap

One way to measure this that isn't moral judgements is the ecological depth of an environment. If one part of the system is destroyed (e.g. a blight on plant A) how devastating to the rest of the system will that be?

One of the hallmarks of human engineered environments is how shallow and fragile they are. Changes, like the reintroduction of wolves, are "good" because they give us deeper and more resilient environments

The story I've heard is that general motors down fall began when they stopped writing loans for other car manufacturers. They used to under write basically every car loan in the US.

I've never looked into how truthful it is, but it smacks of idiotic/arrogant executive tropes so well I almost don't want to discover it's false

I'll throw my hat in the ring for "disagree". Few kinds of work have such clear and unambiguous results as optimization work (its now X% faster!). Few kinds of work have such incredible and detailed tools to guide your hand in finding where to invest your effort (look at this hot loop!).

The fact that sometimes optimization work is tricky or requires some pre-thinking, or is even gasp counter-intuitive is such a hilarious way to say "this is hard". That's just table stakes starting points for so-so-so much work.

Edit: Heck, even deciding whether to prioritize optimization work or feature work is usually a harder problem than the actual optimization work itself.

Sometimes the vibes are wrong, and things go haywire. This is why zero tolerance policies have to be instituted in schools. That doesnt mean the general idea is wrong. Strict adherence to written law will always fail justice. The world is too nuanced and too fractal to handle every edge case well.

Would be really interesting to see what it does with clearly wrong EXIF data

Yeah I'd say kafka is not a great technology if your median and 99ths (or 999ths if volume is large enough) are wildly different which sounds like your situation. I use kafka in contexts where 99ths going awry usually aren't key dependent so I don't have the issues you see.

I tend to prefer other queueing mechanisms in those cases, although I still work hard to make 99ths and medians align as it can still cause issues (especially for monitoring)

Follow on: If you're using kafka to publish messages to multiple consumers, this is even worse as now you're infecting every consumer with data processing issues from every other consumer. Bad juju

I suppose it depends on your message volume. To me, processing 100k messages and then getting a page however long later as the broker (or whatever) falls apart sounds much worse than head of line blocking and seeing the problem directly in my consumer. If I need to not do head of line blocking, I can build whatever failsafe mechanisms I need for the problematic data and defer to some other queueing system (typically, just add an attempt counter and replay the message to the same kafka topic and then if attempts > X, send it off to wherever)

I'd rather debug a worker problem than an infra scaling problem every day of the week and twice on Sundays.

And now imagine how easy dealing with email spam would be if the marginal fiscal cost was not 0 like physical spam. All the technology and tools available and less than 1% of the viable spam surface area

> the goal should be to ensure that anyone who wants to do a thing can, with as few third party requirements as possible.

This is a good starting point, but if you have no barriers then you get abuse problems which is why email is terrible. I remember being horrified in the 90s about attempts to charge 1 cent per email. Now I long for a world where that actually happened.

It wouldn't, but it might preclude using (future) optimizations that forgo those de-optimization hooks?

I suppose serializing the JVM state itself to avoid the cold start problem might take advantage of this?

The past decade is a difficult framing to ask the question in. Notable breakthrough results are usually understood in hindsight and a decade just isn't a lot of time for that context and understanding to develop. Science also doesn't necessarily develop in this way with consistent progress every X timespan. Usually you get lots and lots of breakthroughs all at once as an important paradigm is shattered and a new one is installed. Then observations with tiny differences slowly pile up and a very blurry/messy picture of the problems with the new paradigm takes shape. But none of those things feels like a breakthrough, especially to a layman.

That said: I'll submit the first detection of gravitational waves as two black holes merged together in 2020 as meeting the bar of "notable breakthrough in the last decade".

Ignoring inexperience/incompetence as a reason (which, admittedly, is a likely root cause) domain fuzziness is often a good explanation here. If you aren't extremely familiar with a domain and know the shape of solution you need a-priori all those levels of indirection allow you to keep lots of work "online" while (replacing, refactoring, experimenting) with a particular layer. The intent should be to "find" the right shape with all the indirection in place and then rewrite with a single correct shape without all the indirection. Of course, the rewrite never actually happens =)