NightMKoder

When you measure latency, you’re measuring it based on requests. So in some bucket if you had a request take 2s and a request take 10s, you would say the average is 6s. This answers the question “how long should I expect a single request to take”.

But the articles point is that to the people - it’s not the number of requests that matters - it’s how often they are waiting for them. The question is “how much time am I sitting here waiting?” In that case the 10s request is 5x worse than a 2s request - it takes 5x of the “time spent”.

So you can change the weighing to 1 / 2 2s 1 / 2 10s to 1 * 2/12 & 1 * 10/12 - that gives us 17% and 83%. And the average there is 9.04s.

The difference is the question. If you think about it as road segments, let’s say you have a group of road segments with different lengths. You can ask the “average length of segment” - or you can ask “if you pick a random point among all the segments, how long is the segment that I landed in?” You’re picking very differently there - the second is proportional to the length!

Technically IMO the blog is slightly off - you want to use “mean residual life”. Ie if I pick a random TIME how long do I have to wait for my request to finish. But it’s reasonably close.

This might speak to the craziness of the gstreamer plugin ecosystem - good/bad/ugly might be a fun maintenance mnemonic, but `voaacenc` is actually in `bad` - not `ugly`. Most plugins you'd want to use aren't in `good`. How are you supposed to actually use "well supported plugins" with gstreamer? Is it just to not use gstreamer at all?

Facebook’s wormhole seems like a better approach here - just tailing the MySQL bin log gets you commit safety for messages without running into this kind of locking behavior.

Agreed - concretely with-redefs forces single threaded test execution. So eg you can’t use the eftest multithreaded mode.

Explicit dynamic bindings are better if you need something like this since those are thread local.

Usually the controversial decision for Clojure code highlighting is rainbow parens. This color scheme is horrific and unreadable (on mobile at least).

I don’t know about tires, but for brakes we already know how to make lower dust brakes - use drum brakes instead of disc brakes. The friction material is enclosed on drum brakes so much less of it just flies away.

So I might be putting words in your mouth, so please correct me if this is wrong. It seems like you don’t actually control the SIGTERM handler code. Otherwise you could just write something like:

  sigterm_handler() {
    make_healthcheck_fail();
    sleep(20);
    stop_web_server();
    exit(0);
  }

Technically the server shutdown at the end doesn’t even need to be graceful in this case.

Why the additional SUGUSR1 vs just doing those (failing health, sleeping) on SIGTERM?

Yes sorry for not qualifying - that’s right. IMO the liveness check is only rarely useful - but I've not really run any bleeding edge services on kube. I assume it’s more useful if you actually working on dangerous code - locking, threading, etc. I’ve mostly only run web apps.

This is actually a fascinatingly complex problem. Some notes about the article: * The 20s delay before shutdown is called “lame duck mode.” As implemented it’s close to good, but not perfect. * When in lame duck mode you should fail the pod’s health check. That way you don’t rely on the ALB controller to remove your pod. Your pod is still serving other requests, but gracefully asking everyone to forget about it. * Make an effort to close http keep-alive connections. This is more important if you’re running another proxy that won’t listen to the health checks above (eg AWS -> Node -> kube-proxy -> pod). Note that you can only do that when a request comes in - but it’s as simple as a Connection: close header on the response. * On a fun note, the new-ish kubernetes graceful node shutdown feature won’t remove your pod readiness when shutting down.