compumike
Joined 17 years ago
Karma
4,426
Submissions
594
Member since
Oct 6
Recent submissions
Comment · 1 day ago
Yes it’s a real effect. I’ve just hardcoded offsets on various jobs:
* * * * * sleep 17 && …
That way I still maintain a per-job 60-second cadence consistently (which random would not), but don’t do it right on the top of the minute.
Comment · 2 days ago
Try:
sudo ss -lnp -A mptcp
Comment · 2 months ago
Not PCB layout, but I've been building CircuitLab Assistant (https://www.circuitlab.com/), a chat window 100% integrated with our schematic editor and simulation engine. Can build / modify a schematic, troubleshoot, answer questions, configure simulations, etc.
If anyone wants to try an early beta, reply with your CircuitLab username and I'll get you set up. We know have a lot more to do, but would love some feedback on this early version!
Comment · 4 months ago
I don’t see any other impact-narrowing criteria.
The default kubelet `syncFrequency` is 1 minute (per Pod). (There can be additional event-driven ones, but this is a floor.) Back of the envelope: 30 Pods * 1440 * 7 = 300k calls to startPodSync in a week.
I’d guess a lot of production clusters don’t run the latest release of Kubernetes, and 1.36 was only released in late April, less than two months before I reported the issue.
Given that the issue shows up as linear memory growth on the order of maybe 1 GiB/month per node, there really is a time-based discovery process here. (Could be artificially accelerated for a CI test, I imagine!) And any restarted nodes or clusters that upgraded point releases would restart that clock.
The only unusual contributing cause that made the issue more visible to me was trying to run everything on a tiny 2 GiB RAM node!
Comment · 4 months ago
It does seem like a serious tripping point.
I took a quick look at "containedctx" and it seems like for this case, it would almost be backwards: it would flag the (not-memory-leaking) struct-stored "status.ctx", but wouldn't flag when there is a stored "status.cancelFn" only (which resulted in the memory leak).
Could Go implement a runtime leaked-context detector, like the data race detector? https://go.dev/doc/articles/race_detector
Comment · 4 months ago
What kind of health checks? In my case, the kubelet process was staying alive and responsive to queries, I believe due to:
# cat /proc/$(pgrep kubelet)/oom_score_adj
-999
(from OOMScoreAdjust=-999 in /etc/systemd/system/kubelet.service)
Comment · 4 months ago
Thanks. I know there's a `go vet` tool that's run as part of Kubernetes CI, and one of its checks is:
lostcancel: check cancel func returned by context.WithCancel is called
Comment · 4 months ago
Author here! If you're running a Kubernetes cluster, I recommend you check `kubectl version` and see if you're running "Server Version: v1.36.[0,1,2]". If so, you may want to use the one-liner at the end of the article to check your "process_resident_memory_bytes" on each node, and consider restarting kubelet as a temporary workaround to tame the memory leak until v1.36.3 is released.
Comment · 4 months ago
Here are some N-year rolling total inflation charts to put this datapoint in a longer-term perspective: https://totalrealreturns.com/inflation . Zooming out always smooths the noise.
Comment · 5 months ago
> Redis (and memcache) are memory caches and should be treated like that
If you haven't come across Kvrocks yet, it may be worth a look: https://github.com/apache/kvrocks https://kvrocks.apache.org/ . It's a database with a Redis-compatible wire protocol, but the database is stored on disk. This means your working set is not limited by RAM and can be a few orders of magnitude larger! On modern SSDs this is still very fast. I think it improves the durability story as well. But the big win is the orders of magnitude larger database space.
As I've been improving my side project https://totalrealreturns.com/ recently I've ended up using both Redis and Kvrocks together. Redis is great for small global state that needs to be super fast. Kvrocks is great for larger bulk data storage (large precomputed datasets), but also supports a lot of the Redis data structures as well as Lua scripts.
Comment · 5 months ago
Re: "page for all 500s": there's a world of difference between "page me with a critical alert at 3am" and "notify me on Monday morning when my normal workday starts". At the extremes:
If my DB health check endpoint is returning 500s for N consecutive checks over M minutes, yeah, please wake me up at 3am!
If one user hit a weird edge case in form validation and got a one-off 500, please don't! We can fix that on Monday.
Not always easy to distinguish those clearly or configure those business hours rules, but for my team at https://heyoncall.com/ that is the goal -- otherwise your team burns out fast. Waking up someone at 3am has a real cost, so you better be sure it's worth it.
Comment · 5 months ago
You can cheaply and readably give a lot of clues to both agents and humans with some assertions at the start of a method:
raise ArgumentError.new("...") unless ...
I've also had good results using agents to write Crystal https://crystal-lang.org/ which is Ruby-like but does have the static types and produces blazing fast static binaries. Might be a sweet spot for coding agents if you're building some backend services. But I'd still pick Ruby on Rails for a new full stack project.
Comment · 5 months ago
Check out this map: https://api.phillymesh.net/map for live data from the Philadelphia area.
The edges drawn are between nodes that have been able to hear each other in the last 24 hours, based on observed traceroute packets.
(Even then, it’s only a subset of the actually-connected nodes: the map only shows nodes that have published their position on the public channel, and have set a flag that their data is okay to uplink to a server over MQTT.)
Comment · 6 months ago
It can be dangerous to backfeed (which is why you're supposed to have an interlock for a generator inlet, ensuring utility power is disconnected). But:
1. These grid tie inverters are designed and tested to shut off completely if there's no grid power. (This is a big design tradeoff: it means they don't provide any power during a grid power outage, even if it's very sunny out.)
2. Even if I had a beefy generator that was unsafely backfeeding my house while the utility power was still connected, the generator would be trying to power not just my house, but all my neighbors too! And the circuit breaker and/or inverter on the generator would likely trip and shut down almost instantly.
There's still a possible risk from #2, especially if the downed wire being repaired is relatively local (i.e. your house only).
But I think #1 and #2 mitigate this risk very well.
Co-founder of CircuitLab (YC W13): https://www.circuitlab.com/
Formerly I ran Growth at Triplebyte (YC S15): https://triplebyte.com/