pahae

I set up a fairly large Prom-based architecture which I later on migrated to VictoriaMetrics (VM) so I think I can chime in here.

Both Prom and VM are exceptionally stable in my opinion, even on _very_ large scales. There were times when I had a single (Prom, later VM) and not-overly-large instances scrape 2Mio samples/s without any issues. In addition to fairly spiky query loads.

However, if something does go wrong, the single most impactful difference between VM and Prom is simply the difference in startup time. Prometheus with 2TB of metrics takes _forever_ to start up. We're talking up to 2 hours on SSD while VM just... starts.

Remote Write is a viable alternative in Prometheus and its drop-in replacements. I'm not a massive fan of it myself as I feel the pull-based approach is superior overall but still make heavy use of it.

The pushgateway's documentation itself calls out that there are only very limited cirumstances where it makes sense.

I personally only used it in $old_job and only for batch jobs that could not use the node_exporter's textfile collector. I would not use it again and would even advise against it.

I agree. I think OP has made the mistake of using more than just Grafana for dashboards and perhaps user queries.

I operate a fairly large custom VictoriaMetrics-based Observability platform and have learned early on to only use Grafana as opposed to other Grafana products. Part of the stack used to use Mimir's frontend as caching layer but even that died with Mimir v3.0, now that it can't talk to generic Prometheus APIs anymore (vanilla Prom, VictoriaMetrics, promxy etc.). I went back to Cortex for caching.

Such a custom stack is obviously not for everyone and takes much more time, knowledge and effort to deploy than some helm chart but overall I'd say it did save me some headache. At least when compared to the Google-like deprecation culture Grafana seems to have.

I wish the big providers would offer some sort of trial period where you can evaluate models in a _realistic_ setting yourself (i.e cli tools or IDE integrations). I wouldn't even mind strict limits -- just give me two hours or so of usage and I'd already be happy. Seriously.

My use-case is probably pretty far from the usual tasks: I'm currently implementing a full observability platform based on VictoriaMetrics / Victorialogs + Grafana. It's quite elaborate and has practically no overlap with the usual/cloud solutions you find out there. For example, it uses an authenticated query stack: I use the Grafana oauth token to authenticate queries by injecting matchers via prom-label-proxy and forward that to promxy for fan-out to different datasources (using the label filter to only query some datasources). The IaC stuff is also not mainstream as I'm not using any of the big cloud providers, but the provider I use nonetheless has a terraform provider.

As you can imagine, there's probably not much training data for most of this, so quality of the responses varies widely. From my experience so far Claude (Sonnet 4.5 ) does a _much_ better job than GTP-5 (Codex or normal) with the day-to-day task. Stuff like keeping documentation up to date, spotting inconsistencies, helping me find blind spots in the Alerting rules, etc. It also seems to do better working with provided documentation / links.

I've been using Claude for a couple of weeks now but recently switched to codex after my subscription to Claude ran out. I was really curious after reading a lot of good things about it but I gotta say, so far, I'm not impressed. Compared to Claude it gives wrong answers much more frequently (at least in this domain). The results it produces take much more effort to clean up than Claude's. Probably on a level where I could just invest the time myself. Might be that I do not yet know how to correctly prompt GPT but giving both tools the same prompt, Claude does a better job 90% of the time.

Anyway, I guess this is my long-winded way of saying that the quality of responses "off the beaten track" varies widely and is worth testing several models with. Especially if your work is not 70+% of coding. Even then I guess that many benchmarks have seized being useful by now?

Quadlets are pretty nice but require podman > 4.4 to function properly. Debian 12, for example, still only has podman ~4.3 in its repos.

We used Seafile in $old_job a couple of years back. Still had some rough edges back then (mostly with user management) but the rest was rock solid. Loved working with it, managing and sharing files was pretty easy.

> It is striking that such a backdoor in seemingly safe OSs like Linux and FreeBSD [...]

Not saying these are a 100% secure but you're plying this was _built_ into Linux (and other OSes). It was not. And the PDF makes no such claim either.