diurnalist
Joined 14 years ago
Joined 14 years ago
Unfortunately I think the real sentiment here is "here's the thing I don't want to admit: I don't know how to use AI to actually do that." It's not at all clear to me that nobody knows how to utilize AI practically. It's more clear that you have to be working in an environment that values developing this capability enough to pay a lot for tokens and let employees be free to explore, within reason. Or, you have to be willing to spend a lot of free time and cash doing that yourself.
Some will say that AI can solve this, as is the fashion. But so far the things people are interested in doing with the new potential for synthesizing and translating animal sounds does not inspire any confidence[0].
[0]: https://www.bloomberg.com/news/articles/2026-09-04/ai-s-next...
[1]: https://www.boggsfiles.com/x-files-scripts-by-season/season-... [2]: https://drive.google.com/file/d/1BQbfRmeq3FsajjYcEuqDHv0kf2B...
IMO, with the current tech, it entirely depends on what data you're talking about.
For metrics and traces, I would use the OTel collector personally. You will have much more flexibility and it's pretty easy to write custom processors in Go. Support for traces is quite mature and metrics isn't far off. We've been running collectors for production scale of metric and trace ingest for the past couple of years, on the order of 1m events/sec (metric datapoints or spans). You mentioned low volume so that's less important, but I just wanted to mention in case others find this comment.
Logs are a bit different. We looked in to this in the past year. Vector has emerging support for OTLP but it's pretty early. Still, I bet it's pretty straightforward if your backend can ingest via OTLP. Our main concern with running the otel-collector as the log ingest agent was around throughput/performance. Vector is battle-tested, otel is still a bit early in this space. I imagine over time the gap will be closed but I would probably still reach for Vector for this use-case for higher scale. That said, YMMV and as with any technical decision, empirical data and benchmarking on your workloads will be the best way to determine the tradeoffs.
For your scale you could probably get away with an OTel collector daemonset and maybe a deployment with the Target Allocator (to allocate Prometheus scrapes) and call it a day :)
Eventually I think there was some work to move to GraphQL, which can solve some of the same problems. But GraphQL is a technology, and BFF is more of a pattern. There is a later reflection on that blog that makes this distinction, which I only read today.[1] It makes another observation that I kind of forgot about, because it was hiding right in front of my face as a worker there:
"The defining characteristic of a BFF is that the API used by a client application is part of said application, owned by the same team that owns it, and it is not meant to be used by any other applications or clients."
The ownership model is indeed a big deal. In practice, it helped in many ways to have a sort of intermediate layer between the client applications and the rest of the architecture. For example, in the SoundCloud web application, when you load a page of playlists, only the first 5 tracks in the playlist are visible to the end-user. So the web BFF application had special logic to only partially load all the track metadata past track 5, which had significant impact on scalability and latency, especially when rendering a lot of playlists that had lots of tracks!
[0]: https://philcalcado.com/2015/09/18/the_back_end_for_front_en... [1]: https://philcalcado.com/2019/07/12/some_thoughts_graphql_bff...
[0]: https://github.com/opentracing/specification/issues/142
It is built on a plugin architecture, so plugins are certainly a viable option and this is documented in more detail here[0]. In general I have found the Keycloak docs thorough and well-written. When I operated Keycloak I built a few plugins to solve specific needs/assumptions we had around IdP when migrating to Keycloak from a bespoke solution.
Re: your second point, the docs also describe this in detail[1]. Having the realm data exist in a simple form that can be exported/imported was very useful. However, I would have liked if they thought more about how to do live backup/restore; perhaps that is easier now than it was at the time.
[0]: https://www.keycloak.org/docs/latest/server_development/inde... [1]: https://www.keycloak.org/server/importExport#_importing_a_re...
I find it interesting how this ties in to the "productivity paradox."[0] The idea the author seems to be getting at--that software accelerates the creation of ever more elaborate solutions (often to problems created by prior iterations of software), and in the process leaves a wake of complexity that frustrates and baffles the society it was supposed to serve--is something I'd like to read more about.