_5csa

Not really a public plea when it is behind a signup wall.

Flowers for Algernon by Daniel Keyes.

Read it first when I was 16, at the recommendation of my literature teacher in high school. For the past... - darn, this was long ago! - ...more than 20 years, I've read it at least once a year, and intend to do that for as long as I can read.

That book changed me, like nothing else since.

You know what would be amazing? Combining this with logic programming.

(run 100 [box] (fresh [x y z] (randomo x) (randomo y) (randomo z) (locationo box x y z)))

(Overly simplified and terrible DSL for it, but you get the idea!)

Is there anything like blessed + blessed-contrib for either of those? It'd be fairly trivial to write the bulk of potential-happiness in Java, if there was a display library like blessed + blessed-contrib in the JS world.

Author here. I'm running this in "production" (kinda, on laptop, looking at data from my own servers), and works for me. There are some annoying shortcomings, though, which I will fix, eventually.

Among them is error reporting and handling: there's pretty much none. If the network dies, potential-happiness will not be too happy: it will cry and crash.

Bug reports, feature requests and whatnot are of course appreciated. I wrote this mostly for my own use, so it isn't exactly friendly just yet. (That, and my JavaScript is terrible.)

> you're saying that yours is unconditionally better

I don't think I'm saying that. The article presents two setups and a few related use cases, where I believe binary log storage is superior.

> With the services you run, you might be able to dictate that the log formats are restrictive enough that writing a parser for each one isn't a problematic overhead.

I don't need to dictate all log formats. If I can't parse one, I'll just store it as-is, with some meta-data (timestamp, origin host, and so on). My processed logs do not need to be completely uniform. As long as they have a few common keys, I can work with them.

For some apps or groups of apps, I can create special parsers, but I don't necessarily need that from day one. If I'm ok with only new logs being parsed according to the new rules (and most often, I am), I can add new rules anytime.

> Parsing up-front, assuming you know what data you can safely throw away, might appear to some as a premature optimisation.

>> We have well documented tools and workflows, so anyone new to the system can catch up and start working with the logs within minutes. > It sounds like this is something which could be usefully open-sourced, to show how it's done.

LogStash is a reasonable starting point. Our solution has a lot of common with it, at least on the idea level.

> Pre-parsed binary logs in a locked-down environment might be as flexible as freeform text, but I'd need to see a running system to properly judge.

Only our storage is binary. That is all the article is talking about. Within that binary blob, there are many traces of freeform text, mostly in the MESSAGE keys of application logs which we care less about (and thus, parse no further than basic syslog parsing). You still have the flexibility of freeform text, even if you store it in a binary storage format.

I assume this comment is related to the journal. The article is not.

But, to reply: yes, many organisations have fully functional, well-debugged logging infrastructures. A lot of them also use binary log storage, and have been for over a decade, and are more than satisfied with the solution.

Both rsyslog and syslog-ng have been able to assist with setting such a thing up for about a decade now.

> Where are the rsyslogd and syslog-ng competitors to systemd's journald? Where is the interoperability? Where is the smooth, useful upgrade mechanism?

The journal has a syslog forwarded, but both rsyslog and syslog-ng can read directly from the journal. Interoperability was there from day one. Smooth upgrade mechanism took a while to iron out, but it's there now, too.

I experimented with that, and heard others toying with the idea too. There are even products out there that do something similar.

Agreed. But if I keep my logs in a database, I may aswell use the database to query my logs, instead of grepping in them.

(And voila, you have binary log storage.)