Right, but this isn't an argument about log formats. You're making a bigger argument about workflows, and you're saying that yours is unconditionally better.
In your environment, it might be most appropriate to put in the up-front investment to totally control all your log formats.
Within your context, you get to define a lowest common denominator which isn't text, and it sounds like that makes sense. With the services
you run, you might be able to dictate that the log formats are restrictive enough that writing a parser for each one isn't a problematic overhead.
To get the benefits you're claiming, the storage format of your logs is actually irrelevant. If you're going to have an environment where you have to exert that much control over the output of your applications, when you parse the logs doesn't matter. You could do your parsing with grep and awk as the very last step before the user sees results, and you'd see the same benefits. Parsing up-front, assuming you know what data you can safely throw away, might appear to some as a premature optimisation.
> We have well documented tools and workflows, so anyone new to the system can catch up and start working with the logs within minutes.
It sounds like this is something which could be usefully open-sourced, to show how it's done.
> Our lowest common denominator is not text, because we control the environment, and we can raise the bar. Being able to do that is - I believe - important for any admin.
It's a question of what you choose to optimise for. Pre-parsed binary logs in a locked-down environment might be as flexible as freeform text, but I'd need to see a running system to properly judge.
To get the benefits you're claiming, the storage format of your logs is actually irrelevant. If you're going to have an environment where you have to exert that much control over the output of your applications, when you parse the logs doesn't matter. You could do your parsing with grep and awk as the very last step before the user sees results, and you'd see the same benefits. Parsing up-front, assuming you know what data you can safely throw away, might appear to some as a premature optimisation.
> We have well documented tools and workflows, so anyone new to the system can catch up and start working with the logs within minutes.
It sounds like this is something which could be usefully open-sourced, to show how it's done.
> Our lowest common denominator is not text, because we control the environment, and we can raise the bar. Being able to do that is - I believe - important for any admin.
It's a question of what you choose to optimise for. Pre-parsed binary logs in a locked-down environment might be as flexible as freeform text, but I'd need to see a running system to properly judge.