kiyoto

While I'm happy to hear about a great success story of a great piece of open source software, Elasticsearch has done a great disservice by making application developers lazy about learning the ins and outs of various analytical/transactional/storage backend systems.

Echoing other commenters, Elasticsearch is hardly the best tool for many kinds of analytics. In fact, it is strictly not a good tool for several use cases. For starters:

1. It's not good at joining two or more data sources

2. It's not good at complex analytical processing like window functions (for example to calculating session length based on the deltas of consecutive timestamps partitioned by user_id and ordered by time).

Of course, it's also good at many things like simple filtering and aggregation against "real-time" data. Being in-memory really helps with performance, and with right tools, it's horizontally scalable. Elastic's commercial support is also not to be discounted.

However, as an old OLAP fart who spent years optimizing KDB+ queries, I am deeply concerned about the willful ignorance of data processing systems that I see among Elasticsearch fans. Just take my word for it and study Postgres (with c_store extension) and other real databases, in-memory or otherwise, open-source or proprietary, so that you won't be shooting yourself (or future co-workers) in the foot, trying to shoehorn Elasticsearch and its ilk into suboptimal workloads (To be fair, I see a similar tendency among Splunk zealots).

What surprised me a bit is that pg decided to use the word "bias" without any clarification, considering his background in computer science and AI.

Anyway, I think pg's whole argument is rather moot because the three assumptions that he states are incredibly difficult to measure (Part of the reason why it is very difficult to argue for or against affirmative actions without coming across as "biased").

This reminded me of an interview of David Foster Wallace

"...here’s this fundamental difference that comes up in freshman comp and haunts you all the way through teaching undergrads:　there is a fundamental difference between expressive writing and communicative writing. One of the biggest problems in terms of learning to write, or teaching anybody to write, is getting it in your nerve endings that the reader cannot read your mind. That what you say isn’t interesting simply because you, yourself, say it. Whether that translates to a feeling of obligation to the reader I don’t know, but we’ve all probably sat next to people at dinner or on public transport who are producing communication signals but it’s not communicative expression. It’s expressive expression, right? And actually it’s in conversation that you can feel most vividly how alienating and unpleasant it is to feel as if someone is going through all the motions of communicating with you but in actual fact you don’t even need to be there at all."

"Conversations with David Foster Wallace" (Literary Conversations Series, page 113

A big thing that pg seems to be unaware is that most of us are expressive, not communicative, when we talk. Stylistically, it's true that plain English is the way to go. However, the deeper problem lies in our (in)ability to communicate our thoughts, in writing or in speech.

I actually began using Acme a couple of months ago: it's kind of interesting how much I have accustomed to mouse-driven interaction and lack of syntax highlighting. Humans indeed are creatures of habit.

You are preaching to the choir here. Fluentd (as a proxy for my logging-related beliefs) pushed Docker to have logging long before the logging driver, and now it is one of the officially supported logging drivers.

What you don't seem to realize is that the cost of centralized logging is not always worth it. Machines are ephemeral and so are many application related problems. It's one thing to counter the OP saying that centralized logging has merits (and I believe the OP agrees with that statement) and another to say centralized logging is always a must.

I think you are conflating between the needs of centralizing logs and having them around at all. The OP is saying that always centralizing them might not always make sense, and I tent to agree (and I say this as someone who maintains a popular open source log collector)

If I were to play the devil's advocate, the real needs for raw log data in a centralized location is for folks outside of Ops: data analysts and data scientists.

I think there are two kinds of logging that's conflated into two in the industry: logging for devops and logging for analytics.

For logging for devops, I 100% agree with you. Looking at application metrics rather than raw logs is far more productive, and the raw logs should only be consulted after you have triaged the situation based on the metrics monitored.

However, there is another kind of logging, and that's for data science and analytics. Here, it's hugely helpful to have centralized logging. Hell, it is a must. The last thing you want is to have data scientists with a shaky Linux knowledge to ssh into your prod machines. At the same time, logs are the best source of customer behavior data to inform product insights, etc. By centralizing these logs and making them available on S3 or HDFS or something, you can point them there and have everyone win.

Among Fluentd users, we definitely see both camps. As a matter of fact, one of the reasons that I think people like Fluentd is that because it enables both monitoring and log aggregation within it.