I'm trying to build realtime data infrastructure for logging. For ingestion layer, I'm thinking about using Kafka / Logstash for logging ingestion layer and then after that I can store it in any database and I can easily change the store without changing the ingestion layer in the future.
Any experience using logstash or Kafka in production?
Also, additional question is, I concern a lot for missing data case when we're sending data to logstash or kafka using kind of lightweight shipper like filebeat? Any experience for handling missing data at scale?
I'm trying to build realtime data infrastructure for logging. For ingestion layer, I'm thinking about using Kafka / Logstash for logging ingestion layer and then after that I can store it in any database and I can easily change the store without changing the ingestion layer in the future.
Any experience using logstash or Kafka in production?
Also, additional question is, I concern a lot for missing data case when we're sending data to logstash or kafka using kind of lightweight shipper like filebeat? Any experience for handling missing data at scale?