whost49
Joined 12 years ago
Karma
130
Submissions
18
Member since
Jun 30
Recent submissions
Comment · 3 years ago
I hope that I would be able to finish this quickly because I want to focus on my other project (https://github.com/yousseb/atfal-ai & https://github.com/yousseb/atfal-site) which is an effort to track missing children/people and use AI to help find them. I love meld, but as a person who has lost his brother in a way that he couldn't come back - and I keep losing family in Gaza -, I want to help reunite families who still have the chance to be reunited. Everyone is welcome to help.
Comment · 10 years ago
Drew, thanks for sharing your settings. It looks like you are using a 2GB RAM system.
How many unicorn workers are you using? For 2GB systems, I'd recommend at most 2 rather than the default 3. Some discussion about this is here: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1279
We'll be doing more work in these coming months to profile and reduce the memory usage needed by GitLab so that all your tools can run comfortably in the 2GB range.
Comment · 10 years ago
For your use case, I think Fluentd may work fine. LogZoom currently deals with structured JSON log data received from hundreds of hosts around the world. It could be modified to handle arbitrary logs (and wrap a structure around it) and integrate with Docker, but that was not the goal here.
Comment · 10 years ago
Thanks for that information. There's no doubt Logstash can do a lot, and it sounds like with the multiple pipeline feature Logstash will make it easier to do what we wanted to do in a single process.
In the past, we've also been burned by many Big Data solutions running out of heap space that adding more processes that relied on tuning JVM parameters again did not appeal to us.
Comment · 10 years ago
> If you're around I'd love to hear specifically what you mean by this. Internally Logstash is very thread friendly, we only recommend multiple processes when you want either greater isolation or greater fault tolerance.
Right, we considered using multiple Logstash processes, but we really didn't want to run three instances of Logstash requiring three relatively heavyweight Java VMs. The total memory consumption of a single VM running Logstash is higher than running three different instances of LogZoom.
We looked at the Filebeat Redis output as well. First, it didn't seem to support encryption or client authentication out of the box. But what we really wanted was a way to make Logstash duplicate the data into two independent queues so that Elasticsearch and S3 outputs could work independently.
Comment · 10 years ago
Heka looks good and does a lot more. It doesn't appear to support Redis and S3 out of the box, so we would have probably had to evaluate, learn, and change the third-party plugins had we known about Heka beforehand.
Comment · 10 years ago
No, we did not consider Gollum--it definitely looks like a possible solution and one we might have considered. I think the name of the project makes it hard to find, unfortunately.
Comment · 11 years ago
What does Brubeck do that the collectd statsd plugin can't?
Comment · 11 years ago
Is there a reason why you didn't just use collectd, which is written in C and has lots of plugins, including statsd, InfluxDB, Graphite, etc.?