javiermaestro

The main issue (as usual in big companies) is the large amount of inter-dependencies with internal systems. Scribe as-is today doesn't make much sense outside Facebook. And yeah, it could be cleaned up, mock the internal services and OSS it. But that's a lot of work, both doing it and maintaining it. And having all those mocks, etc "wouldn't be Scribe as-is either"in terms of how e.g. it scales and so on.

It's really hard to believe that people still write stuff online without doing some basic searching. But here we are...

Here's a couple links to illustrate how RAMPANT knife crime is in the UK:

[1] For the year ending September 2019, there were 221 homicides involving a knife or sharp instrument recorded by the police. This figure is for England and Wales only and it excludes data from Greater Manchester Police (GMP).

[2] Recorded knife crime rose by 7% from just above 41,000 in the year to June 2018 to just above 44,000 in the year to June 2019, knifepoint rapes, robberies and assaults logged by police continued to rise.

I think it's pretty clear that your "rampant" is off BY A LOT. Knife crime in the UK is definitely not something to dismiss but guns (and gun crime and gun accidents) in the US tromp pretty much every other crime statistic in the "modern, civilized" world.

In any case, the data is "whatever needs to be logged".

And it's not "server logs", which is what I'm interpreting from your comment. Scribe transports most data at Facebook to be processed by real-time systems (e.g. Puma, Scuba) and also "batch systems" (data warehouse). So, it's quite a lot, being "the ingestion pipe" for Facebook.

Does this answer your question? :-?

Basically, everything that needs logging and post-processing by both real-time systems (e.g. Puma) and batch processing (e.g. all of the data that's ingested and sent to the data warehouse) goes through Scribe.

(disclaimer: I work in Scribe)

Maybe one day we'll have a version available. In any case, one of the larger parts of the system (LogDevice) is open source :)

(disclaimer: I work in Scribe)

I already replied to metheus on Twitter (2) in a thread where we asked for a way to repro their claims. I found their reply and comments very inappropriate, similar to the comment in here. Arrogant and derogatory to OnGres.

Anyway, I was writing this to note that OnGres has replied to Mongo's reply setting an example of how tech discussions should happen: without derogatory and arrogant comments, open to valid criticism (i.e. with something more than words and numbers that cannot be reproduced) and transparency.