AJSDfljff

"So subsecond I would say is a requirement." you do not make any specific point why you came to that conclusion.

You can easily entertain users to show them that the system is doing something in the background without loosing them and if they are collegues who actually need to search, you don't even need to keep them as they have to use your setup.

It will be interesting how long business people overestimate what they can do until the tools get good enough that it is okay again.

Take small companies with eveyrone being business people besides the nerds in the IT department. If you don't have one security aware person, this IT department will write unsecure code without anyone in the company every questiong this every.

You know like 'hidden' api keys in an mobile app everyone can extract, still using plaintext for passwords or md5, having ports open, old operating systems etc. etc.

Whats your usage? The 'around town' triggered me to ask :)

Do they understand the difference between logs and metrics?

Feels like they just log instead of having a separation between logs and metrics.

The question is why would someone need search through TBs of data.

If you are not google cloud and just have your workers ready to stream all data in parallel on x amount of workers in parallel, i would force usefull limitations and for broad searches, i would add a background system.

Start your query, come back later or get streaming results.

On the other hand, if not toooo many people search in parallel constantly and you go with data pods like backblaze, just add a little bit more cpu and memory and use the cpu of the datapods for parallisation. Should still be much cheaper than putting it on s3 / cloud.

I would probably build my own storage pods, keep a day or a week on cloud and move everything over every night.

Yes but audit requirements doesn't mean you need to be able to search everything very fast.

Binance might not have a 24/7 constant load, there might be plenty of time to compact and write audit data away at lower load while leveraging existing infrastructure.

Or extracting audit logging into binary format like protobuff and writing it away highly optimized.

I would question first if the system needs to search with subsecond latency and if the same system needs to be which can handle 10k writes/sec.

Even google cloud and others let you wait for longer search queries. If not business ciritical, you can definitly wait a bit.

And the write system might not need to write it in the endformat. Especially as it also has to handle transformation and filtering.

Nonetheless, as mentioned in my other comment, the interesting details of this is missing.

Would be curious what they are searching exactly.

At this size and cost, aligning what you log should save a lot of money.

Good question. I thought it would be a no brainer to put it on s3 or similiar but thats already way to expensive at 2m/month without api requests.

Backplace storage pods are an initial investment of 5 Million, thats probably the best bet you could do and on that savings level, having 1-3 good people dedicated to this is probably still cheaper.

But you could / should start talking to the big cloud providers to see if they are flexible enough going lower on the price.

I have seen enough companies, including big ones, being absolut shitty in optimizing these types of things. At this level of data, i would optimize everyting including encoding, date format etc.

But i said it in my other comment: the interesting questions are not answered :D

Unfortunate the interesting part is missing.

Its not hard at all to scale to PB. Junk your data based on time, scale horizontally. When you can scale horizontally it doesn't matter how much it is.

Elastic is not something i would use for scaling horizontally basic logs, i would use it for live data which i need live with little latency or if i do constantly a lot of log analysis live again.

Did Binance really needed elastic or did they just start pushing everything into elastic without every looking left and right?

Did they do any log processing and cleanup before?