As someone who has been on that side of things for 10 years, can we stop building really complicated stuff that requires this pile of shit and dedicated staff to run it.
The whole otel, logging and metrics thing is based around "record everything" and dig through the pile of shit looking for things that might be helpful. Fine until you have 50TB of logs/traces online which is our problem. Oh that and the 50 gig prometheus instance. 99.99999% of all information we record is entirely and utterly useless and costs us money
Make simpler things, log only exceptional things, use the rule of silence.
The whole otel, logging and metrics thing is based around "record everything" and dig through the pile of shit looking for things that might be helpful. Fine until you have 50TB of logs/traces online which is our problem. Oh that and the 50 gig prometheus instance. 99.99999% of all information we record is entirely and utterly useless and costs us money
Make simpler things, log only exceptional things, use the rule of silence.