This works well for a while. But eventually you get big, and have little to no idea of what is in your downstream. Then every single format change in any event you write must be treated like open heart surgery, because tracing your data dependencies is unreliable.
Sometimes it seems that it's fixable by 'just having a list of people listening', and then you look and all that some of them do is mildly transform your data and pass it along. It doesn't take long before people realize that. 'just logging some events' is making future promises to other teams you don't know about, and people start being terrified of emitting anything.
This is a story I've seen in at least 4 places in my career. Making data available to other people is not any less scary in kafka than it was back in the days where applications shared a giant database, and you'd see yearlong projects to do some mild changes to a data model, which was originally designed in 5 minutes.
As for kafka being easy, It's not quite as hard as some people say, but it's both a pub sub system and a distributed database. When your clusters get large, it definitely isn't easy.
Sometimes it seems that it's fixable by 'just having a list of people listening', and then you look and all that some of them do is mildly transform your data and pass it along. It doesn't take long before people realize that. 'just logging some events' is making future promises to other teams you don't know about, and people start being terrified of emitting anything.
This is a story I've seen in at least 4 places in my career. Making data available to other people is not any less scary in kafka than it was back in the days where applications shared a giant database, and you'd see yearlong projects to do some mild changes to a data model, which was originally designed in 5 minutes.
As for kafka being easy, It's not quite as hard as some people say, but it's both a pub sub system and a distributed database. When your clusters get large, it definitely isn't easy.