Joined 17 years ago
It used to be that distributed systems were a big trade off. They were operationally complex, they had limited apis (NoSQL), but they scaled. The best solution used to be to build things using a non-scalable but easy to use and run system, and then re-write it later if it needed to scale (often in a big hurry).
This is just not the case any more, though. Why? Two reasons:
1. We’ve gotten much better at distributed systems so the apis aren’t nearly as limited. It’s no longer that you either choose hard-to-use things like Hadoop/NoSQL or elegant but unscalable single-server databases. You can have both good abstractions and scale.
2. The cloud makes it possible to get systems as a service so there should be way less ops than running a single node system yourself
In the case of Kafka, I’m super biased as I’m one of the original authors, but I think the abstractions Kafka gives, stream processing capabilities, connectors, etc are just way better than a lot of the traditional solutions. Using something worse until you “need” Kafka might make sense on premise, but not in the cloud.
Confluent offers a Kafka service which is fully managed so you don’t do any of the upgrades, security patches, midnight pages, etc you just use the APIs. This is super affordable for the kind of simple apps the article describes. The price varies by cloud, but e.g. on GCP it starts at $0.11/GB for reads and $0.10/GB stored. That is a lot cheaper than using a single node system and then rebuilding everything if you need to scale, but not only that, it is also lower operational overhead (effective none) and a better interface/abstraction.
I think this isn’t unique to Kafka, either. There are great managed systems that are built to scale for most of the kinds of data systems you would use—-CockroachDB, Spanner, Aurora, Snowflake, Elasticsearch, Bigquery, etc.
Basically, you can have nice things now, just like the big tech companies.
The point about effective managers need to stay out of the details and don't solve their team's problems can't be overstated.
1. We aren't trying to get cloud providers to license our proprietary features. We run a cloud service of our software.
2. The book analogy is not very accurate. We have an FAQ here the helps clarify interpretation. The limitations it places are extraordinarily small, 99.9999% of users are completely unimpacted, it really only impacts companies wanting to offer, say, KSQL-as-a-service. https://www.confluent.io/confluent-community-license-faq
3. We aren't trying to "co-opt" the community or open terminology. We actually tried super hard both in the license and in the blog post to be honest and upfront. Whatever else you think you have to agree that Confluent's license is _exceptionally_ permissive and the software has a pretty great community of users. How do you describe a license that let's you run, modify, fork, and redistribute the code and do virtually anything other than offer a competing SaaS offering of the product?
4. Bryan Cantrill is an amazing engineer, but, well, as a lawyer, I think ours are probably better. We're quite confident in the enforceability, but it's a bit ironic because I remember this being the FUD around GPL that it was "totally unenforceable".
5. The "open source companies are all failing"-meme isn't factually correct. Many open source companies are actually doing quite well. MongoDB has gone up in value about 3x over the last year, Elastic was the breakout IPO of the year. There are a handful of other really strong businesses a year or so behind, including Confluent. An open source project is not in-and-of-itself a business model, but it is, just empirically, a big part of some of the recent successes in the infrastructure space. Probably worth noting that the reverse is true too: if you look at some of the really cool up-and-coming open source platform data technologies, a lot of them have the support of a company behind them. Of course there are plenty of sucky open source companies, but that is true of every category of startup.
6. I agree that it is silly to moralize about the behavior of the cloud providers. They are following their economic interest. The point is that this behavior does undermine the cycle of investment in some of the more promising hard tech open source projects and to try to change this dynamic.
7. This article has a bit of a tone of "Son, new things aren't possible, trust me, I tried them and have the scars to prove it". I have huge respect for Bryan, and I know that to some extent that is his schtick as a public personality, but I'm not sure that attitude is most likely to lead to improvement. I don't think the current crop of licenses was handed down from the mountain on Stone Tablets by our elders to be revered and not questioned. I think CockroachDB, Elastic, MongoDB, and Confluent are building really innovating technology platforms and building pretty cool companies to help fund that. I don't think we need dogma. And I still don't say "GNU/Linux".
Short answer: 1. AGPL doesn't actually solve this problem. This is why MongoDB just moved away from AGPL. 2. AGPL is quite restrictive for people who want to use the software in proprietary applications but don't want to open source their own code. This is a really large proportion of usage for us and we don't want to restrict people in that way.
The other reason we don't want to use AGPL is that it can be quite aggressive in making you open source your own code if you want to embed our code in a proprietary application. We actually think building proprietary applications is a fine use of our software, and is one of the major use cases for this kind of infrastructure. This is addressed in more depth along with many other questions in this FAQ: https://www.confluent.io/confluent-community-license-faq
