maxwellpeterson

picoservices

Producers (and all clients) can only see and interact with the gateway worker. DOs aren’t accessible from the outside world, and all communication with them happens through the gateway worker. In the case of DO failure, it’s the gateway worker’s job to reconnect with the failed DO. Depending on the nature of the failure, this reconnection request might be routed to a DO instance located on a different machine from the original, failed instance (in which case a leadership change happened internally). From the client perspective, nothing changes, and it can keep using the same gateway worker connection as before.

Regarding 0.8, that’s right, I wanted to start with something as simple as possible. This project wasn’t intended for serious, practical usage, so spending time on extra features added in later versions didn’t seem useful. I’d imagine that newer protocol versions could be implemented without too much difficulty, but whether this thing is fast enough to ever be practical is another larger question.

Thanks Kenton!!

Yep, all the diagrams were created with excalidraw. The exported “source” for the diagrams is stored in the diagrams.excalidraw file in the root of the repository.

Yep, one of the challenges here is that KafkaJS requires Kafka 0.10+, which is true for several other popular clients. For this broker implementation to be somewhat practical, it would need to be extended to 0.10.0 at least. 0.8.0 was a nice, simple starting point for a “proof of concept” project like this.