spetz
Joined 10 years ago
Karma
248
Submissions
80
Member since
Apr 8
Recent submissions
Comment · 12 days ago
Part 3 out of 3 of the blog series about our VSR implementation, this time focusing on the deterministic simulation testing (DST).
Comment · 2 months ago
Sort of, but anyone familiar with Kafka should feel like at home, although there a few subtle differences when it comes to the transport protocol, hierarchy etc.
And VSR clustering is hopefully coming this week!
Comment · 2 months ago
> I am a passenger and I ride and I ride
Comment · 2 months ago
There's an open PR to implement sink + source for MQTT
https://github.com/apache/iggy/issues/3385
Also, since we've been asked about this a lot, there's a high chance, that we might implement MQTT Gateway, just like we currently do for Kafka (proxy on top, as a separate runtime).
Comment · 2 months ago
Awesome! You can find mine too in one of the older blog posts, hence the name :)
Comment · 2 months ago
It's more like Kafka or Pulsar in terms of being the message streaming infrastructure (so an append-only log, not the message broker like, e.g., raw RabbitMQ). I think that the main docs page should give you a good understanding of how the data is stored/organized https://iggy.apache.org/docs/ - for example, on top of the topics, we also have "stream" which is just an extra hierarchy that can be used for something like multi-tenancy isolation or anything else depending on the use case. And yes, there's a built-in RBAC (read/manage particular streams, topics, servers, etc.). As for protocols, we have native support for TCP, QUIC, WebSocket (these 3 are stateful binary protocols), and HTTP as well.
There are a bunch of things that sum up to the overall performance gains - using Rust, thread-per-core architecture + shared-nothing (think of Seastar), custom zero-copy serialization, io_uring for disk * network I/O, and VSR-based consensus (inspired by TigerBeetle) - we simply build all this stuff from the ground up to make the most out of modern hardware and Kernel features, with a custom protocol on top (but the Kafka proxy/gateway is also WiP). And there are also connectors (sink, source), benchmarking runtime, CLI and so on.
Comment · 2 months ago
Apache Iggy™ has officially graduated from the Apache Incubator and is now an Apache Software Foundation Top-Level Project (TLP)
It’s been quite a journey, from a small Rust message-streaming experiment in 2023 to an independent Apache project with a growing community. And its biggest feature yet, being the Viewstamped Replication Revisited (VSR) clustering, is coming soon.
Comment · 11 months ago
Thank you for the mention! BTW, we're currently working on VSR (Viewstamped Replication) to provide the proper clustering :)
Comment · 11 months ago
Thank you, and yes, 100% agreed about Krishna's work! :)
Comment · 11 months ago
Thanks! Actually, the mentioned PR was just an internal merge; the one to the master branch is this one https://github.com/apache/iggy/pull/2299 which happened last week :)
Sans-IO is for clients only, and we'd like to introduce it one day to SDK to support the different runtimes as well as the blocking, sync client.
Comment · 11 months ago
There's a core difference between the message queue and the stream, and one of them is that the append-only log acts as a simple database from which multiple independent consumers can read records at the same time; hence, you don't truncate the append-only log unless specified in its settings via a particular cleanup policy (e.g. based on message expiry or its allowed size).