adesh_nalpet
Joined 3 years ago
Karma
65
Submissions
21
Member since
Jan 28
Recent submissions
Comment · 2 months ago
Comment · 2 months ago
The team at opendata.dev/Responsive is killing it, with a strong, opinionated stance on what works well and what doesn't. I did consider building on SlateDB, but it was forcing an unnecessary primitive on streams. I have documented the design choices here: https://picomq.com/docs/design/overview
Comment · 2 months ago
Precisely this! I might even add a section in the docs, “Not a replacement for Kafka,” under the FAQ.
Where Kafka starts to fall short is routing. If you want to access the data of one user from user-events-topic, that’s expensive to do. Most other streaming technologies are built around the same design, such as Kinesis.
There are other implementations that support the Kafka wire protocol and are cheaper in exchange for latency, e.g., AutoMQ and WarpStream.
That said, I’ll release Disk/EBS-staged WAL soon enough: https://github.com/PicoMQ/picomq/issues/13 as an add-on to cover low-latency needs.
Comment · 2 months ago
I'd say PicoMQ is close to being fully serverless, since nodes are stateless. But would miss out benefits of batching, caching and low-latency.
S2-lite is pretty cool, but it's single-node only and built on SlateDB. So, PicoMQ is closer S2 Cloud.
Comment · 2 months ago
Totally agree! Tigris is awesome, although I haven't used it at scale. PicoMQ (even cluster mode) is a one click deployment on Fly.io and they use Tigris as the Object Store. https://picomq.com/docs/operations/deployment/fly
Comment · 2 months ago
Indeed! The binary even embeds the admin dashboard. A good starting point to get the hang of PicoMQ: https://picomq.com/docs/playground
Comment · 2 months ago
True, there are also a lot of open PRs and issues. I'm surprised no other developers or orgs are stepping in to keep the project afloat. No wonder the Apache Way goes a long way! Either way, their approach of building around an open-protocol was a good decision.
If you'd like to consider using PicoMQ, I have examples to deploy on Fly.io and AWS. ursula.tonbo.io is also a good option to try.
Comment · 2 months ago
PicoMQ works with any S3-compatible object store. But I wasn't aware of GCS Rapid Bucket, it sounds a lot like AWS S3 Express, which is also zonal. And it does help with durability ACK latency quite a bit, keeping it closer to ~50ms.
I'll be setting up a GCP deployment example similar to AWS soon. I'll be sure to try Rapid Bucket as well, thanks for sharing!
Comment · 2 months ago
Your welcome! And feel free to ask any questions as try out PicoMQ or raise on Github issues. The disk-staged WAL is currently tracked here: https://github.com/PicoMQ/picomq/issues/13, I'm planning to ship it as an optional add-on.
Comment · 2 months ago
On the same boat, and honestly, it’s not getting the attention it deserves. Given Databricks acquired ElectricSQL primarily for PGlite, the Durable Streams project is likely going to be abandoned unless new maintainers step in.
I did, in fact, initially write PicoMQ with OpenRaft, similar to Ursala, but I really wanted the operational complexity to be minimal and the nodes to be stateless (at least for most use cases), Pico uses SQL database as a metadata command log, inspired by RisingWave.
But Ursala would, without a doubt, have better durability ACK latency, as it wouldn’t have to wait for an ACK from S3. That said, I do plan on extending disk/EBS-staged WAL to hit similar low-latency durability ACK numbers. But again, most use cases don’t need single-digit-millisecond latency for ACKs.
Comment · 2 months ago
Thank you! I'd say pretty close. But S2 is not open source, and the OSS version (https://github.com/s2-streamstore/s2) is S2-Lite, which is single-node only and uses SlateDB.
SlateDB is great, but it's not purpose-built for streaming workloads. With PicoMQ, the goal is to really do one thing well. It would have been far easier to just extend SlateDB, but instead, the S3Stream engine is built from the ground up with AutoMQ's core primitives.
Not to mention, Pico supports multi-node/cluster deployments and a bunch of other features (no feature gating).
Comment · 2 months ago
Yeah, much of the base cost I mentioned was from the compute, networking and S3 write costs, which could sustain a lot more messages for sure. It also depends on the number of active streams/rooms, the throughput bursts, as opposed to a steady state.
But as we speak, I am in the process of running open-benchmarks on AWS with cost attribution. I'll be posting them in the docs with transparency soon enough.
Read and write through cache already exists today, which would work in favour of both cost, latency, and consumer reads fanout.
I like the HN Mirror over a Discord-like app for simplicity. No doubt that's where my weekend is going!
But because the Pico's semantics are close to Kafka, but not tightly coupled, it's quite feasible to support the Kafka protocol, * with some restrictions *, such as a topic can only ever have one partition, no support for transactions (since they would only really apply when producers are publishing to multiple streams), and more along those lines. As a result, you'd get Kafka with no topic tax and a lot less operational complexity.