matticakes
Joined 18 years ago
Karma
303
Submissions
64
Member since
Nov 7
Recent submissions
Comment · 9 years ago
I'm one of the original authors, happy to answer any questions.
Comment · 11 years ago
NSQ is a full-featured messaging platform out-of-the-box, whereas zeromq and nanomsq are lower level libraries that you could use to build (the same) functionality.
Comment · 11 years ago
You can embed the daemon into your application.
That sentence is missing an important word. No centralized brokers, meaning the typical/recommended deployment topology is an nsqd node on all hosts producing messages.
Comment · 11 years ago
Thanks!
This is something that I've always tried to stress when talking about NSQ...
The "message queue" is the most boring and uninteresting aspect of the system.
It's the combination of out-of-the-box tooling and conceptually simple primitives that really differentiate it from other systems.
Comment · 11 years ago
Yea, in the original announcement blog post (which became the design doc):
Comment · 11 years ago
(project author)
I appreciate all the work that Tyler has put into his series of blog posts on messaging systems (and NSQ bug reports!) but I think this particular article is one that misses the mark [1].
While raw performance is important, I think comparing the guarantees and related operational and development semantics are far more interesting and useful.
These systems vary a great deal in the interfaces and functionality they expose to users building on top of them and operators dealing with them when they're deployed in production (and inevitably break!). This is ultimately what matters most.
P.S. single node (localhost) performance is essentially useless. Most of these tools are designed to be deployed as the backbone of large distributed systems (NSQ in particular), so scalability and performance in that context is a better indication of real-world performance.
[1] He's since posted this follow-up that outlines many of the same things http://www.bravenewgeek.com/benchmark-responsibly/
Comment · 11 years ago
(project author)
NSQ is as much about what it doesn't do as it is what it does. To a certain extent this mirrors, and was inspired by, the language's philosophy (Go) [1].
Also, NSQ was designed to replace an existing home-grown system deployed at scale. This dictated a lot of the initial requirements (and in certain cases excluded off-the-shelf tools).
When we left the experimental phase we realized we had built something that was useful to others, and it turns out that despite not having the features you've identified it can be incredibly effective in lots of use cases that don't need stronger guarantees.
[1] If I'm being honest, NSQ was a vehicle for adoption of Go at bitly as well as the project we used to learn the language. This was a huge risk at the time (almost 3 years ago) but one that has certainly paid off.
Comment · 12 years ago
check out NSQ, designed for (in part) exactly this use case (and also written in Go)
Comment · 13 years ago
NSQ treats the message data as an opaque blob so the format wouldn't directly affect it (except on some lower level related to overall message size I suppose). It would impact your producers (encoding) and consumers (decoding), obviously.
re: de-duping - there are lots of things to consider, I highly recommend reading through http://cs.brown.edu/courses/csci2270/archives/2012/papers/we..., it's a fantastic paper. At a high level the answer is idempotency.
What sort of use case are you thinking of? (context would help answer your de-dupe question)
Comment · 13 years ago
there are a few (growing) number of production installations of NSQ...
our (bitly's) cluster spans a a few datacenters and hits peaks of 80k messages/second.
I can answer any questions you have (one of the authors)
Comment · 14 years ago
You can swap out the "metrics system" described in the post with any downstream system.
I have a lot of experience with NSQ but have not used NewRelic. I can answer any specific questions you might have...
Comment · 14 years ago
Thanks.
For the record... NSQ is a generic messaging platform and is data format agnostic, there is nothing specific about metrics collection (it was just my contrived example).