We build out our architecture using microservices and recommend using NSQ over RabbitMQ when applying a message queue.
At Wiredcraft, we love building microservices. They fit perfectly with our company philosophy and eliminate a lot of the headaches software developers usually had in the past. We think they’re pretty great and we use them for all of our client engagements.
So, what’s the big deal with microservices? Well, there are numerous benefits to building these little guys compared to the traditional monolithic architecture of days long gone:
Easy to enhance and extend
Easy to deploy and scale
I won’t waste your time telling you why you should be using microservices; there are plenty of folks advocating this approach already. You can refer to this dude and these guys for more information about what microservices are and the advantages they have. Despite their extreme usefulness, microservices are certainly not some silver bullet for all of your backend problems. One of their biggest downfalls is the complexity of interaction between the services you build.
We use Node.js and Loopback to build REST API services, which means we mainly rely on the HTTP(S) protocol. As you start to build the backend, it’s not too difficult to keep a clean and graceful structure at the beginning, because you can definitely differentiate the upstream and downstream services clearly and neatly. However, as the requirements increase in volume and complexity and the system evolves, you may find you messed something up and the API callings are now looking like your mother’s spaghetti!
Why we use message queues
The cross-dependency means the system is tightly coupled, so no single service can go it alone without cooperation from other services. We use the message queue as a supplement for decoupling and keeping the the architecture flexible.
Actually, in addition to the decoupling, we expect the following features from the message queue:
A mechanism with retry (and delay retry upon failure)
Pub-sub (publish-subscribe) pattern
We have 2 candidates for serving as our message queue platform: RabbitMQ and NSQ. Both have their own pros and cons, so let’s evaluate.
Introduce exchange between producer and consumer and fledged with lots of patterns (Direct/Worker/Pub-Sub/Route/RPC…)
Cons of RabbitMQ
A bit of steep learning curve
Not easy to implement retry with failure.
Pros of NSQ
Good at distributed topologies with no SPOF, which means highly reliable availability (even if some nodes go down, you can still use the messaging service)
One concise message model
Supports retry with delay naturally
Cons of NSQ
Messages are not durable by default
Why we choose NSQ
First, the message model of NSQ is simple and direct. No middle man, no broker, no garbage. All you need to do is define the producer and the consumer. If you need fanout or broadcast, you can add multiple channels for the topic.
Second, we need retry with a delay mechanism. Though it is doable with RabbitMQ, it’s not that easy to manage. The main approach for this is using an additional dead letter exchange to simulate the delay-attempts. Here’s the gist of how it works. It’s an unnecessary hassle, so we avoid it whenever possible.
If you use NSQ, you merely need to call requeue() with parameter of delay, then you’re done!
The last reason we prefer NSQ is its core is written with Go. Aside from Node.js, we also use Go for some of our backend projects. At Wiredcraft, we are always willing to try out and challenge ourselves with new technologies. :D
Something worth mentioning again, NSQ is not perfect and does have its pain points (e.g. if you want to ensure strong message durability), so you should be aware of the following shortcomings.
No message replication and it’s mainly in memory.
No publisher confirmation. If there’s a failure in the nsqd node when the message happens to arrive, you lost this message.
There is a roadmap of NSQ’s durability and delivery guarantee. To solve the problems above, you can duplicate the message with the same topic to ensure it will be delivered at least once (it’ll duplicate more than once, in most cases). This means you need your client to de-dupe or make the operation idempotent.
How we use NSQ
That’s why we built the library nsq-strategies, a wrapper of the official client library (nsqjs) with different strategies. Currently, it supports round-robin and fanout strategies. For example, you now can transfer the lookup addresses to the producer, which means you don’t need to change the code when the nsqd cluster changes, and the produce would pick one of nsqd nodes in a round-robin way for sending message.
If you’re building microservices, you should consider adopting a message queue as a supplement and giving it a try with NSQ (or RabbitMQ). Use it then improve it, like we did. That’s the way we build apps that matter.
What are your experiences with microservices? Let us know on Twitter (@wiredcraft) what you think and about any cool projects we should check out.