Messaging enables software applications to connect and scale. Applications can connect to each other, as components of a larger application, or to user devices and data. Messaging is asynchronous, decoupling applications by separating sending and receiving data.

A message broker is an intermediary for messaging. It gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.

• RabbitMQ - The most widely deployed open source message broker, written in Erlang, lightweight, easy to deploy, lots of features, and decent performance.
• ActiveMQ - An open source, multi-protocol, Java-based messaging server. It supports industry standard protocols so users get the benefits of client choices across a broad range of languages and platforms.
• Kafka - Written in Java, under the Apache project umbrella, powerful event streaming platform, suitable for real-time processing, and excellent for big data projects.
• Redis - A bit different from the other message brokers. At its core, Redis is an in-memory data store that can be used as a high-performance key-value store. Since Redis 5.0 introduced the Pub-Sub you can use as a message broker.

When choosing a broker for executing your asynchronous operations, you should consider a few things:

• Broker scale - The number of messages sent per second in the system.
• Data persistency - The ability to recover messages.
• Consumer capability - Whether the broker is capable of managing one-to-one and/or one-to-many consumers.

In general, if you want a simple traditional pub-sub message broker, the obvious choice is RabbitMQ, as it will most probably scale more than you will ever need it to scale.

RabbitMQ

RabbitMQ is one of the most popular open source message brokers. It is lightweight, open-source, easy to deploy on premises and in the cloud, supports multiple messaging protocols, and can be deployed in distributed and federated configurations to meet high-scale, high-availability requirements.

RabbitMQ ships in a state where it can be used straight away in simple cases such as development and QA environments - just start the server, enable the necessary plugins and it’s ready to go.

RabbitMQ is officially supported on a number of operating systems and has several official client libraries. In addition, the RabbitMQ community has created numerous clients, adaptors and tools that we list here for your convenience.

Applications interact with RabbitMQ using client libraries. There are client libraries available for many programming languages and platforms. Each protocol has its own set of client libraries. Most client libraries are open source.

All protocols supported by RabbitMQ are TCP-based and assume long-lived connections (a new connection is not opened per protocol operation) for efficiency. One client library connection uses a single TCP connection. In order for a client to successfully connect, target RabbitMQ node must allow for connections on a certain protocol-specific port.

RabbitMQ supports several messaging protocols, directly (AMQP 0-9-1) and through the use of plugins (STOMP, MQTT, AMQP 1.0).

ActiveMQ

ActiveMQ is a Java-based open-source project developed by the Apache Software Foundation.

It’s possible to write ActiveMQ clients in other languages (such as Node.js, Ruby, and Python), but ActiveMQ is built on Java, and is probably best suited for an organization that’s already invested in Java.

ActiveMQ comes with and the suite of protocols that it supports, along with the enterprise flavour and readiness, can place this tool at the top of the list when it comes to choosing the right messaging broker for your infrastructure.

Java Message Service (JMS) is a messaging standard that allows application components based on the Java Platform Enterprise Edition (Java EE) to create, send, receive, and read messages. It enables distributed communication that is loosely coupled, reliable, and asynchronous.

ActiveMQ sends messages between client applications—producers, which create messages and submit them for delivery, and consumers, which receive and process messages. The ActiveMQ broker routes each message through one of two types of destinations: a queue and a topic.

ActiveMQ gives you the flexibility to send messages through both queues and topics using a single broker. In point-to-point messaging, the broker acts as a load balancer by routing each message from the queue to one of the available consumers in a round-robin pattern.

ActiveMQ 5 Classic: Long established, endlessly pluggable architecture serving many generations of applications.

• JMS 1.1 with full client implementation including JNDI
• High availability using shared storage
• Familiar JMS-based addressing model
• “Network of brokers” for distributing load
• KahaDB & JDBC options for persistence

ActiveMQ Artemis: High-performance, non-blocking architecture for the next generation of event-driven messaging applications.

• JMS 1.1 & 2.0 with full client implementation including JNDI
• High availability using shared storage or network replication
• Simple & powerful protocol agnostic addressing model
• Flexible clustering for distributing load
• Advanced journal implementations for low-latency persistence as well as JDBC
• High feature parity with ActiveMQ 5 to ease migration

Kafka

Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Kafka was originally created at LinkedIn, where it played a part in analysing the connections between their millions of professional users in order to build networks between people. It was given open source status and passed to the Apache Foundation – which coordinates and oversees development of open source software – in 2011.

Kafka is a distributed system consisting of servers and clients that communicate via a high-performance TCP network protocol. It can be deployed on bare-metal hardware, virtual machines, and containers in on-premise as well as cloud environments.

For a long time, Kafka was a little unique (some would say odd) as an infrastructure product—neither a database nor a log file collection system nor a traditional messaging system.

• Messaging - Kafka works well as a replacement for a more traditional message broker like ActiveMQ or RabbitMQ. Kafka has good throughput, built-in partitioning, replication, and fault-tolerance which makes it a good solution for large scale message processing applications.
• Website Activity Tracking - The original use case for Kafka was to be able to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds.
• Metrics - Kafka is often used for operational monitoring data. This involves aggregating statistics from distributed applications to produce centralized feeds of operational data.
• Log Aggregation - Kafka can be used as a replacement for a log aggregation solution like Scribe or Flume. Kafka abstracts away the details of files and gives a clean abstraction of log or event data as a stream of messages.
• Stream Processing - Process data in processing pipelines consisting of multiple stages, where raw input data is consumed from Kafka topics and then aggregated, enriched, or otherwise transformed into new topics for further consumption or follow-up processing.
• Event Sourcing - Kafka’s support for very large stored log data makes it an excellent backend for an application built in this style.
• Commit Log - Kafka can serve as a kind of external commit-log for a distributed system. The log helps replicate data between nodes and acts as a re-syncing mechanism for failed nodes to restore their data.

Redis

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker.

The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps.

Redis has built-in replication, Lua scripting, LRU eviction, transactions, and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.

Redis is often referred to as a data structures server. What this means is that Redis provides access to mutable data structures via a set of commands, which are sent using a server-client model with TCP sockets and a simple protocol. So different processes can query and modify the same data structures in a shared way.

Data structures implemented into Redis have a few special properties:

• Redis cares to store them on disk, even if they are always served and modified into the server memory. This means that Redis is fast, but that it is also non-volatile.
• The implementation of data structures emphasizes memory efficiency, so data structures inside Redis will likely use less memory compared to the same data structure modelled using a high-level programming language.
• Redis offers a number of features that are natural to find in a database, like replication, tunable levels of durability, clustering, and high availability.

Another good example is to think of Redis as a more complex version of memcached, where the operations are not just SETs and GETs, but operations that work with complex data types like Lists, Sets, ordered data structures, and so forth.