Bull - Simple Queue System for Node

Bull is a JavaScript library that implements a fast and robust queuing system for Node backed by Redis.

There are many queueing systems out there. Each one of them is different and was created for solving certain problems: ActiveMQ, Amazon MQ, Amazon Simple Queue Service (SQS), Apache Kafka, Kue, Message Bus, RabbitMQ, Sidekiq, Bull, etc.

What Message Queue is best fit for a Node application? You asked! I came across implementations like Kue and Bull. They are redis backed and seem like good solutions. Have you guys tried any of these? I’ve been using Bull for years to handle async jobs and schedule messages and quite satisfied with it!

If you are new to queues you may wonder why they are needed after all. Queues can solve many different problems in an elegant way, from smoothing out processing peaks to creating robust communication channels between microservices or offloading heavy work from one server to many smaller workers, etc.

Features

Although it is possible to implement queues directly using Redis commands, this library provides an API that takes care of all the low-level details and enriches Redis basic functionality so that more complex use-cases can be handled easily.

• Minimal CPU usage due to a polling-free design.
• Robust design based on Redis.
• Delayed jobs.
• Schedule and repeat jobs according to a cron specification.
• Rate limiter for jobs.
• Retries.
• Priority.
• Concurrency.
• Pause/resume—globally or locally.
• Multiple job types per queue.
• Threaded (sandboxed) processing functions.
• Automatic recovery from process crashes.

Comparison

Since there are a few job queue solutions, here is a table comparing them:

• Kue: A priority job queue backed by redis, built for node.js
• Bee Queue: A simple, fast, robust job/task queue for Node.js, backed by Redis.
• Agenda: Lightweight job scheduling for Node.js

Feature (Jul 2019)	Bull	Kue	Bee	Agenda
Backend	redis	redis	redis	mongo
Priorities	✓	✓		✓
Concurrency	✓	✓	✓	✓
Delayed jobs	✓	✓		✓
Global events	✓	✓
Rate Limiter	✓
Pause/Resume	✓	✓
Sandboxed worker	✓
Repeatable jobs	✓			✓
Atomic ops	✓		✓
Persistence	✓	✓	✓	✓
UI	✓	✓		✓
Optimized for	Jobs / Messages	Jobs	Messages	Jobs

A Job’s Lifecycle

In order to use the full potential of Bull queues, it is important to understand the lifecycle of a job. From the moment a producer calls the add method on a queue instance, a job enters a lifecycle where it will be in different states, until its completion or failure (although technically a failed job could be retried and get a new lifecycle).

When a job is added to a queue it can be in one of two states, it can either be in the “wait” status, which is, in fact, a waiting list, where all jobs must enter before they can be processed, or it can be in a “delayed” status: a delayed status implies that the job is waiting for some timeout or to be promoted for being processed, however, a delayed job will not be processed directly, instead it will be placed at the beginning of the waiting list and processed as soon as a worker is idle.

The next state for a job I the “active” state. The active state is represented by a set, and are jobs that are currently being processed, i.e. they are running in the process function explained in the previous chapter. A job can be in the active state for an unlimited amount of time until the process is completed or an exception is thrown so that the job will end in either the “completed” or the “failed” status.

Bull Arena

An intuitive Web GUI for Bee Queue and Bull. Built on Express so you can run Arena standalone, or mounted in another app as middleware. Having following features:

• Check the health of a queue and its jobs at a glance
• Paginate and filter jobs by their state
• View details and stacktraces of jobs with permalinks
• Restart and retry jobs with one click