As a web developer, it’s crucial that your apps perform as quickly as they can. You should build web apps that respond to requests in the fastest time possible.

One of the many technologies that can help you out is task queuing.

So, what is task queuing, and how can you use it to optimize a Node.js application?

What Is Task Queuing?

Message queuing is a means of asynchronous communication between two applications or services, usually referred to as the producer and consumer. It’s a well-known concept employed in serverless and microservice architectures.

The concept of task or job queuing leverages message queuing to improve application performance. It abstracts the complexities of managing messages and enables you to define functions to manage jobs or tasks asynchronously using a queue, thereby reducing the rate of memory usage in some parts of an application.

The most common example of message queue software is RabbitMQ. Task queue tools include Celery and Bull. You can also configure RabbitMQ to work as a task queue. Read on to learn about task queuing in Node.js using Bull.

What Is BullMQ?

The BullMQ logo, a silhouette of a bull poised to charge at the word “BULL”

BullMQ (Bull.js) is a Node.js library used for implementing queues in Node applications. Bull is a Redis-based system (you might be more familiar with Redis as a tool for quick data storage) and it is a fast and reliable option to consider for task queuing in Node.js.

You can use Bull for many tasks such as implementing delayed jobs, scheduled jobs, repeatable jobs, priority queues, and many more.

So, how can you use Bull and Redis to run Node.js tasks asynchronously?

How to Configure Bull and Redis for Task Queuing in Node.js

To get started with task queuing in Node.js with Bull, you need Node.js and Redis installed on your machine. You may follow the Redis labs guide to install Redis if you don’t have it installed.

The first step to implementing Bull is to add it to your project’s dependencies by running npm install bull or yarn add bull in the terminal inside your project’s folder. There are multiple ways to initialize a queue in Bull as shown below: 

        const Queue = require('bull');

// different ways to initialize a queue
// - using redis URL string
const emailQueue = new Queue('Email Queue', 'redis://127.0.0.1:6379');

// - with a redis connection and queue options object
const videoQueue = new Queue('Video Queue', 'redis://127.0.0.1:6379', queueOptions);

// - without a redis connection but with queueOption
const docQueue = new Queue('Document Queue', queueOptions);

// - without a redis connection or queue options
const QueueClient = new Queue('My Queue');

These all use minimal configuration for Bull in Node.js. The options object supports many properties and you can learn about them in the queue options section of Bull’s documentation.

Implementing an Email Task Queue Using BullMQ

To implement a queue for sending emails, you can define your producer function which adds emails to the email queue, and a consumer function to handle the sending of emails.

Firstly, you may initialize your queue in a class using a Redis URL and some queue options as seen below. 

        // queueHandler.js
const Queue = require('bull');

// use a real email handler module here - this is just an example
const emailHandler = require('./emailHandler.js');

// define constants, Redis URL, and queue options
const REDIS_URL = 'redis://127.0.0.1:6379';

const queueOpts = {
    // rate limiter options to avoid overloading the queue
    limiter: {
        // maximum number of tasks queue can take
        max: 100,

        // time to wait in milliseconds before accepting new jobs after
        // reaching limit
        duration: 10000
    },
    prefix: 'EMAIL-TASK', // a prefix to be added to all queue keys
    defaultJobOptions: {  // default options for tasks in the queue
        attempts: 3,       // default number of times to retry a task

        // to remove a task from the queue after completion
        removeOnComplete: true
    }
};

class EmailQueue {
    constructor() {
        this.queue =  new Queue('Email Queue', REDIS_URL, queueOpts);
    }
};

export default EmailQueue; // export the class

Now that you’ve initialized a queue, you can define your producer function (using Bull's add() function) as a method of the EmailQueue class to add emails to the task queue. The following code block demonstrates this: 

        // queueHandler.js

class EmailQueue {
    constructor () {
        // ...
    }

    // producer function to add emails to queue
    async addEmailToQueue(emailData) {
        // add task with name 'email_notification' to queue
        await this.queue.add('email_notification', emailData);
        console.log('the email has been added to the queue...');
    }
};

export default EmailQueue; // export the class

The producer function is ready, and you may now define a consumer function (using Bull's process() function) to process all email tasks in the queue—i.e. call the function to send an email. You should define this consumer function in the constructor of the class. 

        // queueHandler.js
class EmailQueue {
    constructor () {
        // ...

        // consumer function that takes in the assigned name of the task and
       // a callback function
        this.queue.process('email_notification', async (emailJob, done) => {
            console.log('processing email notification task');
            await emailHandler.sendEmail(emailJob);  // send the email
            done();  // complete the task
        })
    }
    // ...
};

export default EmailQueue; // export the class

A job may also have options to define its behavior in the queue or how the consumer function handles it. You can find out more about this in the job options section of Bull’s documentation.

The emailJob argument is an object that contains the properties of the task for the queue to process. It also includes the main data needed to construct the email. For easy understanding, the sendEmail() function would be similar to this example: 

        // emailHandler.js
const sendgridMail = require('@sendgrid/mail');

const apiKey = process.env.SENDGRID_API_KEY

sendgridMail.setApiKey(apiKey);  // set email transporter security credentials

const sendEmail = async (emailJob) => {
    try {
        // extract the email data from the job
        const { name, email } = emailJob.data;

        const message = {
            from: 'me@example.com',
            to: 'you@example.com',
            subject: 'Hi! Welcome',
            text: `Hello ${name}, welcome to MUO`
        };

        await sendgridMail.sendMail(message); // send email

        // mark task as completed in the queue
        await emailJob.moveToCompleted('done', true);
        console.log('Email sent successfully...');
    } catch (error) {
        // move the task to failed jobs
        await emailJob.moveToFailed({ message: 'task processing failed..' });
        console.error(error);  // log the error
    }
}

export default sendEmail;

Now that you have both the producer and consumer functions defined and ready to use, you may now call your producer function anywhere in your application to add an email to the queue for processing.

An example controller would look like this: 

        // userController.js
const EmailQueue = require('../handlers/queueHandler.js')

const signUp = async (req, res) => {
    const { name, email, password } = req.body;

    // -- 
    // a query to add the new user to Database...
    // --

    // add to Email queue
    const emailData = { name, email };
    await EmailQueue.addEmailToQueue(emailData);

    res.status(200).json({
        message: "Sign up successful, kindly check your email"
    })
}

Your queueHandler.js file should now be as follows: 

        // queueHandler.js
const Queue = require('bull');
const emailHandler = require('../handlers/emailHandler.js');

const REDIS_URL = 'redis://127.0.0.1:6379';

const queueOpts = {
    limiter: {          
        max: 100,         
        duration: 10000   
    },

    prefix: 'EMAIL-TASK',

    defaultJobOptions: {
        attempts: 3,       
        removeOnComplete: true
    }
};

class EmailQueue {
    constructor() {
        this.queue =  new Queue('Email Queue', REDIS_URL, queueOpts);

        // consumer
        this.queue.process('email_notification', async (emailJob, done) => {
            console.log('processing email notification task');
            await emailHandler.sendEmail(emailJob);
            done();
        })
    }

    // producer
    async addEmailToQueue(emailData) {
        // add task with name 'email_notification' to queue
        await this.queue.add('email_notification', emailData);
        console.log('the email has been added to the queue...');
    }
};

export default EmailQueue;

When you implement this in a Node.js REST API, you will notice a decrease in the response time of the signup endpoint, and faster email delivery times, compared to the alternative.

Task queues also enabled you to handle signup and email errors independently.

Optimizing Applications Using Task Queues

Message and task queues are a great way to improve the general performance of applications. They are also very cheap and you can use them in as many parts of an application as you need.

Although this tutorial used emails as an example scenario for handling memory-consuming tasks with queues, there are many other cases where you can apply the same concepts. These include heavy read/write operations, rendering high-quality images or documents, and sending out bulk notifications.