programming

Local job queue for NextJS | AgendaJS

Job Queues are not just a great way to build highly resilient and scalable systems, but they are also a great way to maintain clean code. In our day to day App development, we often find some tasks which can benefit from a Job Queue, for example:

Sending out emails to clients whose membership is expiring on within certain days in future
Download data for a user from a third party server, like a CRM
Encoding or converting video files uploaded by the users

How about cloud Job Queues?

Yes, all the major cloud platforms like Amazon Web Service, Google Cloud Platform and Azure have some implementations of the job systems. But for a small web application they seem rather large or pointless, the reason being we often need to do some configuration and setup before we can use them, some of the configuration may include:

Job Deployments, often we need to deploy the code responsible to job execution to the cloud platforms. This by nature needs to be a continuous deployment.
Queue Definitions, not as complex as the last step, but we still need to define the Queue locations and other configuration.
Persistence, we need some way to access some sort of persistence, to track and store the result of the job. Which can get reasonably complex. Like saving a file on S3 Bucket, saving some results in MongoDB, etc.. Generally your NextJS has already access to those, so those access often need to be shared with the cloud jobs...

For the above said reasons, I prefer not to use these cloud platform Job Queues for NextJS applications, especially if they are in early stages. Where any effort spent on the cloud configuration can be spent on developing the features faster. So what is the solution to handle these jobs?

Enter AgendaJS

AgendaJS, this queue system is driven my mongodb, is very robust and highly scalable. I will go through how we can use Agenda with NextJS in this article, also you can checkout the github repository for a working version of this setup.

Architecture

Architecture We will be running Agenda in it's own process, which means that we need to keep a shared instance between NextJs and AgendaJS. NextJS can use this instance to schedule jobs in the MongoDB and the agenda process will pull the job and run it.

Directory Structure

NextJS Project
│   package.json
└───src
|   └───app - the next js app
│   └───tasks - all the tasks configuration
│       │   index.ts - holds all job definitions and schedules
│       │   Job.ts - A single Job
│       │   ...
|   └─── agenda.ts - agenda process 
|   └─── agenda-instance.ts - instance of the agenda

Setup

We are assuming that you already have the NextJS app running. To install agenda and required library

npm i @hokify/agenda nodemon esbuild tsconfig-paths ts-node dotenv

Next, we create the src/agenda-instance.ts file

import { Agenda } from '@hokify/agenda';
// Dotenv
import dotenv from 'dotenv';
dotenv.config();
const mongoConnectionString = process.env.MONGO_URI || '';
if (!mongoConnectionString) {
  console.error('Mongo URI is missing');
  process.exit(1);
}
const agenda = new Agenda({
  db: { address: mongoConnectionString, collection: 'jobs' },
});
export default agenda;

Next, lets create src/agenda.ts file

// Dotenv
import dotenv from 'dotenv';

import agenda from './agenda-instance';
import { registerTasks } from './tasks/index';
dotenv.config();

const mongoConnectionString = process.env.MONGO_URI || '';

if (!mongoConnectionString) {
  console.error('Mongo URI is missing');
  process.exit(1);
}

registerTasks(agenda); // Loads all tasks and schedules them

(async function start() {
  await agenda.start();
  console.log('Agenda started...');
})();

Lets create a job file at src/tasks/HelloJob.ts

import { Job } from '@hokify/agenda';

export default function HelloJob(job: Job): void {
  console.log('Hello Job');
}

Lets now create the src/tasks/index.ts file, this file will register our job with agenda,

import { Agenda } from '@hokify/agenda';

import HelloJob from './HelloJob';

export function registerTasks(agenda: Agenda) {
  // Register tasks
  agenda.define('HelloJob', HelloJob);

  // Schedule tasks
  agenda.every('1 minute', 'HelloJob'); // Example: Run every minute
}

NPM Scripts

Now we have the basic setup ready for agenda, we need a way to run it. To start, we need to define a file called tsconfig.tasks.json

{
  "compilerOptions": {
    "target": "es5", // Keep ES5 for broader compatibility
    "lib": ["dom", "dom.iterable", "esnext"], // Only include required libraries
    "strict": true,
    "esModuleInterop": true, // Ensure compatibility with CommonJS modules
    "module": "commonjs", // Use CommonJS for `ts-node`
    "moduleResolution": "node", // Use Node.js module resolution
    "resolveJsonModule": true, // Allow importing JSON files
    "outDir": "agenda-dist", // Output directory for compiled files
    "baseUrl": ".", // Base directory for paths
    "paths": {
      "@/*": ["./src/*"]
    }
  },
  "include": ["src/tasks/*", "src/agenda.ts"], // Include all task files
  "exclude": ["node_modules", ".next"] // Exclude unnecessary directories
}

Lets create a Nodemon launched for tasks, we will use this launcher to run the task process in development mode, let's create the file nodemon.json

{
  "watch": ["src/**/*"],
  "ext": "ts,json",
  "ignore": ["src/**/*.test.ts", "node_modules"],
  "exec": "ts-node --require tsconfig-paths/register --project tsconfig.tasks.json src/agenda.ts"
}

We finally need esbuild.tasks.js, in order to package our tasks for production

// eslint-disable-next-line @typescript-eslint/no-var-requires
const esbuild = require('esbuild');

esbuild
  .build({
    entryPoints: ['./src/agenda.ts'], // Entry file for Agenda tasks
    bundle: true, // Bundle all dependencies into a single file
    platform: 'node', // Target Node.js environment
    outfile: 'agenda-dist/agenda.js', // Output file
    sourcemap: true, // Optional: Generate source maps
    target: 'node16', // Adjust based on your Node.js version
    minify: true, // Optional: Minify the output
  })
  .catch(() => process.exit(1));

Finally, lets add the scripts to run tasks in development mode and production mode, add following lines in scripts section of package.json

"dev:tasks": "nodemon",
"build:tasks": "node esbuild.tasks.js",

Add MONGO_URI with mongodb connection string in .env. That's it, after you run npm run dev:tasks, you can see the message from our tasks getting logged in the console, which means the jobs are running.

To push a new job from the NextJS side, you can do that with following lines:

import agenda from '@/agenda-instance';

agenda.now('HelloJob', { leadId: lead._id });

Conclusion

We have now learned how to implement Agenda with NextJS and push jobs from NextJS to Agenda. Checkout the working version of the setup at github repository where you can see everything in action.