Handle MongoDB transactions in Node.js using Mongoose
A transaction is an execution of many operations (insert, update, or delete) as a single unit that takes the database from a consistent state and lets it to a consistent one. If one operation fails, all the operations are canceled. The properties of a transaction are Atomicity, Consistency, Isolation, and Durability (ACID).
Transactions are common for Relational databases, while with NoSQL databases, the philosophy of database management goes against the strict ACID rules. MongoDB is one of the most used NoSQL databases, which added support for transactions since version 4.0.
This tutorial will show how to use MongoDB transactions in a Node.js application.
The use case
We will implement the product's ordering on an e-commerce website as a real-world use case. Below is the simplified class diagram of our system.
Let's explain the relationships between the classes:
- A user can make one or many orders
- An order can have one or many payments (when the payment fails and we try with a new method, pay in many times)
- An order has one or many items
- An order item is associated with a product
Additional requirements:
- When the user places his order, if the payment succeeds and exceeds 100 euros, he gets a 5-mark bonus.
- When the order is completed, we update the product quantity in stock.
Here is the sequence diagram of the workflow to implement:
The importance of transaction
In the sequence diagram above, five steps constitute the order workflow. Let's analyze our system if we have a failure at each step:
- Step 1: If the application fails to create the order (duplicate order reference, insufficient permissions, etc.), the application will break, and the other steps will not continue.
- Step 2: If the payment execution fails (uncaught errors, service unavailable, etc.), we have an order created with payment status unknown, meaning if the user checks out the payment, he will have a message like, "We are processing your payment..." for minutes, hours, days if we don't change the status manually.
- Step 3: If the payment execution succeeds but the system fails to update the status (wrong API response parsing, bad update query, etc.), we have an order already paid, but the user receives a message asking him to pay again.
- Step 4: If the application fails to update the quantity of products in stock, users could have a bad experience since they can order an unavailable product.
- Step 5: If the payment is greater than 100€ and the application fails to increment the user bonus mark, you will have angry shouting everywhere that your promotion is a scam.
As we can see, to have a consistent system, all five steps must succeed. If one step fails, we should cancel the previous changes on the system. That is why we need a transaction of these steps as a single unit.
Prerequisites
MongoDB supports transactions since version 4.0; you must use at least this version to follow this tutorial.
For transactions to work on MongoDB, a replica set must be created. I wrote this full post where I guide you on setting it up in your local environment. You can also use MongoDB Atlas, which gives a free instance that comes with a replica set out of the box.
I will use MongoDB with a replica set on my local environment, so you will also need Docker and Docker compose
Finally, Node.js 18+ must be installed since we will use it to build the sample application.
Set up the project
To start, we will use the project created in the tutorial, where I show how to create a replica set. Let's clone it from GitHub.
git clone https://github.com/tericcabrel/blog-tutorials.git
cd blog-tutorials/mongodb-replica-set
Initialize a Node.js project and also Typescript
yarn init -y
yarn add -D typescript
yarn tsc --init
Start the MongoDB server
First, start the MongoDB server with a replica set by running the command below:
./dbstart.sh
To connect to the database, we need the URL to that server. Each MongoDB runs on port 27017 internally; this port is mapped to the host's specific port, respectively: 27021, 27022, and 27023.
The URL will be: mongodb://localhost:27021,localhost:27022,localhost:27023/test?replicaSet=dbrs
The database name is test
is the name of the database and dbrs
is the name of the replica set.
Connect to the replication set
We will use Mongoose, the ORM for MongoDB in Node.js; we also need ts-node to run a .ts file. Let's install them.
yarn add mongoose
yarn add -D ts-node
Create a folder named src
then, create the file called app.ts
and finally, add the code below:
import mongoose, { ConnectOptions } from "mongoose";
const DATABASE_URL = "mongodb://localhost:27021,localhost:27022,localhost:27023/test?replicaSet=dbrs";
const options: ConnectOptions = {
readPreference: 'secondary',
};
(async () => {
try {
await mongoose.connect(DATABASE_URL, options);
console.log("Connected to the database successfully ");
} catch (e) {
console.error("Failed to connect to the database!", e);
}
})();
Execute the file to test the connection:
yarn ts-node ./src/app.ts
The connection should work as expected. The option readPreference
indicates if the read queries should be executed on the primary or secondary replica.
Define the models
We need to create Mongoose models for the application based on our diagram. Be careful! Modeling for NoSQL is not the same as for relational databases.
On an RDBMS, we will create 5 tables.
For MongoDB who is a NoSQL database, we will create 3 collections.
I will not cover the models' creation since it is not the main part of the topic, but you can find their definition in the project repository here.
Create database collections
When Performing MongoDB queries without transactions, you don't need to create the collections at first because they are created automatically when we insert the first document.
On a replica set, you have to do it manually; otherwise, transactions will not work. Let's update the ./src/app.ts
to create the collections from the model.
import mongoose, { ConnectOptions, Model, Document } from "mongoose";
import { models } from "./models";
const DATABASE_URL = "mongodb://localhost:27021,localhost:27022,localhost:27023/test?replicaSet=dbrs";
const options: ConnectOptions = {
readPreference: 'secondary',
};
const createCollections = async (models: Model<Document>[]) => {
await Promise.all(
models.map((model) => model.createCollection())
);
};
(async () => {
try {
await mongoose.connect(DATABASE_URL, options);
await createCollections(models);
console.log("Connected to the database successfully ");
} catch (e) {
console.error("Failed to connect to the database!", e);
}
})();
The collections are created. Let's add some users and products to use in the upcoming part.
Find the source code for the database seed in the GitHub repository in the file src/seed.ts
.
Anatomy of a transaction's execution
The code below shows the template of what a transaction looks like:
const session: ClientSession = await mongoose.startSession();
session.startTransaction();
try {
// TODO Add your statement here
// Commit the changes
await session.commitTransaction();
} catch (error) {
// Rollback any changes made in the database
await session.abortTransaction();
// Rethrow the error
throw error;
} finally {
// Ending the session
session.endSession();
}
- We create a transaction session and then, initialize the transaction
- Inside the "try" block, we will add our operations to the database that must be executed atomically.
- Once done, we commit the changes.
- If an error occurs, the transaction is aborted inside the catch block, and the error is rethrown.
- Finally, we end the session.
Simulate a transaction failure
To see the transaction in action, we will first simulate a failure case where I explicitly make the execution fail. In the function proceedToPayment()
, I intentionally throw an error after 2 seconds to simulate the payment execution failure.
We expect nothing to be saved in the database at the end of the execution.
Inside the folder src, create a file called failure-transaction.ts, then add the code below:
import mongoose, { ClientSession } from "mongoose";
import { Order, OrderInput, OrderStatusEnum, PaymentStatusEnum } from "./models/order.model";
import { Product } from "./models/product.model";
import { User } from "./models/user.model";
const userId = '619052e09f31f6bcc5cf0234'; // ID of "User First" in the users collection
const orderInput: OrderInput = {
amount: 50,
vat: 20,
user: new mongoose.Types.ObjectId(userId),
status: OrderStatusEnum.CREATED,
reference: Math.random().toString(),
payment: {
amount: 50,
reference: Math.random().toString(),
status: PaymentStatusEnum.CREATED,
},
items: [
{
product: '61903d0c3e658edce54265b6', // ID of "Product One" in the products collection
priceUnit: 5,
quantity: 4,
},
{
product: '61903cdabe8df95babed6698', // ID of the "Product Two" in the products collection
priceUnit: 10,
quantity: 3,
},
]
};
const proceedToPayment = async (paymentInput: OrderInput["payment"]) => {
return new Promise((resolve, reject) => {
setTimeout(() => {
reject('Payment failed for unknown reason');
}, 2000);
})
};
export const processUserOrderFail = async () => {
const session: ClientSession = await mongoose.startSession();
session.startTransaction();
try {
// Step 1: create the order
const [createdOrder] = await Order.create([orderInput], { session });
// Step 2: process the payment
const paymentResult: any = await proceedToPayment(orderInput.payment);
if (paymentResult.failed) {
// Step 3: update the payment status
await Order.updateOne({ reference: createdOrder.reference }, { 'payment.status': PaymentStatusEnum.FAILED }, { session });
} else {
// Step 3: update the payment status
await Order.updateOne({ reference: createdOrder.reference }, { 'payment.status': PaymentStatusEnum.SUCCEEDED }, { session });
// Step 4: Update the product quantity in stock
await Promise.all(orderInput.items.map((item) => {
Product.findOneAndUpdate({ _id: item.product }, { $inc: { inStock: item.quantity * -1 } }, { new: true, session });
}));
// Step 5: Update bonus mark
if (orderInput.payment.amount > 100) {
await User.findOneAndUpdate({ _id: orderInput.user }, { $inc: { bonusMark: 5 } }, { new: true, session });
}
}
// Commit the changes
await session.commitTransaction();
} catch (error) {
// Rollback any changes made in the database
await session.abortTransaction();
// logging the error
console.error(error);
// Rethrow the error
throw error;
} finally {
// Ending the session
session.endSession();
}
};
Call the function processUserOrderFail()
in the file src/app.ts
then, run the code with:
yarn ts-node ./src/app.ts
As we can see, there is no document inside the collection orders
.
Execute a transaction successfully
For the preview code to work as we expect, we just need to update the function proceedToPayment()
to return a valid response.
const proceedToPayment = async (paymentInput: OrderInput["payment"]) => {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve({ failed: false });
}, 2000);
})
};
Run the code and see the result.
Bonus: A wrapper for transaction execution
We have seen before that there are many lines of code used to define a transaction statement with Mongoose. If you have to run your code at many places, you will end with many lines of code that look the same. The solution is to create a function that takes charge of the transaction management, and all you have to do is add your instructions.
Here is the function called runInTransaction()
:
import mongoose, { ClientSession } from "mongoose";
type TransactionCallback = (session: ClientSession) => Promise<void>;
export const runInTransaction = async (callback: TransactionCallback) => {
const session: ClientSession = await mongoose.startSession();
session.startTransaction();
try {
await callback(session);
// Commit the changes
await session.commitTransaction();
} catch (error) {
// Rollback any changes made in the database
await session.abortTransaction();
// logging the error
console.error(error);
// Rethrow the error
throw error;
} finally {
// Ending the session
session.endSession();
}
};
Here is how to use it:
await runInTransaction(async (session) => {
await Product.findOneAndUpdate({ _id: '61912208082a0397ff955e53' }, { $inc: { inStock: -5 } }, { session });
await User.findOneAndUpdate({ _id: '61912208082a0397ff955e51' }, { $inc: { bonusMark: 5 } }, { session });
});
No more boilerplate code to add everywhere you need to use transactions.
Wrap up
This tutorial ends, and you now know how to execute a transaction in MongoDB. A transaction can prevent an inconsistent state in your database and your application. It is helpful for a suit of operations that needs to be completed atomically.
You can find the code source on the GitHub repository.
Follow me on Twitter or subscribe to my newsletter to avoid missing the upcoming posts and the tips and tricks I occasionally share.