Implementing event-driven communication using AWS SNS and AWS CDK

AWS provides many services to build event-driven applications; one is AWS SNS, which allows asynchronous communication between services following the pub-sub pattern.

With AWS SNS, a service called publisher publishes a message to an SNS topic; one or many services called subscribers subscribe to the SNS topic to receive the messages and perform custom actions.

In this post, we will learn how to make two applications communicate asynchronously using AWS SNS. We will use the AWS CDK to declare the Infrastructure as Code for the required AWS resources.

Asynchronous communication between Node.js applications using RabbitMQ

In this post, we will send a message from a Node.js application into a RabbitMQ queue and receive this message from another Node.js application.

Teco TutorialsEric Cabrel TIOGO

Use case

We have an electronics e-commerce website where users can add products to their cart and proceed to the purchase. The purchase process is as follows.

1. Create the order on the system.
2. Proceed to the payment of the order.
3. Send a notification to the user through email to confirm the order.

To decouple the order processing from the user notification, we want to split it into two microservices:

• Core service: create the order and process the payment.
• Notification service: notify users through a specific channel (email).

The two services must communicate asynchronously since the core service doesn't need to wait for the notification service's response.

The core service will publish events for the notification service to consume them.

Architecture schema

The whole system is deployed on AWS. Each microservice is represented by an AWS Lambda, and they will communicate together using AWS SNS. We will store the data in MongoDB.

Prerequisites

To follow this tutorial, make sure you meet the requirements below:

• An AWS account with a free tier is enough.
• AWS CLI v2 configured; read my post on how to do it.
• Node.js 20+ - Download link.
• Docker for testing your Lambda function locally.
• The AWS CDK v2 installed locally: npm install -g cdk
  • Check if the installation succeeded by running the following: cdk --version

Set up the project using the AWS CDK

To focus on connecting two applications through events using AWS SNS, I prepared a repository containing two AWS CDK projects:

The Core service project

This project exposes the API routes behind an API Gateway

• [POST] /orders -> Create an order for products purchased by a user. This is where an event must be published to notify the notification service about the order created.
• [GET] /orders/{id} -> Retrieve the order details using its ID.
• [GET] /users/{id} -> Retrieve the user details using its ID.

Each API route triggers a Lambda function connected to a Mongo Database to read and write data.

Create a REST API with AWS Lamda & API Gateway using AWS CDK

This post shows how to define RESTful API routes on AWS API Gateway that invoke Lambda functions. We will write the infrastructure stack using the AWS CDK and SAM for local testing.

Teco TutorialsEric Cabrel TIOGO

The Notification service project

This project contains a Lambda function that must be triggered by an SNS event containing the ID of the order created.

The Lambda function calls the Core service API to retrieve the order and user details and then builds the payload necessary to send the user an email using Amazon SES.

Set up the project locally

Run the following commands to clone and set up the project locally.

git clone https://github.com/tericcabrel/node-aws-sns.git

cd node-aws-sns

cd core-service 

yarn install

cp .env.example .env

cd ../notification-service

# It is npm instead of yarn because of the stack build with AWS CDK.
npm install 

cp .env.example .env

Environment variables

The core service API requires a MongoDB URL; you can create a free instance of MongoDB on MongoDB Cloud.

Open the file "core-service/.env" and add the property "DATABASE_URL".

The notification service requires the core service's API URL and the SMTP settings for sending emails: SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASSWORD, and FROM_EMAIL.

Send email in Node.js using Handlebars and Amazon SES

In this post, we will see how to set up Amazon SES and send emails through SMTP in Node.js using Handlebars and Nodemailer.

Teco TutorialsEric Cabrel TIOGO

You can set the SMTP settings now, but you cannot put the core service API URL because we must deploy the stack first to get the API gateway URL.

Create the SNS topic

In the core service project, we must define the stack to create an SNS topic; the AWS CDK provides a library to achieve that.

Update the file "core-service/lib/core-service-stack.ts" to add the code below:

// Existing import here...
import * as cdk from 'aws-cdk-lib';
import * as sns from "aws-cdk-lib/aws-sns";


export class CoreServiceStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Existing CDK stacks definition here...

    const orderTopic = new sns.Topic(this, `OrderTopic`, {
      displayName: `EcommerceOrderTopic`,
      topicName: `ecommerce-order-topic`,
    });

    const processOrderFn = new lambda.NodejsFunction(this, `ProcessOrderFunction`, {
      // Existing properties...
      environment: {
        DATABASE_URL: process.env.DATABASE_URL,
        ORDER_EVENTS_TOPIC_ARN: orderTopic.topicArn,
      },
      // Existing properties...
    });

    orderTopic.grantPublish(processOrderFn);

    new cdk.CfnOutput(this, `OrderTopicArn`, {
      value: orderTopic.topicArn,
    });
  }
}

We inject the SNS topic ARN in the environment variables of the Lambda function with the key "ORDER_EVENTS_TOPIC_ARN".

How to publish an event on an SNS topic

Publishing an SNS event requires two pieces of information:

• The SNS topic to publish the event in.
• The payload of the event.

Structuring the event payload

An SNS topic represents a business domain in our application. In a domain, various events can happen.

For example, in the order domain, we can have the following events: order created, order shipped, order canceled, etc.

When publishing an event, you must structure it to make it easy to identify and use the power of TypeScript to infer the expected payload from the event type.

This is how we will structure our event payload:

type OrderCreatedEvent = {
  type: 'order.created';
  payload: {
    orderId: string;
  };
}

type OrderShippedEvent = {
  type: 'order.shipped';
  payload: {
    orderId: string;
    shippingId: string;
  };
}

type OrderCancelledEvent = {
  type: 'order.cancelled';
  payload: {
    orderId: string;
    reason: string;
  };
}

export type OrderEvent = OrderCreatedEvent | OrderShippedEvent | OrderCancelledEvent;

const orderCreateEvent: OrderEvent = {
  type: 'order.created',
  payload: {
    orderId: '123',
  }
}

const orderShippedEvent: OrderEvent = {
  type: 'order.shipped',
  payload: {
    orderId: '123',
    shippingId: 'abc',
  }
}

Publish an SNS event from a Lambda Function

We will use the AWS SDK v3 to publish an SNS event from the Lambda function responsible for processing the order.

Let's install the Node.js SDK client for AWS SNS in the core service project:

yarn add @aws-sdk/client-sns

Now, update the Lamda function code handler to publish the SNS event.

import { APIGatewayProxyHandler } from 'aws-lambda';
import { connectToDatabase } from "../utils/db";
import { handleProcessOrder, validateInput } from "../services/orders/process-order";
import { OrderEvent } from "../types/events";
import { SNSClient, PublishCommand } from "@aws-sdk/client-sns"

const snsClient = new SNSClient({ region: process.env.AWS_REGION });

export const handler: APIGatewayProxyHandler = async (event, context) => {
  // Make sure to add this so you can re-use `conn` between function calls.
  // See https://www.mongodb.com/blog/post/serverless-development-with-nodejs-aws-lambda-mongodb-atlas
  context.callbackWaitsForEmptyEventLoop = false;

  await connectToDatabase();

  const payload = JSON.parse(event.body ?? '{}');

  const validationResult = validateInput(payload);

  if (validationResult.success) {
    const processResult = await handleProcessOrder(validationResult.data);

    if (!processResult.success) {
      return {
        statusCode: 400,
        headers: { "Content-Type": "text/json" },
        body: JSON.stringify(processResult.error),
      };
    }

    const snsPayload: OrderEvent = {
      type: 'order.created',
      payload: {
        orderId: processResult.data.orderId,
      }
    };

    const publishCommand = new PublishCommand({
      TopicArn: process.env.ORDER_EVENTS_TOPIC_ARN,
      Message: JSON.stringify(snsPayload),
    });

    await snsClient.send(publishCommand);

    return {
      statusCode: 200,
      headers: { "Content-Type": "text/json" },
      body: JSON.stringify(processResult.data),
    };
  } else {
    return {
      statusCode: 400,
      headers: { "Content-Type": "text/json" },
      // @ts-ignore
      body: JSON.stringify({ message: "Invalid input data", errors: validationResult.error }),
    };
  }
};

Deploy the resources on AWS

The core service project is ready, but before deploying it on AWS, ensure:

• You have configured the AWS CLI and are targeting the correct AWS region.
• You defined the MongoDB URL in the .env file.

Run the following commands to deploy the resources:

yarn cdk bootstrap

export $(grep -v '#' .env | xargs) && yarn cdk synth

export $(grep -v '#' .env | xargs) && yarn cdk deploy

The first command deploys the AWS resources necessary for the AWS CDK to deploy and manage your infrastructure's state.

Wait for the deployment to complete.

You can see the API gateway URL and the order topic ARN. Let's make an API call to create an order.

We successfully created the order, and the event has been published, but there is no subscriber to the SNS topic yet.

Subscribe a Lambda function to an SNS topic

The Lambda function in the notification service must subscribe to the SNS topic to email the user who created the order.

To allow a Lambda function to subscribe to an SNS topic, let's modify the AWS CDK stack "notification-service/lib/notification-service-stack.ts" to add the code below:

// Existing imports here...
import * as sns from 'aws-cdk-lib/aws-sns';
import * as snsSubscriptions from 'aws-cdk-lib/aws-sns-subscriptions';

export class NotificationServiceStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const notificationFn = new lambda.NodejsFunction(this, 'NotificationFunction', {
      functionName: 'notification-service-notify-customers'
      // Existing Lambda function properties here...
    });
    
    const orderTopic = sns.Topic.fromTopicArn(this, 'OrderTopic', process.env.ORDER_EVENTS_TOPIC_ARN);

    orderTopic.addSubscription(new snsSubscriptions.LambdaSubscription(notificationFn));
    
    orderTopic.grantSubscribe(notificationFn);
  }
}

We retrieve the SNS topic from its ARN that we outputted after deploying the core service resources on AWS. Then, we grant the Lambda function the subscription to this SNS topic.

The topic ARN is read from the environment variables, so update the ".env" file to include this value and the core service API URL.

CORE_SERVICE_API_URL=<core_service_api_gateway_url>
ORDER_EVENTS_TOPIC_ARN=<order_sns_topic_arn>

These two values are printed after deploying the AWS CDK stack of the core service project.

Handle an SNS event from a Lambda function

A Lambda function handler can be triggered by various services, and each service has a different event payload.

When triggered by an SNS subscription, the event payload shape can be imported from the Node.js package @types/aws-lambda.

Let's install it as a development dependency.

npm install --save-dev @types/aws-lambda

Let's update the handler code "src/handlers/notify-customers.ts" to extract the order ID from the SNS event.

import { SNSHandler } from "aws-lambda";
import { OrderEvent } from "../types/events";
import { handleOrderCreated } from "../event-handlers/order-created";

export const handler: SNSHandler = async (event) => {
  const [record] = event.Records;
  const orderEvent = JSON.parse(record.Sns.Message) as OrderEvent;

  console.log("Sending notifications to customers...", orderEvent);

  switch (orderEvent.type) {
    case 'order.created':
      return handleOrderCreated(orderEvent.payload.orderId);
    case 'order.shipped':
    case "order.cancelled":
      console.log('To be implemented');
    default:
      console.error(`Unsupported event type: ${orderEvent.type}`);
      break;
  }
};

We call the associated function to be executed depending on the order event type.

Create a file "event-handlers/order-created.ts" and add the code below:

import * as path from "node:path";
import { findOrder } from "../services/find-order";
import { findUser } from "../services/find-user";
import { generateOrderSummary, OrderSummary } from "../utils/email-data-generator";
import { sendEmail } from "../utils/email-client";

export const handleOrderCreated = async (orderId: string) => {
  const order = await findOrder(orderId);

  if (!order) {
    throw new Error(`Order with ID "${orderId}" not found`);
  }

  const user = await findUser(order.user);

  if (!user) {
    throw new Error(`User with ID "${order.user}" not found`);
  }

  const templateData = generateOrderSummary(order, user);
  const templatePath = path.resolve(__dirname, './templates/order-created.html');

  await sendEmail<OrderSummary>({
    subject: 'Your order was created',
    templateData,
    templatePath,
    to: user.email
  });
}

Test the Lambda function locally

You may want to test locally that your handler works as expected for a faster feedback loop. When working with the AWS CDK, you can combine it with the AWS SAM to test your Lambda function locally.

You can use the SAM CLI to generate an SNS event payload to invoke your Lambda function. Run the command below to generate a sample of an SNS event:

sam local generate-event sns notification

Use the command sam local generate-event --help to see the available AWS services for which you can generate sample events.

You get the following output:

Copy the JSON and paste it into a file called "order-created.json" then update the "Message" property to the following value.

"{\"type\":\"order.created\",\"payload\":{\"orderId\":\"order_id\"}}"

Replace the "order_id" with an existing order ID in your Mongo database.

Run the following commands to test your Lambda function locally.

export $(grep -v '#' .env | xargs) && npm run cdk synth

sam local invoke NotificationFunction -t ./cdk.out/NotificationServiceStack.template.json -e ./src/handlers/test-data/order-created.json

You get a similar output

Our Lambda function successfully extracts the order event type and payload and then sends the email to the user.

Deploy the resources on AWS

To deploy the notification service on AWS, run the following commands:

npm run cdk bootstrap

export $(grep -v '#' .env | xargs) && npm run cdk synth

export $(grep -v '#' .env | xargs) && npm run cdk deploy

Wait for the deployment to complete.

Test the complete workflow

The core service creates an order and publishes an SNS topic, while the notification service subscribes to the SNS topic and sends an email.

Here is the complete demo of the workflow:

Best practices when working with AWS SNS

When working with AWS SNS on real-world applications, it is essential to ensure reliability between services. Here are some best practices to enforce:

• Connect your SNS topic to an SQS queue to prevent message loss forever and avoid overloading your Lambda function in pick season.
• Attach a Dead Letter Queue(DLQ) to your SNS topic to prevent losing messages that have failed to be delivered. This will help debug and improve the retry mechanism.
• Avoid creating unnecessary SNS topics and encourage using the SNS message filtering feature to notify only subscribers matching the event payload in their filtering policy.
• Centralize the TypeScript types of the SNS event payload and share them between services publishing/subscribing to SNS events. This will maintain type consistency with your services.

Destroy the stack

A significant advantage of using IaC like the AWS CDK is the ability to destroy the AWS resources in a single command.

To destroy the stack of the core and notification service, run the command below:

cd notification-service

npm run cdk destroy

cd ../core-service

yarn cdk destroy 

The notification service stack must be deleted first because it depends on the core service stack.

Wait for the stack to be deleted on AWS and go to the console to ensure they no longer exist.

Wrap up

AWS SNS allows asynchronous communication between many services, encouraging business domain decoupling and allowing applications to evolve autonomously.

The steps for making two services communicate through AWS SNS are as follows:

• Create the SNS Topic resource using the AWS CDK.
• Grant the publication to the service publishing events on the SNS topic.
• Publish an SNS event using the AWS SDK client for SNS.
• In the subscriber service, grant the subscription to the SNS topic.
• Handle the SNS event and execute the business logic associated with the event.

Remember the best practices when building real-world applications using AWS SNS to have reliable services.

You can find the code source on the GitHub repository.

Follow me on Twitter or subscribe to my newsletter to avoid missing the upcoming posts and the tips and tricks I occasionally share.