The Simple Notification Service is often overlooked when it comes to Managing Scale. It is actually one of the most useful services in the AWS backbone. This article will show you how you can leverage SNS to meet the demand on your Apps, as well as combining with CloudWatch.

What SNS is about?

Simply put, SNS is a Publish-Subscribe Middleware in Amazon Web Services. As such, it offers topics under which clients can create subscriptions and receive notifications from. Current Protocols include:

• E-mail
• HTTP / HTTPS (they’re considered distinct endpoints)
• SMS (yes, text messages for your phone)
• And Simple Queue Service

One of the most common uses of SNS comes around when you either create Notification Topics for Elastic Beanstalk, as well as S3’ Reduced Redundancy Storage (RRS) Notifications. The most common use is related to CloudWatch, by which a simple query into your mailbox will attest:

But SNS is not for mailing you alerts. In fact, you can easily create SNS endpoints for automating most of your tasks, when combined with CloudWatch.

Learn how to Enable Notifications for Lost Objects in (Reduced Redundancy Storage) RRS

A Real Case: DynamoDB

What if we could use SNS to automatically manage our scaling needs for DynamoDB? That was the first idea which crossed my mind. In particular, it is very common for me to forget DynamoDB tables with more throughput for too long a period.

That deserves a CloudWatch alarm, doesn’t it? In particular, one that triggers when a table has Consumed Units equal to Zero for more than 12 hours. In fact, two alarms: One for Read, another for Write.

So here’s what I did: I went into CloudWatch and picked a couple of tables. Created alarms for the following situations for the same table:

• ConsumedWriteCapacityUnits < 5 for 1,440 minutes
• ConsumedReadCapacityUnits < 5 for 1,440 minutes

I added email notifications in the same topic, with the table name (“ca-asset-grant”) and added my email. So there’s a single point of notification created.

Actually, your context might be a bit different. No problem, CloudWatch is easy to tinker with, so spend your time and pick your best solution.

Then back to SNS

It is time to look more closely at SNS. If you received the confirmation email, click on the link, and you’ll get into a page like this one:

This means a topic was created, and there’s already one subscriber. Go into the “SNS” tab of the AWS Console and look under the “ca-asset-grant” topic:

Some ideas on what you can do:

• Request a subscription for someone else (not me! J)
• Publish messages to try and see if it works
• Create filters in your email inbox to classify it, so your inbox does not get cluttered
• Create SMS alerts, if your carrier supports it (see its FAQ)

However, let’s take a different route: Postbin

Postbin with SNS

Postbin is a play on the concept of Paste Bins. Quoting Wikipedia (here):

A pastebin is a type of web application where anyone can store text for a certain period of time. This type of website is mainly used by programmers to store pieces of source code or configuration information, but anyone can basically share any type of text. The idea behind pastebins is to make it more convenient for people to share large amounts of text online. A vast number of pastebin related websites exist on the Internet, suiting a number of different needs and providing features tailored towards the crowd they focus on most.

A postbin is like a pastebin, instead it stores HTTP Requests. There are several, so one can easily google for “Postbin”. I instead took requestb.in and created one for myself. Simply log in and click on ‘create one for me’, and it will redirect you to an address you can copy and paste into a new HTTP subscription for SNS.

Then I clicked “Subscribe”. SNS showed the subscription as “Pending Confirmation”. If I switched back to RequestBin, I’d see that SNS actually sent a message to my endpoint, to confirm this:

So SNS always requires confirmation in order to accept, copy and paste the “SubscribeURL” value onto your browser. You’ll get a confirmation message as usual, but this time, the response will be in XML. At this point, the SNS console will reflect the change by showing a valid “Subscription ID” where it was reflecting “Pending Confirmation”.

I published a “Hello World” message. Requestbin confirmed by showing what it received from SNS:

Subscribing SQS Tables

Simple Queue Service is the point-to-point version of SNS, and it enables you to store messages created out of notifications into a Durable, Persistent Queue. This is useful, for instead of processing a message on demand, you can place it on a queue and assign several parallel workers to poll and peek. Or just troubleshoot all the messages as Notification Received without clogging your mailbox.

So what if you wanted to subscribe to Amazon SQS Tables? No problem: Jump into SQS and create a queue. In my case, I just set a name and accepted the details:

Once I created the Queue, I asked to subscribe to the SNS Topic:

Notice you could also subscribe from the SNS Console. So why did we subscribe from the SQS Console instead of the SNS? The answer is simple: The SQS table requires a special AWS policy to accept SNS subscriptions. The support to create SQS Queues with the right policy was added a few months ago. But if you’re calling the API, here is the generated policy to accepting Queue Messages from a Topic:

{
  "Version": "2008-10-17",
  "Id": "arn:aws:sqs:us-east-1:235368163414:idle-dynamodb-tables/SQSDefaultPolicy",
  "Statement": [
    {
      "Sid": "Sid1358129412837",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:235368163414:idle-dynamodb-tables",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:us-east-1:235368163414:idle-dynamodb-table"
        }
      }
    }
  ]
}

Publish a message and confirm you’re receiving it, that’s all we need.

Confirming SNS Message Authenticity

Each SNS message is digitally signed by a certificate hosted in AWS, and it is described in each call by the “SigningCertURL” JSON Property. Here are some details, courtesy of Keystore Explorer:

The AWS FAQ states it cryptically (source):

“Signature: Base64-encoded “SHA1withRSA” signature of the Message, MessageId, Subject (if present), Type, Timestamp, and Topic values.”

But here is the deal: The AWS SDK for Java already includes it, in the form of the “SignatureChecker” class. Here’s a Unit Test to confirm a message.

Note it requires Jackson – Which the AWS SDK already declares as dependent on it, so if you’re using Maven, it is not a concern.

package br.com.ingenieux.services.web;
import java.net.URL;
import java.security.PublicKey;
import java.security.cert.Certificate;
import java.security.cert.CertificateFactory;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.node.ObjectNode;
import org.junit.Before;
import org.junit.Test;
import com.amazonaws.services.sns.util.SignatureChecker;
public class SignatureTest {
	ObjectMapper objectMapper;
	ObjectNode messageObj;
	String originalMessage;
	@Before
	public void before() throws Exception {
		objectMapper = new ObjectMapper();
		this.originalMessage = "{ \"Type\" : \"Notification\", \"MessageId\" : \"ee945a43-582c-55c9-a360-132e334f24b4\", \"TopicArn\" : \"arn:aws:sns:us-east-1:235368163414:idle-dynamodb-table\",  \"Subject\" : \"Test Message\",  \"Message\" : \"Hello, World!\",  \"Timestamp\" : \"2022-01-14T02:10:43.261Z\", \"SignatureVersion\" : \"1\", \"Signature\" : \"QhrMr8wxpxPi2izCevjihxJJw11Zn5CFmDvdV7Qp+J/IOAPdSuYrmencSGk9eVbPlZLG+OEZKnjp5G3udYCysFzsPqauwsAF3gmdKbxn4VaN4F8KMNi1Sw+fIbfAwj+MyPXh3reZV/R/rIuY2yBKXaym04bNujUxrMspKLBWhFI=\", \"SigningCertURL\" : \"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem\", \"UnsubscribeURL\" : \"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:235368163414:idle-dynamodb-table:0c7c26fc-726e-4986-a132-276754757cbc\"}";
		this.messageObj = (ObjectNode) objectMapper.readTree(originalMessage);
	}
	@Test
	public void testSignature() throws Exception {
		SignatureChecker sigChecker = new SignatureChecker();
		String signature = messageObj.get("Signature").getTextValue();
		URL signingCertURL = new URL(messageObj.get("SigningCertURL").getTextValue());
		Certificate x509Cert = CertificateFactory.getInstance("X.509").generateCertificate(signingCertURL.openStream());
		PublicKey publicKey = x509Cert.getPublicKey();
		sigChecker.verifySignature(originalMessage, signature, publicKey);
	}
}

I’ve updated my ElasticBeanstalk Archetype to do the signature validation part (those changes were needed in BaseSNSResource.java), and it is going to be published in the next release. Regardless of that, there’s a sample project encompassing all the needs for SNS publishing.

Your Own SNS Listener

You can import the project in the previous link into your IDE (don’t forget to review the README.md at its root path), and Debug the “ServerRule” class – It will create an Embedded Web Container. The tricky part is publishing an Endpoint in the Open.

You can rely on localhost tunnel services, open up your DSL Modem, or create an SSH Tunnel with:

$ ssh user@host –R8099:127.0.0.1:8080 –g

Don’t forget the ‘-g’ part of the command above. In the examples below, replace <your-remote-address> to the host you’ve used above, and <port> for ‘8099’ (or whichever port you found open).

Regardless of that, test if <your-remote-address>:<port>/services/api/v1/debug returns a message, then we’re fine to go on to the next step.

The critical part is the BaseSNSResource class. So place a breakpoint in the onSNSMessage method, then ask SNS to create a HTTP subscription to:

http://<your-remote-address>:<port>/services/api/v1/sns/<your-dynamodb-table>

Once you submit the subscription request, you’ll notice the flow, which means:

• You receive a subscription request
• You confirm the source of the certificate, and validate the request
• You call the API to confirm the subscription using the provided token
• You receive another informational message, which gets discarded (but it is validated anyway)

Then, publish into the Topic. If you look at the source of SNSResource.java source code, you’ll notice it uses the DynamoDB table name as a parameter, and resizes it to one:

package com.newvem.sns.resource;
import javax.inject.Inject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import org.codehaus.jackson.node.ObjectNode;
import com.amazonaws.services.dynamodb.AmazonDynamoDB;
import com.amazonaws.services.dynamodb.model.ProvisionedThroughput;
import com.amazonaws.services.dynamodb.model.UpdateTableRequest;
import com.amazonaws.services.sns.model.ListTopicsResult;
import com.amazonaws.services.sns.model.PublishRequest;
import com.amazonaws.services.sns.model.PublishResult;
import com.amazonaws.services.sns.model.Topic;
/**
 * Represents a Minimalistic SNS Resource Template
 * 
 * @author aldrin
 */
@Path("/sns")
public class SNSResource extends BaseSNSResource {
	@Inject
	AmazonDynamoDB dynamoDB;
	@GET
	@Produces("text/plain")
	@Path("/test/" + ID_MASK)
	public String sendMessage(@PathParam("id") String tableId)
			throws Exception {
		PublishRequest publishRequest = new PublishRequest();
		publishRequest.setMessage("{ \"Trigger\":{ \"Dimensions\":[ { \"name\":\"TableName\", \"value\":" + tableId + " } ] } }");
		publishRequest.setSubject("Reduce Request");
		ListTopicsResult listTopics = snsClient.listTopics();
		for (Topic t : listTopics.getTopics()) {
			if (t.getTopicArn().endsWith(tableId)) {
				publishRequest.setTopicArn(t.getTopicArn());
				break;
			}
		}
		PublishResult result = snsClient.publish(publishRequest);
		return result.getMessageId();
	}
	@Override
	public void handleNotification(String endpointId, ObjectNode bodyNode)
			throws Exception {
		ProvisionedThroughput provisionedThroughput = new ProvisionedThroughput();
		provisionedThroughput.setReadCapacityUnits(1L);
		provisionedThroughput.setWriteCapacityUnits(1L);
		ObjectNode messageData = (ObjectNode) objectMapper.readTree(bodyNode.get("Message").getTextValue());
		String tableName = messageData.get("Trigger").get("Dimensions").get(0).get("value").getTextValue();
		UpdateTableRequest updateTableRequest = new UpdateTableRequest()
				.withTableName(tableName)
				.withProvisionedThroughput(provisionedThroughput);
		try {
			dynamoDB.updateTable(updateTableRequest);
		} catch (Exception exc) {
			if (logger.isWarnEnabled())
				logger.warn("Error", exc);
			// Comment-out the next section when in production
			throw exc;
		}
	}
}

Publishing Messages

I added one resource handler to look at the existing tables, and publish with the topic ARN ending with the table name, so you can call with:

http://<your-remote-address>:<port>/services/api/v1/sns/test/<your-dynamodb-table>

This pretty much sums up everything you need to publish into an SNS Topic, which includes:

• The Topic’s Amazon Resource Name (ARN)
• The message subject
• And its content

If you look at its source, you’ll notice it is plain JSON. It will get escaped and wrapped. This explains why we decode the JSON payload again into the SNS Handler.

Testing the DynamoDB Alarm

Some situations I’ve run into, with solutions:

• No alarms were sent: Create another one, of 5 minutes length. Also, simply click “Explore” in the DynamoDB Console, as the CloudWatch alarm needs real usage data to trigger
• I didn’t get my message: Look into the postbin. If you notice, the payload is within the message. Open, copy the message, paste into an editor, unescape (i.e., remove the “\”), copy and send as a new message
• My handler failed and I keep receiving the same request: In DynamoDB, there are limits on reducing tables (once per day). Comment the “throw exc” line in SNSResource.java and test again

The SNS Retry Logic

When debugging, you might have noticed that once you’ve debugged through a handler, another call is made. If you compare with the requestbin, the same request was received just once by the requestbin. Why?

The reason is that the request took longer, so AWS trashed it and considered it worth another retry. Let me quote what the SNS Getting Started Guide says (source):

Amazon SNS only attempts a retry after a failed delivery attempt. Amazon SNS considers the following situations as a failed delivery attempt.

• HTTP status in the range 500-599.
• HTTP status outside the range 200-599.
• A request timeout (15 seconds). Note that if a request timeout occurs, the next retry will occur at the specified interval after the timeout. For example, if the retry interval is 20 seconds and a request times out, the start of the next request will be 35 seconds after the start of the request that timed out.
• Any connection error such as connection timeout, endpoint unreachable, bad SSL certificate, etc.

As it took longer than 15 seconds, it tried after 35 seconds, and so forth.

For more information, read the Getting Started Guide. The defaults are pretty sane, but the hard limit is: A message delivery gets queued until one hour, and your settings must reflect it.

Moving Forward

My Resolution for 2022 was to care less. This involved less maintenance, as well as automating everything I’ve wanted – and learning in the process.

SNS is such a great thing, as it helps me automate those parts. Some applications I’ve envisioned include:

• Setting Up Nagios on my Servers – and publishing the Metrics into CloudWatch, especially Disk I/O as well as Free Space
• Making a single handler for my BitBucket Projects to get built into my Jenkins with minimal to no configuration
• Combining with tools like “ifttt”, especially feed and calendaring needs
• Integrating it with my mobile (Pushover comes to mind) and ditch my email – or at least filtering what is really important
• Adding some scripting in my SNS Handlers, for things like writing SNS Handlers on the fly and avoiding re-deployment altogether

In a nutshell, the plan is to build a “Personal ESB” of some sorts, and try to achieve world domination with that. I, for one, welcome our new Robot overlords.

---

[Newvem analytics tracks you AWS cloud utilization:

• Hourly Utilization Pattern Analysis 
• Reserved Instances Decision Tool 
• Resource Resizing Opportunities

Get Started for Free or Learn More]

---

About the Author

Aldrin Leal, Cloud Architect and Partner at ingenieux Aldrin Leal works as an Architect and QA Consultant, specially for Cloud and Big Data cases. Besides his share of years worth between the trenches in projects ranging from Telecom, Aerospatial, Government and Mining Segments, he is also fond with a passion to meet new paradigms and figure a way to bring them into new and existing endeavours.

Contact Aldrin

---

Keywords: Amazon web services, Amazon AWS console, Amazon Cloud Services, DynamoDB, Dynamo db, AWS CloudWatch, CloudWatch, AWS SNS, Simple Notification Service, CloudWatch Alarm, Simple Queue Service, Amazon SQS, RRS, Reduce Redundant Storage, AWS API, Monitoring, Alerts, Cost Efficiency, NoSQL

Content Disclaimer