Can SQS scale up to 1,000,000 queues for a single account?

Question

I need a messaging service that allows me to create a channel for each user in order to facilitate real-time notifications. If I have somewhere between 100,000 and 1 million users, does it make sense to create an SQS queue for each of these users?

According to the SQS pricing documentation it would only cost $0.40 to create 1 million queues, but would I run into scaling problems?

Also, is there a way to set an expiration date on a queue? If a user deletes their account, then their queue no longer needs to exist.

Antler

How would a system with a queue for each user possibly work? You might be asking the wrong questions here – Pulverulent

@Pulverulent Each notification that is generated will belong to a single user. A user will need to subscribe to a channel that has notifications only for that user. Thus, each user needs their own channel – Antler

Use a pub/sub setup like XMPP or PubNub. This is a terrible way to solve the problem. – Schnabel

To implement a real-time solution using SQS would require constant polling. This will be very very expensive for a million queues. Create a web page and have the user subscribe to an SNS topic. This could be done in a few hours with the AWS PHP SDK. However, managing a million subscribers is another matter. – Jahdol

@Antler you should look into node.js as a solution for the problem you are trying to solve. Its highly optimized for quick I/O (perfect for simple messages in real time) and scales horizontally very well – Pulverulent

@Schnabel PubNub sounds like it could be the solution I'm looking for. The only concern I have is that the pricing is done per device. Per device pricing I think might work well for IoT, but I'm working on an web application where users may have several devices. – Antler

@JohnHanley I disagree that this solution would require constant polling. With long polling, the user will only need to poll once until a notification is received. – Antler

Where did you read that? With long polling the max time is 20 seconds. This will again require constant polling. docs.aws.amazon.com/AWSSimpleQueueService/latest/… – Jahdol

@JohnHanley Oh wow. Good point. I didn't realize the max time was 20 seconds. – Antler

Another factor: time to write to a million queues. This would use a fair amount of bandwidth due to TCP/IP overhead even for small message but take a lot of time. This article talks about the number of writes per second to SQS: warski.org/blog/2014/06/benchmarking-sqs – Jahdol

Answer 1

Creating queues is not an issue here. Polling or even long polling the queue is going to be really expensive for you. In order to process real-time notifications, you need to poll every queue, 1M of them for lets say every 5 seconds.

Based on SQS Pricing, Price per 1 Million Requests after free tier is $0.00000040 per request.

That means you will be calling the ReceiveMessage API for about:

1000000 queues * 17280 (1 day in seconds / 5 seconds) = 17280000000 times.

Which is about $6912.00 per day for the worst case scenarios.

You need to architect the solution in a better way.

Trantham

A user will only need to poll the queue when they are logged in. So not all 1,000,000 queues will be polled everyday. Also, with long polling I doubt the a user would need to poll his or her queue every 5 seconds – Antler

@JohnHanley brought up a good point. That the maximum timeout for a long poll on SQS is 20 seconds. Although I would most likely set the timeout to 20 seconds rather than 5 seconds, 5 seconds is still a reasonable estimate. Accepting this answer. – Antler

Answer 2

"a channel for each user in order to facilitate real-time notifications" - you don't need a dedicated queue per user for this - you can do this with one main messaging queue, and depending on your traffic patterns, probably a few overflow queues to deal with ultra-high-traffic users.

"One queue" you say? "How on earth will that scale to 1M users?"

The number of users doesn't matter. What matters is that your message consumption can keep up with message production (incoming messages per second). If you can do that, it'll seem like realtime to your users.

• Message consumption can scale to as high as you're willing to spend - just spawn up a thread to handle each incoming message (use a thread pool!)
  • of course, you'll need to limit each host to X processing threads, based on how many it can handle (hence 'as high as you're willing')
  • the overflow queues are to keep costs under control - if you're scaled to handle 10K messages per second, you don't want a user to come along and send you 1M messages per second, knocking out your service and the rest of your customers - throttle them to some reasonable limit, and process the rest of those messages at a lower priority.

"But... millions." - Yes. SQS can handle a lot. And a single multi-tenant queue will scale much better in terms of architecture and cost than multiple single-tenant channels ever will.

Kennethkennett

Answer 3

Most AWS resource volumes are limited, and while I'm not finding any account limits on numbers of queues, I may have missed it or it may just not be published. I definitely wouldn't be excited about the queue per notification destination architecture you're pitching here, if my co-workers brought it to me. I would be concerned about the cost of putting the same notification to all the listeners' queues, and then reading them back out.

What you're describing sounds more like pub sub. Or, if you want better delivery guarantees, maybe a stream like kinesis or kafka. I've also heard of folks using Redis to implement this kind of thing.

Bul

Thanks for the response! I should mention that each notification does not need to be sent to each user. A notification will only be sent to one user - making the cost of sending that notification only $0.00000040 – Antler

I was surprised to actually find that "You can create any number of message queues." but then I realized... queues don't really cost AWS anything to provision -- it's the traffic that has costs for them (and revenue) and for a given number of send/poll/receive/delete cycles, it would not really matter whether you hit 100 queues 100 times or 1000 queues 10 times, etc. "Queues" are just string tags for traffic, when you peel away the layers deep enough. – Riha

Answer 4

Could you potentially design a queue consumer that pauses after a certain period of idle time to prevent unnecessary API calls?

Something like:

const AWS = require('aws-sdk');

// Set the AWS region
AWS.config.update({ region: 'YOUR_REGION' });

// Set the parameters for the consumer
const IDLE_TIMEOUT = 300; // Stop Polling after 300 seconds of idle time
const POLLING_INTERVAL = 10; // Poll the queue every 10 seconds

// Create an SQS client
const sqs = new AWS.SQS();

// Set a flag to control the Polling loop
let isPolling = false;

function startPolling() {
  isPolling = true;
}

function stopPolling() {
  isPolling = false;
}

async function processMessage(message) {
  // Do something with the message here
  console.log(message);
}

async function pollQueue() {
  // Set the idle timer to 0
  let idleTimer = 0;

  while (isPolling) {
    // Check if the idle timer has reached the timeout
    if (idleTimer > IDLE_TIMEOUT) {
      // Stop Polling
      stopPolling();
      break;
    }

    // Poll the queue for messages
    const params = {
      QueueUrl: 'YOUR_QUEUE_URL',
      MaxNumberOfMessages: 10,
      WaitTimeSeconds: POLLING_INTERVAL,
    };
    const data = await sqs.receiveMessage(params).promise();

    // Get the messages from the response
    const messages = data.Messages || [];

    // Process the messages
    for (const message of messages) {
      await processMessage(message);

      // Delete the message from the queue
      const deleteParams = {
        QueueUrl: 'YOUR_QUEUE_URL',
        ReceiptHandle: message.ReceiptHandle,
      };
      await sqs.deleteMessage(deleteParams).promise();

      // Reset the idle timer
      idleTimer = 0;
    }

    // Increment the idle timer
    idleTimer += POLLING_INTERVAL;

    // Sleep for the Polling interval
    await new Promise((resolve) => setTimeout(resolve, POLLING_INTERVAL * 1000));
  }
}

// Start the Polling loop when the startPolling function is called
startPolling();
pollQueue();

This way you could only activate the consumer after some process begins, and avoid constant polling of the queue when the service is inactive.

Lugsail