Home Cloud Computing Asserting throughput enhance and useless letter queue redrive assist for Amazon SQS FIFO queues

Asserting throughput enhance and useless letter queue redrive assist for Amazon SQS FIFO queues

0
Asserting throughput enhance and useless letter queue redrive assist for Amazon SQS FIFO queues

[ad_1]

Voiced by Polly

With Amazon Easy Queue Service (Amazon SQS), you’ll be able to ship, retailer, and obtain messages between software program parts at any quantity. In the present day, Amazon SQS has launched two new capabilities for first-in, first-out (FIFO) queues:

  • Most throughput has been elevated as much as 70,000 transactions per second (TPS) per API motion in chosen AWS Areas, supporting sending or receiving as much as 700,000 messages per second with batching.
  • Lifeless letter queue (DLQ) redrive assist to deal with messages that aren’t consumed after a selected variety of retries in a method just like what was already obtainable for normal queues.

Let’s take a extra in-depth take a look at how these work in observe.

FIFO queues throughput enhance as much as 70K TPS
FIFO queues are designed for functions that require messages to be processed precisely as soon as and within the order during which they’re despatched. Whereas commonplace queues have an infinite throughput, FIFO queues have an higher quota within the variety of TPS per API motion.

Commonplace and FIFO queues assist batch actions that may ship and obtain as much as 10 messages with a single API name (as much as a most complete payload of 256 KB). Which means a FIFO queue can course of as much as 10 occasions extra messages per second than its most throughput.

At launch in 2016, FIFO queues supported as much as 300 TPS per API motion (3,000 messages per second with batching). This was sufficient for a lot of use instances, however some clients requested for extra throughput.

With excessive throughput mode launched in 2021, FIFO queues launched a tenfold enhance of the utmost throughput and will course of as much as 3,000 TPS per API motion, relying on the Area. One 12 months later, that quota was doubled to as much as 6,000 TPS per API motion.

This 12 months, Amazon SQS has already elevated FIFO queue throughput quota two occasions, to as much as 9,000 TPS per API motion in August and as much as 18,000 TPS per API motion in October (relying on the Area).

In the present day, the Amazon SQS workforce has been capable of enhance the FIFO queue throughput quota once more, permitting you to course of as much as 70,000 TPS per API motion (as much as 700,000 messages per second with batching) within the US East (N. Virginia), US West (Oregon), and Europe (Eire) Areas. That is greater than 200 occasions the utmost throughput at launch.

DLQ redrive assist for FIFO queues
With Amazon SQS, messages that aren’t consumed after a selected variety of retries can mechanically be moved to a DLQ. There, messages may be analyzed to know the rationale why they haven’t been processed accurately. Generally there’s a bug or a misconfiguration within the shopper software. Different occasions the messages include invalid knowledge from the supply functions that must be mounted to permit the messages to be processed once more.

Both method, you’ll be able to outline a plan to reprocess these messages. For instance, you’ll be able to repair the buyer software and redrive all messages to the supply queue. Or you’ll be able to create a devoted queue the place a customized software receives the messages, fixes their content material, after which sends them to the supply queue.

To simplify shifting the messages again to the supply queue or to a unique queue, Amazon SQS means that you can create a redrive job. Redrive duties are already obtainable for normal queues. Beginning at the moment, it’s also possible to begin a redrive job for FIFO queues.

Utilizing the Amazon SQS console, I create a primary queue (my-dlq.fifo) for use as a DLQ. To redrive messages again to the supply FIFO queue, the queue sort should match, so that is additionally a FIFO queue.

Then, I create a supply FIFO queue (my-source-queue.fifo) to deal with messages as ordinary. Once I create the supply queue, I configure the primary queue (my-dlq.fifo) because the DLQ and specify 3 because the Most receives situation below which messages are moved from the supply queue to the DLQ.

Console screenshot.

When a message has been acquired by a shopper for greater than the variety of occasions specified by this situation, Amazon SQS strikes the message to the DLQ. The unique message ID is retained and can be utilized to uniquely observe the message.

To check this setup, I take advantage of the console to ship a message to the supply queue. Then, I take advantage of the AWS Command Line Interface (AWS CLI) to obtain the message a number of occasions with out deleting it.

aws sqs receive-message --queue-url https://sqs.eu-west-1.amazonaws.com/123412341234/my-source-queue.fifo

{
    "Messages": [
        {
            "MessageId": "ef2f1c72-4bfe-4093-a451-03fe2dbd4d0f",
            "ReceiptHandle": "...",
            "MD5OfBody": "0f445a578fbcb0c06ca8aeb90a36fcfb",
            "Body": "My important message."
        }
    ]
}

To obtain the identical message greater than as soon as, I look forward to the time specified within the queue visibility timeout to cross (30 seconds by default).

After the third time, the message is just not within the supply queue as a result of it has been moved to the DLQ. When I attempt to obtain messages from the supply queue, the record is empty.

aws sqs receive-message --queue-url https://sqs.eu-west-1.amazonaws.com/123412341234/my-source-queue.fifo

{
    "Messages": []
}

To verify that the message has been moved, I ballot the DLQ to see if the message is there.

aws sqs receive-message --queue-url https://sqs.eu-west-1.amazonaws.com/123412341234/my-dlq.fifo  

{
    "Messages": [
        {
            "MessageId": "ef2f1c72-4bfe-4093-a451-03fe2dbd4d0f",
            "ReceiptHandle": "...",
            "MD5OfBody": "0f445a578fbcb0c06ca8aeb90a36fcfb",
            "Body": "My important message."
        }
    ]
}

Now that the message is within the DLQ, I can examine why the message has not been processed (nicely, I do know the rationale this time) and determine whether or not to redrive messages from the DLQ utilizing the Amazon SQS console or the brand new redrive API that was launched a couple of months in the past. For this instance, I take advantage of the console. Again on the Amazon SQS console, I choose the DLQ queue and select Begin DLQ redrive.

In Redrive configuration, I select to redrive the messages to the supply queue. Optionally, I can specify one other FIFO queue as a customized vacation spot. I take advantage of System optimized in Velocity management settings to redrive messages with the utmost variety of messages per second optimized by Amazon SQS. Optionally, if there’s numerous messages within the DLQ, I can configure a customized most fee of messages per second to keep away from overloading shoppers.

Console screenshot.

Earlier than beginning the redrive job, I can use the Examine messages part to ballot and test messages. I already determined what to do, so I select DLQ redrive to begin the duty. I’ve just one message to course of, so the redrive job completes in a short time.

Console screenshot.

As anticipated, the message is again within the supply queue and is able to be processed once more.

Console screenshot.

Issues to know
Lifeless letter queue (DLQ) assist for FIFO queues is obtainable at the moment in all AWS Areas the place Amazon SQS is obtainable except GovCloud Areas and people based mostly in China.

Within the DLQ configuration, the utmost variety of receives ought to be between 1 and 1,000.

There isn’t a further value for utilizing excessive throughput mode or a DLQ. Each Amazon SQS motion counts as a request. A single request can ship or obtain from 1 to 10 messages, as much as a most complete payload of 256 KB. You pay based mostly on the variety of requests, and requests are priced in a different way between commonplace and FIFO queues.

As a part of the AWS Free Tier, there is no such thing as a value for the primary million requests per 30 days for normal queues and for the primary million requests per 30 days for FIFO queues. For extra data, see Amazon SQS pricing.

With these updates and the elevated throughput, you’ll be able to cowl the overwhelming majority of use instances with FIFO queues.

Use Amazon SQS FIFO queues to have excessive throughput, exactly-once processing, and first-in-first-out supply.

Danilo



[ad_2]

Supply hyperlink

LEAVE A REPLY

Please enter your comment!
Please enter your name here