How to route dead letter messages back to the original topic?

Question

I have a Google Cloud Pub/Sub subscription that is using a dead-letter topic. I recently had an outage that was preventing a number of messages from being processed & they ended up in the dead-letter topic.

The outage was resolved and I'd like to easily send the contents of dead letter subscription back to the original subscription. They're all present in the queue still (I have nothing consuming the dead letter sub) so I just need to route them somewhere.

This is an admin task so I'd like it to be manually initiated, if that makes any difference. Ideally via the UI but I can't see anything there.

Friedrick

How are you processing your messages ? You can manually start a new batch job consuming message from dead letter topic. – Muley

have you created a pull subscription on your deadletter topic? In addition, is it a problem is you re-process already processed messages? – Zoller

Was your problem solved? – Shenashenan

No pull subscription has been configured, the messages stay in the queue for manual inspection without ack. – Friedrick

Keep in mind that if you deliver back to the topic, all subscribers will get the message again which, if you're consumers are not designed to be idempotent, could be problematic. I wish there were a way to redeliver to the original subscription which is a request I put in with the product team a couple times now. I think in the meanwhile, the "best practice" is actually to process the dead-letter subscription before switching to the main subscription on startup. – Armalda

Answer 1

You have a few options:

• Use a Dataflow pipeline to move messages from dead letter topic to your topic.
• Update existing pipeline to read from both original topic and dead letter topic based on configuration
• Create a new system that, when enabled, moves messages from one topic to another.

The right answer might depend on your system design and requirements.

In case your use case for dead letter topic always includes moving the messages back to the primary topic after a delay, you might want to use configurable exponential backoff in Cloud Pub/Sub. This feature will have general availability towards the end of Q2 2020.

Samalla

Interesting - I like the RetryPolicy, would that be configurable through the UI once GA? In this case, the messages were undelivered because of an issue that lasted ~hours, so they were shunted off to the DLQ. I think your first option is the best, if I can define and run them ad-hoc. Thanks for the direct link. – Friedrick

RetryPolicy will be configurable through the UI once GA. – Samalla

You can now configure a retry policy in your Cloud Pub/Sub subscription. Retry policy allows users to have per message exponential backoff in case of failures. More information can be found here. – Samalla

Sounds to me like one could implement your "moving the messages back to the primary topic after a delay" in a way that requires no manual intervention by setting the dead letter topic of the dead letter subscription to the primary topic. An application whose only purpose would be to nack messages from the dead letter subscription would be running always. It would be an infinite loop, but a long delay in the retry policy of the dead letter subscription would mean after a long period of time, the messages would eventually end up back in the primary subscription to be reprocessed. – Allopath

hmmm I have to say that this seems pretty friggin' nuts that the base case use of a DLQ must be handled in such a convoluted way... I'm wondering if someone has made a CLI script to accomplish this, I've seen it for AWS's SQS in their docs – Vender

Actually here looks like a good reference to craft a script from: github.com/samswen/gcp-utils/blob/master/src/index.js – Vender

okay - I went ahead and made a script here: #61650587 – Vender

Answer 2

For the frustrated few who may come across this issue, I went ahead and made a Node script here for republishing messages from subscription back to topic that you can hopefully easily adapt for your own use.

It can of course be used this DLQ --> main republish scenario but also any other scenario where you may want to manually republish from subscriber to topic.

https://github.com/sirrodgepodge/gcp-pubsub-republish/blob/main/index.ts

Note that the Dataflow pipeline approach mentioned in Mahesh Gattani's answer does work but was going to cost $120/month per queue and that felt like a bit much for such a basic need.

Vender

Answer 3

There is another option not mentioned here yet:

• Create a temporary subscription and then replay the messages that were moved to the Dead Letter Topic using seek on this temporary subscription.

Use seek with a timestamp that goes back before the first message was moved to the DLT.

After messages have been delivered through the temporary subscription, delete it. Also purge all messages from the pull subscription on the DLT.

This solution requires no extra code and reuses existing infrastructure as much as possible.

Mcgarry

This might be useful -- to confirm, you're suggesting that one could 1) create a temporary subscription that subscribes to the DLQ topic but uses the same push endpoint configuration as the original, 2) on the existing DLQ subscription, create a snapshot of DLQ messages awaiting acknowledgement, 3) replay messages on the existing DLQ subscription, and 4) delete the temp subscription? (one risk being that new messages that get delivered to the DLQ between 1 and 4 will immediately go to the temp subscription?) – Strabismus