NiFi after calling HTTP API, how to get original flowfile before calling HTTP API

Question

I have the below sequence for calling rest API.

Generate FLowfile processor-->jsonpath processor-->text replace processor(for post data creation)---> InvokeHTTP--->XPATH processor for attribute--->original flow file that is generated by generate flow file.

SO, after text replace processor the original data will be replaced with new data. So, how can I get the original data and use the attributes produced after calling API.

Rupee

Answer 1

You can either maintain the original flowfile with a direct output relationship from GenerateFlowFile and merge the flowfiles later with MergeContent with mode Defragment and Keep All Unique Attributes, or if the original flowfile content is small enough, you can move it to an attribute before changing the flowfile content and then recombine them after you receive the new data with Update Attribute/ReplaceText.

Ridley

can you give please me an example or a link as for this – Rupee

Answer 2

at the point where you have original file insert UpdateAttribute with evaluation of some unique attribute.

for example MyUID = ${UUID()}

the success connection after UpdateAttribute should go to preparation flow to invoke http, and the copy of this connection should go to MergeContent that should combine original and evaluated content & attributes.

flow:

UUID & split:

merge content:

Sarnen

Answer 3

The two generic answers are:

1. MergeContent as explained by dagget
2. Wait/Notify as described here (originally by Andy)

These solutions should scale well and be considered in the first place. I personally find them a bit complex, so therefore I also propose a worarkound here.

If you are doing HTTP requests, perhaps you may only be processing a small amount of very small messages. In this case you can consider the following 'trick' to avoid complexity.

Rather than splitting, consider (ab)using an attribute

The easiest fix is by keeping all data together. Rather than splitting your message, just put a copy of the original content in an attribute. This attribute will still be available after your HTTP request, regardless of what is returned in the content.

The simple solution would be to use an ExtractText right before the HTTP request, and create an attribute called something like original with the whole content.

Mandatory warning: Attributes are designed to be small and are therefore stored in memory. Hence putting large content in an attribute may consume your memory quickly.

---

A final workaround for possible future readers: If you control the HTTP service, or at least know the specs, consider whether it is desirable to let the output contain the input as well. Often not, but sometimes you get it anyway!

Vardon