How Can I View FlowFile or STDOUT of Processor?

Question

When things fail, I'd like to view the flow file, or the output (stdout+stderr) from the problematic processor. Is there an easy way to dump out all of the Flowfile's properties, or to just browse a Flowfile?

Jasso

Answer 1

Processors usually have one or more relationships for failures and it is up to the data flow designer to determine what to do with these.

Some failures are due to temporary conditions like if a destination system was down, for those they would typically be looped back to the same processor to keep retrying until the destination comes back up.

Other failures are due to issues related to the data and likely don't make sense to retry because they will continue to fail. This set of failures you can route to PutFile processor to write them out to directory somewhere, or a PutEmail processor to notify you. Either of those would give you access to the raw data. If you want to see the flow file attributes you could use data provenance to look at all the flow files that passed through the PutFile/PutEmail processor.

Pops

In addition to Bryan's answer, if you want to manually inspect the flow files (let's say along a "failure" relationship as he mentioned), you can send the flow files to a processor that you do not start, thereby causing the flow files to queue up. If you right-click on the connection you can List Queue, then inspect any flow files in that queue – Gamble

Thanks guys, that was excellent advice. @BryanBende the PutFile task was exactly was I was looking for. That task clearly reported a permissions error to me, and after fixing that I was able to see my output. – Jasso

@Gamble Your gists on NiFi/HDF are awesome, and I have already stolen/modified a few of your templates. – Jasso

Answer 2

You can look at the Flowfile by uncompressing the flow.xml.gz file that is in the configuration directory. Of course, there is a lot of extraneous information there so it is not useful for debugging.

Looking at the STDOUT and STDERR of a processor would not help debugging either. Look instead at the flowfile content and attributes in the queues.

Prize