Import Modules in Nifi ExecuteScript

Question

I am new to Nifi and python

i want to execute my python script. So used ExecuteScript and tried to import certain modules. I have imported like this:

import json, sftp, paramiko

Though i have sftp installed, When i import it in Executescript, it says "Failed to process session. No module named sftp at line number 1"

which -a sftp
/usr/bin/sftp

When importing paramiko also, got the same error.

Paulson

Answer 1

The "python" engine used by ExecuteScript and InvokeScriptedProcessor is actually Jython, not pure Python. This means it cannot load native modules (.so files, compiled C files, etc.). According to this SO post, paramiko uses Crypto which has native libraries, so cannot be used in Jython (see the bottom of this post for my comment on that). My guess is that the sftp library does the same.

Jython can make use of pure Python modules, there is a discussion on the NiFi mailing list about how to point at (and include) those kinds of modules.

Unilobed

How should I set the "Module Directory" property so that ExecuteScript will use pip-installed modules? – Profanity

Set it to the site-packages directory under your Python install – Unilobed

Answer 2

ExecuteScript processor uses its own Jython Engine to execute your python scripts. As the libraries which you are importing are not available in NIFI inbuild Jython Engine its throwing error.

SOLUTION:

If python is already installed on our machine with all those libraries (the same machine where your NIFI is installed) you can use that python engine to execute your script. you can execute your python code using ExecuteProcess processor. see the configuration of ExecuteProcess.

Canzone

Answer 3

If its really important that you use python. You can use ExecuteStreamCommand. It will run the python code using the python engine installed on your machine.

The disadvantage is that you cannot access attributes of flowfile inside your python code. Only its content.

To access the content,

import sys
data = sys.stdin.readlines()

and to pass the content to the next processor, just print your output.

print("THIS IS MY OUTPUT, IT WILL BE PASSED AS CONTENT TO THE NEXT PROCESSOR")

otherwise, if you need to stick with ExecuteScript, Use groovy, that will save you a lot of headache.

Transmigrant