automating NIFI template deployment

Question

I'm new to nifi and am trying to understand (since it looks many GUI based) if there is a way to automate scale up and down on Nifi and how one would take an xml Nifi template and deploy it to a cluster.

Essentially what we're trying to do is use Nifi to collect JMX and log files off of kafka servers as they come up in an automated fashion so logging and JMX counters start flowing to, lets say and Elasticsearch cluster.

For example, right now we've automated deployment of the kafka servers using Ansible, so the hope was to configure and deploy Nifi xml teamplates along with the cluster deployment so no manual intervention is required to collect the kafka jmx counters and log files.

How would one automate this in Nifi?

Lyso

Answer 1

There are a number of tools and techniques available to automate deployment of a template, including:

• curl, see SO: Post a NIFI template via REST? - most basic API template deployment.
• nifi-api-deploy - performs template deployment plus some migration tasks.
• nifi-deploy-process-group - performs process group migration.

The hard part is migrating from the existing flow to the new flow, especially where an existing flow may have queued flowfiles, and provenance history references the existing flow. But if you are automating the entire deployment of the NiFi cluster and deploying the template to an empty canvas, this might not be a problem.

You can read a full treatment of the issues involved and future plans in Configuration Management of Flows.

Gass

Answer 2

For automating NIFI template deployment, there is a tool that works well : https://github.com/hermannpencole/nifi-config

• Prepare your nifi development

  1. Create a template on nifi
  2. and download it
  3. Extract a sample configuration with the tools
• Deploy it on production
  1. undeploy the old version with the tools
  2. deploy the template with the tools
  3. update the production configuration with the tools

Crept

Answer 3

Basically you need to import your downloaded templates to target production instance; you can use nifi-rest-api.

After creating a nifi template and downloaded to a folder manually than you should use a script or a tool for import your template to your target nifi instance and change some settings after importing.

There are some tools that you can use:

1. this is a script based project for deployment.
2. this one is java based

If there are not enough for you; you can create a tool that smiliar with above ones. You can use nifi-rest-api at all steps. The steps of deployment should be something like below:

• Get existing templates (GET /flow/templates)
• Delete existing template (the template that has same name with your template)
• Delete related controller services (DB Connection Pools, Http Context)
• Upload your template
• Delete Dataflow that created by your template before
• Create Dataflow from your new updated template

Probaply you there are some more sub steps exists during importing process depeneds your templates and dataflow properties. All steps have a rest-api service call that you can find at first link above.

Antipyretic

Answer 4

You can use Nifi-Toolkit which provides CLI to interact with NIFI instance and NIFI Registry instance.

A Powershell script can than be written to automate the whole deployment process.

You need to register registry clients to you Nifi instance. Once you create the appropriate buckets per your environment in your registry instance and connect this instance to your Nifi flows, you can use Nifi toolkit which provides commands to export flow from one environment and import it to other via registry buckets.

e.g. run the following command from the Cli.sh or cli.bat which is found in nifi-toolkit code base.

registry export-flow-version -f $fromFlowidentifier -fv $flowversion -o temp-flow.json -ot json -u $registrylink ;

here fromflowidentifier is the flow identifier that is found in your registry bucket. temp.json is the file where it will export the flow. Once exported you can import the flow using bellow command.

registry import-flow-version -f $toflowidentifier -i temp-flow.json -u $registrylink ;

Here it is assumed you have linked the nifi instances with the buckets in registryclient .

Here is my sample autodeployment powershell script to import nifi flow: It will not only deploy the flow but will also set the flow variables. If deployment fails for some reason it will also send an email. This assume you are using Nifi-registry and have installed Nifi-toolkit. Please go through the readme and script is also commented for proper understanding.

Here is the example powershell script: https://anp8729.github.io/Nifi_AutoDeploymentScript/

Wrenn

Please add the relevant commands to your answer; saying the answer is in another castle does us no good when your link changes or expires. Also, the content of links can't be found by the SO search. – Berger