How to create new flow using Apache Nifi [closed]

Question

How can I create new flows using Apache Nifi. Is there only one flow?

Also what about authentication? Can different users have different flows?

Mountaintop

Answer 1

There is only one top-level canvas, but you can construct as many logical flows as you like. Typically to organize things process groups are used, so the top-level canvas can have several process groups each representing a logical flow, but not necessarily connected to each other.

In the current version of NiFi users have roles, so if a user is a data flow manager they can see and change any part of the flow, and if they are a monitor role they have read only access. In a future release being worked on right now this will change to more fine grained access control per component. In the model, if there were two process groups on the root canvas, you could give one set of users access to only process group 1 and another set of users access to only process group 2.

Overstock

So just to confirm, you are saying that there is only one root process group per Nifi cluster? So if you have 200 dataflow processes running on the cluster, you'll have 200 Process Groups on the root canvas? – Westfalen

@Westfalen if you want them to be organized individually yes, you could also have 10 process groups on the root canvas with 20 sub-process groups under each, depends if you have some way you want to organize them. – Overstock

Hello, Has there been any further UI enhancements. Is it still the best practice to create say 100 process groups for 100 dataflow/etl pipelines each of which has multiple processors in each pipeline. Wont 100 process groups be difficult to see on a single canvas? Or any better way so we can easily see and search the 100 ETL pipelines using some filter like name, userid, date etc. to narrow down the list. – Dombrowski

Nothing has changed in terms of the canvas and process groups. You can organize it however you like, maybe 10 top-level PGs for categories of flows, and then each of those has 10 PGs inside for individual flows. Should be able to search for components/PGs in the top-right search box. – Overstock

Answer 2

How can I create new flows using Apache Nifi. Is there only one flow?

You can create as many flows as you want, however, you can't have 'tabs'. The general consensus is to use Process Groups to separate flows and tidy up the view.

Can different users have different flows?

Not right now, but it is part of their backlog to implement something. Keep an eye out on the Release Notes for more information, or monitor the Dev Mailing list for an upcoming information.

Also what about authentication?

I have set up NiFi to use authentication through LDAP, see the NiFi Docs for more information on how this works and other methods.

Hope this helps!

Spinelli

Answer 3

It has been some time since this thread has opened and newer versions have come out. This below is based off 1.19.1 release.

Yes, there is only 1 logical canvas but if we use Process groups its possible to structure into layers. Providing a very simple example to convey this via screenshots. Scenario : -I need to organize some flows related to Kafka and some related to purely MongoDb and MySql.

[A] I will start with a blank "root canvas" identified as "Ni flow" , see red arrow

[B] I added 4 high level process groups

[C] Now double click the Kafka process group

[D] Nifi will now go into the "Kafka process group" and show me the child process groups. The 2 child groups are "Producer" and "Consumer" Note how the status bar shows at what level I'm at. I'm now 1 level below the "Nifi flow".

[E] Double click the "Producer" process group and now I'm at level 3. Here I have 2 version of the producers - Keyed and Unkeyed

[F]Double click the "unkeyed" processor group and now I have my data flow specific for loading data from json array in a file on disk to a Kafka topic

This was a very simple example to demonstrate the processor group nesting, special care needs to be taken based om your requirements on how you to want to trigger flows, control access to users, maintain versions etc.

Hyoscyamus

Answer 4

You can create multiple process groups. Each process group is isolated unless connected. You can work on individual process group, by double clicking on it. (You can see from the bottom, this is sort of the new tab, that you can start working on).

Sparhawk