Does Apache NiFi support version control

Question

I am trying to explore Apache NiFi. So far haven't seen any ways to version control flows.

Is there a way to version control flows when multiple users are trying to develop in the same instance?

What about code merge from multiple users?

Any help in these regards will help me to continue my exploration.

Chancy

Answer 1

In addition to James's great answer I'll also point out that this approach to flow management has leveraged external version control systems and put the task on the user to perform. What I mean is that users (or automated processes) could initiate the production of a template and then store that template into a VCS. This has worked well but it is also insufficient. The other direction is also important where given a versioned flow one would like that to be automatically reflected on another cluster/system/environment. Think of the software development lifecycle one might go through when building flows in a development environment and proving/vetting into and through production. Or think of a production case where behavior is not as expected. While NiFi offers a really powerful interactive command and control model sometimes people want to be able to test new approaches and theories in another environment. As a result, we're working now on a really awesome capability.

Come join the conversation. We'd like to hear your thoughts.

Thanks

Lamee

Thanks Joe. While I tend to agree (and I se you support that too) with template approach of VCS, please see my comments to the answer proposed by James. – Chancy

I also would be very interested to know if there are plans for Continuous Integration (CI) capabilities in NiFi's roadmap and/or coming anytime sooner. – Chancy

@CalRanjan Yes that is exactly the sort of thing the above linked configuration management of flow outline is designed to address. The solution enables automated or controlled uptake of flow versions based on committed changes from other environments. As a result one can use a more traditional development, staging, production type progression/lifecycle to compliment the power of interactive command and control NiFi already has. – Lamee

Answer 2

NiFi Templates are a great format for configuration management of a NiFi flow. You can define templates for everything from small example snippets up to large nested process group structures, essentially your entire flow. Templates will include processors, queues, and controller services, but will not contain sensitive values like passwords. Templates are stored as XML files friendly to source control (since NiFi v1.0).

Templates provide a way for individual developers to separately build parts of a flow, then merge the parts together in a single NiFi. If you match templates with process groups, swapping out the old one with the new one can be fairly easy and intuitive.

Villagomez

Thank you James. This is good stuff. When I exported a process group template and imported back, preserving the flow.xml before and after, I see "diff" reports some changes between the two xmls. Some group ids are reportedly changed. Doesn't this behavior defeats the purpose of version control? – Chancy

Restoring the template creates a copy. I typically restore a new flow side-by-side with the old, cut over gracefully, then eventually remove the old. Up to you, depending on your needs. – Villagomez

I'm not sure what your version control expectations are. I consider NiFi more analogous to a database than a code module with respect to version control. Yes, stuff should be checked in. But operators need to be able to touch, feel, and work with the running system. – Villagomez

Answer 3

The answer to this question is YES, you can use NiFi Registry to have version control. Below you can see a how it looks like.

The project page is: https://nifi.apache.org/registry.html

Schilling