Separate microservice just for microservices orchestration?
Asked Answered
J

3

6

I have a few microservices where each microservice has REST endpoints for CRUD operations. I have to create a workflow that will start from one microservice with some initial input, but later outputs from a microservice can be used as input to other microservices. There can be some synchronous and asynchronous calls to these REST APIs.

I have looked for some of the workflow-engines but I do not think that I can create my workflow without writing any java code.

Should I write a separate microservice just for microservices orchestration? This orchestration microservice will know the exact workflow and can be configurable for inputs required to start the workflow, and it can also use some third-party workflow engines like Camunda to store the definition of the workflow.

Is this correct thinking to have a separate microservice just for microservices orchestration? Till now the existing microservices have no idea about the other microservices. There could be a chance that output from one microservice needs to be massaged before using as input for other microservice.

Jadda answered 4/11, 2020 at 8:28 Comment(2)
What do you mean by saying "but later outputs from a micro-service can be used as input to other micro-services". Can you be more specific how output from one micro-service can be used as input to another micro-service and how exactly is this problematic for your scenario?Scud
Suppose a REST endpoint returning some JSON for a GET method and the output is used in POST message to other microservice. My basic question is "should I create a separate microservice which will do all the orchestration work because I am not seeing that a workflow engine can solve the problem of all interactions between microservices?"Jadda
S
3

I have looked for some of the workflow-engines but I do not think that I can create my workflow without writing any java code.

This depends on your business processes and the complexity of your workflow. Usually yes you will need to write some code to achieve it.

Should I write a separate microservice just for microservices orchestration? This orchestration microservice will know the exact workflow and can be configurable for inputs required to start the workflow, and it can also use some third-party workflow engines like Camunda to store the definition of the workflow.

Yes you can do that. I did something similar on a system using micro-services. This would be a very good Idea on the long run as you could configure your workflow based on environments as well. For example on your development machine you would have a little different workflow/configuration. This is practical for Developers or QA's testing their solutions. On the other hand on Staging/Production you can pre-define Customer setups/orchestration which you can reuse any time if you get new customers or users.

Is this correct thinking to have a separate microservice just for microservices orchestration? Till now the existing microservices have no idea about the other microservices. There could be a chance that output from one microservice needs to be massaged before using as input for other microservice.

Yes you can do that without problems although I would be careful with the name orchestration as this has another meaning in context in micro-service architecture(Docker, Docker-Swarm, Kubernetes). Similar examples would be some kind of EndToEndTest or Cross micro-service testing-micro-service. That would test cross micro-service business operations and assert the results. Usually business operations involve more then 1 micro-service so in order to test that you can use this approach. This micro-service would call APIs from multiple micro-services and test the results and scenarios based on your Business rules. Another example would be something like seeder-micro-service(which seems to be very similar to what you are trying to do here). This seeder-micro-service would be responsible for seeding(creating) test data to your micro-services. This test data is some basic setup/configuration data which you need in order to have your micro-service business processes to work. This is very handy for development machines or some test environments where you need to quickly setup an environment. Using this seeder-micro-service you can easily setup do your work or tests and dispose the environment(data) as you need it. This is especially useful for development machines setups but it can also be used on shared test environments and etc. Both of those examples are micro-services which server your needs and make your life easier to work with your system.

One final note regarding this:

Till now the existing microservices have no idea about the other microservices.

They should be abstracted from each other in a way that they are not aware of internal implementation or data(separate databases) but they should communicate between each other in order to perform business operations which sometimes are cross micro-services. Like the typical example of payment-micro-service and order-micro-service from an online shop example. So it is fine that they know about each other and communicate but this communication has to be very carefully designed in order to avoid some common pitfalls. They usually communicate with each other with direct calls over HTTP or some other protocol or through some message queue like Apache Kafka or RabbitMq or others. You can read more about it in this answer.

Scud answered 5/11, 2020 at 16:16 Comment(0)
N
0

Yes, you should cover the orchestration part in a separate service. And, yes, go with a BPMN 2 process engine as orchestrator, as you already suspected. Yes, this may include writing a little code mostly for data mapping or connectors. Benefits include for instance ootb support for:

  • state management and long running processes / persistence for data
  • versioning (!)
  • retries and error handling
  • tooling to modify state and date in case something went wrong
  • timeouts, parallel execution (if necessary)
  • scalability
  • graphical process model
  • audit trail
  • and end to end visibility in monitoring tools based on BPMN 2 model
  • ability to include business rules tasks (DMN) for more complex rules
  • combination of push and pull communication pattern and a/sync communication
  • business-IT alignment via BPMN 2
  • support for the various BPMN 2 events
  • standardization (skills, security, software quality, features) ...

This is a great related article about the WHY using an airline ticket booking as an example: https://blog.bernd-ruecker.com/3-common-pitfalls-in-microservice-integration-and-how-to-avoid-them-3f27a442cd07

This is about the important design consideration in case you go with a process engine: https://blog.bernd-ruecker.com/the-microservice-workflow-automation-cheat-sheet-fc0a80dc25aa

Nicholenicholl answered 6/11, 2020 at 2:1 Comment(0)
W
-1

I faced a similar problem as in the original question. I'd a few microservices with simple dependencies that needed to be managed and did go down the path of writing my own microservice https://github.com/pedro-r-marques/workflow to manage the dependencies and do the orchestration. It uses a yaml definition file to describe the dependencies and rabbitmq for message passing. One could also replace the usage of rabbitmq by REST API calls front-ended by a load-balancer.

Wendalyn answered 28/7, 2021 at 8:4 Comment(3)
There a many solutions for services / process orchestration out there (see for insatnce blog.bernd-ruecker.com/… ) and several are open source. Several support graphical process modelling, some in the BPMN2 standard. Many companies started with home grown solutions 10+ years back and are replacing them with standard-based solutions now. Standardization leads to standard skill requirements, lower learning curve and higher code quality and security. Why reinvent the wheel?Nicholenicholl
How does you solution deal with more complex typologies including loops, parallel execution, error handling, timer or conditional events,... ? What about persistence for audit and reporting requirements?Nicholenicholl
Not all scenarios have the same level of complexity. In the scenarios I've faced I needed simple at-least-once semantics which are an order of magnitude simpler that what BPMN2 provides. For simple semantics, being able to configure the workflow with a simple yaml file and use any language that can speak to the message bus is imho a significant advantage over complex solutions.Wendalyn

© 2022 - 2024 — McMap. All rights reserved.