REST APIs and messaging

Question

I have a system that exposes a REST API with a rich set of CRUD endpoints to manage different resources. The REST API is used also by a front-end application that executes calls by using Ajax.

I would like to make some of these calls asynchronous and add reliability.

The obvious choice seems a message broker (ActiveMQ, RabbitMQ, etc...).

Never used message brokers before and I am wondering if they can be "put in front of" the REST API without having to rewrite them.

I do not want to access the REST API only through the messaging system: for some endpoints, a call must always be synchronous and the reliability is less important (mainly because in case of error the user receives an immediate feedback).

Would a full ESB be a better option for this use case?

Butterwort

Is your intention to 'hide' the Rest API by redeveloping your front end application to put a message on a queue? Or to have a new implementation of the Rest API which uses queuing for its implementation? – Sik

I would like to hide the REST API from the front end application. The messaging system could be used also for server-to-server communications. The API will be accessible directly but some of its endpoints could be enhanced adding reliability. Ideally, I would like also to define subscribers on the browser side using WebSockets. – Butterwort

Depending on your backend configuration, new requests are usually handled within their own thread. Therefore I guess async invocation from JS to your backend shouldn't be a real issue. Though, could you clarify what reliability constraints you need? Something along the line of at-most-once consumption or losing messages issued from the frontend to the backend (or the responses)? Usually your FrontEnd would fill a queue which triggers some consumption logic in the backend (similar to your API which is invoked via HTTP request) I therefore wouldn't put the MQ in front of the API – Motherwort

@RomanVottner Probably I asked too many questions. I am mostly interested in adding reliability and asyncronous messages to an existing REST API. The API will be called not only by the frontend. I have some endpoints that needs a long time to complete the required task. I want for the client to receive an immediate feedback and, in the same time, I want to be sure that if the message has been sent and received by the message broker, it will be delivered (sooner or later). – Butterwort

You are probably looking for an HTTP way (not REST; plenty of people confuse the latter for the former). One easy way to immediately return a response is by i.e. sending an initial POST request that triggers a long running calculation on the server in a new thread. Meanwhile the server will acknowledge the receipt with a 202 Accepted containing a Link Header with an URI the client can invoke to poll for status updates. In the meantime the server can proceed on its task with the client asking for status updates from time to time. Once the request finished a 200 OK with the result is returned – Motherwort

Answer 1

If I understand your question, you would like to "register" an API endpoint as a subscriber so that it could receive the messages sent to a given queue.

I do not think that a message broker can be configured to do this.

For example, if you want to use a message broker, both your producers and subscribers need to use the JMS API.

I do not know if a solution can be to implement a subscriber that will execute the corresponding API call. In this case, the reliability is compromised because the message will be dequeued before the API call is executed. It can make sense if the subscriber is running in the same process of the API, but in this case it is not clear why you should use a REST API instead of a library.

Booking

Answer 2

IMO @EligioEleuterioFontana you have a misunderstanding of the roles of:

• an RESTful Api
• a message broker

These are two different subsystems which provide different services.

Now, let's explain their roles with respect to your requirements:

• You have clients (desktop browsers, mobile phone browsers or apps) which need to get/push data to your system. (Assumption from the REST API mention).
• Requests from the clients are using HTTP/HTTPS (that's the REST API part of your requirement).
• Any data that is pushed, you wish to make this more responsive, quicker, reliable.

If I've gotten that right, then I would answer it as:

• All clients need to push requests to a REST API because this does just more than simple CRUD. The Api also handles things like security (authentication and authorization), caching, possibly even request throttling, etc.

• REST API should always been the front end to clients as this also 'hides' the subsystems that the API uses. Users should never see/know about any of your subsystem choices (eg. what DB you are using. Are you caching? if so, with what? etc).

• Message Brokers are great for offloading the work that was requested now and handling the work later. There's heaps of ways this can be done (queues or pub/sub, etc) but the point here is this is a decision the clients should never see or know about.

• MB's are also great for resilience (as you noted). If something fails, the message on a queue would be re-attempted after 'x' time ... etc. (no, I'm not going to mention poison queues, dead letter queue, etc).

You can have some endpoints of the Api that are synchronous. Sure! Then have others that leverage some eventual consistency (i.e. for that request, I'll deal with it later (even if later in 5 secs later) and just return the response to the client saying "thanks! got it! I'll do it soon"). This is the asynchronous workflow you are after.

The API endpoints needs to be simple, concise and hopefully pretty stable. What you do behind the scenes as you change things hopefully will be hidden away from the clients. This includes the use of message brokers.

---

Anyway, that my take on how I see REST APIs and Message Brokers and how they related to each other.

Knute

The message broker like its bigger brother the service bus is an integration middleware. It is used to integrate applications or components of an application. – Butterwort

In the most common use, as far as I know, it integrates services. A service can be a soap service, a rest service and even a database or a file system. – Butterwort

Answer 3

It might be worth looking into the Google Cloud sub/pub? - https://cloud.google.com/pubsub/docs/overview

Prunelle