Is it possible to run queues working synchronously with Laravel

Question

I am trying to set up an API system that synchronously communicates with a number of workers in Laravel. I use Laravel 5.4 and, if possible, would like to use its functionality whenever possible without too many plugins.

What I had in mind are two servers. The first one with a Laravel instance – let’s call it APP – receiving and answering requests from and to a user. The second one runs different workers, each a Laravel instance. This is how I see the workflow:

• APP receives a request from user
• APP puts request on a queue
• Workers look for jobs on the queue and eventually finds one.
• Worker resolves job
• Worker responses to APP OR APP finds out somehow that job is resolved
• APP sends response to user

My first idea was to work with queues and beanstalkd. The problem is that this all seem to work asynchronously. Is there a way for the APP to wait for the result of one of the workers?

After some more research I stumbled upon Guzzle. Would this be a way to go?

EDIT: Some extra info on the project. I am talking about a Restful API. E.g. a user sends a request in the form of "https://our.domain/article/1" and their API token in the header. What the user receives is a JSON formatted string like {"id":1,"name":"article_name",etc.}

The reason for using two sides is twofold. At one hand there is the use of different workers. On the other hand we want all the logic of the API as secure as possible. When a hack attack is made, only the APP side would be compromised.

Perhaps I am making things all to difficult with the queues and all that? If you have a better approach to meet the same ends, that would of course also help.

Dorian

I would say that in this setup, it might be good to put a request in a "pool" in a database with a status indicator of some sort (pending/inprogress/complete). Then, your worker can do some continuous polling against the DB to perform whatever actions exist. The APP also polls the DB, waiting for the action to complete, before responding with the result. – Zygophyllaceous

How you wanna place the request and serve the request ?? Using REST endpoints or what??:\ – Polad

Apps typically don't wait for a result from a queue worker - they fire the queued job and move on. Is the queue job very slow? Why not run the job then, after the job has run, raise an event to notify the user, via webhooks or open web sockets. Or, simply set the laravel queue driver to 'sync' and then it will do the work immediately and give you a response without the job queueing. – Logorrhea

I added some more info to my original post. Perhaps this can help you in your answers? – Dorian

Answer 1

I know your question was how you could run this synchronously, I think that the problem that you are facing is that you are not able to update the first server after the worker is done. The way you could achieve this is with broadcasting.

I have done something similar with uploads in our application. We use a Redis queue but beanstalk will do the same job. On top of that we use pusher which the uses sockets that the user can subscribe to and it looks great.

• User loads the web app, connecting to the pusher server
• User uploads file (at this point you could show something to tell the user that the file is processing)
• Worker sees that there is a file
• Worker processes file
• Worker triggers and event when done or on fail
• This event is broadcasted to the pusher server
• Since the user is listening to the pusher server the event is received via javascript
• You can now show a popup or update the table with javascript (works even if the user has navigated away)

We used pusher for this but you could use redis, beanstalk and many other solutions to do this. Read about Event Broadcasting in the Laravel documentation.

Subgenus

I have been looking into broadcasting before, but from what I understand it is something used between front and back-end? I added some more information on the project. With that knowledge, would your method still work? Also I am all new to things like Redis, Pusher, Beanstalkd and the like, so it is all quite blurry on what to use on what terms. From link and the link in the second answer in this thread it seems that Beanstalkd is the way to go when multiple workers are desired? – Dorian

@RubenColpaert It all depends on the job that needs to be processed, if it is quick enough that you can keep the user waiting with a HTTP request then do it, no need to use workers. Otherwise you might want to consider using a socket api where a user could connect and send messages that are later received. You could also make the api two sided, maybe the api server could send a post request to the frontend server (like a webhook) when the worker is done and trigger an event, same concept as above with an extra step added. – Subgenus

I'm still having problems with the issue. I created a new topic to explain the issue in more detail. Perhaps this will give a better view of what the issue is: #43845841. – Dorian