I am trying to set up an API which uses a queue system on another server to handle the requests. Let me start what I am trying to accomplish without the queue system (no authorization to keep it simple): Using Postman for example making a GET request to the URL https://example.com/products would return a JSON string like
[
{
"id": 1,
"name": "some name",
...
},
{
"id": 2,
"name": "some other name",
...
}.
...
]
The code in routes/api.php would be something like:
<php
Route::get('/products', ProductController@index');
And the code in app/Http/Controllers/ProductController.php:
<?php
namespace App\Http\Controllers;
class ProductController extends Controller
{
/**
* Return the products.
*
* @return \Illuminate\Http\Response
*/
public function index()
{
// Logic to get the products.
return $products->toJson();
}
}
What I would like to accomplish is that all the business logic is processed on another server which runs multiple workers. What follows is my reasoning behind this.
- Security: In the case we would get hacked, it will most probably be the client server and not the worker server. Since the last one has all the business logic, the hacker will, in worst case scenario, only be able to get incoming and outgoing request data.
- Multiple workers: Getting the products will probably not take a long time, but there might be other requests which need more time to process. The user making the request will have to wait for the result in most situations. However, other users making a call should not have to wait for this. Therefore, another worker can take this request and handle the job.
This would be the workflow how I see it:
All free workers are constantly polling for a job on the queue
- User makes request
- Client Server takes request data and put it on the queue
- A worker takes a job from the queue and handles it
- The worker returns the result to the client server
- The client server returns the result to the user
Below A small drawing to clear things out.
User 1
_ \
| \
\ \ 1.
\ \ request
\ \ -------------
result \ \ / \
5. \ \ | Worker |
\ _| | Server |
\ | --------- |
------------- | / \ |
/ \ | | Worker1 | |
| Client | / | | | | \
| Server 2. | / | \ / | \
| --------- | / | --------- | \
| / \ | / | --------- | \
| | Queue | | / | / \ | \ ---------
| | | | |_ | | Worker2 | | _| / \
| | Job A | | | | | | | DB |
| | Job B | | 3. <----- | \ / | -----> | Server |
| | | | _ | --------- | _ | |
| \ / | | | ... | | \ /
| --------- | \ | --------- | / ---------
\ / \ | / \ | / ---------
------------- \ | | WorkerN | | / / \
_ 4. ? \ | | | | / | Other |
| | \ / | | Servers |
/ / | --------- | | |
1. / / \ / \ /
request / / ------------- ---------
/ /
/ / result
/ / 5.
/ /
|_
User 2
In the documentation of Laravel I came across queues, which I thought would easily do the trick. I started experimenting with Beanstalkd, but I assume any queue driver would do. The issue I stumbled upon is that the queue system works asynchronously. As a consequence, the Client server simply carries on without waiting for a result. Unless I am missing something, there seems to be no option to make the queue system work synchronously.
When looking further into the Laravel documentation I came across broadcasting. I am not sure if I understand the concept of broadcasting a 100%, but from what I do understand is that the receiving seems to happen in Javascript. I am a backend developer and would like to stay clear from Javascript. For some reason it feels wrong to me use javascript here, however I am not sure if that feeling is justifiable.
Looking further in the documentation I came across Redis. I was mainly intrigued by the Pub/Sub functionality. I was thinking that the Client server could generate a unique value, send it with the request data to the queue and subscribe to it. Once the worker is finished it can publish the result with this unique value. I was thinking that this could cover the missing part for step 4. I am still not sure how this would work in code, if this logic would work in the first place. I am mainly stuck with the part where the client should listen and receive the data from Redis.
I might be missing something very simple. Know that I am relatively new to programming with PHP and to the concept of programming over the world wide web. Therefore, if you find that the logic is flawed, or it is all too farfetched, please give me some pointers on other/better methods.
An extra FYI, I have heard of Gearman, which seem to able to work both synchronously and asynchronously. I would like to stay clear from that, however, since my aim is to use the tools provided by Laravel to the fullest. I am still learning and am not confident enough to use too many external plugins.
Edit: This is how far I got. What am I still missing? Or is what I am asking (near) impossible?
User calls http://my.domain.com/worker?message=whoop
The user should receive a JSON response
{"message":"you said whoop"}
Note that in the header of the response the content type should be "application/json" and not "text/html; charset=UTF-8"
This is what I have until now:
Two servers API-server and WORKER-server. The API-server receives the requests and push them on the queue (local Redis). The workers on a WORKER-server process the jobs on the API-server. Once a worker processed a job, the result of this job is broadcasted to the API-server. The API-server listens for the broadcasted response and sends it to the user. This is done with Redis and socket.io. My problem is that at this point, in order to send the result, I send a blade file with some Javascript that listens for the response. This results in a "text/html; charset=UTF-8" content type, with an element that is updated once the result from the worker is broadcasted. Is there a way to, rather than returning a view, make the return 'wait' until the result is broadcasted?
API-server: routes\web.php:
<?php
Route::get('/worker', 'ApiController@workerHello');
API-server: app\Http\Controllers\ApiController.php
<?php
namespace App\Http\Controllers;
use App\Jobs\WorkerHello;
use Illuminate\Http\Request;
class ApiController extends Controller
{
/**
* Dispatch on queue and open the response page
*
* @return string
*/
public function workerHello(Request $request)
{
// Dispatch the request to the queue
$jobId = $this->dispatch(new WorkerHello($request->message));
return view('response', compact('jobId'));
}
}
API-server: app\Jobs\WorkerHello.php
<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Support\Facades\Log;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
class WorkerHello implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public $message;
/**
* Create a new job instance.
*
* @param string $message the message
* @return void
*/
public function __construct($message = null)
{
$this->message = $message;
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
}
}
WORKER-server: app\Jobs\WorkerHello.php
<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use App\Events\WorkerResponse;
use Illuminate\Support\Facades\Log;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
class WorkerHello implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public $message;
/**
* Create a new job instance.
*
* @param string $message the message
* @return void
*/
public function __construct($message = null)
{
$this->message = $message;
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
// Do stuff
$message = json_encode(['message' => 'you said ' . $this->message]);
event(new WorkerResponse($this->job->getJobId(), $message));
}
}
WORKER-server: app\Events\WorkerResponse.php
<?php
namespace App\Events;
use Illuminate\Broadcasting\Channel;
use Illuminate\Queue\SerializesModels;
use Illuminate\Broadcasting\PrivateChannel;
use Illuminate\Broadcasting\PresenceChannel;
use Illuminate\Foundation\Events\Dispatchable;
use Illuminate\Broadcasting\InteractsWithSockets;
use Illuminate\Contracts\Broadcasting\ShouldBroadcast;
class WorkerResponse implements ShouldBroadcast
{
use Dispatchable, InteractsWithSockets, SerializesModels;
protected $jobId;
public $message;
/**
* Create a new event instance.
*
* @return void
*/
public function __construct($jobId, $message)
{
$this->jobId = $jobId;
$this->message = $message;
}
/**
* Get the channels the event should broadcast on.
*
* @return Channel|array
*/
public function broadcastOn()
{
return new Channel('worker-response.' . $this->jobId);
}
}
API-server: socket.js (runs with node)
var server = require('http').Server();
var io = require('socket.io')(server);
var Redis = require('ioredis');
var redis = new Redis();
redis.psubscribe('worker-response.*');
redis.on('pmessage', function(pattern, channel, message) {
message = JSON.parse(message);
io.emit(channel + ':' + message.event, channel, message.data);
});
server.listen(3000);
API-server: resources\views\response.blade.php
<!doctype html>
<html lang="{{ config('app.locale') }}">
<head>
</head>
<body>
<div id="app">
<p>@{{ message }}</p>
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/vue/2.3.4/vue.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/2.0.3/socket.io.js"></script>
<script>
var socket = io('http://192.168.10.10:3000');
new Vue({
el: '#app',
data: {
message: '',
},
mounted: function() {
socket.on('worker-response.{{ $jobId }}:App\\Events\\WorkerResponse', function (channel, data) {
this.message = data.message;
}.bind(this));
}
});
</script>
</body>
</html>