Performance issue with HTTP request in Julia API

Question

Exploring the Oxygen package to build APIs I found this I haven't been able to solve

1. When I run this API (with a 150 ms sleep to simulate the time taken by a model):

using Oxygen

@get "/model" function()
    sleep(0.15)
    return json("ok")
end

serve(port = 8001)

I reach about 560 RPS (with concurrency = 100)

ab -n1000 -c100 'http://localhost:8001/model'

Server Software:        
Server Hostname:        localhost
Server Port:            8001

Document Path:          /model
Document Length:        4 bytes

Concurrency Level:      100
Time taken for tests:   1.756 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      83000 bytes
HTML transferred:       4000 bytes
Requests per second:    569.62 [#/sec] (mean)
Time per request:       175.556 [ms] (mean)
Time per request:       1.756 [ms] (mean, across all concurrent requests)
Transfer rate:          46.17 [Kbytes/sec] received

When I build another Oxygen API consuming the previous one

using Oxygen
using HTTP

@get "/test" function()
    response = HTTP.get("http://localhost:8001/model") # sleep(0.15) - Just this line is different
    return json("ok")
end

serve(port = 8002)

I reach only around 100 RPS even using 4 threads with serveparallel. I would expect to see an RPS number closer to 560.

ab -n1000 -c100 'http://localhost:8002/test'

Server Software:        
Server Hostname:        localhost
Server Port:            8002

Document Path:          /test
Document Length:        4 bytes

Concurrency Level:      100
Time taken for tests:   9.815 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      83000 bytes
HTML transferred:       4000 bytes
Requests per second:    101.89 [#/sec] (mean)
Time per request:       981.455 [ms] (mean)
Time per request:       9.815 [ms] (mean, across all concurrent requests)
Transfer rate:          8.26 [Kbytes/sec] received

I think the issue is in the HTTP.get request on the Julia API maybe there is a better way to do it or ways to speed it up. Any hint is welcome.

I tested it in: Julia version: 1.9.4 and 1.10.1 Oxygen version: 1.4.9 and 1.5.0 HTTP version: 1.10.2

If I do the same with python using FastAPI and request

import requests
from fastapi import FastAPI

app = FastAPI()

@app.get("/test")
def test():
    response = requests.get("http://localhost:8001/model")
    return "ok"

With just one worker (uvicorn api_test_nested:app --host 0.0.0.0 --port 8003 --workers 1) I get around 220 RPS with 4 workers I almost reach the 450 RPS

ab -n1000 -c100 'http://localhost:8003/test'

Server Software:        uvicorn
Server Hostname:        localhost
Server Port:            8003

Document Path:          /test
Document Length:        6 bytes

Concurrency Level:      100
Time taken for tests:   4.436 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      149000 bytes
HTML transferred:       6000 bytes
Requests per second:    225.42 [#/sec] (mean)
Time per request:       443.618 [ms] (mean)
Time per request:       4.436 [ms] (mean, across all concurrent requests)
Transfer rate:          32.80 [Kbytes/sec] received

Progress:

I changed the /test API to use curl instead of HTTP.jl and I got a huge improvement so I'm almost sure there is something strange with HTTP.jl

using Oxygen

@get "/test" function()
    response = run(`curl http://localhost:8001/model`) # instead of HTTP.request("GET", "http://localhost:8001/model")
    return json("ok")
end

serve(port = 8002)

With this small change I go from 100 RPS to about 440 RPS, much closer to 560 RPS I get from /model and twice the RPS from FastAPI with one worker.

Any suggestions?

Risible

Answer 1

Edit: Tried running the code with various concurrent requests and machines, and turns out that your case only happens when there's two julia servers running on the same machine. If you were to HTTP.get from a server from another machine, the result will be much faster than when run on the same machine, and multiprocessing only slows it down. I think this is a special case, as other conditions I've tested(allocating, invoking IO on files & external servers) shows better performance using single process approach than multiprocessing.

Old Answer (WRONG)

TL;DR: 1. Use curl or LibCURL.jl 2. Use multiprocessing instead of multithreading

It seems that you are wondering about 2 things;

1. Why HTTP.get is slower than the alternatives

Unfortunately, HTTP.jl isn't as actively developed or used as its counterparts, so some shortcomings on performance is inevitable for the moment. If you need the performance, running curl via external command like you did, or using LibCURL.jl would be an option.

2. Why isn't the performance improving as you add more threads.

Julia Multithreading is slowing down when the number of threads is high

The link above deals with some similar situation. It says that Julia's garbage collector is inefficient when it comes to many threads, so multiprocessing is preferred if your code will allocate a lot.

It holds true for this case too. I tried running multiprocessed verion of your code on my machine using Distributed.jl, and the performance scaled up nicely as I added more processes.

# multiprocess.jl
using Oxygen, HTTP, Distributed 
@get "/test" function()
       future=Distributed.@spawnat :any begin
           response = HTTP.get("http://localhost:8001/model")
       end
       fetch(future)
       return json("ok")
end
serve(port=8002)
 

With 1 process only

$ julia multiprocess.jl
$ ab -n 1000 -c 100 http://localhost:8002
Server Software:        
Server Hostname:        localhost
Server Port:            8002

Document Path:          /test
Document Length:        4 bytes

Concurrency Level:      100
Time taken for tests:   9.825 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      89000 bytes
HTML transferred:       4000 bytes
Requests per second:    101.78 [#/sec] (mean)
Time per request:       982.481 [ms] (mean)
Time per request:       9.825 [ms] (mean, across all concurrent requests)
Transfer rate:          8.85 [Kbytes/sec] received

With 4 processes

$ julia -p 4 multiprocess.jl
$ ab -n 1000 -c 100 http://localhost:8002
Server Software:        
Server Hostname:        localhost
Server Port:            8002

Document Path:          /test
Document Length:        4 bytes

Concurrency Level:      100
Time taken for tests:   2.603 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      89000 bytes
HTML transferred:       4000 bytes
Requests per second:    384.24 [#/sec] (mean)
Time per request:       260.252 [ms] (mean)
Time per request:       2.603 [ms] (mean, across all concurrent requests)
Transfer rate:          33.40 [Kbytes/sec] received

Halation