Difference between open-ended and close-ended load testing tools

Question

What is the difference between open-ended and close-ended load testing?
Thread-based load-testing tool comes in which category?
Which are some examples of load testing tools in each category?

Banausic

I've been doing load testing for 17 years, and I never heard of the term "open ended load testing tool" or "close ended load testing tool". Can you reference some definition maybe? – Allcot

the terms may be misleading, but the concepts exists, have a look here: users.cms.caltech.edu/~adamw/papers/openvsclosed.pdf – Ossian

Answer 1

Looks like you are speaking about closed vs. open workload models.

This classification based on closed/open systems separation:

• in case of closed system model a new request is only triggered by the completion of a previous request, following by a think time. System has negative feedback that makes it impossible to bury-out the service, so users wait for the responses before making new requests;
• in case of open system model new requests arrival independently of completions, e.g. according to a stochastic process or fixed trace. System has no negative feedback.

Based on this classification we can divide load-testing tools into following categories based on workload model used:

• thread-based model where each thread will wait for a response before sending another request - so request rate depends on response rate (load generator drives the load as fast as system can take it):
  • thread-based approach is VU [virtual user]-oriented while trying to simulate N users working together;
  • thread-based tools require a separate thread to simulate a user;
  • examples: JMeter;
• hit-based model
  • hit-based approach is RPS [request per second]-oriented while trying to produce N hits per second load;
  • hit-based tools use async approach and will always attempt to create defined request rate, queuing requests if necessary;
  • examples: tsung, Yandex.Tank.

Venterea

I'll mention another open source tool: Gatling lets you generate load either way - both open and closed loops – Arlon

Actually, hit-based load can be generated by JMeter too, using Throughput Shaping Timer – Goings

Answer 2

I basically agree with the explanation of closed/open system from Aliaksandr and give you an example.

For example: you have 3 users, each iterating over a sequence of requests. Before a new iteration can start, the previous iteration has to finish. The previous iteration finishes, if all requests are finished. If the System-under-Test takes longer to respond, the request/s rate drops. So load-generator and system-under-test are in a closed loop.

In open loops, the request rate is constant, regardless of the response times - there is no feedback.

Both models reveal different performance characteristics of the system under test, i.e. capacity/throughput limits with closed-loop, queue sizes with open loops. It's more easy to overload a system with an open-loop

Regarding threads, usually thread-based tools define a close-loop, but, you could model open loops as well.

Take JMeter for example, if you want a constant rate of 2 users/s over a period of 3600 seconds, you use a rampup time of 3600 seconds and use a thread number of 7200, without looping.

Gatling is another free tool that is not thread-based but event based. You can model closed-loops (repeat, during) or open-loops (constantUsersPerSecs)

Tap