How to regard Hyper-Threading when deciding thread pool size?

Question

I've read several Questions and articles regarding how to decide on a size for a thread pool. Questions include:

• How to choose thread pool size?
• What is the relationship between number of CPU cores and number of threads in an app in java?
• Java thread pool size (Executors)

Articles include:

• Tuning the Size of Your Thread Pool by Kirk Pepperdine

However, none of these directly address the issue of Hyper-Threading on Intel chips.

On a computer with Hyper-Threading enabled, should I consider the virtual cores when deciding thread pool size?

Take for example Brian Goetz’ advice in the Java Concurrency In Practice book that, generally speaking, for a CPU-bound app one might consider using ( # of cores + 1 ) as the number of threads. For an Intel Core i7 chip with 4 real cores and 8 virtual (Hyper-Threading) cores, would that formula be ( 4 + 1 ) or ( 8 + 1 )?

Also, how much of difference does the nature of the app matter in how to consider the Hyper-Threading cores?

Contrary to the mention above, my own app is not CPU bound. Instead my app is a server-side Vaadin app where the threads are making Internet connections and hitting a local database through JDBC, several times a minute. Given that Hyper-Threading is basically a second set of registers attached to the same core, perhaps a CPU-bound app should consider only real cores while a network/IO-bound app should consider virtual cores?

Lastly, does the type of Intel chip affect Hyper-Threading and therefore the calculation of thread pool size? Specifically, is there a difference regarding this issue between a Xeon and a Core i7/i5? For example, the difference between a current MacBook (Core i7) and Mac Pro (Xeon).

I realize there are many variables involved, and this is a complex topic. No perfectly precise answers are possible. I'm just looking for some general rules and advice to help out programmers like me who are not as savvy about such hardware matters.

Flawed

A network-bound APP doesn't benefit from more threads and should only choose the number needed to deal with things like unexpected page faults or how many I/O is can pend usefully at once. The CPU is almost irrelevant to an app that's not CPU bound. – Sitka

@DavidSchwartz, that's a rather broad claim, i'd say that in general, when you're exposed to external latencies (i.e. memory, network, etc), the more threads you have the more work you can achieve, assuming your app can parallelize its work. HT could help here by increasing the effective core count without the need of context switching. Of course, you also have caching considerations, this paper discusses the tradeoffs - ieeexplore.ieee.org/xpl/… – Abbott

For performance issues, the first step is always to PROFILE! – God

Your app didn't need nor worth to pay effort to consider whether true core or hyper-threaded core will benefit or not while it's 1/ not CPU bounded; 2/ abstracted by the Java VM implementation; 3/ Depends on the efficiency of the framework; 4/ More likely the database will face the bottleneck before the Java code running inside the framework. – Cloudscape

Answer 1

How to regard Hyper-Threading when deciding thread pool size?

The short answer is Don't.

The longer answer is that Goetz's "formula" is actually just a rule of thumb. The language used here

... "generally speaking, for a CPU-bound app one might consider using ( # of cores + 1 ) as the number of threads"

makes that clear. There are all sorts of reasons why that "rule of thumb" number might give you suboptimal performance.

The correct approach is:

1. Pick a number
2. Measure performance
3. Adjust number and go to step 2.

.... until you arrive at a thread pool size that gives roughly the best answer for your use-case.

The other thing to note is that when you are building a server-based system is that performance is only one of many considerations. Another is how your system performs under extreme load. If you optimize your performance based on a "best case" workload without considering behaviour under overload situations, then you can be in for a nasty shock if something goes wrong.

Optimizing solely for maximum throughput can have bad consequences...

Incumbent

The approach "Pick a number / Measure performance" makes sense when you are building a server / service which runs in a single or few instances on a more or less stable HW. When you create an application which may be run on varied HW (e.g. something distributed to thousands or millions of users), this is almost impossible - and you still want to use the scale to the HW capabilities. – Dwarfism

So provide the users with documentation on how to do the tuning. Here's the thing: in such a scenario you have no idea what your users' tuning requirements will be. If they want to run the application AND read email, browse the web, play FPS games at the same time, they won't appreciate the app hogging the system resources. – Incumbent

Assume the application is a consumer software (e.g. a computer or mobile game). It should not expect users to perform any kind of tuning, it should Just Work. – Dwarfism

So you are talking about hypothetical applications? Well sure, there are hypothetical cases where hand tuning for the platform may not be an option. But the flipside is that there is no universal "advice" on whether or how to take account of hyper-threads will work for all applications. If you are a developer who is tasked to implement an application that is required to perform optimally on all platforms ... without any tuning ... you are are in a difficult place. – Incumbent

I am talking about an application I am developing, but at the same time I try to keep discussion general and useful for other - and yes, I think developers of user application, be it desktop and mobile, are in a difficult place regarding HW compatibility. – Dwarfism

You are not talking about a hardware compatibility issue. You are talking about a "how to get the last 10% of performance out of the hardware" issue. If you really think that is an important issue, you should be writing in a language that gives you more control over that kind of thing. As I said before, there are no good general answers to this problem. IMO, there is nothing more to discuss ... and this isn't a discussion forum anyway. – Incumbent