Variable and degrading performance when using jbcrypt

Question

I'm using jbcrypt to hash passwords in a project. Performance is about 500 ms when validating passwords on the hardware I am using (log_rounds set to 12). However, after a while with regular use the performance time suddenly drops to a whopping 15 seconds. The drop is very sudden with no buildup and stays constant until the process is restarted.

Profiling shows that the extra time is used in the key(..) method.

Source: http://jbcrypt.googlecode.com/svn/tags/jbcrypt-0.3m/src/main/java/org/mindrot/jbcrypt/BCrypt.java

This method only calculates the hash using basic functions like xor, and, shift etc. There is no object assignments, usage of external resources, random number generation etc.

Performance does not drop for other functionality in the same process. Memory allocation is stable and low. Full GC is not involved.

Has anyone seen this before or any clue to why this happens? I could understand a variable performance that to some degree was dependent on other circumstances, but this is a very sudden and stable drop from about 500ms. to about 15000 ms.

Sansen

What function are you calling to validate the passwords? – Sandlin

key itself is simply fixed-length loops over calls to streamtoword and encipher. Did your profiling identify the aggregate time spent in those methods? – Holub

Could you try tool like visualvm? It is light weight, may help to identify bottleneck. – Unpractical

@joe Not exactly those two methods, but I agree that it is likely that the time is used there. – Sansen

Answer 1

It turned out that this had something to do with classloading. The library was loaded in many different classloaders. The problem disappeared when we loaded the library in the system classloader.

Sansen

How did you avoid it? Why is it a problem? – Starlike

Answer 2

It's possible that SecureRandom is running out of entropy and causing this issue.

See How to solve performance problem with Java SecureRandom?

Sandlin

That could have been the reason, but SecureRandom is only used by the library when it generates a new salt, and that part is not used when validating passwords. – Sansen

Answer 3

The problem is indeed related to class loading as @sstendal mentioned, but only indirectly.

I had the same issue and in my case, the real culprit was the JIT-Compiler.

BCrypt.encipher() is very slow if it does not get compiled. After I switched on JIT compilation logging with -XX:+PrintCompilation, it turned out that the code cache was too small. After I increased its size with -XX:ReservedCodeCacheSize, the problem vanished.

Saltzman