Huge amount of JAR files in jboss/server/web/tmp/vfs-nested.tmp directory

Question

Sometimes we have huge amount of JAR files in jboss/server/web/tmp/vfs-nested.tmp directory.
For example today this directory contained over 350k jar files.
But on other hosts there are only 2 jar files in this directory.
What can be the root cause of this problem?

We use JBoss 5.1

UPDATE:
I found following information in release notes for JBoss 5.1.0.GA:

JBoss VFS provides a set of different switches to control it's internal behavior. JBoss AS sets jboss.vfs.forceCopy=true by default. To see all the provided VFS flags check out the code of the VFSUtils.java class.

So I do not understand what should I set?
Should I set -Djboss.vfs.forceNoCopy=true or -Djboss.vfs.forceCopy=false?
Or should I set both of them?

UPDATE 1:
I have read entire thread http://community.jboss.org/thread/2148?start=0&tstart=0 and now I am not shure that I should change either jboss.vfs.forceCopy or jboss.vfs.forceNoCopy.
According to this thread I will have OutOfMemory error instead of huge amount of files in tmp dir.

Malpractice

Have the same issue, so I'm putting a bounty on this question. – Stipe

Did you ever get a good answer to this? I am having the same issue. Two identical(?) servers, one filling up with tmp files, the other seemingly fine. And I believe that creating these files every minute is also slowing the server down. – Pegmatite

@DanielWilliams We are also having this problem, and, while we have not found a way to stop JBoss from producing these files, we have found that we can safely delete the older files from this folder while JBoss is running without affecting the application(s). The command we use is 'find . -ctime +1 -exec rm {} \;" from inside the vfs-nested.tmp folder to delete files older than 24 hours (YMMV). We are pretty sure that the problem is associated with the use of symlinks to point towards our deployables, perhaps in association with the use of Twiddle to do our deployments. See JBoss forums too! – Ursola

Relates to serverfault.com/questions/128458/… – Dominicadominical

Answer 1

From here: http://sourceforge.net/project/shownotes.php?release_id=575410

"Excessive nestedjarNNN.tmp files in the tmp directory. The VFS unwraps nested jars by extracting the nested jar into a tmp file in the java tmp directory. This can result in a large number of files that fill up the tmp directory. You can disable this behavior by setting -Djboss.vfs.forceNoCopy=true on command line used to start jboss. This will be enabled by default in a future release, JBAS-4389."

Flaviaflavian

Our application works on different hosts and only on 2 we have this problem. Why don't we have this problem on all hosts? – Malpractice

The problem here is that forceNoCopy has documented impact on memory consumption and istead of out-of-disk-space you get OutOfMemoryError – Lutetium

Answer 2

jskaggz has a good answer. In addition, I have this in the beginning of my run.bat file:

rmdir /s /q c:\apps\jboss-5.1.0.ga\server\default\tmp
rmdir /s /q c:\apps\jboss-5.1.0.ga\server\default\work
rmdir /s /q c:\apps\jboss-5.1.0.ga\server\default\log
mkdir       c:\apps\jboss-5.1.0.ga\server\default\tmp
mkdir       c:\apps\jboss-5.1.0.ga\server\default\work
mkdir       c:\apps\jboss-5.1.0.ga\server\default\log
echo --- Cleared temp folders ---

I've had problems with old copies of classes hanging around, so this seems to help.

Dominicadominical

Our application works on different hosts and only on 2 we have this problem. Why don't we have this problem on all hosts? – Malpractice

Answer 3

I had the same issue described above in production and resolved it with the following solution.

Added java options

    -Djboss.vfs.cache=org.jboss.virtual.plugins.cache.IterableTimedVFSCache
    -Djboss.vfs.cache.TimedPolicyCaching.lifetime=1440

My setup also defines additional deployment directories so I needed to add these additional directories to vfs.xml file located in $JBOSS_SERVER_HOME/conf/bootstrap/ in order to see the benefit.

The lifetime setting I think is in minutes so I set it to a day as I have a scheduled restart of the server overnight.

Prior to finding this solution I had also tried using -Djboss.vfs.forceNoCopy=true and -Djboss.vfs.forceCopy=false

This appeared to work but I noticed the application ran a lot slower - presumably because these settings turn vfs caching off.

My Jboss version is jboss-5.1.0.GA and my application runs in a cluster on production.

Kazue

Answer 4

We have solved this problem by exploded deployment ( works for war and ear) as described in jboss documentation http://docs.jboss.org/jbossas/docs/Administration_And_Configuration_Guide/5/html/ch03s01.html

That's way vfs is not used.

Selassie

Answer 5

Found a lot others having the same problem running in cluster (or farm) environments. https://issues.jboss.org/browse/JBAS-7126 desribes to solve the problem having a farm directory as deployment directory.

I had the same problem using a 2nd deploy directory. The jar files out of my applications coming from this 2nd deploy directory got copied until the disk was full.

Tried adding the 2nd deploy directory the same way as at https://issues.jboss.org/browse/JBAS-7126 described for the farm directory.

It works well!

Kreiker

Answer 6

We were facing the same issue and were able to circumvent the issue by using a farm directory as deployment directory.

After putting that process in place we were facing one more issue due to the nature of our DEV environment ( We have clustered environment and we have many developers deploying on the shared DEV environment ) of not getting a consistent results while we were deploying the EARs and WARs that way .We circumvented the issue by making sure that the EARs and JARs that are being deployed are TOUCHED (http://en.wikipedia.org/wiki/Touch_(Unix) ) on the servers to make sure that inconsistencies are avoided .

Modestomodesty