Interpreting jstat results

Question

I am new to jstat tool. Therefore I did a sample as below.

./jstat -gcutil -t 4001 5000
Timestamp         S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
       565088.4   0.00   0.89  75.86  40.59  84.80    405    3.822     4    0.549    4.371
       565093.4   0.00   0.89  77.81  40.59  84.80    405    3.822     4    0.549    4.371
       565098.4   0.00   0.89  77.81  40.59  84.80    405    3.822     4    0.549    4.371
       565103.5   0.00   0.89  77.85  40.59  84.80    405    3.822     4    0.549    4.371
       565108.5   0.00   0.89  77.85  40.59  84.80    405    3.822     4    0.549    4.371
       565113.4   0.00   0.89  77.85  40.59  84.80    405    3.822     4    0.549    4.371


jstat -gc output

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
704.0  704.0   0.4    0.0    6080.0   4013.8   14928.0     6335.2   21248.0 18019.6    436    3.957   4      0.549    4.506
704.0  704.0   0.4    0.0    6080.0   4016.6   14928.0     6335.2   21248.0 18019.6    436    3.957   4      0.549    4.506
704.0  704.0   0.4    0.0    6080.0   4135.4   14928.0     6335.2   21248.0 18019.6    436    3.957   4      0.549    4.506
704.0  704.0   0.4    0.0    6080.0   4135.4   14928.0     6335.2   21248.0 18019.6    436    3.957   4      0.549    4.506
704.0  704.0   0.4    0.0    6080.0   4135.4   14928.0     6335.2   21248.0 18019.6    436    3.957   4      0.549    4.506
704.0  704.0   0.4    0.0    6080.0   4135.4   14928.0     6335.2   21248.0 18019.6    436    3.957   4      0.549    4.506

What does this results indicates? Which are the columns to look out for possible memory problem e.g. memory leak etc.

Hebe

Answer 1

See the documentation:

https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html

Basically one row is one point in time. The columns show data about the JVM memory areas (Survivor, Eden, ...), understanding them is impossible without knowing how the JVM works.

For example in the article JVM garbage collection in young generation there is some explanation.

Here is the excerpt how JVM object generation works:

Eden is a place where new objects created. When the Eden is full, a small GC is run: if an object has no reference to it, it will be deleted, otherwise it will survive, and move to the Survivor space (only one of the survivor spaces in use at a time, all objects from the other space is copied there).

If an object survives a certain number of back-and-forth copying, it is moved to Old space. If the Old space is full, a Full GC is run, which affects all objects in the JVM, so it is much heavier operation.

Also, there is the Permanent space, where the "metadata" (class descriptors, field, method, ... descriptors) are stored.

Jefferson

I have google too but I dont quite get this part S0C Current survivor space 0 capacity (KB). S1C Current survivor space 1 capacity (KB). as most of the sites discuss regarding this. – Hebe

This is not the size, it's the capacity (in %). Have a look at this: #13661371 There is som explanation about the object generation. – Jefferson

so in my case the eden space is 77.85 which soon will be having a minor GC is it? So at present the eden is now nearly 94% what does this indicate when the eden grow meaning new stuff is added is it? – Hebe

Does it mean that my old stufed are not cleared automatically and can this lead to memory leakage? – Hebe

Eden reach 100 then the S0 becomes 0.89 and S1 is 0.00 and then E start over and S0 is 0.00 and S1 back to 0.89? – Hebe

You don't have to worry, GC runs quite frequently. See update for some explanation. – Jefferson

In your case I would add more PermGen space, 84% is a bit high. And yes, if PermGen is full, it can cause OutOfMemoryError. – Jefferson

now I notice is 80.87% there is a drop. How will it get a drop? How much PermGen to be added? – Hebe

THe PermGen itself also has a minimum size, actual size, and maximum size. Probably when it was 84%, the size was smaller. Later it has been extended, do the percent ws dropped. – Jefferson

I have accepted it but I want to learn and confirm more things from you. How do I do the percent vs dropped? What else should be on the looked out? – Hebe

When I run gccause it shows me "Allocation Failure No GC". What does this means Allocation Failure? – Hebe

If you run jstat -gc you will get the capacities and the utilisation percentage as well. There you can see the numbers. I cannot give you a course on jstat :) but I can tell what I would see: Old and PermGen size, the frequency of the small GC (it tells about the object usage, if it's frequent I'd add more to Eden). Also, if it's a critical app, I would check it more frequently and compared the result (as users use the app, how the memory spaces are growing) – Jefferson

On Allocation Failure No GC" : look at ibm.com/developerworks/library/i-gctroub and see "Avoiding heap trashing" - there is a solution there. – Jefferson

I updated my question with jstat -gc output so what should I look is the allocated and utilisation ? Yes I understand you cant give me a course hehe but just highlight to me what should be on the look out as my app is a critical one. – Hebe

I have visited the link you gave so that means I must increate a bigger heap size to avoid this problem is it? Actually my application is quite small but why is it having allocation failure – Hebe

Maybe because the GC always frees some memory, but it's not enough for your app's needs. E.g. you want to allocate a large array (e.g. a String has a char[] inside), and there is not enough memory for that. You cannot say that the memory is full, it's just not enough. – Jefferson

Yes I use quite a number of places String variables. Another thing I have also few lines to monitor the memory in the application itself. System.out.println("Used Memory:" + (runtime.totalMemory() - runtime.freeMemory()) / mb); I notice there is some small increase like one or 2 mb after sometime. – Hebe

It is interesting if you want to store a String with 1M (e.g. even with an implicit stringBuilder.toString()). If the GC runs, it will free up some memory, probably that's what you can see. 2M is quite small. From the latest stats I can see that you should increase the PermGen space, because that is almost full. – Jefferson

I dont get you when you say "if you want to store a String with 1M". How much should I add the PermGen? So during GC wouldnt the PermGen get clearer. – Hebe

These are 2 different stories. Actually, both the -Xmx and the -XX:MaxPermSize parameters should be raised. – Jefferson

so for a 4Gb machine what are the best parameters set for both -Xmx and -XX:MaxPermSize? – Hebe

It depends. What are the current settings? Is it desktop or server app? What OS you have? What other apps are running on it? What user load is expected (compared to the current load)? Generally, you can give 3G to -Xmx and 512M - 768M to permsize as a maximum. – Jefferson

Please consider using a room on Stack Overflow Chat; it's really much easier to hold a conversation there. – Nariko

@MichaelMyers you're right, but unfortunately new14 has not enough reputation to use it – Jefferson

I am using Centos 6.3. I am not sure how to check out the current settings. The app is socket listener where it is a server and clients will connect and pass data to it. The cients here devices which will connect and pass data. – Hebe

Check the script where you start the app, probably there are the current options. If you find it, you can modify them. If not, you use your JVM's default settings (which is usually about 256M), in this case add the params you want and start playing around. Try 1G / 128M for start. – Jefferson

Ok I will try out with Xmx as 1G and MaxPermSize as 128M. Another thing System.out.println("Max Memory:" + runtime.maxMemory() / mb); gives the values as 1735? IS this the Xmx value. – Hebe

@GaborSch I notice there is an increase of 1Mb over some few hours. What else tools can I use to confirm if I have some memory issues? – Hebe

1MB is not much. In fact, that's very low. I don't know that libraries you use, but imagine that every bit of code you use in this JVM can add something (classes, objects, caches, factories, sessions, pools, whatever construct they use) to the allocated memory. So if you have 1G, calculate that when would it reach the maximum available memory - weeks. But that's another time scale, you will see that the rate of the increase is dropping - it's kind of warm-up. – Jefferson

I also recommend to look around that tools you find in the $JAVA_HOME/bin. I recommend jmap - docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jmap.html – Jefferson

@GaborSch ok basically my application is socket listener and there after it have a number select,insert and update statement are run. In addition I am also using bonecp for connection pooling. I have tried using the jmap and manage to take the output. So what are key things I should look out in the jmap. I will post a sample of it? – Hebe

Yes, I'd recommend to post a new question about that. But before posting a question, please look around if there are similar (already answered) questions there, otherwise you will be downvoted. – Jefferson

@GaborSch actually I have managed to build snapshots using this method ./jmap -dump:format=b,file=snapshot2.bin 12671 and view it via visual vm tool and compare snapshot is that good enough to indicate? – Hebe

Look for oddities. For example there are too many instances of (YourClass). – Jefferson

I saw this java.lang.outofMemoryError 8 instance but the bigger instances are of of char =32367 and java.lang.String =28761. So what does this number indicte that this many characacter have been use or in use ? – Hebe

Answer 2

gcutil gives stats in terms of percentage utilization

-gcutil Option
Summary of Garbage Collection Statistics 
Column  Description
S0      Survivor space 0 utilization as a percentage of the space's current capacity.
S1      Survivor space 1 utilization as a percentage of the space's current capacity.
E       Eden space utilization as a percentage of the space's current capacity.
O       Old space utilization as a percentage of the space's current capacity.
P       Permanent space utilization as a percentage of the space's current capacity.
YGC     Number of young generation GC events.
YGCT    Young generation garbage collection time.
FGC     Number of full GC events.
FGCT    Full garbage collection time.
GCT     Total garbage collection time.

gc gives statistics in terms of alloted space and utilized space.

-gc Option
Garbage-collected heap statistics 
Column  Description
S0C     Current survivor space 0 capacity (KB).
S1C     Current survivor space 1 capacity (KB).
S0U     Survivor space 0 utilization (KB).
S1U     Survivor space 1 utilization (KB).
EC      Current eden space capacity (KB).
EU      Eden space utilization (KB).
OC      Current old space capacity (KB).
OU      Old space utilization (KB).
PC      Current permanent space capacity (KB).
PU      Permanent space utilization (KB).
YGC     Number of young generation GC Events.
YGCT    Young generation garbage collection time.
FGC     Number of full GC events.
FGCT    Full garbage collection time.
GCT     Total garbage collection time.

Source : Docs

Fraley

Answer 3

Use this simple online jstat visualizer tool to plot jstat GC statistics.

Pilar

Rather than post links (which can die), you should provide the necessary information in your answer. – Fernferna