Analyze GC logs for Sun Hotspots, JVM 6

Question

I'm trying to analyze GC behaviour for our application (running in Tomcat, under Sun's Hotspots, JVM 1.6).

So far I've Instructed the JVM to emit GC logs to a separate file using...

-Xloggc:gc.log 
-XX:+PrintGCApplicationStoppedTime 
-XX:+PrintGCApplicationConcurrentTime 
-XX:+PrintGC 
-XX:+PrintGCTimeStamps 
-XX:+PrintGCDetails

...and used jstat to output logs using...

jstat -gc -t 29045 5s > jstat.gc

I am seeing interesting information, but haven't found a tool to help me analyze/visualize these logs. I was pointed to GCViewer by this question, but it only parses a few log lines from gc.log and then crashes with an exception. Is there a better or more up-to-date tool for parsing these specific logs, for the specific JVM I'm using?

Vole

Answer 1

gcviewer hasn't been updated in a number of years, so it's hit or miss - some gc files will work fine, others get exceptions.

IBM's gc log parser works acceptably, http://www.alphaworks.ibm.com/tech/pmat/faq

Sun has something called GCPortal, but it requires:

• install into a web app server
• install of awk and perl
• install of a RDBMS with a JDBC driver and configuration
• and the real killer, third-party graph/chart software that used to be free and now isn't. Amazing.

gchisto has died, there's no longer anything in the project.

HPJmeter doesn't understand IBM gc files.

Molybdate

gcviewer for jdk 1.6/1.7 is being updated at following site https://github.com/chewiebug/GCViewer – Extraditable

Answer 2

The tools available in the IBM Support Assistant do a pretty good job.

Travail

Answer 3

I personally use HP JMeter for a lot of GC visualization. it works "ok" on SUN JRE's, and exceptionally well on HP's JRE (go figure).

With Sun HotSpot 1.6 (on non-HP platforms), I use these GC Options to produce the logs for analysis:

-Xloggc:/path/to/vgc/log/location/logfile.vgc  --XX:+PrintHeapAtGC 

Ossein

I remember using it, But now-a-days HPJmeter doesn't seems to be compatible with Oracle JRE gc.log – Bleacher

I've not had issues using HP-JMeter on any version of of oracle's latest JVMs (up through 1.7 anyway) using the options I've listed above. – Ossein

Answer 4

Have you looked at jvisualvm? Comes with the latest JDK, and allows you to watch the JVM. Sample output (using the visualGC plugin). Sample output - alt text http://g4u0420c.houston.hp.com/en/5992-4687/img/visualgc_2.png

Hautemarne

I saw it, but it can only tell me how the JVM is performing right now (or some limited timed in the past). I want to be able to know what happen 10,20,100 hours ago. – Vole

Answer 5

Here is a $0.00 log scraper good for 1.5 CMS collector which gives you a high level view of GC pauses.

You may need to change positional parameter $7 argument to the timestamp function in order to match your log line syntax (my .out gets "enhanced" by Tanuki Wrapper).

#! /usr/bin/awk -f
# Awk script to parse .out logs and print total of
# stop-the-world GC pause times in ten minute intervals

BEGIN {print "t\timark\tmark\tremark\tfullgc"}

/CMS-initial-mark:/ {
  t=timestamp($7);
  imark[t] += $(NF-1);
}

/\[CMS-concurrent-mark:/ {
  t=timestamp($7);
  split($(NF-1), b, "/");
#  print t" NF="NF" val="b[1];
  mark[t] += b[1];
}

/CMS-remark/ {
  t=timestamp($7);
  remark[t] += $(NF-1);
}

/\[Full GC / {
  t=timestamp($7);
  level=0;
  for (i=1; i<=NF; i++) {
      if ($i ~ /\[/) {
        level++;
      } else if ($i ~ /\]/) {
        level--;
      }
  }
  while (level > 0) {
    getline;
    for (i=1; i<=NF; i++) {
      if ($i ~ /\[/) {
        level++;
      } else if ( $i ~ /\]/ ) {
        level-- ;
      }
    }
  }
  if ( $(NF) ~ /secs\]/ ) {
    full[t] += $(NF-1) ;
  }
}

function timestamp(str) {
  split(str, a, ":");
  return a[1]":"substr(a[2],0,length(a[2])-1)"0";
}

# print out UK+US trading hours

END {
  for (hour = 5; hour <= 16; hour++) {
    for (minute = 0; minute <= 59; minute+=10) {
      t = sprintf("%02d:%02d", hour, minute);
      printf "%s\t%d\t%d\t%d\t%d\n", t, imark[t], mark[t], remark[t], full[t];
    }
  }
}

Humiliate

Answer 6

I tried to use Visual GC, but it seems to work with an id process (the jvm one ou jstatd one). I can't use it with a jstat.gc, I mean a file and not a stream Is it right?

Lunsford

Answer 7

Try using gchisto (gchisto.dev.java.net). It can understand the GC log output (I am not sure whether it has been updated to work with the G1 GC). You have to get the sources from CVS (you need a dev.java.net account for that) and build this yourself

Expect