Java Flight Recorder - Continuous rolling recording

Question

Question

How to get a rolling recording into disk, with a maximum age?

Context

When something goes bad in my server, I want to be able to dump the profiling information of the hours prior and analyse it, to know what went wrong.

1. As I don't know when things will go bad, the JDK should be continuously saving the events to disk.
2. As the server is not rebooted often, to avoid the files growing unbounded, I need to set some sort of cap (either age, or size).

So, in other words, I wanted the JDK to save the recordings continuously to disk, but remove the older files/recordings such that the total amount remains under a certain threshold (age or size).

To that end, these are the options I have for version Oracle JDK 1.8.0_144:

-XX:+UnlockCommercialFeatures
-XX:+FlightRecorder
-XX:StartFlightRecording
   name=<foo-bar>
-XX:FlightRecorderOptions
   defaultrecording=true   // what does this do even?
   disk=true
   maxage=1h // this is what I thought would solve my problem! 
   repository=<path-to-where-I-want-the-recording>
   maxchunksize=5M

I would have thought that setting maxage=1h would only keep the last 1 hour of recording on disk. But no! Its been past 1 day, and the files are not being capped.

At the same time the maxchunksize appears to work. The various .jfr files have approximately 5M. Of which there are many such files, since the the age capped is not being enforced.

What am I doing wrong?

Dehlia

Why don't you make a cron job that removes old files? – Barham

@Barham Suggesting a workaround doesn't help much with resolving the root cause. – Underhand

Asking questions that include // what does this do even? don't exactly bring out the best of my customer support qualities out. – Barham

@Barham cron jobs add complexity to the system. An extra thing to remember, maintain, document, which results in a brittle system. Additionally, this is a simple behaviour which one would think is expected and in fact needed by most users of JFR, so a "native" solution most likely exists. – Dehlia

1. That is what I am doing by posting here. 2. No need to be rude. My question is clear, succinct, well curated. Your frustrations are ill-grounded. 3. I have found no documentation on the meaning of defaultrecording. Have you? – Dehlia

Answer 1

The problem, I think, is that you are starting two recordings, one with-XX:StartFlightRecording and one with -XX:FlightRecorderOptions=defaultrecording=true.

The one with -XX:StartFlightRecording is unbounded. I think the following would be appropriate option for Oracle JDK 1.8.0_144 and your use case:

-XX:+UnlockCommercialFeatures
-XX:FlightRecorderOptions=repository=<path>
-XX:StartFlightRecording=maxage=1h,name=<name>

-XX:+UnlockCommercialFeatures is needed because JFR is a commercial feature in Oracle JDK 8. From JDK 11, it's no longer needed.

-XX:+FlightRecorder is not needed with -XX:StartFlightRecoding for JDK 8u40 or later. The JFR buffers are now setup when the first recording is started, not when the JVM starts. If a recording is started with defaultrecording=true, -XX:+FlightRecorder is still needed.

-XX:FlightRecorderOptions=defaultrecording=true does a lot of things, mostly for historical reasons, but only needed when doing in-memory recordings. From JDK 9, the option is never needed and has been removed.

-XX:FlightRecorderOptions=disk=true,maxage=1h isn't needed if -XX:StartFlightRecording is used, which is the recommended way to start JFR.

Unless you have an issue, I would keep the maxchunksize at the default (12 MB). It's the chunk files size JFR has been optimized for and tested against.

Testa

Answer 2

I am accepting Kire Haglin's answer.

Adding a bit more value for what worked for me on this JDK:

-XX:+UnlockCommercialFeatures
-XX:StartFlightRecording
  name=<foo-bar>
  maxage=12h
  dumponexit=true
-XX:FlightRecorderOptions
  dumponexitpath=<path-to-file>.jfr
  disk=true
  repository=<some-folder-path>

Notice the extra parameters dumponexit and dumponexitpath, not present on my original question. I ended up also needing those.
After trial and error, it appears dumponexit must exist within the XX:StartFlightRecording argument, and dumponexitpath on the FlightRecorderOptions argument. No other arrangement seems to work.

Notice also that removing -XX:+FlightRecorder and defaultrecording=true (as Kire suggested) still worked. That being said, I don't think the presence of defaultrecording=true was triggering a double recording.
I am saying this because when issuing command jcmd <PID> JFR.check <name> I only got one entry.

Dehlia

Did you try "jcmd <PID> JFR.check" without the <name>? If you want a fixed file name (not an autogenerated), you should be able to use -XX:StartFlightRecording=filename=dump.jfr,dumponexit=true,maxage=1h and then only have -XX:FlightRecorderOptions=repository=<some-folder-path>. If you want the filename autogenerated, then -XX:StartFlightRecording can't be used and defaultrecording=true path is needed with dumponexitpath. All this mess was fixed in JDK 9. – Testa

That's a good point about the command without the <name>. I no longer remember. If I did issue it with the name, maybe I was indeed triggering two recordings after all. Thanks about the other info. What a mess indeed. xD – Dehlia