How can I suspend appstats for a single request on App Engine/Java?

Question

I normally run appstats fulltime on my sandbox appid. However, I have one complicated operation (basically rebuilding the stock database) that causes appstats to blow up my instance, throwing OutOfMemoryErrors. Even with larger instance sizes, it still fails. Appstats just wants too much RAM.

I don't need appstats on this request. Ideally I will call a method on whatever ThreadLocal object is responsible for appstats collection and tell it to twiddle its thumbs for a few minutes.

I've considered extending the AppstatsFilter to ignore certain URLs, but the offending request executes as a deferred task and identifying it by path is somewhat complicated.

How can I tell appstats to pause?

Just in case it isn't clear: Uploading a version of my app with appstats disabled, running my task, then uploading a version with appstats enabled is what I'm doing now. I don't want to do this.

Astigmatism

I'm assuming just opting in for the urls you want monitored isn't an option? FWIW I've wanted to be able to do what you're asking for more than once but I bit the bullet and just made of list of the urls we needed to monitor and put them into the config. – Normalie

Unfortunately the URL I want to exclude is /_ah/queue/__deferred__. I could probably mount the DeferredTaskServlet on an additional URL and route this specific request to it, but this seems like going pretty far off the reservation. – Astigmatism

Answer 1

What I did is write my own CustomAppstatsFilter and exclude certain url's.

public class CustomAppstatsFilter extends AppstatsFilter {

  @Override
  public void doFilter(ServletRequest request, ServletResponse response,
        FilterChain chain) throws IOException, ServletException {

    if (request instanceof HttpServletRequest) {
        String url = ((HttpServletRequest) request).getRequestURL().toString();

        // We do not want these to show up in appstats
        if ((url.contains("appstats"))
                || url.contains("_ah/")
                || url.contains("cron/")
                || url.contains("batch/")) {
            chain.doFilter(request, response);
            return;
        } else {
            super.doFilter(request, response, chain);
        }
    }
  }
}

EDIT - This can be combined with ZiglioNZ great answer.

<!-- Appstats filter for profiling application -->
<filter>
    <filter-name>appstats</filter-name>
    <filter-class>my.company.filter.CustomAppstatsFilter</filter-class>
    <init-param>
        <param-name>maxLinesOfStackTrace</param-name>
        <param-value>5</param-value>
    </init-param>
</filter>
<filter-mapping>
    <!-- excludes are in CustomAppstatsFilter -->
    <filter-name>appstats</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>

Dmso

This almost does the trick, but the URL is the deferred task execution URL, which applies to all deferred tasks. What I need is something equivalent to Python's dont_record() that I can run in my code. – Astigmatism

Like this article explains, if you need complete control over how tasks are queued and executed, you need to use the Task Queue API. ;-) – Dmso

I don't need or want complete control over how tasks are queued and executed; that is swatting at flies with sledgehammers. I just want appstats to fail more gracefully. Python has an API for it, but apparently Java doesn't :( – Astigmatism

Answer 2

Good question. For Python, the answer is simple:

from google.appengine.ext.appstats import recording

class ...(webapp.RequestHandler):
  def get(self):
    recording.dont_record()
    ...

Maybe there's a similar API in Java?

Alternatively, again the Python version has a flexible way of filtering out which requests to record; I think in the Java version you can accomplish a similar thing by using the and entries in your web.xml. (See the Java appstats docs.)

Palliative

I can't find a Java equivalent in the SDK source code, and unfortunately the relevant url is the standard deferred task url. A method like this is exactly what I'm looking for though! – Astigmatism

Answer 3

Jeff, I wonder whether this can help you reducing the occurence of OutOfMemoryErrors: How to reduce the memory usage of Appstats on Google App Engine Java

<filter>
  <filter-name>appstats</filter-name>
    <filter-class>com.google.appengine.tools.appstats.AppstatsFilter</filter-class>
    <init-param>
      <param-name>maxLinesOfStackTrace</param-name>
      <param-value>16</param-value>
    </init-param>
</filter>

Literacy