Logs are Not Nested Under Requests in Flexible VM

Question

I have a module running in App Engine Classic and another module running in App Engine Flexible. When viewing logs running in App Engine Classic, the log entries are nested within each request.

However, when viewing logs running in App Engine Flexible, they are not nested. Each log entry appears to be unconnected from the request. This makes it very difficult to determine which log is associated with which request.

Is it possible to get the logs in App Engine Flexible to be nested under each request (as they are with App Engine Classic)?

I cannot find anything in the documentation that explains this discrepancy.

Melanoma

Which runtime are you using? Java 8, Python, ... or are you using a custom Docker image? – Nakesha

I am using a custom runtime (runtime: custom, vm: true), but my base image is gcr.io/google_appengine/python-compat-multicore, which should be compatible with the standard App Engine APIs. – Melanoma

Answer 1

You can now write nested application logs that behave similarly to App Engine Standard by following the instructions here: https://cloud.google.com/logging/docs/view/service/appengine-logs#linking_application_logs_and_requests

Trichloride

Yup, just noticed this feature was turned on, glad to see it finally happen. A side note: in order to see these logs, you need to manually select the app logs in the log type dropdown in the log viewer. – Melanoma

Thanks! Also when it says "trace must be the same in the parent and its child(ren)" it also means "trace must be set". – Zook

Could someone give an example on how this can be used. For instance Flexible environment already generates "parent" log entry (for the request itself). How do I get values of resource.type, resource.labels and trace for that parent entry, to propagate them to child entry? – Instill

Answer 2

As you might imagine, App Engine Flexible uses a good bit of different machinery from traditional App Engine, and these different environments do log differently. It is unfortunately not currently possible to induce Flexible to log in the same data format as Classic.

If this is sufficiently important to you, you can always directly log via the Cloud Logging API, which is what the App Engine logging API ends up passing through to today. You can infer the data format to log by looking at traditional App Engine entries, though we have open a feature request to publicly document this more explicitly.

Also, as a passing note, on the Stackdriver Logging (formerly Cloud Logging) front, we're definitely thinking about ways to support this kind of denormalization of log data more generally, rather than just special-casing for App Engine logging in a specific format, as this would be broadly useful functionality. However, we don't have any specific plans or timelines that we can share at this point—just noting that this is on our radar.

Kauri

Thanks Ben. Is it true that these denormalized logs only appear when using a Flexible Custom Runtime (i.e., runtime: custom, vm: true)? Do users get nested logs when using the standard Flexible Runtime (i.e., runtime: python-compat, vm: true)? – Melanoma

Chiming in on the different runtimes, on runtime: python we do not get the nested logs. No idea about python-compat. – Inedible

Hi Ben, do you an update on that the documentation of how to use the logging API to emulate app engine standard log grouping by request? – Hereunto

Chiming in that it's been nearly two years now. Would love to see this documented and official. – Zook

Here's a GAE LogEntry: gist.github.com/6c9df73d4e168eedfb7f14276c7bd1c1 – Zook

Answer 3

The suggestion by Marc Unangst to link application logs and requests sounds great, unfortunately the documentation on how to achieve this is not very clear (https://cloud.google.com/appengine/articles/logging#linking_app_logs_and_requests).

Using the "Google App Engine flexible Log Handler" from the Google Cloud Client Libraries for Python (https://google-cloud-python.readthedocs.io/en/latest/logging/handlers-app-engine.html) I have been able to log to the 'app' log that is part of the log-stack of the app engine flexible app itself. So I am currently in the same boat as the topic starter.

I noticed that, as required for log linking, the resource.labels and resource.type properties are automatically set correctly this way. However, trace information is absent, instructions seem to be missing on how to include this.

Next, the timing seems to always be off as well. The instructions require the timestamp of the app log entries to be in between parent.timestamp - parent.httpRequest.latency of the corresponding HTTP request. I'm noticing that the timestamps of my app log entries are always slightly later then the timestamp of the corresponding request. Which is weird since these entries are made during the request.

To me it currently looks like the log handler is unaware of an HTTP request cycle going on while writing log entries (I am using this with Django), hence the missing trace id's? Not sure on how to continue from here, love to hear what others may have achieved.

Flory

Hi Raj, and welcome to Stack Overflow! I don't fully understand your answer - did you provide additional information on how to solve this? Or are you just saying that you're stuck on the same problem as the original poster? – Sabba

Hi Max, both cases apply. I am stuck on the same problem as the original poster and I wanted to expand on his post by providing information on my experience in trying to solve this. For example, logging to the 'app' log by itself requires quite some searching through the various ways of using the StackDriver API's. As mentioned and felt by other posters as well, log linking is a nice feature but could benefit from better (or better findable) documentation. I considered placing a comment a few levels up but was restricted (new account). – Flory

Answer 4

I had an exact problem just like yours. My application was running on the Django framework, So I created middleware to manage logs nesting and started logging via Google Cloud Logging API using "google-cloud-logging" library.

Google Cloud Loggin API documentation does not have a good document on this.

While creating a log you need to add "TRACE" This trace should be pointing to its parent log.

I was able to achieve the following results:

Please check my step by step blog on implementing this solution here

Please check the repository source code here on Github.

Stilwell

Please try to avoid links that in the future could disappear and it results in broken links. – Palpable

Yes, actually you are right, the "accepted answer" link which referees to solutions from Google Cloud Document is expired and leading somewhere else. I had been through this issue and thought I can share solutions which I created and share it here quickly :). Thanks for your thoughts :) – Stilwell