Retrieve list of log names from Google Cloud Stackdriver API with Python

Question

I'm using Google's Stackdriver Logging Client Libraries for Python to programmatically retrieve log entries, similar to using gcloud beta logging read.

Stackdriver also does provide an API to retrieve a list of log names, which is most probably what gcloud beta logging logs list uses.

How can I use that API with the Python client libraries? I couldn't find anything in the docs.

Abcoulomb

Answer 1

You can work with the Stackdriver Logging Client Libraries for Python. You can install them using the command pip install --upgrade google-cloud-logging, and after setting up authentication, you will be able to run a simple program such as the one I have quickly developed and share below.

Before getting into the code itself, let me share with you some interesting documentation pages that will help you develop your own code to retrieve log entries programatically using these Client Libraries:

• First, there is the general Stackdriver Logging Python Client Library documentation. You will find all kind of information here: retrieving, writing and deleting logs, exporting logs, etc.
  • In detail, you will be interested in how to retrieve log entries, listing them from a single or multiple projects, and also applying advanced filters.
• Also have a look at how the entry class is defined, in order to access the fields that you are interested in (in my example, I only check the timestamp and the severity fields).
• A set of examples that might also be useful.

---

Now that you have all the data you need, let's get into some easy coding:

# Import the Google Cloud Python client library
from google.cloud import logging
from google.cloud.logging import DESCENDING

# Instantiate a client
logging_client = logging.Client(project = "<YOUR_PROJECT_ID>")

# Set the filter to apply to the logs
FILTER = 'resource.type:gae_app and resource.labels.module_id:default and severity>=WARNING'

i = 0 
# List the entries in DESCENDING order and applying the FILTER
for entry in logging_client.list_entries(order_by=DESCENDING, filter_=FILTER):  # API call
    print('{} - Severity: {}'.format(entry.timestamp, entry.severity))
    if (i >= 5):
        break
    i += 1

This small snippet imports the Client Libraries, instantiates a client at your project (with Project ID equal to YOUR_PROJECT_ID), sets a filter that only looks for log entries with a severity higher than WARNING, and finally lists the 6 most recent logs matching the filter.

The results of running this code are the following:

my-console:python/logs$ python example_log.py
2018-01-25 09:57:51.524603+00:00 - Severity: ERROR
2018-01-25 09:57:44.696807+00:00 - Severity: WARNING
2018-01-25 09:57:44.661957+00:00 - Severity: ERROR
2018-01-25 09:57:37.948483+00:00 - Severity: WARNING
2018-01-25 09:57:19.632910+00:00 - Severity: ERROR
2018-01-25 09:54:39.334199+00:00 - Severity: ERROR

Which are the entries that exactly correspond to the logs matching the filter I established (note they are shown in inverse order in this screenshot):

I hope this small piece of code (accompanied by all the documentation pages I shared) can be useful for you to retrieve logs programmatically using the Stackdriver Client Libraries for Python.

---

As pointed out by @otto.poellath, it might also be interesting to list all the log names available in your project. However, there is currently not a Python Client Library method available for that purpose, so we will have to work with the old Python API Client Library (not the same as Python Client Library) for that. It can be installed with the command pip install --upgrade google-api-python-client, and it makes easier to use the REST API (which as you shared in your question does indeed include a method to list log names) by providing a library for Python. It is not as easy to work with it as it is with the new Client Libraries, but it implements all (or almost all) methods that are available through the REST API itself. Below I share another code snippet that lists all the log names with any written log in your project:

from apiclient.discovery import build
from oauth2client.client import GoogleCredentials
import json

credentials = GoogleCredentials.get_application_default()
service = build('logging', 'v2', credentials=credentials)

# Methods available in: https://developers.google.com/resources/api-libraries/documentation/logging/v2/python/latest/index.html
collection = service.logs()

# Build the request and execute it
request = collection.list(parent='projects/<YOUR_PROJECT_ID>')
res = request.execute()

print(json.dumps(res, sort_keys=True, indent=4))

It prints a result such as this one:

my-console:python/logs$ python list_logs.py
{
    "logNames": [
        "projects/<YOUR_PROJECT_ID>/logs/my-log",
        "projects/<YOUR_PROJECT_ID>/logs/my-test-log",
        "projects/<YOUR_PROJECT_ID>/logs/python",
        "projects/<YOUR_PROJECT_ID>/logs/requests"
    ]
}

I know this is not exactly what you ask in the question, as it is not using Python Client Libraries specifically, but I think it might be also interesting for you, knowing that this feature is not available in the new Client Libraries, and the result is similar, as you can access the log names list programmatically using Python.

Kristopherkristos

Thanks for your answer! But as I wrote in my question, retrieving log entries works just fine. What I'm asking about is how to use the python libraries to retrieve the list of logs, like what 'gcloud logging logs list` does. – Abcoulomb

Oh sorry, I missunderstood your question, let me have a look in case I find the information you require. – Kristopherkristos

I have edited my question adding a workaround to find the log names list you need. It is not exactly using Python Client Libraries (as the method you require is not available in those libraries), but it is a good way to retrieve the list programmatically with Python. I hope it helps. – Kristopherkristos

Answer 2

from google.cloud import logging

# Instantiate a client
logging_client = logging.Client(project="projectID")

# Set the filter to apply to the logs
FILTER = 'resource.labels.function_name="func_name" severity="DEBUG"'

for entry in logging_client.list_entries(filter_=FILTER):  # API call
    print(entry.severity, entry.timestamp, entry.payload)

This is with respect to Cloud Function!

Violence