Python OAuth after few days fails refreshing tokens with "invalid_grant" error

Question

I have a Python app running on Raspberry Pi that starts a livestream to a YouTube channel that I manage. This is the code that I use to authenticate:

import google_auth_oauthlib.flow
import googleapiclient.discovery
import googleapiclient.errors
import google.auth.transport.requests
import google.oauth2.credentials
import requests

CLIENT_SECRETS_FILE = "client_secrets.json"
YOUTUBE_READ_WRITE_SCOPE = "https://www.googleapis.com/auth/youtube"
YOUTUBE_API_SERVICE_NAME = "youtube"
YOUTUBE_API_VERSION = "v3"

def get_authenticated_service(args):
    credentials = None
    credentials_json_file = "youtube-%s.json" % slugify(args.oauth2_name)
    if os.path.exists(credentials_json_file):
        # load credentials from file
        with open(credentials_json_file, encoding='utf-8') as f:
            credentials_json = json.load(f)
        credentials = google.oauth2.credentials.Credentials.from_authorized_user_info(credentials_json)
    if not credentials or not credentials.valid:
        # no credentials file or invalid credentials
        if credentials and credentials.expired and credentials.refresh_token:
            # refresh
            request = google.auth.transport.requests.Request()
            credentials.refresh(request)
        else:
            # re-authenticate
            flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(CLIENT_SECRETS_FILE, [YOUTUBE_READ_WRITE_SCOPE])
            credentials = flow.run_console()
        # save credentials to file
        credentials_json = credentials.to_json()
        with open(credentials_json_file, 'w', encoding='utf-8') as f:
            f.write(credentials_json)
    return googleapiclient.discovery.build(
        YOUTUBE_API_SERVICE_NAME, YOUTUBE_API_VERSION, credentials=credentials)

When I run my app a first time, I must authenticate. The credentials are stored in a JSON file that looks like this:

{
  "token": "...", 
  "refresh_token": "...", 
  "token_uri": "https://oauth2.googleapis.com/token", 
  "client_id": "....apps.googleusercontent.com", 
  "client_secret": "...", 
  "scopes": ["https://www.googleapis.com/auth/youtube"],    
  "expiry": "2021-02-28T09:27:44.221302Z"
}

When I re-run the app later on, I don't have to re-authenticate. That works fine.

But after 2-3 days, I get this error:

Traceback (most recent call last):
  File "./create_broadcast.py", line 188, in <module>
    youtube = get_authenticated_service(args)
  File "./create_broadcast.py", line 83, in get_authenticated_service
    credentials.refresh(request)
  File "/home/pi/.local/lib/python3.7/site-packages/google/oauth2/credentials.py", line 214, in refresh
    scopes,
  File "/home/pi/.local/lib/python3.7/site-packages/google/oauth2/_client.py", line 248, in refresh_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/home/pi/.local/lib/python3.7/site-packages/google/oauth2/_client.py", line 124, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/home/pi/.local/lib/python3.7/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Token has been expired or revoked.', '{\n  "error": "invalid_grant",\n  "error_description": "Token has been expired or revoked."\n}')

The workaround is to delete the credentials file and re-authenticate. But I'd expect the credentials refresh to still work after a couple of days!

I do have NTP installed and running. I didn't manually revoke the token. I didn't change my Google password. I didn't generate a lot of other tokens elsewhere. I did none of the things that are told elsewhere to cause this error.

One thing to note: the app is not verified, because it's only meant for internal use. Still this shouldn't impact the lifespan of the refresh token, should it?

What could make that refreshing works after 1 day or after 2 days, but not anymore after 3 days?!

Best regards, Vic

Arlindaarline

Are you authorizing this more than once? When you refresh access does it return the same refresh token or a different one. You can have a limit of 50 refresh tokens and then the first tone will expire. – Inflame

When I re-authenticate after less than an hour, no credentials are updated. When I re-authenticate after more than an hour, both the token and the refresh token in the credentials file are new. Older tokens are not stored. – Arlindaarline

Answer 1

I very much presume that your corresponding Google project shows -- from within the Developers Console, on the page OAuth consent screen -- your app to have its Publishing status set as Testing.

According to the official documentation, your refresh tokens are subject to the following restrictions:

Refresh token expiration

You must write your code to anticipate the possibility that a granted refresh token might no longer work. A refresh token might stop working for one of these reasons:

[...]

A Google Cloud Platform project with an OAuth consent screen configured for an external user type and a publishing status of "Testing" is issued a refresh token expiring in 7 days.

Until your app's publishing status becomes set to in production -- i.e. your app gets audited by Google --, the above restriction implies that you have to run to successful completion an OAuth authentication/authorization flow every week for to obtain new refresh tokens (that have a limited lifespan of only 7 days).

Hurlburt

Is that new? thats not something i have seen before. – Inflame

I had a long exchange of comments recently on this matter (the thread got moved to chat). Initially I knew that desktop apps need not be verified; but have created a test project of this type and the Console requires verification for that app to change publishing status to in production. (Try it yourself. It's always better to double-check things.) – Hurlburt

Yes indeed, the publishing status is testing. I didn't read about this restriction before. So I'll have to do the verification dance, even if I'm the only user who has to authenticate in the app? Or did I overlook some very obvious single-user-authentication mechanism? – Arlindaarline

No, you did not overlooked any other authentication mechanisms, because OAuth is the only one. Emphasizing it once more: verification for to step in production is a very new requirement Google is imposing to its API users. – Hurlburt

I tried another approach. Apps in a Google Workspace or Cloud Identity account can be set as "internal" and need no verification. I created a new app in a Cloud Identity account and I made the admin user of this account Administrator of our YouTube channel, but when I want to authenticate to the app by selecting the admin user and then as brand account the YouTube channel, it's refused because "This client is restricted to users within its organization." Confusing, because the user is within the organization, only he's managing a YouTube channel outside the organization... back to square one. – Arlindaarline

Referring to my previous comment, here's a confirmation that the "internal user" approach won't work for apps that require the YouTube Data API: https://mcmap.net/q/1769639/-youtube-data-api-upload-video-to-youtube-brand-account-39-s-channel-using-manager-account – Arlindaarline

I got my app verified by Google. At first they suggested that I should use a Cloud Identity account for internal users, but when I explained the above, the approval came through. – Arlindaarline