How to fix the storage exceeded issue for Google Drive when uploading using Service Accounts

Question

If you're using Google Apps for Work (also called G-Suite) and uploading files to your Google Drive via the Drive API you'll eventually find yourself with an error message that looks like this:

Error: The user's Drive storage quota has been exceeded.

Read below to learn why this happens and how to solve it.

Fidelfidela

Answer 1

Background:

Google Drive API is authorized using "service accounts", not regular "human users". Google does not currently have a way to increase the 15GB limit for service accounts, and there is no mechanism for buying additional space. Once you've consumed your standard space allocation you'll be prevented from uploading any additional files.

Human/regular G-Suite users also get the standard 15GB but with simple options to increase it to 100GB or more for mere pennies.

So how do you upload files against the storage quota of a human account rather than a service account so you can have as much space as you want? It's actually really simple! Just not super well documented...

Solution:

This solution is to configure your service account to impersonate a human account. This means you still authenticate to the Drive API using the service account but the service account is authorized to act on behalf of a human account and the storage space will be allocated from that user's quota.

Prep: This is important. Carefully review the instructions here to make sure your service account is set up correctly for impersonation. Any deviation from this will give you random errors that will frustrate you and waste time. The service account needs to have DwD (domain-wide-delegation) enabled in the Google IAM Console AND it needs to be permitted to access the Drive scope in the API client access page of the Google Admin site.

Now that your service account is properly set up, to impersonate a regular account you simply need to specify the human account email address when creating the JWT client.

In NodeJS it looks like this (note the last parameter):

   let jwtClient = new GoogleApi.auth.JWT(
   service_account_email, 
   null, 
   private_key, 
   ['https://www.googleapis.com/auth/drive'],
   '[email protected]');

For other languages, please refer to the official Google documentation for how to impersonate a user.

When your Drive API script now uploads files they will be owned by this regular user and the space will be allocated from their quota. Increasing this is trivial and can be done by any administrator or by contacting G-Suite support.

Fidelfidela

Hi - I find it odd that this only has 7 upvotes over 5 years. We have just hit this same issue, is this really the only way around it? Or is there a newer option that has not been added here? (Thanks for the post - we may end up going down this route and if so, it will have saved us lots of time!) – Stannum

Any updates David? – Tory

Very good explained here qawithexperts.com/article/asp-net/… – Rica

For python, after setting up domain-wide delegation, use credentials.with_subject('[email protected]'): developers.google.com/identity/protocols/oauth2/…. – Correspondent

Just a remark: the account that is used for impersonation must have access to resources you want to manipulate with – Leadbelly

"Google does not currently have a way to increase the 15GB limit for service accounts" you mean they capped it on purpose, right? why? what's the point of this limit? – Collapse

Answer 2

for python:

from google.oauth2 import service_account
from googleapiclient.discovery import build
from googleapiclient.http import MediaFileUpload, MediaIoBaseDownload
from googleapiclient.errors import HttpError

def getDriveService(impersonate=False):
    SCOPES = ['https://www.googleapis.com/auth/spreadsheets', 'https://www.googleapis.com/auth/drive']
    credentials = service_account.Credentials.from_service_account_file(
            'flowautokey.json', scopes=SCOPES,
            )
    if impersonate:
        credentials = credentials.with_subject('[email protected]')
    
    drive = build('drive', 'v3', credentials=credentials)    
    return drive

You could also pass the "subject" parameter into "from_service_account_file" directly:

credentials = service_account.Credentials.from_service_account_file(
            'flowautokey.json', scopes=SCOPES, subject='email_to_impersonate@#gmail.com'
            )

Correspondent

Answer 3

In case if you use Google One instead of Google Workspace, Use this script to delete all old files spanning more than 7 days.

def delete_old_files(service, folder_id):
    # Calculate the date 7 days ago
    seven_days_ago = datetime.datetime.utcnow() - datetime.timedelta(days=7)
    seven_days_ago_str = seven_days_ago.isoformat() + 'Z'  # Format for Google Drive API

    # List all files in the specified folder
    query = f"'{folder_id}' in parents and trashed=false and createdTime < '{seven_days_ago_str}'"
    results = service.files().list(q=query, fields='files(id, name, createdTime)').execute()
    files = results.get('files', [])

    # Delete files found
    for file in files:
        try:
            service.files().delete(fileId=file['id']).execute()
            print(f"Deleted {file['name']} (ID: {file['id']})")
        except Exception as e:
            print(f"Could not delete {file['name']} (ID: {file['id']}): {e}")

Unequaled