Google Apps Script To Copy Entire Google Drive File Structure; How To Avoid Timeouts?

Question

My organization is switching to a Google Business account, and everyone needs to transfer their Drive files to their new accounts. Drive will not allow transfer of ownership between these accounts, so I've created a script to copy files and folders from the old account to the new account. (The old account's contents have been moved into a folder shared with the new account.)

Here's what I have so far:

function copyDrive() {
  var originFolder = DriveApp.getFolderById(originFolderID);
  var destination = DriveApp.getFolderById(destinationID);
  copyFiles(originFolder, destination);

};

function copyFiles(passedFolder, targetFolder) {
  var fileContents = passedFolder.getFiles();
  var file;
  var fileName;

  while(fileContents.hasNext()) {
    file = fileContents.next();
    fileName = file.getName();
    file.makeCopy(fileName, targetFolder);
  }
  copySubFolders(passedFolder, targetFolder);
};

function copySubFolders(passedFolder, targetFolder) {
  var folderContents = passedFolder.getFolders();
  var folder;
  var folderName;

  while(folderContents.hasNext()) {
    folder = folderContents.next();
    folderName = folder.getName();
    var subFolderCopy = targetFolder.createFolder(folderName);
    copyFiles(folder, subFolderCopy);
  }
};

Please pardon any inelegance; I am new at this. The script actually works great and preserves the folder structure, but it times out after copying ~150 files and folders. I've been looking into how to use continuation tokens, and I've read this post closely. I think I'm stuck on a conceptual level, because I'm not sure how the continuation tokens will interact with the recursive functions I've set up. It seems like I will end up with a stack of my copySubFolders function, and they will each need their own continuation tokens. Of course they all use the same variable name for their iterators, so I really have no idea how to set that up.

Any thoughts? Sorry for posting such a helpless newbie question; I hope it will at least be an interesting problem for someone.

Grout

There are questions that deal with similar issues. to get you started, the key part you need is triggers. You can have a 5minute repearing trigger that will process as much as it can and somehow remember where it it so the next trigger continues where it left off. You may store your position using script properties. Its not trivial but doable – Beneficial

@ZigMandel triggers are trivial, and not really necessary. I am capable of telling the script to execute repeatedly until it is done, and I don't think it would have to run very many times for most of our users. I know that the way to store the position is by saving "continuation tokens" as script properties, but my questions was about how to make those tokens work properly with the recursive functions I've created. Does that part make sense? – Grout

It will timeout after 6 min so if you dont use triggers it will fail unless you have few files overall. If the issue is about a problem using continuation tokens, post the code you tried using them. – Beneficial

Triggers won't stop it from failing; they'll only restart it periodically. I don't have code with tokens because I need to solve this problem about how to use them. I included a link to a project that does use them. When I have a whole call stack going I'm going to have multiple folder iterators in progress, all named folderContents. If I want to stop the script and be able to pick up where I left off, I have to save a different continuation token for each instance of the function and figure out how to get that stack going again and assign the correct tokens to each level. Make sense? – Grout

You can try my Folder Copy Sheet Add-on. If you have ideas to make it more robust, I have been looking for a partner to help develop it more. – Reta

It is not free, but it is robust, you can look at gMigrate from Promevo: promevo.com/#gmigrate – Reta

@BjornBehrendt The video is compelling, but it looks like it is several years old. Does it still work with the latest versions of Google Apps? Also, in what ways do you want to make it more robust? How much does it cost? – Grout

@bill, Folder copy add-on is free and works with the latest version of GA. It is very slow, and prone to time-out and doesn't handle files that are in multiple folders. It is good for smaller folders, and keeps the structure. I have only heard of gMigrate and never used it myself. I believe each user can use it once for free. – Reta

Answer 1

I think I have solved the conceptual problem, though I am getting

We're sorry, a server error occurred. Please wait a bit and try again. (line 9, file "Code")

when I try to execute it.

Basically, I set it up to only try to copy one top-level folder at a time, and for each one of those it uses the recursive functions I had before. It should save continuation tokens for that first level of folders and any files in the root folder so it can pick up in the next execution where it left off. This way, the tokens are not involved in my recursive stack of functions.

function copyDrive() {


  var originFolder = DriveApp.getFolderById(originFolderID);
  var destination = DriveApp.getFolderById(destinationID);

  var scriptProperties = PropertiesService.getScriptProperties();
  var fileContinuationToken = scriptProperties.getProperty('FILE_CONTINUATION_TOKEN');
  var fileIterator = fileContinuationToken == null ?
    originFolder.getFiles() : DriveApp.continueFileIterator(fileContinuationToken);
  var folderContinuationToken = scriptProperties.getProperty('FOLDER_CONTINUATION_TOKEN');
  var folderIterator = folderContinuationToken == null ?
    originFolder.getFolders() : DriveApp.continueFolderIterator(folderContinuationToken);

  try {
    var rootFileName;
    while(fileIterator.hasNext()) {
      var rootFile = fileIterator.next();
      rootFileName = rootFile.getName();
      rootFile.makeCopy(rootFileName, destination);
      }

    var folder = folderIterator.next();
    var folderName = folder.getName();
    var folderCopy = folder.makeCopy(folderName, destination);


    copyFiles(folder, folderCopy);

  } catch(err) {
    Logger.log(err);
  }

  if(fileIterator.hasNext()) {
    scriptProperties.setProperty('FILE_CONTINUATION_TOKEN', fileIterator.getContinuationToken());
  } else {
    scriptProperties.deleteProperty('FILE_CONTINUATION_TOKEN');
  }
  if(folderIterator.hasNext()) {
    scriptProperties.setProperty('FOLDER_CONTINUATION_TOKEN', folderIterator.getContinuationToken());
  } else {
    scriptProperties.deleteProperty('FOLDER_CONTINUATION_TOKEN');
  }

};

function copyFiles(passedFolder, targetFolder) {
  var fileContents = passedFolder.getFiles();
  var file;
  var fileName;

  while(fileContents.hasNext()) {
    file = fileContents.next();
    fileName = file.getName();
    file.makeCopy(fileName, targetFolder);
  }
  copySubFolders(passedFolder, targetFolder);
};

function copySubFolders(passedFolder, targetFolder) {
  var subFolderContents = passedFolder.getFolders();
  var subFolder;
  var subFolderName;

  while(folderContents.hasNext()) {
    subFolder = subFolderContents.next();
    subFolderName = subFolder.getName();
    var subFolderCopy = targetFolder.createFolder(folderName);
    copyFiles(subFolder, subFolderCopy);
  }
};

Grout

I resolved the above error by manually deleting the script properties. Now, though, the continuation token saved for the folder iterator at the end is the same as the one that is loaded from the script properties at the beginning. I have no idea why that's happening, and I guess it still fits the original question. – Grout

Good job! Looks like it ask to wait a bit because of API quotas. Script has reached limit for 1000 queries per 100 seconds per user or has reached overall day quota. You can sleep your script or update short limits per user. – Theocritus

It looks like this solution is only storing a single folder iterator. To recursively iterate over all folders, you're going to need an array at a minimum. See this answer for more details. – Flippant

Answer 2

I know you would like a easy, programmatic way to do this, but it may be easiest to install Google Drive for Desktop and have them right-click, copy, paste.

The idea:

1. Create a single folder in which the user puts every item of their Drive. (I see you have already done that.)
2. Share that folder with their new account. (I see you have already done that, as well.)
3. Sign into their new account with Drive for Desktop.
4. Copy the folder in Drive for Desktop and paste it right back in. Ownership gets transferred to the new account.

Just a thought.

Wisp

I agree that using Drive for Desktop should be a simple solution, but several people have reported severe crashes after installing it. I have no idea why that is happening, and I'm not really in a position to get to the bottom of it. I'm not in IT or anything; I'm just a random employee who thought he could help. – Grout

This wont work unless you don't mind that none of your google files (soreadsheets, docs etc) wont get copied. – Beneficial

Bill, I understand your situation. Your initial question is very valid, on how to avoid the time-out. I'll keep digging into a solution for that. Zig, this works perfectly for Google files. I've been doing it for awhile now. Why do you think it would not? – Wisp

Thanks GM! Zig, this does indeed work for Google files, as I mentioned in the original question. – Grout

Cool. Om surprised this works for google files like spreadsheets. – Beneficial

I confirm this does in fact work. The only bummer is that the file URLs change so any links will need to be updated, but I guess that's unavoidable. – Overcloud

FWIW, it seems like, at least on macOS, that this no longer works in Drive for Desktop. It likely has to do with Google shifting to shortcuts instead of their previous symlink-like folder structures. All you have in the destination drive is a shortcut to the shared folder owned by the source drive, and copying and pasting that (or drag-and-drop copying) just results in either another shortcut or a file that seems to be a corrupted shortcut. Selecting the contents of the shared folder on the destination drive and copy-pasting only copies subfolders, but none of the actual Google Docs themselves – Aircrew

@SethBattis thank you, I thought I was missing something obvious. – Orianna

Answer 3

You're going to need to store an array of folder iterators and file iterators since each folder could have a nested array of folders. If you're reusing the same folder iterator as in the accepted solution, you won't be able to resume on more top level folders.

Take a look at my answer here for a template that you can use to recursively iterate over all the files in a drive with resume functionality built-in.

Flippant