How do I force Robocopy to overwrite files?
Asked Answered
M

4

50

In general, Robocopy ignores files for which lastwrittendate and filesize are the same. How can we escape this design? I'd like to force overwriting with Robocopy.

I expected that dst\sample.txt should be written test001. But these file are recognized as the same files by Robocopy and not overwritten. The "/IS" option is not effective in this case.

New-Item src -itemType Directory
New-Item dst -itemType Directory
New-Item src\sample.txt -itemType File -Value "test001"
New-Item dst\sample.txt -itemType File -Value "test002"
Set-ItemProperty src\sample.txt -Name LastWriteTime -Value "2016/1/1 15:00:00"
Set-ItemProperty dst\sample.txt -Name LastWriteTime -Value "2016/1/1 15:00:00"

ROBOCOPY.exe src dst /COPYALL /MIR
Get-Content src\sample.txt, dst\sample.txt
> test001
> test002

ROBOCOPY.exe src dst /COPYALL /MIR /IS
Get-Content src\sample.txt, dst\sample.txt
> test001
> test002
Mellen answered 22/11, 2016 at 14:22 Comment(0)
G
45

From the documentation:

/is Includes the same files. /it Includes "tweaked" files.

"Same files" means files that are identical (name, size, times, attributes). "Tweaked files" means files that have the same name, size, and times, but different attributes.

robocopy src dst sample.txt /is      # copy if attributes are equal
robocopy src dst sample.txt /it      # copy if attributes differ
robocopy src dst sample.txt /is /it  # copy irrespective of attributes

This answer on Super User has a good explanation of what kind of files the selection parameters match.

With that said, I could reproduce the behavior you describe, but from my understanding of the documentation and the output robocopy generated in my tests I would consider this a bug.

PS C:\temp> New-Item src -Type Directory >$null
PS C:\temp> New-Item dst -Type Directory >$null
PS C:\temp> New-Item src\sample.txt -Type File -Value "test001" >$null
PS C:\temp> New-Item dst\sample.txt -Type File -Value "test002" >$null
PS C:\temp> Set-ItemProperty src\sample.txt -Name LastWriteTime -Value "2016/1/1 15:00:00"
PS C:\temp> Set-ItemProperty dst\sample.txt -Name LastWriteTime -Value "2016/1/1 15:00:00"
PS C:\temp> robocopy src dst sample.txt /is /it /copyall /mir
...
  Options : /S /E /COPYALL /PURGE /MIR /IS /IT /R:1000000 /W:30

------------------------------------------------------------------------------

                           1    C:\temp\src\
            Modified                   7        sample.txt

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         1         0         0         0         0         0
   Files :         1         1         0         0         0         0
   Bytes :         7         7         0         0         0         0
...
PS C:\temp> robocopy src dst sample.txt /is /it /copyall /mir
...
  Options : /S /E /COPYALL /PURGE /MIR /IS /IT /R:1000000 /W:30

------------------------------------------------------------------------------

                           1    C:\temp\src\
            Same                       7        sample.txt

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         1         0         0         0         0         0
   Files :         1         1         0         0         0         0
   Bytes :         7         7         0         0         0         0
...
PS C:\temp> Get-Content .\src\sample.txt
test001
PS C:\temp> Get-Content .\dst\sample.txt
test002

The file is listed as copied, and since it becomes a same file after the first robocopy run at least the times are synced. However, even though seven bytes have been copied according to the output no data was actually written to the destination file in both cases despite the data flag being set (via /copyall). The behavior also doesn't change if the data flag is set explicitly (/copy:d).

I had to modify the last write time to get robocopy to actually synchronize the data.

PS C:\temp> Set-ItemProperty src\sample.txt -Name LastWriteTime -Value (Get-Date)
PS C:\temp> robocopy src dst sample.txt /is /it /copyall /mir
...
  Options : /S /E /COPYALL /PURGE /MIR /IS /IT /R:1000000 /W:30

------------------------------------------------------------------------------

                           1    C:\temp\src\
100%        Newer                      7        sample.txt

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         1         0         0         0         0         0
   Files :         1         1         0         0         0         0
   Bytes :         7         7         0         0         0         0
...
PS C:\temp> Get-Content .\dst\sample.txt
test001

An admittedly ugly workaround would be to change the last write time of same/tweaked files to force robocopy to copy the data:

& robocopy src dst /is /it /l /ndl /njh /njs /ns /nc |
  Where-Object { $_.Trim() } |
  ForEach-Object {
    $f = Get-Item $_
    $f.LastWriteTime = $f.LastWriteTime.AddSeconds(1)
  }
& robocopy src dst /copyall /mir

Switching to xcopy is probably your best option:

& xcopy src dst /k/r/e/i/s/c/h/f/o/x/y
Geosyncline answered 22/11, 2016 at 19:39 Comment(5)
OK. I'm going to use xcopy.Mellen
I concur and 3 years later the /is switch doesn't seem to force the file to copy again. It says it's copied (i.e. copy = 1) but it clearly doesn't as I'm testing with a 1GB and it's instantGasteropod
Just tried this on Windows Server 2019 and robocopy source destn file /is does copy the file again. It's v10.0.17763 on there which is the same version that shipped with Windows 10 1809Gasteropod
Windows Server 2019 Robocopy 10.0.17763.1 from local to local, overwrites files as expected. Same command from local to mapped network drive, overwrite does not happen. I suspect DFS (Distributed File System). Same command from local to UNC of the mapped network drive, overwrite does not work. Same command from local to UNC of another development computer, overwrite works as expected. I made sure share caching was on and tried again, overwrite worked. Therefore, I believe DFS is the issue or need to use /ZB.Erupt
Worked for me :)Magnetostriction
L
37

This is really weird, why nobody is mentioning the /IM switch ?! I've been using it for a long time in backup jobs. But I tried googling just now and I couldn't land on a single web page that says anything about it even on MS website !!! Also found so many user posts complaining about the same issue!!

Anyway.. to use Robocopy to overwrite EVERYTHING what ever size or time in source or distination you must include these three switches in your command (/IS /IT /IM)

/IS :: Include Same files. (Includes same size files)
/IT :: Include Tweaked files. (Includes same files with different Attributes)
/IM :: Include Modified files (Includes same files with different times).

This is the exact command I use to transfer few TeraBytes of mostly 1GB+ files (ISOs - Disk Images - 4K Videos):

robocopy B:\Source D:\Destination /E /J /COPYALL /MT:1 /DCOPY:DATE /IS /IT /IM /X /V /NP /LOG:A:\ROBOCOPY.LOG

I did a small test for you .. and here is the result:

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :      1028      1028         0         0         0       169
   Files :      8053      8053         0         0         0         1
   Bytes : 649.666 g 649.666 g         0         0         0   1.707 g
   Times :   2:46:53   0:41:43                       0:00:00   0:41:44


   Speed :           278653398 Bytes/sec.
   Speed :           15944.675 MegaBytes/min.
   Ended : Friday, August 21, 2020 7:34:33 AM

Dest, Disk: WD Gold 6TB (Compare the write speed with my result)

Even with those "Extras", that's for reporting only because of the "/X" switch. As you can see nothing was Skipped and Total number and size of all files are equal to the Copied. Sometimes It will show small number of skipped files when I abuse it and cancel it multiple times during operation but even with that the values in the first 2 columns are always Equal. I also confirmed that once before by running a PowerShell script that scans all files in destination and generate a report of all time-stamps.

Some performance tips from my history with it and so many tests & troubles!:

. Despite of what most users online advise to use maximum threads "/MT:128" like it's a general trick to get the best performance ... PLEASE DON'T USE "/MT:128" WITH VERY LARGE FILES ... that's a big mistake and it will decrease your drive performance dramatically after several runs .. it will create very high fragmentation or even cause the files system to fail in some cases and you end up spending valuable time trying to recover a RAW partition and all that nonsense. And above all that, It will perform 4-6 times slower!!

For very large files:

  1. Use Only "One" thread "/MT:1" | Impact: BIG
  2. Must use "/J" to disable buffering. | Impact: High
  3. Use "/NP" with "/LOG:file" and Don't output to the console by "/TEE" | Impact: Medium.
  4. Put the "/LOG:file" on a separate drive from the source or destination | Impact: Low.

For regular big files:

  1. Use multi threads, I would not exceed "/MT:4" | Impact: BIG
  2. IF destination disk has low Cache specs use "/J" to disable buffering | Impact: High
  3. & 4 same as above.

For thousands of tiny files:

  1. Go nuts :) with Multi threads, at first I would start with 16 and multibly by 2 while monitoring the disk performance. Once it starts dropping I'll fall back to the prevouse value and stik with it | Impact: BIG
  2. Don't use "/J" | Impact: High
  3. Use "/NP" with "/LOG:file" and Don't output to the console by "/TEE" | Impact: HIGH.
  4. Put the "/LOG:file" on a separate drive from the source or destination | Impact: HIGH.
Lacrimatory answered 21/8, 2020 at 7:44 Comment(3)
I tried an otherwise working robocopy call, and it complained that "/IM" isn't recognized. Specifically, it reported "/IM" as "Invalid parameter".Depilatory
This site lists the parameters, including /IM /IS /IT. Or simply try Robocopy /?.Degust
@Depilatory /IM is a later addition to ROBOCOPY but I think can be omitted in this caseBailly
A
1

I did this for a home folder where all the folders are on the desktops of the corresponding users, reachable through a shortcut which did not have the appropriate permissions, so that users couldn't see it even if it was there. So I used Robocopy with the parameter to overwrite the file with the right settings:

FOR /F "tokens=*" %G IN ('dir /b') DO robocopy  "\\server02\Folder with shortcut" "\\server02\home\%G\Desktop" /S /A /V /log+:C:\RobocopyShortcut.txt /XF *.url *.mp3 *.hta *.htm *.mht *.js *.IE5 *.css *.temp *.html *.svg *.ocx *.3gp *.opus *.zzzzz *.avi *.bin *.cab *.mp4 *.mov *.mkv *.flv *.tiff *.tif *.asf *.webm *.exe *.dll *.dl_ *.oc_ *.ex_ *.sy_ *.sys *.msi *.inf *.ini *.bmp *.png *.gif *.jpeg *.jpg *.mpg *.db *.wav *.wma *.wmv *.mpeg *.tmp *.old *.vbs *.log *.bat *.cmd *.zip /SEC /IT /ZB /R:0

As you see there are many file types which I set to ignore (just in case), just set them for your needs or your case scenario.

It was tested on Windows Server 2012, and every switch is documented on Microsoft's sites and others.

Age answered 20/2, 2018 at 21:28 Comment(1)
Hi, welcome to SO. I have edited your answer in an attempt to make it more understandable, but I'm not sure this is what you meant. Anyway, you have described your scenario (which I haven't really understood, sorry), but you haven't explained what your command does, and since it is a complex one, figuring it out isn't trivial. I suggest to edit your answer and provide at least a general description of what this command does and how. Besides, are you sure you are answering the original question, about how to force Robocopy to overwrite files that look identical?Deaf
T
0

Complementing FahadBA's answer here, I was able to solve this using Robocopy's /IM Switch, with one minor change:

robocopy /IM /IS /TIMFIX "Source" "Destination" "Files"

Because I had already copied files to the directory before this, the /IS switch alone did not work, and I wanted to ensure the files kept the same modified time stamp since this matters for my application. Adding the /TIMFIX switch will force the same timestamps across to the all paths, including UNC paths (\\servername\ nomenclature) thus resolving the issue reported by @AMissico for the /IS switch in the comments section of the answer posted here, giving a simplified command for robocopy (and bypassing the cludgy workaround w/ xcopy from that answer) shown above.

Thulium answered 12/10, 2023 at 23:10 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.