Copy a folder in go

Question

Is there an easy way to copy a directory in go? I have the following function:

err = CopyDir("sourceFolder","destinationFolder")

Nothing so far has worked, including libraries such as github.com/cf-guardian/guardian/kernel/fileutils

One important thing to note is that I need to preserve directory structure, including the sourceFolder itself, not simply copy all contents of the folder.

Presidentship

Can you show us what your CopyDir function does? Then we might be able to help you. – Slobber

On which system should this work? You could use a system program for example cp, which copies your folder. You can call that program via the os/exec package. – Bolin

Answer 1

I believe that docker implementation can be considered as complete solution for handling edge cases: https://github.com/moby/moby/blob/master/daemon/graphdriver/copy/copy.go

There are following good things:

• unsupported file type rise error
• preserving permissions and ownership
• preserving extended attributes
• preserving timestamp

but because of a lot of imports your tiny application becomes huge.

I've tried to combine several solutions but use stdlib and for Linux only:

func CopyDirectory(scrDir, dest string) error {
    entries, err := os.ReadDir(scrDir)
    if err != nil {
        return err
    }
    for _, entry := range entries {
        sourcePath := filepath.Join(scrDir, entry.Name())
        destPath := filepath.Join(dest, entry.Name())

        fileInfo, err := os.Stat(sourcePath)
        if err != nil {
            return err
        }

        stat, ok := fileInfo.Sys().(*syscall.Stat_t)
        if !ok {
            return fmt.Errorf("failed to get raw syscall.Stat_t data for '%s'", sourcePath)
        }

        switch fileInfo.Mode() & os.ModeType{
        case os.ModeDir:
            if err := CreateIfNotExists(destPath, 0755); err != nil {
                return err
            }
            if err := CopyDirectory(sourcePath, destPath); err != nil {
                return err
            }
        case os.ModeSymlink:
            if err := CopySymLink(sourcePath, destPath); err != nil {
                return err
            }
        default:
            if err := Copy(sourcePath, destPath); err != nil {
                return err
            }
        }

        if err := os.Lchown(destPath, int(stat.Uid), int(stat.Gid)); err != nil {
            return err
        }

        fInfo, err := entry.Info()
        if err != nil {
            return err
        }

        isSymlink := fInfo.Mode()&os.ModeSymlink != 0
        if !isSymlink {
            if err := os.Chmod(destPath, fInfo.Mode()); err != nil {
                return err
            }
        }
    }
    return nil
}

func Copy(srcFile, dstFile string) error {
    out, err := os.Create(dstFile)
    if err != nil {
        return err
    }

    defer out.Close()

    in, err := os.Open(srcFile)
    if err != nil {
        return err
    }

    defer in.Close()

    _, err = io.Copy(out, in)
    if err != nil {
        return err
    }

    return nil
}

func Exists(filePath string) bool {
    if _, err := os.Stat(filePath); os.IsNotExist(err) {
        return false
    }

    return true
}

func CreateIfNotExists(dir string, perm os.FileMode) error {
    if Exists(dir) {
        return nil
    }

    if err := os.MkdirAll(dir, perm); err != nil {
        return fmt.Errorf("failed to create directory: '%s', error: '%s'", dir, err.Error())
    }

    return nil
}

func CopySymLink(source, dest string) error {
    link, err := os.Readlink(source)
    if err != nil {
        return err
    }
    return os.Symlink(link, dest)
}

Channel

You can use the main idea of Oleg's solution with this package: github.com/plus3it/gorecurcopy. To save compatibility with Windows, it doesn't do the os.Lchown() or syscall.Stat_t parts. – Houck

You should do defer out.Close() after if err != nil. And the code does not compile on Windows at all. – Headlight

@Headlight Thank you for your feedback and catch with defer. there is comment above your about why it's not windows compatible. – Channel

The upto date link for the docker copy github github.com/moby/moby/blob/master/daemon/graphdriver/copy/… – Gown

I found a bug: fileInfo, err := os.Stat(sourcePath) should use os.Lstat() instead. As it is, it follows symlinks, meaning if you have a symlink pointing to a file that doesn't exist, it'll fail. – Cupid

The security note: copying symlinks to non existing files can lead to security issues. – Channel

Answer 2

This package seems to do exactly what you want to do, give it a try.

From the readme:

err := Copy("your/source/directory", "your/destination/directory")

Slobber

@Bolin there are libraries consisting of multiple packages ;) – Nettienetting

Please dont post just link, post a little exampe and description. – Oneiric

I have actually tried using this one specifically, and it's problematic because it does not preserve directory tree structure, it just copies the contents of the folder and dumps all of them into the destination. – Presidentship

Then your best bet is to try the method that the other answer proposes, to execute the cp command using exec.Command – Slobber

Hi @PASH, I am the author of that package. I don't get your point and I'm interested in what you pointed out. Could you please make an issue or just explain the problem you are talking here. Thanks. – Isador

@Isador I was referring to nested folders -- folder structure was not being preserved when I tried your library at the time. It seemed like the library would recursively get all of the files and then dump them into one shared folder, as opposed to keeping the nested directory structure the same as the original – Presidentship

@PASH I've got your point but still am interested in how you concluded like that. When did you try and do you still find that problem? I am interested. – Isador

@otiai10, @PASH I was able to successfully use the github.com/otiai10/copy module in version v1.2.0 – Peg

Answer 3

Not satisfied with the already listed options which include using sketchy libraries, or vastly bloated libraries.

In my case, I opted to do things the old fashioned way. With shell commands!

import (
    "os/exec"
)

func main() {
    // completely arbitrary paths
    oldDir := "/home/arshbot/"
    newDir := "/tmp/"

    cmd := exec.Command("cp", "--recursive", oldDir, newDir)
    cmd.Run()
}

Whitethorn

I think this method should be avoided for a) it only works on *nix systems, not on windows and b) it may be attackable like: newDir = "somedir ; sudo rm -rf /" – Prefab

This is a good answer for cases which don't take user input and are likely to run in nix environments. Other environs should have accomodating features. – Whitethorn

Answer 4

Considering that golang/go issue 62484 is "likely accept", Go 1.23 (Q3 2024) will include:

err = os.CopyFS(destDir, os.DirFS(srcDir))

From the new API:

// CopyFS copies the file system fsys into the directory dir,
// creating dir if necessary.
//
// Newly created directories and files have their default modes
// according to the current umask, except that the execute bits
// are copied from the file in fsys when creating a local file.
//
// If a file name in fsys does not satisfy filepath.IsLocal,
// an error is returned for that file.
//
// Copying stops at and returns the first error encountered.
func CopyFS(dir string, fsys fs.FS) error

Mustachio

Answer 5

This solution copies a directory recursively, including symbolic links. Trying to be efficient in the actual copy stage using streams. Also it's fairly easy to handle more of irregular files if needed.

// CopyDir copies the content of src to dst. src should be a full path.
func CopyDir(dst, src string) error {

    return filepath.Walk(src, func(path string, info fs.FileInfo, err error) error {
        if err != nil {
            return err
        }

        // copy to this path
        outpath := filepath.Join(dst, strings.TrimPrefix(path, src))

        if info.IsDir() {
            os.MkdirAll(outpath, info.Mode())
            return nil // means recursive
        }

        // handle irregular files
        if !info.Mode().IsRegular() {
            switch info.Mode().Type() & os.ModeType {
            case os.ModeSymlink:
                link, err := os.Readlink(path)
                if err != nil {
                    return err
                }
                return os.Symlink(link, outpath)
            }
            return nil
        }

        // copy contents of regular file efficiently

        // open input
        in, _ := os.Open(path)
        if err != nil {
            return err
        }
        defer in.Close()

        // create output
        fh, err := os.Create(outpath)
        if err != nil {
            return err
        }
        defer fh.Close()

        // make it the same
        fh.Chmod(info.Mode())

        // copy content
        _, err = io.Copy(fh, in)
        return err
    })
}

Spevek

Answer 6

I've come up with a relatively shorter answer which uses path/filepath's Walk method:

import (
    "io/ioutil"
    "path/filepath"
    "os"
    "strings"
)

func copy(source, destination string) error {
    var err error = filepath.Walk(source, func(path string, info os.FileInfo, err error) error {
        var relPath string = strings.Replace(path, source, "", 1)
        if relPath == "" {
            return nil
        }
        if info.IsDir() {
            return os.Mkdir(filepath.Join(destination, relPath), 0755)
        } else {
            var data, err1 = ioutil.ReadFile(filepath.Join(source, relPath))
            if err1 != nil {
                return err1
            }
            return ioutil.WriteFile(filepath.Join(destination, relPath), data, 0777)
        }
    })
    return err
}

Discretion

The if relPath == "" {return nil} does not work properly since it has to create the directory whenever info.IsDir is true – Burglarious

This loads every file in memory fully, instead of streaming the content. – Skiagraph

That's easy to fix, just use out.ReadFrom(in) to do the copying, then this answer is probably one of the best here. Also, instead of using magic numbers, use constants like os.ModePerm. – Adhesive

Answer 7

Also this might be a solution:

available on github.com/floscodes/golang-tools

import (
    "fmt"
    "io/ioutil"
    "os"
)

func CopyDir(src string, dest string) error {

    if dest[:len(src)] == src {
        return fmt.Errorf("Cannot copy a folder into the folder itself!")
    }

    f, err := os.Open(src)
    if err != nil {
        return err
    }

    file, err := f.Stat()
    if err != nil {
        return err
    }
    if !file.IsDir() {
        return fmt.Errorf("Source " + file.Name() + " is not a directory!")
    }

    err = os.Mkdir(dest, 0755)
    if err != nil {
        return err
    }

    files, err := ioutil.ReadDir(src)
    if err != nil {
        return err
    }

    for _, f := range files {

        if f.IsDir() {

            err = CopyDir(src+"/"+f.Name(), dest+"/"+f.Name())
            if err != nil {
                return err
            }

        }

        if !f.IsDir() {

            content, err := ioutil.ReadFile(src + "/" + f.Name())
            if err != nil {
                return err

            }

            err = ioutil.WriteFile(dest+"/"+f.Name(), content, 0755)
            if err != nil {
                return err

            }

        }

    }

    return nil
}

Wallflower