How can I open files relative to my GOPATH?

Question

I'm using io/ioutil to read a small text file:

fileBytes, err := ioutil.ReadFile("/absolute/path/to/file.txt")

And that works fine, but this isn't exactly portable. In my case, the files I want to open are in my GOPATH, for example:

/Users/matt/Dev/go/src/github.com/mholt/mypackage/data/file.txt

Since the data folder rides right alongside the source code, I'd love to just specify the relative path:

data/file.txt

But then I get this error:

panic: open data/file.txt: no such file or directory

How can I open files using their relative path, especially if they live alongside my Go code?

(Note that my question is specifically about opening files relative to the GOPATH. Opening files using any relative path in Go is as easy as giving the relative path instead of an absolute path; files are opened relative to the compiled binary's working directory. In my case, I want to open files relative to where the binary was compiled. In hindsight, this is a bad design decision.)

Thorium

The GOPATH hasn't a big meaning once your program is compiled, and even less when you distribute it. – Fisher

What you seem to want looks more like some embedding of the files in your compiled program. – Fisher

Kind of... except I want the data files separate from the source. The data files are vital to the program's functionality. So when somebody pulls down my source code (with the data files along side it) and compiles and runs it, the data files are loaded using a relative path because they exist near the source code, or near where the program is executing. – Thorium

The the compiled binary should have no dependence on the location of the source files, but it would be nice if there was a way to create an executable bundle that contained a copy of external resources upon which packages may depend. – Githens

@CrazyTrain Sounds good to me... I'm somewhat new to Go, and my typical approach to this (like Python or PHP) is to bundle the data files with the source code and reference them relatively. So for Go, my real question might be: how do I access these external resources? (If they are missing, of course, I'd handle accordingly.) – Thorium

@CrazyTrain is that what fileembed-go does? (bitbucket.org/rj/fileembed-go) – Bergstrom

Here's a related question about bundling resources which may be sufficient, though this isn't my preferred method in my case: #13904941 -- or this one: https://mcmap.net/q/245811/-how-to-access-resource-files-after-the-39-go-39-tool-installed-the-executable/1048862 – Thorium

Check using pwd , _ := os.Getwd() your working dir then traverse to the path like opening a file "content, err := os.OpenFile(("..\\..\\pkg\\service\\profile\\TTL.json"), os.O_RDONLY, 0755)" – Richly

Answer 1

Hmm... the path/filepath package has Abs() which does what I need (so far) though it's a bit inconvenient:

absPath, _ := filepath.Abs("../mypackage/data/file.txt")

Then I use absPath to load the file and it works fine.

Note that, in my case, the data files are in a package separate from the main package from which I'm running the program. If it was all in the same package, I'd remove the leading ../mypackage/. Since this path is obviously relative, different programs will have different structures and need this adjusted accordingly.

If there's a better way to use external resources with a Go program and keep it portable, feel free to contribute another answer.

Thorium

Ha, nearly a year later into Go, and I have never had a problem opening files by relative path since, nor have I had to use this little trick. I guess that means I've learned something. – Thorium

What have you learned? :) Should we not share files like this across packages? – Fling

@Matt, share the love. Don't want to use this "little trick" if you've got something better for us. You're the top result on google. – Metro

This question was from my first weeks programming in Go. The better question would have been how to bundle required data with a program. My answer here is for my very specific situation, I hope people aren't going off and using this haphazardly. – Thorium

@Thorium I am facing this issue because depending on how I start the program/tests in GoLand the working directory changes and hence the path to my file. If that is consistent then there probably isn't a reason to find the absolute path on every execution – Louanne

I read this and thought I had to expand relative paths for ioutil.ReadFile to work correctly, but it is completely unecessary. This works fine: ioutil.ReadFile("../mypackage/data/file.txt"). There's probably many other cases where you want to construct an absolute path, but it appears use with io.ReadFile is not one of them. – Aciniform

@Aciniform I think relative paths are resolved relative to the cwd. What will happen if you run the program from a different directory? Like 'go run ./some/directory/main.go' instead of 'go run main.go' – Gambier

@ArunaHerath you can verify that by trying it out – Aciniform

@Aciniform Verified. If you run your program from a different directory it wont work. Because files are resolved relative to the current working directory. – Gambier

Thanks @Matt. I am hoping this was from the early days of Caddy. Let us know if this is still the solution. May I also add that in Matt's case "../mypackage" is where the *.go is opening the file. It could also be a folder inside a package. Then the folder name becomes the new mypackage. – Isbella

@Isbella This whole question and answer should not be considered as relevant anymore. The GOPATH has long-been deprecated, and resources that ship with the binary should be embedded into it with go:embed. (It wasn't for Caddy though, which I didn't start developing until 4 months later.) – Thorium

Answer 2

this seems to work pretty well:

import "os"
import "io/ioutil"

pwd, _ := os.Getwd()
txt, _ := ioutil.ReadFile(pwd+"/path/to/file.txt")

Eremite

This is bad form because / is os specific, better to use filepath.join() when concatenating paths because it will use os.PathSeperator – Wingless

Let me save the time for those willing to follow the (good) advice from the previous comment: use import "path/filepath" and call filepath.Join(...) (with capital J). :-) – Plasmasol

os.Open(pwd + "../mocks/product/default.json") how do you handle this case? it is taking this as literal. – Helmick

The os.Open is fine for me: csvfile, err := os.Open("/mnt/c/Users/dmotta/Downloads/archivos_csv/tb_desp_new_201904081633.csv") – Inculpate

Answer 3

I wrote gobundle to solve exactly this problem. It generates Go source code from data files, which you then compile into your binary. You can then access the file data through a VFS-like layer. It's completely portable, supports adding entire file trees, compression, etc.

The downside is that you need an intermediate step to build the Go files from the source data. I usually use make for this.

Here's how you'd iterate over all files in a bundle, reading the bytes:

for _, name := range bundle.Files() {
    r, _ := bundle.Open(name)
    b, _ := ioutil.ReadAll(r)
    fmt.Printf("file %s has length %d\n", name, len(b))
}

You can see a real example of its use in my GeoIP package. The Makefile generates the code, and geoip.go uses the VFS.

Toilworn

@Anthracosilicosis has a good point; my data files are hundreds of MB in size. This is cool, but I don't think that compiling the plain text files into the binary is a feasible option. – Thorium

Answer 4

Starting from Go 1.16, you can use the embed package. This allows you to embed the files in the running go program.

Given the file structure:

-- main.go
-- data
  \- file.txt

You can reference the file using a go directive

package main

import (
  "embed"
  "fmt"
)

//go:embed data/file.txt
var content embed.FS

func main() {
  text, _ := content.ReadFile("data/file.txt")
  fmt.Println(string(text))
}

This program will run successfully regardless of where the program is executed. This is useful in case the file could be called from multiple different locations, for instance, from a test directory.

Unless

IMHO, this answer should be nominated as correct answer. Works perfect for my use case here: github.com/VOU-folks/HoGo/blob/main/apps/hogo-manager/… – Deutero

Answer 5

I think Alec Thomas has provided The Answer, but in my experience it isn't foolproof. One problem I had with compiling resources into the binary is that compiling may require a lot of memory depending on the size of your assets. If they're small, then it's probably nothing to worry about. In my particular scenario, a 1MB font file was causing compilation to require somewhere around 1GB of memory to compile. It was a problem because I wanted it to be go gettable on a Raspberry Pi. This was with Go 1.0; things may have improved in Go 1.1.

So in that particular case, I opt to just use the go/build package to find the source directory of the program based on the import path. Of course, this requires that your targets have a GOPATH set up and that the source is available. So it isn't an ideal solution in all cases.

Anthracosilicosis