How to embed files into Go binaries

Question

I have some text file that I read from my Go program. I'd like to ship a single executable, without supplying that text file additionally. How do I embed it into compilation on Windows and Linux?

Barre

Answer 1

Use go-bindata. From the README:

This tool converts any file into managable Go source code. Useful for embedding binary data into a go program. The file data is optionally gzip compressed before being converted to a raw byte slice.

Zinovievsk

This is now sort of obsolete. The generate code does not pass golint and the owner of the repo is not interacting with pull requests. I would be wary of this library – Convertible

A new project called statik seems to do this and has been active more recently. – Acupuncture

This tool seems to be migrated to the new dedicated project - github.com/go-bindata/go-bindata, but there are other (and may be better) alternatives mentioned in my answer. – Progress

Answer 2

===== Edit Jan 2021 =====

Starting with Go 1.16, released in Feb 2021, you can use the go:embed directive:

import "embed"

//go:embed hello.txt
var s string
print(s)

//go:embed hello.txt
var b []byte
print(string(b))

//go:embed hello.txt
var f embed.FS
data, _ := f.ReadFile("hello.txt")
print(string(data))

===== Original answer ======

Since Go 1.4, you can use go generate if you need more flexibility.

If you have more than one text file or the text file may change you might not want to hardcode the text file but include it at compile time.

If you have the following files:

main.go
scripts/includetxt.go
a.txt
b.txt

And want to have access to the contents of all .txt files in main.go, you can include a special comment containing a go generate command.

main.go

package main

import "fmt"

//go:generate go run scripts/includetxt.go

func main() {
    fmt.Println(a)
    fmt.Println(b)
}

The go generate command will run the script after go:generate. In this case it runs a go script which reads all text files and outputs them as string literals into a new file. I skipped the error handling for shorter code.

script/includetxt.go

package main

import (
    "io"
    "io/ioutil"
    "os"
    "strings"
)

// Reads all .txt files in the current folder
// and encodes them as strings literals in textfiles.go
func main() {
    fs, _ := ioutil.ReadDir(".")
    out, _ := os.Create("textfiles.go")
    out.Write([]byte("package main \n\nconst (\n"))
    for _, f := range fs {
        if strings.HasSuffix(f.Name(), ".txt") {
            out.Write([]byte(strings.TrimSuffix(f.Name(), ".txt") + " = `"))
            f, _ := os.Open(f.Name())
            io.Copy(out, f)
            out.Write([]byte("`\n"))
        }
    }
    out.Write([]byte(")\n"))
}

To compile all .txt files into your exectutable:

$ go generate
$ go build -o main

Now your directory structure will look like:

main.go
main
scripts/includetxt.go
textfiles.go
a.txt
b.txt

Where textfiles.go was generated by go generate and script/includetxt.go

textfiles.go

package main 

const (
a = `hello`
b = `world`
)

And running main gives

$ ./main
hello
world

This will work fine as long as you're encoding UTF8 encoded files. If you want to encode other files you have the full power of the go language (or any other tool) to do so. I used this technique to hex encode png:s into a single executable. That requires a minor change to includetxt.go.

Zoa

Nice! I had no idea that function existed. This is so much cleaner than dumping in a string literal in the source. – Willock

Just curious, why do we have to write preprocessor command in comments and then separately run go generate? Couldn't go build take care of running command when it notices //go:generate ... comment? – Froude

As of 2019-07-23, this code does not work for text files that contain backticks. They will either produce syntax errors or allow for code injection, neither of which is desired. – Perea

I'm Go newpie, it gave me undefined a and b by running the main file, the includedtext.go file had been generated successfuly. – Apposite

@Bunyk, I found this explanation in a Go Blog article: The latest Go release, 1.4, includes a new command that makes it easier to run such tools. It's called go generate, and it works by scanning for special comments in Go source code that identify general commands to run. It's important to understand that go generate is not part of go build. It contains no dependency analysis and must be run explicitly before running go build. It is intended to be used by the author of the Go package, not its clients. – Gillie

Offical docs suggests importing embed package with underscore (blank identifier) as import _ "embed" for using go:embed directive to load content into a string or []byte type of variable. Discussion on github – Flong

Answer 3

Updated 2023-10-19

From the author of previously recommended in original answer mjibson/esc:

Stop using this and switch to embed now in the standard library.

Original answer

Was looking for the same thing and came across esc: Embedding Static Assets in Go (by 19 Nov 2014) where author, Matt Jibson, is evaluating 3 other popular packages that claims to do file embedding:

1. rakyll/statik
2. jteeuwen/go-bindata (and the new official go-bindata/go-bindata and another improved one kevinburke/go-bindata)
3. GeertJohan/go.rice

and explain why he eventually come up with his own package:

4. mjibson/esc

So after briefly trying them all (in that order) I've naturally settled on Matt's esc as it was the only one that was working out of the box with necessary for me functionality (HTTPS service in a single executable), namely:

1. Can take some directories and recursively embed all files in them in a way that was compatible with http.FileSystem
2. Can optionally be disabled for use with the local file system for local development without changing the client's code
3. Will not change the output file on subsequent runs has reasonable-sized diffs when files changed
4. Capable of doing the work via //go:generate instead of forcing you to manually write additional Go code

The point #2 was important for me and the rest of the packages for one reason or another didn't work out that well.

From esc's README:

esc embeds files into go programs and provides http.FileSystem interfaces to them.

It adds all named files or files recursively under named directories at the path specified. The output file provides an http.FileSystem interface with zero dependencies on packages outside the standard library.

Progress

Answer 4

check packr, its quite friendly to use

package main

import (
  "net/http"

  "github.com/gobuffalo/packr"
)

func main() {
  box := packr.NewBox("./templates")

  http.Handle("/", http.FileServer(box))
  http.ListenAndServe(":3000", nil)
}

Lounging

If you're looking at the most stars as a measure for software then this is the one. – Boulevard

Which is going to be superseded by pkger – Progress

Answer 5

With go1.16, you can start using embed, this is standard package which helps you embed static non-go files into your binary

Documentation: https://pkg.go.dev/embed
Example: https://blog.carlmjohnson.net/post/2021/how-to-use-go-embed/

for go < 1.16, you can use packr. It's an awesome tool, you can check out more about this at https://github.com/gobuffalo/packr

Euchologion

Answer 6

You can use a string literal to define the text as a constant or variable. String literals are defined by enclosing the string with back-quotes. e.g. `string`.

For example:

package main

import "fmt"

func main() {
    const text = `
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec a diam lectus. Sed sit  
amet ipsum mauris. Maecenas congue ligula ac quam viverra nec consectetur ante 
hendrerit. Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet 
vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut 
libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, 
consectetur adipiscing elit. Aenean ut gravida lorem. Ut turpis felis, pulvinar a 
semper sed, adipiscing id dolor. Pellentesque auctor nisi id magna consequat sagittis. 
Curabitur dapibus enim sit amet elit pharetra tincidunt feugiat nisl imperdiet. Ut 
convallis libero in urna ultrices accumsan. Donec sed odio eros. Donec viverra mi quis 
quam pulvinar at malesuada arcu rhoncus. Cum sociis natoque penatibus et magnis dis 
parturient montes, nascetur ridiculus mus. In rutrum accumsan ultricies. Mauris vitae 
nisi at sem facilisis semper ac in est.
`

    fmt.Println(text)
}

Laszlo

Although simpler it doesn't work if your 'file' contains characters like %. – Rhetorician

Answer 7

I used a simple function to read an external template in a go generate run and to generate Go code from it. A function returning the template as a string will be generated. One can then parse the returned template string using tpl, err := template.New("myname").Parse(mynameTemplate())

I did put that code to github. You might want to try https://github.com/wlbr/templify

Very simple, but works for me quite well.

Zayas

Answer 8

Based on @CoreyOgburn comment and this Github comment, the following snippet was created:

//go:generate statik -src=./html

package main

import (
    _ "./statik"
    "github.com/rakyll/statik/fs"
)

func statikFile() {
    s, _ := fs.New()
    f, _ := s.Open("/tmpl/login.html")
    b, _ := ioutil.ReadAll(f)
    t, _ := template.New("login").Parse(string(b))
    t.Execute(w, nil)
}

and run

go generate

and subsequently

go build

should create a binary that contains the files

Fess

Nowadays I would recommend packr, see this answer. – Fess