How do you use Go 1.16 embed features in subfolders/packages?

Question

Go 1.16 is out and I want to use the new embed features. I can get it to work if everything is in the main package. But it's not clear how to handle accessing resources from subfolders/packages. Trying to do it with embed.FS support.

e.g. I have a main.go and also an HTTP handler in a handlers package/folder

If I put the handler in the main, it works. If I put it in the handlers package, it can't find the templates. I get:

handlers/indexHandler.go:11:12: pattern templates: no matching files found exit status 1

Similarly, I can get it to serve an image from the static folder if I serve it from /. But I can't serve both a handler from / and the static/images from /. If I put images on /static/ it can't find the images.

I think it has to do with relative paths. But I can't find the right combination through trial and error... Could it be too early to rely on these features?

Previously I was using go-rice and I did not have these problems. But I would like to use the std library as much as possible.

main.go:

package main

import (...)

//go:embed static
var static embed.FS

func main() {

    fsRoot, _ := fs.Sub(static, "static")
    fsStatic := http.FileServer(http.FS(fsRoot))
    http.Handle("/", fsStatic)
    http.HandleFunc("/index", Index)
    http.ListenAndServe(":8080", nil)
}

handlers/indexHandler.go:

package handlers

import (...)

//go:embed templates
var templates embed.FS

// Index handler
func Index(w http.ResponseWriter, r *http.Request) {

    tmpl := template.New("")
    var err error
    if tmpl, err = tmpl.ParseFS(templates, "simple.gohtml"); err != nil {
        fmt.Println(err)
    }
    if err = tmpl.ExecuteTemplate(w, "simple", nil); err != nil {
        log.Print(err)
    }    
}

Structure is as follows...

.
├── go.mod
├── go.sum
├── handlers
│   └── indexHandler.go
├── main.go
├── static
│   ├── css
│   │   └── layout.css
│   └── images
│       └── logo.png
└── templates
    └── simple.gohtml

Burgher

Please add details of your directory structure (where is your simple.gohtml). Ref the second part of your question - you probably need StripPrefix (see the example). – Sirloin

Made some progress on getting inside the static folder and removing the static from the path. But still can't have a handler on the same / for the indexHandler... – Burgher

Sorry - I'm not clear on what issue you are still having (perhaps show updated code). Ref simple./html; unless you are also moving the templates to handlers/templates you will need to use //go:embed ../templates (the embed path is " relative to the package directory containing the source file"). – Sirloin

That's what I think too. Except the ".." Is not allowed and generates an error. So maybe templates needs to be under the package... – Burgher

Apologies - you are correct ("Patterns may not contain ‘.’ or ‘..’ or empty path elements, nor may they begin or end with a slash"). – Sirloin

I have put this aside for now and gone back to using rice. I will revisit it after it matures a bit. Thanks... – Burgher

I am using it in production now and it works fine. However you do need to change your directory structure moving the files underneath the handlers folder. – Sirloin

Moved the templates folder under handlers. Wow is that ever easier than using go-rice. Thanks Go! – Burgher

Answer 1

I finally figured it out...

You can keep the templates folder in the main folder and embed them from there. Then you need to inject the FS variable into the other handler package. It's always easy after you figure it out.

e.g.

package main

//go:embed templates/*
var templateFs embed.FS

func main() {
    handlers.TemplateFs = templateFs
...

package handlers

var TemplateFs embed.FS

func handlerIndex() {
    ...
    tmpl, err = tmpl.ParseFS(TemplateFs, "templates/layout.gohtml",...
...

Burgher

Thanks for this solution. It's quite annoying that it is not possible to use relative paths. – Castillo

@Jørgen that is actually precisely false. In fact, if you read the 2nd paragraph under the example here it literally says the opposite: "The patterns are interpreted relative to the package directory containing the source file". – Xeres

@Xeres yeah I probably didn't use the right phrasing, but what I meant is to use ../ If you have the cmd/main.go you can not do ../templates/* But thanks for your reply. – Castillo

Answer 2

Currently in Go 1.17 , embed is not supporting subfolders/packages, see https://github.com/golang/go/issues/41191#issuecomment-686616556

Asymptote

Answer 3

I've encountered the same problem while creating integration tests relying on migration files. The migration files are outside the tests folder. What I did was make a Golang file inside the migration folder and import this Golang file inside my testing files as below:

myproject/
├── migrations/
│   ├── 001_initial.up.sql
│   └── migrations.go
├── internal/
│   ├── core/
│   │   └── domain/
│   │       └── single_test.go
└── go.mod

migrations.go

package migrations

import (
    "embed"
    "path"
)

//go:embed *.sql
var migrations embed.FS

func GetMigrationFiles() ([]string, error){
    dirEntries, err := migrations.ReadDir(".")
    if err != nil {
        return nil, err
    }

    migrationContents := make([]string, 0, len(dirEntries))

    for _, dirEntry := range dirEntries {
        if !dirEntry.IsDir() {
            migrationContent, err := migrations.ReadFile(path.Join(".", dirEntry.Name()))
            if err != nil {
                return nil, err
            }

            migrationContents = append(migrationContents, string(migrationContent))
        }
    }

    return migrationContents, nil
}

single_test.go

func TestCreate(t *testing.T) {
    migrationContents, err := migrations.GetMigrationFiles()
    
    // loop over each migration content and exec db.Migrate()...
}

Deflation