Type converting slices of interfaces

Question

I'm curious why Go does't implicitly convert []T to []interface{} when it will implicitly convert T to interface{}. Is there something non-trivial about this conversion that I'm missing?

Example:

func foo([]interface{}) { /* do something */ }

func main() {
    var a []string = []string{"hello", "world"}
    foo(a)
}

go build complains

cannot use a (type []string) as type []interface {} in function argument

And if I try to do it explicitly, same thing: b := []interface{}(a) complains

cannot convert a (type []string) to type []interface {}

So every time I need to do this conversion (which seems to come up a lot), I've been doing something like this:

b = make([]interface{}, len(a), len(a))
for i := range a {
    b[i] = a[i]
}

Is there a better way to do this, or standard library functions to help with these conversions? It seems kind of silly to write 4 extra lines of code every time I want to call a function that can take a list of e.g. ints or strings.

Tabling

so you define "implicitly convert []T to []interface{}" to mean "allocating a new slice and copying all the elements over". That is inconsistent with what "implicitly convert T to interface{}" does, which is just a view of the same value as a more generic static type; nothing is copied; and if you type-assert it back to type T, you would still get the same thing back. – Handed

wasn't thinking in too much detail about how it would be implemented, just that at a higher level it would be convenient to be able to easily convert a whole list to another type as a workaround for not having generic types. – Tabling

Many years later there's hope: Go Generics are on their way, and there' already a playground where you can see and try them in action go2goplay.golang.org (Background: go.dev/blog/generics-next-step) – Hemelytron

Answer 1

In Go, there is a general rule that syntax should not hide complex/costly operations.

Converting a string to an interface{} is done in O(1) time. Converting a []string to an interface{} is also done in O(1) time since a slice is still one value. However, converting a []string to an []interface{} is O(n) time because each element of the slice must be converted to an interface{}.

The one exception to this rule is converting strings. When converting a string to and from a []byte or a []rune, Go does O(n) work even though conversions are "syntax".

There is no standard library function that will do this conversion for you. Your best option though is just to use the lines of code you gave in your question:

b := make([]interface{}, len(a))
for i := range a {
    b[i] = a[i]
}

Otherwise, you could make one with reflect, but it would be slower than the three line option. Example with reflection:

func InterfaceSlice(slice interface{}) []interface{} {
    s := reflect.ValueOf(slice)
    if s.Kind() != reflect.Slice {
        panic("InterfaceSlice() given a non-slice type")
    }

    // Keep the distinction between nil and empty slice input
    if s.IsNil() {
        return nil
    }

    ret := make([]interface{}, s.Len())

    for i:=0; i<s.Len(); i++ {
        ret[i] = s.Index(i).Interface()
    }

    return ret
}

Dol

it might be slower, but it would work generically with any type of slice – Handed

This whole answer applies to maps too btw. – Skin

This also applies to channels as well. – Trollope

Thanks for clear explanation, if possible can you add ref. for Converting a []string to an interface{} is also done in O(1) time? – Luster

Added a benchmark for this answer: stackoverflow.com/a/69309292. You're right - its 33% faster to use a loop. – Shellfish

"syntax should not hide complex/costly operations." - And this is why I like Python, not Go... – Cathleencathlene

Converting each element to interface{} is not just O(N) time, but more significantly, requires N allocations. – Wortman

Answer 2

The thing you are missing is that T and interface{} which holds a value of T have different representations in memory so can't be trivially converted.

A variable of type T is just its value in memory. There is no associated type information (in Go every variable has a single type known at compile time not at run time). It is represented in memory like this:

• value

An interface{} holding a variable of type T is represented in memory like this

• pointer to type T
• value

So coming back to your original question: why go does't implicitly convert []T to []interface{}?

Converting []T to []interface{} would involve creating a new slice of interface {} values which is a non-trivial operation since the in-memory layout is completely different.

Fenestration

This is informative and well written (+1), but you're not addressing the other part of his question: "Is there a better way to do this..." (-1). – Despondency

According to the Golang wiki article "InterfaceSlice", the data layout in the interface is slightly different: Each interface{} takes up two words (one word for the type of what is contained, the other word for either the contained data or a pointer to it). – Monastic

Answer 3

Here is the official explanation: https://github.com/golang/go/wiki/InterfaceSlice

var dataSlice []int = foo()
var interfaceSlice []interface{} = make([]interface{}, len(dataSlice))
for i, d := range dataSlice {
    interfaceSlice[i] = d
}

Fornix

Good info but it's not an official explanation. It's a wiki where everyone can edit. – Riddance

How should I convert []interface to the type I expect such as []map[string]interface{} ? – Gi

Answer 4

In Go 1.18 or later, use the following function to convert an arbitrary slice type to []interface{} or its alias any:

func ToSliceOfAny[T any](s []T) []any {
    result := make([]any, len(s))
    for i, v := range s {
        result[i] = v
    }
    return result
}

The Go 1.18 generics feature does not eliminate the need to convert an arbitrary slice to []any. Here's an example of where the conversion is required: The application wants to query a database using the elements of a []string as the variadic query arguments declared as args ...any. The function in this answer allows the application to query the database in a convenient one-liner:

rows, err := db.Query(qs, ToSliceOfAny(stringArgs)...)

Persiflage

Answer 5

Try interface{} instead. To cast back as slice, try

func foo(bar interface{}) {
    s := bar.([]string)
    // ...
}

Frisky

This begs for the next question: how the OP is then supposed to iterate over bar so as to interpret it as "a slice of any type"? Note that his three-liner creates a []interface{}, not []string or a slice of other concrete type. – Craggie

-1: This only works if bar is []string, in which case, you may as well write: func foo(bar []string) { /* ... */ } – Despondency

use a type switch, or use reflection as in the accepted answer. – Frisky

OP is going to have to figure out his type at run time any way. Using non slice interface{} will make the compiler not complain when passing an argument as in foo(a). He could either use reflect or type switching (golang.org/doc/effective_go.html#type_switch) inside foo. – Dank

Answer 6

In case you need more shorting your code, you can creating new type for helper

type Strings []string

func (ss Strings) ToInterfaceSlice() []interface{} {
    iface := make([]interface{}, len(ss))
    for i := range ss {
        iface[i] = ss[i]
    }
    return iface
}

then

a := []strings{"a", "b", "c", "d"}
sliceIFace := Strings(a).ToInterfaceSlice()

Becoming

Answer 7

The reason []T (a slice of a specific type) cannot be directly converted to []interface{} is because the memory layout of the underlying array is not compatible with storing elements of different types.

When you have a slice []T, the memory layout of the underlying array is designed to store elements of type T. Each element in the array occupies a specific amount of memory based on its type T.

If you were allowed to convert []T to []interface{}, it would mean that the memory layout of the underlying array should be able to accommodate elements of any type, including interface{}. However, the original underlying array was designed to store elements of type T, not interface{} or any other type.

The issue is that the memory layout of the underlying array cannot simultaneously satisfy the requirements of different types. For example, if the original slice was []string, the memory layout of the underlying array is optimized for storing strings.

If you were able to convert []string to []interface{}, and then try to store elements of a different type (e.g., int) in the same underlying array, it would lead to an invalid memory layout. The memory allocated for strings would not be appropriate for storing int values, and accessing those values as interface{} could result in unexpected behavior or crashes.

To preserve type safety and maintain the integrity of the memory layout, Go requires you to manually create a new slice of []interface{} and copy the elements from the original slice, performing the necessary conversions. This ensures that the memory layout of the new slice is appropriate for interface{} values and avoids conflicting memory layouts or type mismatches.

So, in summary, the conversion from []T to []interface{} is not allowed because the memory layout of the underlying array is specific to the element type T, and it cannot be reconfigured to accommodate elements of different types without causing memory layout conflicts or type mismatches.

Millais

Answer 8

I was curious how much slower it is convert interface arrays via reflection vs. doing it inside a loop, as described in Stephen's answer. Here's a benchmark comparison of the two approaches:

benchmark                             iter      time/iter   bytes alloc         allocs
---------                             ----      ---------   -----------         ------
BenchmarkLoopConversion-12         2285820   522.30 ns/op      400 B/op   11 allocs/op
BenchmarkReflectionConversion-12   1780002   669.00 ns/op      584 B/op   13 allocs/op

So using a loop is ~20% faster than doing it via reflection.

Here's my test code in case you'd like to verify if I did things correctly:

    import (
        "math/rand"
        "reflect"
        "testing"
        "time"
    )
    
    func InterfaceSlice(slice interface{}) []interface{} {
        s := reflect.ValueOf(slice)
        if s.Kind() != reflect.Slice {
            panic("InterfaceSlice() given a non-slice type")
        }
    
        // Keep the distinction between nil and empty slice input
        if s.IsNil() {
            return nil
        }
    
        ret := make([]interface{}, s.Len())
    
        for i := 0; i < s.Len(); i++ {
            ret[i] = s.Index(i).Interface()
        }
    
        return ret
    }
    
    type TestStruct struct {
        name string
        age  int
    }
    
    var letters = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")
    
    func randSeq(n int) string {
        b := make([]rune, n)
        for i := range b {
            b[i] = letters[rand.Intn(len(letters))]
        }
        return string(b)
    }
    
    func randTestStruct(lenArray int, lenMap int) map[int][]TestStruct {
        randomStructMap := make(map[int][]TestStruct, lenMap)
        for i := 0; i < lenMap; i++ {
            var testStructs = make([]TestStruct, 0)
            for k := 0; k < lenArray; k++ {
                rand.Seed(time.Now().UnixNano())
                randomString := randSeq(10)
                randomInt := rand.Intn(100)
                testStructs = append(testStructs, TestStruct{name: randomString, age: randomInt})
            }
            randomStructMap[i] = testStructs
        }
        return randomStructMap
    }
    
    func BenchmarkLoopConversion(b *testing.B) {
        var testStructMap = randTestStruct(10, 100)
        b.ResetTimer()
    
        for i := 0; i < b.N; i++ {
            obj := make([]interface{}, len(testStructMap[i%100]))
            for k := range testStructMap[i%100] {
                obj[k] = testStructMap[i%100][k]
            }
        }
    }
    
    func BenchmarkReflectionConversion(b *testing.B) {
        var testStructMap = randTestStruct(10, 100)
        b.ResetTimer()
    
        for i := 0; i < b.N; i++ {
            obj := make([]interface{}, len(testStructMap[i%100]))
            obj = InterfaceSlice(testStructMap[i%100])
            _ = obj
        }
    }

Shellfish

The language provides all the facilities to write reproducible benchmarks, profile the code, to effectively understand what is going on under the hood. See also prettybench or benchcmp – Hadwin

@mh-cbon ok, benchmark updated with -bench. – Shellfish

Answer 9

Though you can use a generic function to convert a slice to a slice of interface{}, it may be most appropriate and cheapest in terms of execution time to change foo to a generic function if possible.

For example:

func foo[T any](slice []T) { /* do something */ }

func main() {
    var a []string = []string{"hello", "world"}
    foo(a)
}

Now there is no conversion necessary at all.

Wortman