How to create an array or a slice from an array unsafe.Pointer in golang?

Question

A pointer to an array, let's say:

p := uintptr(unsafe.Pointer(&array))
size := 5

I can't access to the variable array, the code above is used to make it more clear.

Also, I know the size of the array, but the size is not constant, it changes according to the runtime.

Now, I want to initialize a slice or an array with the known pointer, size and of course the data type.

I come up with the following code:

data := make([]byte, size)
stepSize := unsafe.Sizeof(data[0])
for i := 0; i < size; i++ {
    data[i] = *(*byte)(unsafe.Pointer(p))
    p += stepSize
}
fmt.println(data)

but this method does memory copy, which might be inefficient, is there anyway without doing the copy?

P.S. I also tried the following two methods,

// method 1
data := *(*[]byte)(unsafe.Pointer(p))
// method 2
data := *(*[size]byte)(unsafe.Pointer(p))

but it will fail at runtime and I know its reason now.

Monk

Answer 1

Foreword:

You should know: if you get the pointer as a value of uintptr type, that does not prevent the original array to get garbage collected (an uintptr value does not count as a reference). So be careful when using such value, there is no guarantee it will point to a valid value / memory area.

Quoting from package unsafe.Pointer:

A uintptr is an integer, not a reference. Converting a Pointer to a uintptr creates an integer value with no pointer semantics. Even if a uintptr holds the address of some object, the garbage collector will not update that uintptr's value if the object moves, nor will that uintptr keep the object from being reclaimed.

General advice: stay away from package unsafe as much as possible. Stay inside Go's type safety.

---

Declare a variable of slice type, and use unsafe conversions to obtain its reflect.SliceHeader descriptor.

Then you can modify its fields, using the pointer as the SliceHeader.Data value, and the size as SliceHeader.Len and SliceHeader.Cap.

Once you're done with this, the slice variable will point to the same array as your initial pointer.

arr := [10]byte{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}

size := len(arr)
p := uintptr(unsafe.Pointer(&arr))

var data []byte

sh := (*reflect.SliceHeader)(unsafe.Pointer(&data))
sh.Data = p
sh.Len = size
sh.Cap = size

fmt.Println(data)

runtime.KeepAlive(arr)

Output (try it on the Go Playground):

[0 1 2 3 4 5 6 7 8 9]

_{Note that I used runtime.KeepAlive(). This is because after taking the address of arr and getting its length, we don't refer to arr anymore (p being uintptr does not count as a reference), and an aggressive GC might–rightfully–erase arr before we get to the point to print data (pointing to arr). Placing a runtime.KeepAlive() to the end of main() will ensure that arr will not be garbage collected before this call. For details, see In Go, when will a variable become unreachable? You do not need to call runtime.KeepAlive() in your code if the supplier of the pointer ensures it will not be garbage collected.}

Alternatively you can create a reflect.SliceHeader with a composite literal, and use unsafe conversions to obtain a slice from it, like this:

sh := &reflect.SliceHeader{
    Data: p,
    Len:  size,
    Cap:  size,
}

data := *(*[]byte)(unsafe.Pointer(sh))

fmt.Println(data)

runtime.KeepAlive(arr)

Output will be the same. Try this one on the Go Playground.

This possibility / use case is documented at unsafe.Pointer, with the caveats and warnings:

(6) Conversion of a reflect.SliceHeader or reflect.StringHeader Data field to or from Pointer.

As in the previous case, the reflect data structures SliceHeader and StringHeader declare the field Data as a uintptr to keep callers from changing the result to an arbitrary type without first importing "unsafe". However, this means that SliceHeader and StringHeader are only valid when interpreting the content of an actual slice or string value.

var s string
hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) // case 1
hdr.Data = uintptr(unsafe.Pointer(p))              // case 6 (this case)
hdr.Len = n

In this usage hdr.Data is really an alternate way to refer to the underlying pointer in the slice header, not a uintptr variable itself.

In general, reflect.SliceHeader and reflect.StringHeader should be used only as *reflect.SliceHeader and *reflect.StringHeader pointing at actual slices or strings, never as plain structs. A program should not declare or allocate variables of these struct types.

// INVALID: a directly-declared header will not hold Data as a reference.
var hdr reflect.StringHeader
hdr.Data = uintptr(unsafe.Pointer(p))
hdr.Len = n
s := *(*string)(unsafe.Pointer(&hdr)) // p possibly already lost

Contraption

Works like a charm! Great answer, thanks! I have to interact with dll written in C, so I have to use unsafe. – Monk

@Allenlu Added an alternative way and included runtime.KeepAlive() in my answer. – Contraption

Your quote from unsafe.Pointer's docs mentions that reflect.SliceHeader should be used only as *reflect.SliceHeader pointing to actual slices, never as plain structs. There is also the note that a directly declared header won't hold Data as a reference. Is the use of new(reflect.SliceHeader) or &reflect.SliceHeader{} an exception? I ran the example where you used the composite literal on playground and go vet shows the warning "possible misuse of reflect.SliceHeader". Is this a false positive? I'm just curious about the validity of the alternative method. – Carrero

@Carrero The vet warning is rightful because creating a reflect.SliceHeader will not count as a reference and thus not keeping the object pointed by its Data field from getting garbage collected. However, in my case that wasn't the intention. To still ensure the pointed object (by the Data field) is valid, I used runtime.KeepAlive(). – Contraption

Answer 2

With Go 1.17, we now have unsafe.Slice(ptr *ArbitraryType, len IntegerType).

You can use this function to create a slice whose underlying array starts at ptr and whose length and capacity are len.

Say you have an array:

arr := [10]byte{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}

If you have a unsafe.Pointer named p:

p := unsafe.Pointer(&arr); // Or unsafe.Pointer(&arr[0])
data := unsafe.Slice((*byte)(p), len(arr)); 

But if you have access to the arr directly, you can simplify this to:

data := unsafe.Slice(&arr[0], len(arr)); 

A word of caution: be sure to use &arr[0] and not &arr if you are passing the pointer directly. &arr is a *[10]byte, not *byte. Using &arr would create a [][10]byte of length and capacity 10, rather than a []byte of length and capacity 10.

---

With Go 1.17, we can also conveniently cast from a slice to an array pointer. To put it simply, your method 2 would work if you used a slice instead of unsafe.Pointer(p):

array := *(*[len(arr)]byte)(data)

Carrero