How to safely reinterpret Vec<f64> as Vec<num_complex::Complex<f64>> with half the size?

Question

I have complex number data filled into a Vec<f64> by an external C library (prefer not to change) in the form [i_0_real, i_0_imag, i_1_real, i_1_imag, ...] and it appears that this Vec<f64> has the same memory layout as a Vec<num_complex::Complex<f64>> of half the length would be, given that num_complex::Complex<f64>'s data structure is memory-layout compatible with [f64; 2] as documented here. I'd like to use it as such without needing a re-allocation of a potentially large buffer.

I'm assuming that it's valid to use from_raw_parts() in std::vec::Vec to fake a new Vec that takes ownership of the old Vec's memory (by forgetting the old Vec) and use size / 2 and capacity / 2, but that requires unsafe code. Is there a "safe" way to do this kind of data re-interpretation?

The Vec is allocated in Rust as a Vec<f64> and is populated by a C function using .as_mut_ptr() that fills in the Vec<f64>.

My current compiling unsafe implementation:

extern crate num_complex;

pub fn convert_to_complex_unsafe(mut buffer: Vec<f64>) -> Vec<num_complex::Complex<f64>> {
    let new_vec = unsafe {
        Vec::from_raw_parts(
            buffer.as_mut_ptr() as *mut num_complex::Complex<f64>,
            buffer.len() / 2,
            buffer.capacity() / 2,
        )
    };
    std::mem::forget(buffer);
    return new_vec;
}

fn main() {
    println!(
        "Converted vector: {:?}",
        convert_to_complex_unsafe(vec![3.0, 4.0, 5.0, 6.0])
    );
}

Karyotype

f64 is not a complex in C – Lysimachus

Answer 1

Is there a "safe" way to do this kind of data re-interpretation?

No. At the very least, this is because the information you need to know is not expressed in the Rust type system but is expressed via prose (a.k.a. the docs):

Complex<T> is memory layout compatible with an array [T; 2].

— Complex docs

If a Vec has allocated memory, then [...] its pointer points to len initialized, contiguous elements in order (what you would see if you coerced it to a slice),

— Vec docs

Arrays coerce to slices ([T])

— Array docs

Since a Complex is memory-compatible with an array, an array's data is memory-compatible with a slice, and a Vec's data is memory-compatible with a slice, this transformation should be safe, even though the compiler cannot tell this.

This information should be attached (via a comment) to your unsafe block.

I would make some small tweaks to your function:

• Having two Vecs at the same time pointing to the same data makes me very nervous. This can be trivially avoided by introducing some variables and forgetting one before creating the other.

• Remove the return keyword to be more idiomatic

• Add some asserts that the starting length of the data is a multiple of two.

• As rodrigo points out, the capacity could easily be an odd number. To attempt to avoid this, we call shrink_to_fit. This has the downside that the Vec may need to reallocate and copy the memory, depending on the implementation.

• Expand the unsafe block to cover all of the related code that is required to ensure that the safety invariants are upheld.

pub fn convert_to_complex(mut buffer: Vec<f64>) -> Vec<num_complex::Complex<f64>> {
    // This is where I'd put the rationale for why this `unsafe` block
    // upholds the guarantees that I must ensure. Too bad I
    // copy-and-pasted from Stack Overflow without reading this comment!
    unsafe {
        buffer.shrink_to_fit();

        let ptr = buffer.as_mut_ptr() as *mut num_complex::Complex<f64>;
        let len = buffer.len();
        let cap = buffer.capacity();

        assert!(len % 2 == 0);
        assert!(cap % 2 == 0);

        std::mem::forget(buffer);

        Vec::from_raw_parts(ptr, len / 2, cap / 2)
    }
}

To avoid all the worrying about the capacity, you could just convert a slice into the Vec. This also doesn't have any extra memory allocation. It's simpler because we can "lose" any odd trailing values because the Vec still maintains them.

pub fn convert_to_complex(buffer: &[f64]) -> &[num_complex::Complex<f64>] {
    // This is where I'd put the rationale for why this `unsafe` block
    // upholds the guarantees that I must ensure. Too bad I
    // copy-and-pasted from Stack Overflow without reading this comment!
    unsafe {
        let ptr = buffer.as_ptr() as *const num_complex::Complex<f64>;
        let len = buffer.len();

        assert!(len % 2 == 0);
        
        std::slice::from_raw_parts(ptr, len / 2)
    }
}

Hydrophyte

Wish I could give another +1 for the codes comment. – Cutcheon

What makes me slightly nervous is the assert!(cap % 2 == 0); because the capacity is not usually managed by the user, and it could inadvertently be odd and crash the program. I would prefer a function such as as_complex_unsafe(buffer: &[f64] -> &[Complex<f64>] so that no capacity is involved. – Threadfin

@Threadfin great points. I skipped the slice answer because OP specifically requested about Vec, but you are right that it should be included. Updated to handle odd capacities as well. – Hydrophyte

Good points all on odd capacity - I did built parallel implementations for &[f64] and &mut [f64], perhaps it's better to make these the only conversions allowed. – Karyotype

Say @Shepmaster, is the reason why you don't want two Vecs pointing to the same memory because if a terminate/panic happens elsewhere, there's a chance that unwinding from this thread if the execution is before the std::mem::forget() will cause a double-free? – Karyotype

@Karyotype in this specific case, yes. The original code can't have this problem as there's no code between the creation and the forget, but it could easily happen with a bit of refactoring over time. – Hydrophyte