How to return array from a PHP extension, without copying it in memory?

Question

I'm developing a PHP-extension, where an object method needs to return an array zval.

The method looks like:

ZEND_METHOD(myObject, myMethod)
{
    zval **myArrayProperty;
    if (zend_hash_find(Z_OBJPROP_P(getThis()), "myArrayProperty", sizeof("myArrayProperty"), (void **) &myArrayProperty) == FAILURE) {
        RETURN_FALSE;
    }
    RETURN_ZVAL(*myArrayProperty, 1, 0);
}

The code works fine and does the expected thing - it returns object's myArrayProperty. However, I'd like to optimize the process.

myArrayProperty stores an array, which can be quite big. And the RETURN_ZVAL() macro duplicates that array in order to return the value. The duplication process takes good amount of time to acquire memory and copy all the array values. At the same time, the returned array is usually used for read-only operations. So a nice optimization would be to use PHP's mechanism with reference counting and do not duplicate myArrayProperty contents. Rather I'd increase refcount of myArrayProperty and just return pointer to it. This is the same tactic as usually used, when working with variables in a PHP extension.

However, seems, there is no way to do it - you have to duplicate value in order to return it from a PHP extension function. Changing function signature to return value by reference, is not an option, because it links the property and returned value - i.e. changing returned value later, changes the property as well. That is not an acceptable behavior.

The inability to engage reference counting looks strange, because same code in PHP:

function myMethod() {
{
    return $this->myArrayProperty;
}

is optimized by the reference counting mechanism. That's why I'm asking this question at StackOverflow in case I missed something.

So, is there a way to return an array from a function in PHP extension, without copying the array in memory?

Boorman

Answer 1

If your function returns by-value this is only possible as of PHP 5.6 (current master) using the RETURN_ZVAL_FAST macro:

RETURN_ZVAL_FAST(*myArrayProperty);

If your function returns by-reference (return_reference=1 in the arginfo) you can return using the following code:

zval_ptr_dtor(&return_value);
SEPARATE_ZVAL_TO_MAKE_IS_REF(myArrayProperty);
Z_ADDREF_PP(myArrayProperty);
*return_value_ptr = *myArrayProperty;

If your function returns by-value and you're on PHP 5.5 or older you can still optimize the refcount=1 case:

if (Z_REFCOUNT_PP(myArrayProperty) == 1) {
    RETVAL_ZVAL(*myArrayProperty, 0, 1);
    Z_ADDREF_P(return_value);
    *myArrayProperty = return_value;
} else {
    RETVAL_ZVAL(*myArrayProperty, 1, 0);
}

Googly

Well, as it is said in the description - the function doesn't return value by reference. So it is not a solution, unfortunately. – Boorman

Sorry, I missed that. In this case what you want is not possible. – Googly

Though I don't see an immediate reason why we couldn't pass in return_value_ptr even without ACC_RETURN_REFERENCE (apart from the fact that this would allow you to return an is_ref=1 zval from a non-ref function). You might want to ask on internals@ about this. – Googly

It doesn't work, because return_value_ptr is initialized by engine only when return by reference is declared for a function/method. – Boorman

@AndreyTserkus I'm aware of that :) Just saying that this is something we might want to change. It's not clear to me why we couldn't just always set return_value_ptr. – Googly

Got it, @Googly so it were just the thoughts on possible future improvements, not the solution. – Boorman

@Googly has just pushed the changes to PHP to address this issue, so it should be included in the next PHP 5.5.3 release. – Paralysis

@Paralysis I applied the patch only to master, so this is 5.6 only. I'm not sure this can be applied to 5.5 because it changes the binary API (it changes the expectations towards implementers of zend_execute_internal). – Googly

@NikiC, thanks for the clarification / correction, which the readers here need. But I've already regressed it into my local 5.5.3 so I can benchmark it :-) – Paralysis

@Paralysis I'm pretty sure you will not get any measurable difference for "normal" code. The only thing that might have some impact is the change to __call. But that only if it's used a lot to return large arrays... – Googly

@NikiC, Not so sure that's the case, but here isn't the place to have this chat. Let me post back on the Internals DL when I've collected the stats if there is anything material to report or I can use the PHP chatroom. TTFN (Ta-Ta for now) – Paralysis

Answer 2

I don't have access to PHP < 5.6 but I think the problem is that the value is copied only. To be absolutely sure you should search the code for the defines in question.

That means you might be able to try:

 zval *arr;
 MAKE_STD_ZVAL(arr);
 array_init(arr);
 // Do things to the array.
 RETVAL_ZVAL(arr, 0, 0);
 efree(arr);

This is dangerous if used unwisely. If used with your own temporary containers I don't know of any problems.

You can also probably work on return value directly which might be a better approach. You would likely initialise it and pass it around as a pointer at the start.

You can wrap your return result like this. You can also experiment with references.

Hydrotherapy

Answer 3

It's been awhile, since I coded something like this…

So, what I do in code below: 1). explicitly increasing refcounter 2). returning zval without copying it

ZEND_METHOD(myObject, myMethod)
{
    zval **myArrayProperty;

    if (zend_hash_find(Z_OBJPROP_P(getThis()), "myArrayProperty", sizeof("myArrayProperty"), (void **) &myArrayProperty) == FAILURE) {
        RETURN_FALSE;
    }

    Z_ADDREF_PP(myArrayProperty);
    RETURN_ZVAL(*myArrayProperty, 0, 0);
}

Poliomyelitis

But isn't it going to lead to a memory leak or segfault (whichever comes first)? Memory leak will happen, when all the references to the property are cleared, but the memory, occupied by its zval container cannot be freed, because the refcount would still remain 1. Segfault will happen, when the returned value is disposed, thus its zval container is cleared together with the array (shared HashTable, which is referenced both from that container and from property zval container), so using property later will lead to unpredictable, but definitely wrong effects. – Boorman

Property references zval — that's refcount=1. this code increases refcount, as zval is returned and will be referenced by both object and the caller. if the object doesn't need property it will decrease refcount, so it will be owned only by caller. so, this code looks sane to me. but, again, all of this is strictly theoretical — I don't even have sources of PHP unpacked right now – Poliomyelitis

unfortunately, as predicted, the code doesn't work - confirmed in practice: pastebin.com/FRfaJZvL . The issue is as said above: the object and the caller reference different memory locations. – Boorman