Convert JSON string to array WITHOUT json_decode

Question

I am using PHP on shared server to access external site via API that is returning JSON containing 2 levels of data (Level 1: Performer & Level 2: Category array inside performer). I want to convert this to multidimensional associative array WITHOUT USING json_decode function (it uses too much memory for this usage!!!)

Example of JSON data:

[
{
    "performerId": 99999,
    "name": " Any performer name",
    "category": {
        "categoryId": 99,
        "name": "Some category name",
        "eventType": "Category Event"
    },
    "eventType": "Performer Event",
    "url": "http://www.novalidsite.com/something/performerspage.html",
    "priority": 0
},
{
    "performerId": 88888,
    "name": " Second performer name",
    "category": {
        "categoryId": 88,
        "name": "Second Category name",
        "eventType": "Category Event 2"
    },
    "eventType": "Performer Event 2",
    "url": "http://www.novalidsite.com/somethingelse/performerspage2.html",
    "priority": 7
}
]

I have tried to use substr and strip the "[" and "]".

Then performed the call:

preg_match_all('/\{([^}]+)\}/', $input, $matches);

This gives me the string for each row BUT truncates after the trailing "}" of the category data.

How can I return the FULL ROW of data AS AN ARRAY using something like preg_split, preg_match_all, etc. INSTEAD of the heavy handed calls like json_decode on the overall JSON string?

Once I have the array with each row identified correctly, I CAN THEN perform json_decode on that string without overtaxing the memory on the shared server.

---

For those wanting more detail about json_decode usage causing error:

$aryPerformersfile[ ] = file_get_contents('https://subdomain.domain.com/dir/getresults?id=1234');
$aryPerformers = $aryPerformersfile[0];
unset($aryPerformersfile);
$mytmpvar = json_decode($aryPerformers);
print_r($mytmpvar);
exit;

Hydrometer

Can this link https://mcmap.net/q/21104/-regex-to-validate-json be helpful? – Amphithecium

So rather than using json_decode, you're instead calling undoubtedly a less efficient workaround. Yep. That'll go swimmingly. @AbraCadaver check out phantomJS – Propagandism

how does it use too much memory? can you show us how its using too much, whats your definition of too much? my guess is not that the function is using to much is that your not garbage collecting properly with unset(), or you need to increase your memory allotment in your app – Electrochemistry

I am really interested how did you calculate that json_decode use too much memory? how big is your json sring? how much does json_decode takes in your case? what tool dod you use to scale memory usage? – Catholicize

Thanks for the ridicule without evidence. It is so appreciated! On a website currently running Wordpress (PHP and MySQL) on shared hosting. I make the API call which returns the JSON content into a variable. When I call json_decode on the variable, I receive the following error: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 10 bytes) in /home/mysite/public_html/subdir/myfile.php on line 3. – Hydrometer

so maybe your WP code is not optimaized? do you understand that json_decode is just last command was send before the error and was trying to allocate 10 bytes? ;-) does it prove that json_decode overusing the memory? – Catholicize

Actually the error is on line 4. My mistake. – Hydrometer

Increase php memory_limit. "preg_match_all" will still load all the matches into memory, so you will probably run into the same problem. Also If the data the API sending back is too big, and you have control over it, consider offset and limit. – Moonshiner

My site is running on the server. I MUST have this code run within the same account on this server. json_decode is triggering the memory error. I am able to run preg_match_all and/or preg_split on the variable just fine! – Hydrometer

The point is that almost ALL of the allowed memory is being used BEFORE the call that generates that error. – Absolute

could you do var_dump($aryPerformers); exit(); before json_decode? what size is your string? could it be included in your post? is there something really private? – Catholicize

The problem here is most likely data size of what you're trying to decode. Implementing your own decode won't fix your problem. – Sedate

@BeachCarolina is it the only site that use this server? does your WP running well since 3month? 6 month several years? do you have control panel to check memory usage even when you have no visitors on your site? – Catholicize

Answer 1

If you have a limited amount of memory, you could read the data as a stream and parse the JSON one piece at a time, instead of parsing everything at once.

getresults.json:

[
    {
        "performerId": 99999,
        "name": " Any performer name",
        "category": {
            "categoryId": 99,
            "name": "Some category name",
            "eventType": "Category Event"
        },
        "eventType": "Performer Event",
        "url": "http://www.novalidsite.com/something/performerspage.html",
        "priority": 0
    },
    {
        "performerId": 88888,
        "name": " Second performer name",
        "category": {
            "categoryId": 88,
            "name": "Second Category name",
            "eventType": "Category Event 2"
        },
        "eventType": "Performer Event 2",
        "url": "http://www.novalidsite.com/somethingelse/performerspage2.html",
        "priority": 7
    }
]

PHP:

$stream = fopen('getresults.json', 'rb');

// Read one character at a time from $stream until
// $count number of $char characters is read
function readUpTo($stream, $char, $count)
{
    $str = '';
    $foundCount = 0;
    while (!feof($stream)) {
        $readChar = stream_get_contents($stream, 1);

        $str .= $readChar;
        if ($readChar == $char && ++$foundCount == $count)
            return $str;
    }
    return false;
}

// Read one JSON performer object
function readOneJsonPerformer($stream)
{
    if ($json = readUpTo($stream, '{', 1))
        return '{' . readUpTo($stream, '}', 2);
    return false;
}

while ($json = readOneJsonPerformer($stream)) {
    $performer = json_decode($json);

    echo 'Performer with ID ' . $performer->performerId
        . ' has category ' . $performer->category->name, PHP_EOL;
}
fclose($stream);

Output:

Performer with ID 99999 has category Some category name
Performer with ID 88888 has category Second Category name

This code could of course be improved by using a buffer for faster reads, take into account that string values may themselves include { and } chars etc.

Quintana

Thanks @mhall! much appreciated – Hydrometer

This helped me out a ton. Thanks! For other using this, be sure to adjust the $count parameter according to how many closing brackets you're working with. – Episcopalism

Answer 2

You have two options here, and neither of them include you writing your own decoder; don't over-complicate the solution with an unnecessary work-around.

1) Decrease the size of the json that is being decoded, or 2) Increase the allowed memory on your server.

The first option would require access to the json that is being created. This may or may not be possible depending on if you're the one originally creating the json. The easiest way to do this is to unset() any useless data. For example, maybe there is some debug info you won't need, so you can do unset($json_array['debug']); on the useless data. http://php.net/manual/en/function.unset.php

The second option requires you to have access to the php.ini file on your server. You need to find the line with something like memory_limit = 128M and make the 128M part larger. Try increasing this to double the value already within the file (so it would be 256M in this case). This might not solve your problem though, since large json data could still be the core of your problem; this only provides a work-around for inefficient code.

Sedate

It's also worth mentioning that the problem could be unrelated to the json data. The problem might be caused by a large variable that could be caused before the decode ever takes place, making the decode "the straw that broke the camel's back" in this case. – Sedate