Iterate through an array in batches using php

Question

I've got an API I'm working with that accepts a call every 5 seconds, any more than that it wont respond. In each call it will accept 5 records in a batch. I've got a list of 1000s of records that I need to check using the api, so what I'm trying to do is send it my list of records in broken up into batches of 5 every 5 seconds.

I can get most of it to work, but the bit I can't figure is breaking down the list of records which is an array in batches, any idea how to do this?

This is the code I was using below, but it's outputting each individual part of the array every 5 seconds, rather than in batches of 5.

$my_array = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20];

    foreach ($my_array as $key => $value) {
        sleep (5);
        echo $value;
    }

Steak

So far you're not doing anything to split up your array. You need to break it into chunks somehow. Hmm, I wonder if PHP has some sort of array chunk function... – Trudi

Answer 1

you could have a second loop of

$batch_of = 5;
$batch = array_chunk($my_array, $batch_of);
foreach($batch as $b) {

    foreach ($b as $key => $value) {

        echo $value;
    }
   sleep (5);
}

Which would work as intended

Christabelle

Answer 2

In case it helps anyone else, I wrote a function that will allow you to process an array in chunks. The description and details are here:

https://totaldev.com/php-process-arrays-batches/

The main difference between mine and array_chunk is that mine doesn't return an array of smaller arrays. It takes a user-defined function as a closure that will handle the small batches. Here is the function:

// Iterate through an array and pass batches to a Closure
function arrayBatch($arr, $batchSize, $closure) {
    $batch = [];
    foreach($arr as $i) {
        $batch[] = $i;

        // See if we have the right amount in the batch
        if(count($batch) === $batchSize) {
            // Pass the batch into the Closure
            $closure($batch);

            // Reset the batch
            $batch = [];
        }
    }

    // See if we have any leftover ids to process
    if(count($batch)) $closure($batch);
}

You can use it like this:

// Use array in batches
arrayBatch($my_array, 5, function($batch) {
    // Do whataver you need to with the $batch of 5 items here...

    sleep (5);
});

Advancement

This would really benefit from using php's array_chunk function. – Hyrax

Answer 3

use

if(($key + 1) % 5 == 0){ sleep(5);} 

in your loop

Wsan

Could you give a bit more context about why and where this condition must be used? – Superintendency

Answer 4

You can create a specific iterator to iterate over batches :

class BatchIterator extends \IteratorIterator
{
    private array $batch = [];

    public function __construct(
        \Traversable $iterator,
        private readonly int $batchSize,
    ) {
        parent::__construct($iterator);
    }

    public function rewind(): void
    {
        parent::rewind();
        $this->next();
    }

    public function next(): void
    {
        $this->batch = [];
        for ($i = 0; $i < $this->batchSize && parent::valid(); $i++) {
            $this->batch[] = parent::current();
            parent::next();
        }
    }

    public function current(): array
    {
        return $this->batch;
    }

    public function valid(): bool
    {
        return (bool) $this->batch;
    }
}

And use it to iterate over your data:

$size = 5;
foreach (new BatchIterator(new ArrayIterator($my_array), $size) as $value) {
    // Here $value is an array of max. 5 items
}

Code sample created from Guzzle3 library: https://github.com/Guzzle3/iterator/blob/master/ChunkedIterator.php

Gnash