How to remove duplicate values from a multi-dimensional array in PHP

Question

How can I remove duplicate values from a multi-dimensional array in PHP?

Example array:

Array
(
    [0] => Array
    (
        [0] => abc
        [1] => def
    )

    [1] => Array
    (
        [0] => ghi
        [1] => jkl
    )

    [2] => Array
    (
        [0] => mno
        [1] => pql
    )

    [3] => Array
    (
        [0] => abc
        [1] => def
    )

    [4] => Array
    (
        [0] => ghi
        [1] => jkl
    )

    [5] => Array
    (
        [0] => mno
        [1] => pql
    )

)

Ppi

This question does not have a minimal reproducible example. We have input, but no expression of the exact desired output. This leads to divergent answers that respect both columns for uniqueness or respect only one of the columns for uniqueness. – Moreno

See Filter/Remove rows where column value is found more than once in a multidimensional array for a comprehensive list of techniques accommodating different scenarios with nuanced behavior. – Moreno

Answer 1

Here is another way. No intermediate variables are saved.

We used this to de-duplicate results from a variety of overlapping queries.

$input = array_map("unserialize", array_unique(array_map("serialize", $input)));

Uredium

Because of unserialize this is slower and slower the larger and more complex the array is. There is a reason I used array_intersect_key (half a year before this answer). – Gargantua

@Gargantua they wanted a one liner i guess. should have just rewritten as $no-duplicates = array_intersect_key( $array , array_unique( array_map('serialize') , $array ) ); – Minneapolis

@Gargantua well just tested it, had a typo but it works.. thanks dude!: $no_duplicates = array_intersect_key( $array , array_unique( array_map('serialize' , $array ) ) ); – Minneapolis

if you want the index continuous, use array_values i.e. $input = array_values(array_map("unserialize", array_unique(array_map("serialize", $input)))); – Dragelin

Nowadays you probably would opt for json_encode and json_decode instead of PHP serialization. should have benefits for the values provided and you don't run into PHP serialization details that serialize/unserialize ships with and most likely are unwanted. – Impair

$input = array_values($input); To fix missing array keys inside the array – Easting

Although cool I don't want to depend on serialize and unserialize. The way PHP has implemented these has changed behind the scenes a few times. For the longest time references was not stored. They are now, but only to references within the serialized array. (I know the question didn't have objects). The better way would be array_unique($input, SORT_REGULAR) or some custom function. – Keiko

Beware that serialize(array('a' => '1', 'b' => '1')) is different from serialize(array('b' => '1', 'a' => '1')). This option will fail for arrays used as sets or (hash)maps. – Inhume

Even better with single commas! $input = array_map('unserialize', array_unique(array_map('serialize', $input))); – Hialeah

@Keiko - From the current PHP doc: "Note that array_unique() is not intended to work on multi dimensional arrays." – Trisaccharide

Good answer but it does not support entries with different order like [['foo', 'bar'], ['bar', 'foo']]. To use your snippet, I sorted the child arrays. – Shillelagh

@dave when you don't include an explanation with your snippet, the page suffers from receiving an exact duplicate snippet which INCLUDES an explanation. You could have prevented this by explaining your answer. – Moreno

Answer 2

Since 5.2.9 you can use array_unique() if you use the SORT_REGULAR flag like so:

array_unique($array, SORT_REGULAR);

This makes the function compare elements for equality as if $a == $b were being used, which is perfect for your case.

Output

Array
(
    [0] => Array
        (
            [0] => abc
            [1] => def
        )

    [1] => Array
        (
            [0] => ghi
            [1] => jkl
        )

    [2] => Array
        (
            [0] => mno
            [1] => pql
        )

)

Keep in mind, though, that the documentation states:

array_unique() is not intended to work on multi dimensional arrays.

Fourfold

I guess this is more quick and more clear solution, than the accepted one! lets vote for this one! :) Hmmm on php site we can see that it is not so quick, as I thought... – Russo

Strange that using the SORT_REGULAR flag just doesn't work for me, to remove duplicate arrays. – Bizerte

@Bizerte then you may wish to consider asking a new question. – Cioffi

There are already other, working answers to this question which I have verified (see for example the accepted answer). I just cannot verify this particular answer, for my particular example. – Bizerte

@Bizerte could you provide a pastebin for your particular issue then? Just telling me it doesn't work is not helping me understand. – Cioffi

@Bizerte You're right; it doesn't seem to give the correct results, but it's probably a bug because it works with PHP 7 =/ – Cioffi

Very interesting... Thank you very much Jack, for your attention to this - amazing response. Anyway, since I cannot change the php version on the server I will just have to use one of the other solutions. – Bizerte

This also appears to work in my case but is anybody else bothered by this note in the array_unique() doc? php.net/manual/en/… – Frederickfredericka

@Jack You're right this is a bug as of PHP 5.6.23: eval.in/645675 but is fixed as of PHP 7.0.8: eval.in/645676 – Couplet

Beware array(1, 2) is different from array(2, 1). This will fail for arrays used as sets. – Inhume

@AndrasGyomrey if you want to express sets with arrays you should normalise them. – Cioffi

Either you use in_array or you make sure your elements are usable as array keys (serializable) and you use some dummy array value. It depends on your implementation. For the first implementation, you can't go with this approach. – Inhume

This is proper solution compared to serialize/unserialize answer. Please note, that array unique is rather fast compared to serialize/deserialize. This might be issue if you do hundreds/thousands of array unique. – Rhodes

Note this causes strange behavior with assoc arrays that have boolean true values (those disappear). Example: 3v4l.org/Yj6Y1 (show_in_configurator disappeared). – Melvin

Answer 3

Another way. Will preserve keys as well.

function array_unique_multidimensional($input)
{
    $serialized = array_map('serialize', $input);
    $unique = array_unique($serialized);
    return array_intersect_key($input, $unique);
}

Gargantua

For large arrays, this method is often at least 50% faster than the accepted answer. – Educational

Answer 4

I had a similar problem but I found a 100% working solution for it.

<?php
    function super_unique($array,$key)
    {
       $temp_array = [];
       foreach ($array as &$v) {
           if (!isset($temp_array[$v[$key]]))
           $temp_array[$v[$key]] =& $v;
       }
       $array = array_values($temp_array);
       return $array;

    }


$arr="";
$arr[0]['id']=0;
$arr[0]['titel']="ABC";
$arr[1]['id']=1;
$arr[1]['titel']="DEF";
$arr[2]['id']=2;
$arr[2]['titel']="ABC";
$arr[3]['id']=3;
$arr[3]['titel']="XYZ";

echo "<pre>";
print_r($arr);
echo "unique*********************<br/>";
print_r(super_unique($arr,'titel'));

?>

Floatstone

This answers a different question. See here: #4585708 – Gargantua

Great function! and in case you're dealing with objects: if(!isset($array->$v->$key)) $array[$v->$key] =& $v; – Scrounge

The modification by reference in this answer is entirely unnecessary. I agree with OIS, this answer is not respecting both columns when determining uniqueness. This is the correct answer to a different question. – Moreno

Answer 5

Array
(
    [0] => Array
        (
            [id] => 1
            [name] => john
        )

    [1] => Array
        (
            [id] => 2
            [name] => smith
        )

    [2] => Array
        (
            [id] => 3
            [name] => john
        )

    [3] => Array
        (
            [id] => 4
            [name] => robert
        )

)

$temp = array_unique(array_column($array, 'name'));
$unique_arr = array_intersect_key($array, $temp);

This will remove the duplicate names from array. unique by key

Elyseelysee

Make sure that $array's keys start at "0". It's possible $array's keys start at another number if $array is the result of a prior array manipulation. Use array_values to reset the keys back to "0" – Carollcarolle

This answer is not respecting both columns when determining uniqueness. This is the correct answer to a different question. – Moreno

Working as expected. Thank you. – Bonanza

Answer 6

If "remove duplicates" means "remove duplicates, but let one there", a solution might be to apply the array_unique(...) on the "identifier column" first and then to remove in the original array all the keys, that have been removed from the column array:

$array = [
    [
        'id' => '123',
        'foo' => 'aaa',
        'bar' => 'bbb'
    ],
    [
        'id' => '123',
        'foo' => 'ccc',
        'bar' => 'ddd'
    ],
    [
        'id' => '567',
        'foo' => 'eee',
        'bar' => 'fff'
    ]
];

$ids = array_column($array, 'id');
$ids = array_unique($ids);
$array = array_filter($array, function ($key, $value) use ($ids) {
    return in_array($value, array_keys($ids));
}, ARRAY_FILTER_USE_BOTH);

The result is:

Array
(
    [0] => Array
        (
            [id] => 123
            [foo] => aaa
            [bar] => bbb
        )

    [2] => Array
        (
            [id] => 567
            [foo] => eee
            [bar] => fff
        )

)

Krone

This answer is not respecting both columns when determining uniqueness. This is the correct answer to a different question. – Moreno

Answer 7

The user comments on the array_unique() documentation have many solutions to this. Here is one of them:

kenrbnsn at rbnsn dot com
27-Sep-2005 12:09

Yet another Array_Unique for multi-demensioned arrays. I've only tested this on two-demensioned arrays, but it could probably be generalized for more, or made to use recursion.

This function uses the serialize, array_unique, and unserialize functions to do the work.

function multi_unique($array) { foreach ($array as $k=>$na) $new[$k] = serialize($na); $uniq = array_unique($new); foreach($uniq as $k=>$ser) $new1[$k] = unserialize($ser); return ($new1); }

This is from https://www.php.net/manual/en/function.array-unique.php#57202.

Aguiar

Answer 8

if you need to eliminate duplicates on specific keys, such as a mysqli id, here's a simple funciton

function search_array_compact($data,$key){
    $compact = [];
    foreach($data as $row){
        if(!in_array($row[$key],$compact)){
            $compact[] = $row;
        }
    }
    return $compact;
}

Bonus Points You can pass an array of keys and add an outer foreach, but it will be 2x slower per additional key.

Neptunian

Iterated calls of in_array() will be one of the worst possible performer on this whole page. !in_array() has a big O of n, therefore we are looking at O(n * n) -- no bueno. – Moreno

@Moreno fair points, thanks for the feedback. i will try to think of a more efficient solution – Neptunian

Answer 9

if you have an array like this:

(users is the name of the array)

Array=>
 [0] => (array)
   'user' => 'john'
   'age' => '23'
 [1] => (array)
  'user' => 'jane'
  'age' => '20'
 [2]=> (array)
  'user' => 'john'
  'age' => '23'

and you want to delete duplicates...then:

$serialized = array();
for ($i=0; $i < sizeof($users); $i++) { 
  $test = in_array($users['user'], $serialized);
    if ($test == false) {
      $serialized[] = $users['user'];
    }
 }

can be a solution :P

Theophrastus

This answer doesn't respect both columns when determining uniqueness. Iterated calls of in_array() will be one of the worst possible performer on this whole page. !in_array() has a big O of n, therefore we are looking at O(n * n) -- no bueno. This answer is merely r3wt's answer written with an unnecessary, single-use variable ($test). – Moreno

Answer 10

Lots of person asked me how to make Unique multidimensional array. I have taken reference from your comment and it helps me.

First of All, Thanks to @jeromegamez @daveilers for your solution. But every time i gave the answer, they asked me how this 'serialize' and 'unserialize' works. That's why i want to share the reason of this with you so that it will help more people to understand the concept behind this.

I am explaining why we use 'serialize' and 'unserialize' in steps :

Step 1: Convert the multidimensional array to one-dimensional array

To convert the multidimensional array to a one-dimensional array, first generate byte stream representation of all the elements (including nested arrays) inside the array. serialize() function can generate byte stream representation of a value. To generate byte stream representation of all the elements, call serialize() function inside array_map() function as a callback function. The result will be a one dimensional array no matter how many levels the multidimensional array has.

Step 2: Make the values unique

To make this one dimensional array unique, use array_unique() function.

Step 3: Revert it to the multidimensional array

Though the array is now unique, the values looks like byte stream representation. To revert it back to the multidimensional array, use unserialize() function.

$input = array_map("unserialize", array_unique(array_map("serialize", $input)));

Thanks again for all this.

Tellurium

Answer 11

A very easy and logical way to Unique a multi dimension array is as follows,

If you have array like this:

Array
(
    [Key1] => Array
        (
            [0] => Value1
            [1] => Value2
            [2] => Value1
            [3] => Value3
            [4] => Value1
        )
    [Key2] => Array
        (
            [0] => Value1
            [1] => Value2
            [2] => Value1
            [3] => Value3
            [4] => Value4
        )
)

use foreach to solve this:

foreach($array as $k=>$v){
    $unique=array_unique($v);
    $array[$k]=$unique;
}

it will give you following result:

Array
(
    [Key1] => Array
        (
            [0] => Value1
            [1] => Value2
            [3] => Value3
        )
    [Key2] => Array
        (
            [0] => Value1
            [1] => Value2
            [3] => Value3
            [4] => Value4
        )
)

and if you want to rearrange the order of the keys,

foreach($array as $k=>$v){
    $unique= array_values(array_unique($v));
    $array[$k]=$unique;
}

This operation will give you arranged key values like this:

Array
(
    [Key1] => Array
        (
            [0] => Value1
            [1] => Value2
            [2] => Value3
        )
    [Key2] => Array
        (
            [0] => Value1
            [1] => Value2
            [2] => Value3
            [3] => Value4
        )
)

I hope this will clear everything.

Incubator

This answer completely ignores the asker's sample data and deviates WAAAAAAY beyond the scope of this question. I'd argue that this answer doesn't get close to answering the question asked. The asker is not seeking a way to remove duplicate values within each subarray. – Moreno

Answer 12

An easy to read solution, probably not the most efficient:

function arrayUnique($myArray){
    if(!is_array($myArray))
        return $myArray;

    foreach ($myArray as &$myvalue){
        $myvalue=serialize($myvalue);
    }

    $myArray=array_unique($myArray);

    foreach ($myArray as &$myvalue){
        $myvalue=unserialize($myvalue);
    }

    return $myArray;

} 

Persephone

Answer 13

As people are saying array_unique() is very slow, here is a snippet I use for one level multidimensional array.

$serialized_array = array_map("serialize", $input);

foreach ($serialized_array as $key => $val) {
     $result[$val] = true;
}

$output = array_map("unserialize", (array_keys($result)));

Reference first user contributed note of array_unique() function page in php.net

Irade

Answer 14

This solution is relevant only when uniqueness is needed for one array column, for example here if we need the uniqueness in index #0 of the arrays.

Solution #1:

Using array_filter with an anonymous function and a static variable:

<?php

$list = [
    ['abc', 'def'],
    ['ghi', 'jkl'],
    ['mno', 'pql'],
    ['abc', 'def'],
    ['ghi', 'jkl'],
    ['mno', 'pql']
];

$list = array_filter($list, function ($item) {
    static $values = [];
    if (!in_array($item[0], $values)) {
        $values[] = $item[0];
        return true;
    } else {
        return false;
    }
});

var_dump($list);

Solution #2:

Since the value when we want the uniqueness are of string type, we can remap the original global array to use these values as keys, which will remove duplicates as we remap it:

<?php

$list = [
    ['abc', 'def'],
    ['ghi', 'jkl'],
    ['mno', 'pql'],
    ['abc', 'def'],
    ['ghi', 'jkl'],
    ['mno', 'pql']
];

$unique = [];

foreach ($list as $item) {
    $unique[$item[0]] = $item;
}

// Remap again to integers
$unique = array_values($unique);

var_dump($unique);

Onto

Answer 15

An alternative to serialize and unique

$test = [
    ['abc','def'],
    ['ghi','jkl'],
    ['mno','pql'],
    ['abc','def'],
    ['ghi','jkl'],
    ['mno','pql'],
];

$result = array_reduce(
    $test,
    function($carry,$item){
        if(!in_array($item,$carry)) {
            array_push($carry,$item);
        }
        return $carry;
    },
    []
);

var_dump($result);

/*
 php unique.php
array(3) {
    [0] =>
        array(2) {
            [0] =>
                string(3) "abc"
            [1] =>
                string(3) "def"
        }
    [1] =>
        array(2) {
            [0] =>
                string(3) "ghi"
            [1] =>
                string(3) "jkl"
        }
    [2] =>
        array(2) {
              [0] =>
                  string(3) "mno"
              [1] =>
                  string(3) "pql"
        }
}

*/

Hinkley

Iterated calls of !in_array() will suffer exponentially worse performance as the size of the input array increases. – Moreno

Answer 16

I've given this problem a lot of thought and have determined that the optimal solution should follow two rules.

1. For scalability, modify the array in place; no copying to a new array
2. For performance, each comparison should be made only once

With that in mind and given all of PHP's quirks, below is the solution I came up with. Unlike some of the other answers, it has the ability to remove elements based on whatever key(s) you want. The input array is expected to be numeric keys.

$count_array = count($input);
for ($i = 0; $i < $count_array; $i++) {
    if (isset($input[$i])) {
        for ($j = $i+1; $j < $count_array; $j++) {
            if (isset($input[$j])) {
                //this is where you do your comparison for dupes
                if ($input[$i]['checksum'] == $input[$j]['checksum']) {
                    unset($input[$j]);
                }
            }
        }
    }
}

The only drawback is that the keys are not in order when the iteration completes. This isn't a problem if you're subsequently using only foreach loops, but if you need to use a for loop, you can put $input = array_values($input); after the above to renumber the keys.

Byler

There is no performance boost to your nested, conditional unset() calls because you are making value comparisons instead of key comparison (which PHP performs much faster with). – Moreno

Answer 17

try this solution for (n) Dimensional array with non-restricted length
for example this array

$arr= [
0 => [0=>"a" , 1=>"b" ,  2=>"c" ] ,
1 => [0=>"x" , 1=>"b" , 2=>"a", 3=>"p"],
2=>   [
       [ 
          0=>"y" ,
          1=>"b" ,
          2=> [0=>"x" , 1=>"m" , 2=>"a"]
       ],
       1=>"z" ,
       2=>"v"
       ]
     ];

This would be the solution

$ar2=[];
$ar3=[];
function test($arr){
    
    global $ar2,$ar3;
    if(is_array($arr)){
       return array_map("test",$arr);
    }
    if(!isset($ar2[$arr])){
        $ar2[$arr]=1;
        $ar3[]=$arr;
    }
}
array_map("test",$arr);
print_r($ar3);

Beaton

Answer 18

Based on the Answer marked as correct, adding my answer. Small code added just to reset the indices-

$input = array_values(array_map("unserialize", array_unique(array_map("serialize", $inputArray))));

Deathblow

@milic's answer from 6 years earlier shows how to wrap the nested function calls with array_values(). I don't see the need to add this answer. You could have just added a comment under the accepted answer to say "if you want to re-index the result, just call array_values()". – Moreno