Remove Duplicates from an Array of GeoFire Objects [duplicate]

Question

I am working with Geofire and Firebase on Angular 6 to store locations and unfortunately it's storing a lot of duplicates this is an example (console logging my variable currentHits):

0: {location: Array(2), distance: "48.84", url: "assets/imgs/fix.png"}
1: {location: Array(2), distance: "48.84", url: "assets/imgs/fix.png"}
2: {location: Array(2), distance: "48.84", url: "assets/imgs/fix.png"}
3: {location: Array(2), distance: "48.85", url: "assets/imgs/free.png"}
4: {location: Array(2), distance: "48.85", url: "assets/imgs/free.png"}
5: {location: Array(2), distance: "48.85", url: "assets/imgs/free.png"}
6: {location: Array(2), distance: "48.87", url: "assets/imgs/low.png"}
7: {location: Array(2), distance: "48.87", url: "assets/imgs/low.png"}
8: {location: Array(2), distance: "48.87", url: "assets/imgs/low.png"}

Location basically is an array of latitude and longitude used to calculate distance, in id 0, 1 and 2 its the same coordinates, and 3,4 and 5 are also the same, ...

This is what I want to get:

0: {location: Array(2), distance: "48.84", url: "assets/imgs/fix.png"}
1: {location: Array(2), distance: "48.85", url: "assets/imgs/free.png"}
2: {location: Array(2), distance: "48.87", url: "assets/imgs/low.png"}

(Optional) this is how It stores these locations:

  ...
  hits = new BehaviorSubject([])

  ...
  queryHits(...){
 ....
 let hit = {
          location: location,
          distance: distance.toFixed(2),
          url:img
        }

        let currentHits = this.hits.value
        currentHits.push(hit)
        this.hits.next(currentHits)
....
}

It's true that this question has probably already been asked and I have been digging through all the similar questions and found these functions:

1. RemoveDuplicates()

function removeDuplicates(arr){
    let unique_array = []
    for(let i = 0;i < arr.length; i++){
        if(unique_array.indexOf(arr[i]) == -1){
            unique_array.push(arr[i])
        }
    }
    return unique_array
}

var newlist = removeDuplicates(list)

It didn't work I get the same list with duplicates.

2. arrUnique:

function arrUnique(arr) {
    var cleaned = [];
    arr.forEach(function(itm) {
        var unique = true;
        cleaned.forEach(function(itm2) {
            if (_.isEqual(itm, itm2)) unique = false;
        });
        if (unique)  cleaned.push(itm);
    });
    return cleaned;
}

var newlist= arrUnique(list);

Also, it didn't work..

3. onlyUnique

  onlyUnique(value, index, self) { 
    return self.indexOf(value) === index;
  }

var newlist = list.filter(onlyUnique)

Unfortunately it didn't work...

These are some of the answers given to similar problem to remove duplicates from an array and none of them worked. I don't understand why they won't work for my type of array, If anyone has an idea or knows why would be very helpful.

Wolcott

Your approaches didn't work because all objects, even if it shares all the same key/values, are different instances (unless Singleton which this isn't) – Homebrew

Can you share sample JSON. That would be better for us – Cleavland

would it be also a solution for you to not use incremental id's (0, 1, 2, 3,...) as the prop name but use object related id's ? This will ensure you, that you have no duplicated records. – Gallenz

you can't compare objects this way : {} === {} return false – Clammy

This is not a duplicate and you are right @RobG I changed to Geofire objects – Wolcott

"Location basically is an array of latitude and longitude…" your example doesn't show any latitude or longitude values, what are you actually trying to compare? – Masonite

As I said location is an array for id 0,1,2, is the same thats why I get same distance because its calculated from the coordinates – Wolcott

Sure, but do you want to detect duplicate lat/long, duplicate distance, or where all properties and values are the same? – Masonite

Please rob this question is not a duplicate I have made a research before making this question and none worked for me. – Wolcott

Answer 1

You could use a set to store and check for duplicate values.

const removeDuplicates = arr => {
    let matches = new Set();
    return arr.filter(elem => {
        const {distance} = elem;
        if(matches.has(distance)){
            return false;
        } else {
            matches.add(distance);
            return true;
        }
    })   
}

Bear in mind that using this approach you may remove results where the distance is the same but the co-ordinates differ. If that causes an issue for you then you'd need to also check against the lat/lng pair.

Breeden

The location is a unique element so can we change instead distance use location? – Wolcott

Answer 2

Problem here is comparing Objects. Two objects are never equal unless both are referencing to same Object.

Example:

{} === {} // false

// Two objects are equal only if they are referencing to same object
var a = {};
a === a; // true

It is clear from your problem that you are facing the first case. Among the solutions you tested Solution 1 and Solution 3 are failing because of this reason as indexOf does === comparision.

But Solution 2 should have worked on your example as it does a deep comparision as explained here. https://lodash.com/docs#isEqual.

PS: It might be a simple typo i have observed in Solution 2 cleaned.,push(itm);, there is an extra comma. Hoping that is not the case I am moving ahead

So i guess the issue is inside your location array, if you can give the contents of location array we should be able to provide better solution. Or as others suggested you can filter based on a single key of the object like id or distance, instead of comparing the whole object

Aile

Yes, though solution 2 could be written a little simpler using if (!_.isEqual(itm, itm2)) cleaned.push(itm);. ;-) – Masonite

Answer 3

You could always check before you add the hits to make sure there are no repeats.

edit: You cannot compare objects unless they have the same reference object. So, you could compare objects by a unique ID

use rxjs filter() this will return an array

// store history of objs for comparison
addedObjs = [];
this.hits.pipe(filter(obj => {
    // check if id is an index of the previous objs
    if (addObjs.indexOf(obj.id) === -1) {
        this.addedObjs.push(obj.id)
        return obj
    });

here is the working stackblitz using some of your code

Crossly

This approach is soo gooad but I am getting error ts] The 'this' context of type 'void' is not assignable to method's 'this' of type 'Observable<{}> – Wolcott

the list I showd on my question actually can be logged from console.log(this.hits.value) or console.log(currentHits) – Wolcott

@ProgrammerMan it's probably a bad import from rxjs. If you are using rxjs 6, import { distinctUntilChanged } from 'rxjs/operators' – Crossly

Thank you for checking I have changed the import as you said and now getting this Argument of type '(value: any[]) => any[]' is not assignable to parameter of type '(x: any[], y: any[]) => boolean'. Type 'any[]' is not assignable to type 'boolean'. – Wolcott

@ProgrammerMan my apologies, actually you don't need to pass value in the operator, i'll update the answer – Crossly

still getting errors the thing is this.hits is BehaviorSubject([]) Object – Wolcott

hmm, you would have to post what you are doing for more help, or add me in a chat. because you can use distinctUntilChanged on a behavior subject #39667633 – Crossly

Let us continue this discussion in chat. – Crossly

Answer 4

uniqWith https://lodash.com/docs/#uniqWith can be used to specify the method to compare by :

var arr = [ { location: [1, 2], distance: "48.84", url: "assets/imgs/fix.png" },
            { location: [1, 2], distance: "48.84", url: "assets/imgs/fix.png" },
            { location: [1, 2], distance: "48.84", url: "assets/imgs/fix.png" },
            { location: [3, 4], distance: "48.85", url: "assets/imgs/free.png"},
            { location: [3, 4], distance: "48.85", url: "assets/imgs/free.png"},
            { location: [3, 4], distance: "48.85", url: "assets/imgs/free.png"},
            { location: [5, 6], distance: "48.87", url: "assets/imgs/low.png" },
            { location: [5, 6], distance: "48.87", url: "assets/imgs/low.png" },
            { location: [5, 6], distance: "48.87", url: "assets/imgs/low.png" } ]
          
var result = _.uniqWith(arr, (a, b) => _.isEqual(a.location, b.location));

console.log( JSON.stringify({...result}).replace(/},/g, '},\n ') );

<script src="https://cdn.jsdelivr.net/npm/[email protected]/lodash.min.js"></script>

Igal

This won't work since it doesn't take in consideration the unique id each object has. – Wolcott

Also, you can't guarantee the order of object properties, so can't guarantee that the JSON.stringify will be a useful comparison. – Masonite

Answer 5

You can use following approach:

Idea:

• You can create your own data structure and have a hashMap to save values.
• Since you have location data, you can use longitude|latitude as your key name as it will be unique.
• Then expose some functions say, add that will check if value exists, override else add.
• Also create a property, say value that would return the list of locations.

Note: Above behavior can be achieved using Set as well. If you cannot use ES6 features, then this is one way that is extensible and easy.

function MyList() {
  var locations = {};

  this.add = function(value) {
    var key = value.location.join('|');
    locations[key] = value;
  }

  Object.defineProperty(this, 'value', {
    get: function() {
      return Object.keys(locations).map(function(key) {return locations[key] })
    }
  })
}

var locations = new MyList();

locations.add({location: [123.12, 456.23], name: 'test 1' });
locations.add({location: [123.16, 451.23], name: 'test 2' });
locations.add({location: [123.12, 456.23], name: 'test 1' });
locations.add({location: [100.12, 456.23], name: 'test 3' });
locations.add({location: [123.12, 456.23], name: 'test 1' });
locations.add({location: [123.12, 400.23], name: 'test 4' });

console.log(locations.value)

Typescript version for more readability:

interface ILocation {
  location: Array<number>
  [key: string]: any;
}

interface IList {
  [key: string]: ILocation
}

class MyList {
  private locations: IList = {};
  
  public add(value: ILocation) {
    const key: string = value.location.join('|');
    this.locations[key] = value;
  }
  
  public get value(): Array<ILocation> {
    return Object.keys(locations).map(function(key) {return locations[key] })
  }
}

Cleavland

This looks so nice can you translate it to typescript?? because when you say this.add where do you declare add? – Wolcott

@ProgrammerMan I have added a typescript code sample. Its not runnable though. Hope it helps you! – Cleavland

Answer 6

Perhaps you would want to use a library such as lodash which has a wide set of functions regarding all types of collections.

let newArr = _.uniqWith(myArr, _.isEqual);

uniqWith with the help of isEqual can get what you want.

Here is fiddle to that solution

Tatum

Answer 7

I think it is your comparison that may not be working correctly. You can try this:

var uniqueHits = currentHits.reduce((acc, curr) => { 
    if (acc.length === 0) {
        acc.push(curr);
    } else if (!acc.some((item) => 
        item.location[0] === curr.location[0]
        && item.location[1] === curr.location[1]
        && item.distance === curr.distance
        && item.url === curr.url)) {
        acc.push(curr);
    }
    return accumulator;
}, []);;

Nita