How do I remove duplicate values from a Coldfusion array?

Question

I have a function that receives a string of tags. In order to save the tags individually, the function transforms the string into an array:

this.tags = listToArray(this.tags, ", ");

How do I remove duplicate values in the event that there are any?

Sheridan

The "set" abstract data type, available in Java, is specifically designed to solve this sort of problem. – Coiffure

Answer 1

An easy way to remove duplicates from a list is to convert the list to a struct first, and then conver the struct to an array. However if the order of items in the list is important this may not be appropriate as the elements in the struct will be sorted.

If the order of items is important you would need to build the array manually rather than using the listToArray feature.

<!--- CF9 --->
<cfset tags = "apples,oranges,bananas,pears,APPLES" />
<cfset tagArray = arrayNew(1) />

<cfloop list="#tags#" index="tag" delimiters=",">
    <cfif not ArrayFindNoCase(tagArray,tag)>
        <cfset arrayAppend(tagArray, tag) />
    </cfif>
</cfloop>

Notepaper

Answer 2

I like to use Java for this kind of task:

<cfset tags = "apples,oranges,bananas,pears,apples" />

<cfset tagsArray = createObject("java", "java.util.ArrayList").init(
createObject("java", "java.util.HashSet").init(ListToArray(tags))
) />

<cfdump var="#tags#" />
<cfdump var="#tagsArray#" />

Only problem is it takes case into account, so thinks "apples" & "APPLES" are different things (which technically yes, depending on your system may well be different). Way round that is to lower case everything in the list first. (NOTE: Added java.util.ArrayList function so that the array is identified & reusable by Adobe ColdFusion; otherwise functions like arraysort will throw an error.)

Myrick

thanks for that. Have you done any testing/know of the top of your head whether this is faster than looping the values into a ColdFusion struct? – Sheridan

@Sheridan - From a sheer speed standpoint, my guess would be HashSet is faster. But .. unless you are dealing with huge lists the differences are usually small. I usually pick the method that does the right job and worry about speed only if it becomes an issue. – Holliman

the beauty of this solution is that it takes ONE LINE! In my case I needed to find out if a particular field in a coldfusion query had more than one unique value. While other solutions outlined here would have worked, this one liner made my code very simple: if (ArrayLen(createObject("java", "java.util.HashSet").init(cfQuery[colName]).toArray()) GT 1) – Morgue

Jason, thanks for the code. After pulling data from another system I have an array of structures that had duplicate values. I was able to pass the array through the HashSet and end up with an de-duped array set. – Gorcock

I updated the example to include java.util.ArrayList based on example here barneyb.com/barneyblog/2008/05/08/use-coldfusion-use-java otherwise the "array" couldn't be reused as an CFML array. – Tana

Answer 3

Since you're really starting with a string/list that you're then converting to an array, you can pass the string through ListRemoveDuplicates before converting the to an array. ListRemoveDuplicates was introduced in Coldfusion 10; the input parameters are (list, delimiter=",", ignoreCase=FALSE).

this.tags = listToArray(listRemoveDuplicates(arrayInput,", ",TRUE));

If you were actually starting with an array, you would need to convert it to a list first, then back again after.

this.tags = listToArray(listRemoveDuplicates(arrayToList(arrayInput),", ",TRUE) );

Peatroy

Code dumps are not answers. Please edit your answer and explain what this code is, how it works, and how it answers the question. – Dailey

I'd say that code speaks for itself. Just read it and it will be clear what's happening. – Perceivable

Hint: ListRemoveDuplicates got added with ColdFusion 10. Don't try it on earlier versions. – Lingcod

Answer 4

based on idea of Jason Haritou, but you can do it in pure CF using Struct! (keys matching will be case-insensitive)

this.tags = listToArray(this.tags, ", ");
var tmpStruct = {};

for (var t in this.tags)
    tmpStruct[t] = "";

return structKeyArray(tmpStruct);

However, for small lists, I prefer Antony's solution.

Forlini

just curious, why bother with the if statement? Structs will not allow duplicate values any way, so isn't using a condition a bit redundant here? – Sheridan

Answer 5

In Coldfusion 10 or Railo 4, you could use Underscore.cfc's uniq() function:

_ = new Underscore();

uniqueArray = _.uniq(arrayWithDuplicates);

One advantage of uniq() is that it allows you to pass a transformation function, if necessary.

Note: I wrote Underscore.cfc

Pyxis

Answer 6

I just had to de-dup a very large list (5k+entries) and found a much faster way than using a loop. I feel the need to share.

1. convert list to array (you already have array, so skip)<cfset thisArray = ListToArray(thisList)>
2. Create a query with queryNew("") <cfset thisQuery = QueryNew("")>
3. Add column to that query with the array from step1 <cfset temp = QueryAddColumn(thisQuery,"items","varChar",thisArray)>
4. Query for Distinct values <cfquery name="qItems" dbtype="query">SELECT DISTINCT items FROM thisQuery</cfquery>
5. convert result to list <cfset returnString = ValueList(qItems.items)>
6. It's an easy step for you to convert this list back to an array

I wrote this into a function for easy use:

<cffunction name="deDupList" output="no" returntype="string">
    <cfargument name="thisList" required="yes">
    <cfargument name="thisDelimeter" required="yes" default=",">
    <cfset var loc = StructNew()>

    <cfset loc.thisArray = ListToArray(thisList,thisDelimeter)>
    <cfset loc.thisQuery = QueryNew("")>
    <cfset loc.temp = QueryAddColumn(loc.thisQuery,"items","varChar",loc.thisArray)>
    <cfquery name="qItems" dbtype="query">
        SELECT DISTINCT items FROM loc.thisQuery
    </cfquery>
    <cfset loc.returnString = ValueList(qItems.items)>
    <cfreturn loc.returnString>
</cffunction>

I bench-marked it against a few other methods and here are the results in milliseconds:
Looping over List checking for > 1 instance: 6265
Using Henry's struct method: 2969
The above method: 31
Jason's Method: 30

Hhd

(Edit) I could see how looping would be slow. But I am very surprised you said Jason's method was slow ie using HashSet. It is lightening fast for me with over 10K items. I even cheated and passed the results back into a Vector to make CF happy. – Holliman

I messed up my bench-marking script when I was implementing Jason's method, stupid mistake. You are right his is pretty damn fast, corrected my bench-mark results. Thanks. – Hhd

Thanks. I would probably go with HashSet. But interesting that a QoQ is faster than I would have expected. – Holliman

Answer 7

Taking jason's answer just a little bit further, here is an arrayDistinct function.

function arrayDistinct (required array data) {
    var output = arrayNew(1);
    output.addAll(createObject("java", "java.util.HashSet").init(arguments.data));
    return output;
}

You can test it here: https://trycf.com/gist/62ff904d4500519e3144fc9564d2bce7/acf

Amarelle

Answer 8

Just put the array into a Struct and then copy it back to an array ;)

http://www.bennadel.com/blog/432-Using-ColdFusion-Structures-To-Remove-Duplicate-List-Values.htm

Xerox

Answer 9

I'm posting another neat solution I like.

arrayReduce(arrayWithDuplicates, function(resultArr, item) {
   if(!arrayFind(resultArr, item)){
      arrayAppend(resultArr, item);
   }
   return resultArr;
}, [])

ArrayReduce is available from ColdFusion 11.

But to answer better for the question, from the ColdFusion 10, we have ListRemoveDuplicates function available.

So the final code may looks like this:

this.tags = listToArray(listRemoveDuplicates(this.tags, ", "), ", ");

Walleyed

Answer 10

There are a couple of UDF's on CFLib that do this, ArrayyDiff (http://www.cflib.org/udf/arrayDiff) and ArrayCompare (http://www.cflib.org/udf/arrayCompare).

hth, larry

Cl