Is there any solution to get score of similarity between lists of words?

Question

I want to calculate the similarity between lists of words, for example :

import math,re
from collections import Counter

test = ['address','ip']
list_a = ['identifiant', 'ip', 'address', 'fixe', 'horadatee', 'cookie', 'mac', 'machine', 'network', 'cable']
list_b = ['address','city']

def counter_cosine_similarity(c1, c2):
    terms = set(c1).union(c2)
    print(c2.get('ip',0)**2)
    dotprod = sum(c1.get(k, 0) * c2.get(k, 0) for k in terms)
    magA = math.sqrt(sum(c1.get(k, 0)**2 for k in terms))
    magB = math.sqrt(sum(c2.get(k, 0)**2 for k in terms))
    return dotprod / (magA * magB)

counter1 = Counter(test)
counter2 = Counter(list_a)
counter3 = Counter(list_b)
score = counter_cosine_similarity(counter1,counter2)
print(score) # output : 0.4472135954999579
score = counter_cosine_similarity(counter1,counter3)
print(score) # output : 0.4999999999999999

For me it's not exactly the score I want to get, the score must be the opposite because list_a contains address and ip so it's a 100% test match I know that cosine similarity does the comparison in this case with test and list_a so since there is some element on the list_a which is not in test it is for that the score is low, so that I will do exactly it is compared that test compared to list_a in one way not in the two way.

Desired output

score = counter_cosine_similarity(counter1,counter2)
print(score) # output : score higher than list_b = 1.0 may be
score = counter_cosine_similarity(counter1,counter3)
print(score) # output : score less the list_a = 0.5 may be

Rupee

Where is the source code for Counter? – Remnant

@Aaron Digulla just edited you can check the code now – Rupee

Why do you use Counter? It just tells you how often each word appears in a list. So in your case, that will be 1 for each term. How does that help to determine "distance"? – Remnant

@Aaron Digulla sometimes i want to test = ['address','address'] to compare with list_a and list_b so for me address address i'ts about location so the score must be higher when i compare test with list_b, but what i need to do in this cas to have the correct result address in list_b to have that mean the frequence of address is high in list_b. – Rupee

This is not a question on python or numpy. You need to define, in a mathematically strict sense, what the "similarity" is for your current purpose. Then you can implement it in python or any other programming language. – Predicative

Answer 1

If you want a higher value the more terms are the same, use this code:

 score = len(set(test).intersection(set(list_x)))

That will tell you how many common terms the two lists have. If you want to score repetitions higher, then try

 commonTerms = set(test).intersection(set(list_x))
 counter = Counter(list_x)
 score = sum((counter.get(term) for term in commonTerms)) #edited

If you need scaling the score to [0..1], I need to know more about your data sets.

Remnant