Creating a dictionary for each word in a file and counting the frequency of words that follow it

Question

I am trying to solve a difficult problem and am getting lost.

Here's what I'm supposed to do:

INPUT: file
OUTPUT: dictionary

Return a dictionary whose keys are all the words in the file (broken by
whitespace). The value for each word is a dictionary containing each word
that can follow the key and a count for the number of times it follows it.

You should lowercase everything.
Use strip and string.punctuation to strip the punctuation from the words.

Example:
>>> #example.txt is a file containing: "The cat chased the dog."
>>> with open('../data/example.txt') as f:
...     word_counts(f)
{'the': {'dog': 1, 'cat': 1}, 'chased': {'the': 1}, 'cat': {'chased': 1}}

Here's all I have so far, in trying to at least pull out the correct words:

def word_counts(f):
    i = 0
    orgwordlist = f.split()
    for word in orgwordlist:
        if i<len(orgwordlist)-1:
            print orgwordlist[i]
            print orgwordlist[i+1]

with open('../data/example.txt') as f:
    word_counts(f)

I'm thinking I need to somehow use the .count method and eventually zip some dictionaries together, but I'm not sure how to count the second words for each first word.

I know I'm nowhere near solving the problem, but trying to take it one step at a time. Any help is appreciated, even just tips pointing in the right direction.

Healthful

f.split(). f is a file handler, not a string. – Diaspora

Answer 1

We can solve this in two passes:

1. in a first pass, we construct a Counter and count the tuples of two consecutive words using zip(..); and
2. then we turn that Counter in a dictionary of dictionaries.

This results in the following code:

from collections import Counter, defaultdict

def word_counts(f):
    st = f.read().lower().split()
    ctr = Counter(zip(st,st[1:]))
    dc = defaultdict(dict)
    for (k1,k2),v in ctr.items():
        dc[k1][k2] = v
    return dict(dc)

Diaspora

Nit: not stripping punctuation. – Capitalize

Answer 2

We can do this in one pass:

1. Use a defaultdict as a counter.
2. Iterate over bigrams, counting in-place

So... For the sake of brevity, we'll leave the normalization and cleaning out:

>>> from collections import defaultdict
>>> counter = defaultdict(lambda: defaultdict(int))
>>> s = 'the dog chased the cat'
>>> tokens = s.split()
>>> from itertools import islice
>>> for a, b in zip(tokens, islice(tokens, 1, None)):
...     counter[a][b] += 1
...
>>> counter
defaultdict(<function <lambda> at 0x102078950>, {'the': defaultdict(<class 'int'>, {'cat': 1, 'dog': 1}), 'dog': defaultdict(<class 'int'>, {'chased': 1}), 'chased': defaultdict(<class 'int'>, {'the': 1})})

And a more readable output:

>>> {k:dict(v) for k,v in counter.items()}
{'the': {'cat': 1, 'dog': 1}, 'dog': {'chased': 1}, 'chased': {'the': 1}}
>>>

Hyposensitize

the cat chased the dog is the correct sentence :D – Thermae

Answer 3

Firstly that is some brave cat who chased a dog! Secondly it is a little tricky because we don't interact with this type of parsing every day. Here's the code:

k = "The cat chased the dog."
sp = k.split()
res = {}
prev = ''
for w in sp:
    word = w.lower().replace('.', '')
    if prev in res:
        if word.lower() in res[prev]:
            res[prev][word] += 1
        else:
            res[prev][word] = 1
    elif not prev == '':
        res[prev] = {word: 1}
    prev = word
print res

Distichous

"because we don't interact with this type of parsing every day". We dont? This is basic processing in natural language processing. This is called a 2-gram. – Diaspora

Thanks! Now I know! – Distichous

@WillemVanOnsem aka Bigram – Hyposensitize

Answer 4

You could:

1. Create a list of stripped words;
2. Create word pairs with either zip(list_, list_[1:]) or any method that iterates by pairs;
3. Create a dict of first words in the pair followed by a list of the second word of the pair;
4. Count the words in the list.

Like so:

from collections import Counter
s="The cat chased the dog."
li=[w.lower().strip('.,') for w in s.split()] # list of the words
di={}                                         
for a,b in zip(li,li[1:]):                    # words by pairs
    di.setdefault(a,[]).append(b)             # list of the words following first

di={k:dict(Counter(v)) for k,v in di.items()} # count the words
>>> di
{'the': {'dog': 1, 'cat': 1}, 'chased': {'the': 1}, 'cat': {'chased': 1}}

If you have a file, just read from the file into a string and proceed.

---

Alternatively, you could

1. Same first two steps
2. Use a defaultdict with a Counter as a factory.

Like so:

from collections import Counter, defaultdict
li=[w.lower().strip('.,') for w in s.split()]
dd=defaultdict(Counter)
for a,b in zip(li, li[1:]):
    dd[a][b]+=1

>>> dict(dd)
{'the': Counter({'dog': 1, 'cat': 1}), 'chased': Counter({'the': 1}), 'cat': Counter({'chased': 1})}

Or,

>>> {k:dict(v) for k,v in dd.items()}   
{'the': {'dog': 1, 'cat': 1}, 'chased': {'the': 1}, 'cat': {'chased': 1}}

Capitalize

Answer 5

I think this is a one pass solution without importing defaultdict. Also it has punctuation stripping. I have tried to optimize it for large files or repeated opening of files

from itertools import islice

class defaultdictint(dict):
    def __missing__(self,k):
        r = self[k] = 0
        return r

class defaultdictdict(dict):
    def __missing__(self,k):
        r = self[k] = defaultdictint()
        return r

keep = set('1234567890abcdefghijklmnopqrstuvwxy ABCDEFGHIJKLMNOPQRSTUVWXYZ')

def count_words(file):
    d = defaultdictdict()
    with open(file,"r") as f:
        for line in f:
            line = ''.join(filter(keep.__contains__,line)).strip().lower().split()
            for one,two in zip(line,islice(line,1,None)):
                d[one][two] += 1
    return d

print (count_words("example.txt"))

will output:

{'chased': {'the': 1}, 'cat': {'chased': 1}, 'the': {'dog': 1, 'cat': 1}}

Seavir

You are not closing the open file – Literacy

@MatiasCicero ah I was wondering if that was the case, fixed now I think – Seavir