Can't get Scrapy pipeline to work

Question

I have spider that I have written using the Scrapy framework. I am having some trouble getting any pipelines to work. I have the following code in my pipelines.py:

class FilePipeline(object):

    def __init__(self):
        self.file = open('items.txt', 'wb')

    def process_item(self, item, spider):
        line = item['title'] + '\n'
        self.file.write(line)
        return item

and my CrawlSpider subclass has this line to activate the pipeline for this class.

ITEM_PIPELINES = [
        'event.pipelines.FilePipeline'
    ]

However when I run it using

scrapy crawl my_spider

I get a line that says

2010-11-03 20:24:06+0000 [scrapy] DEBUG: Enabled item pipelines:

with no pipelines (I presume this is where the logging should output them).

I have tried looking through the documentation but there doesn't seem to be any full examples of a whole project to see if I have missed anything.

Any suggestions on what to try next? or where to look for further documentation?

Rayerayfield

Answer 1

Got it! The line needs to go in the settings module for the project. Now it works!

Rayerayfield

No, my client may have put it up somewhere, but not my place to open source it I'm afraid. – Rayerayfield

i had to use ITEM_PIPELINES as a dict instead of a list in the settings file – Faust

Answer 2

I'm willing to bet that it's a capitalisation difference in the word pipeline somewhere:

Pipeline vs. PipeLine

I notice 'event.pipelines.FilePipeline' uses the former, whereas your code uses the latter: which do your filenames use?

(I have fallen victim to this spelling mistake many times!)

Interpol

Thanks for spotting that (can't believe I missed it!), but it still doesn't work unfortunately – Rayerayfield

It's still my bet at what's wrong, keep checking! :P (if not, I'm out of ideas...) – Interpol

I'll have another look. Cheers! – Rayerayfield