Huge amount of plaintext data for parsing experiment

Question

I am developing a parser in ruby which parses some nonuniform text data. Can anybody tell me, where I can get a good number of plaintext data for that?

Roundly

Answer 1

Here's you'll get a list of many:

http://www.quora.com/Data/Where-can-I-get-large-datasets-open-to-the-public

And my fav is:

http://ftp.sunet.se/mirror/archive/ftp.sunet.se/pub/tv+movies/imdb/

Migrate

As long as amazon us-east-1d is up :) – Bier

Answer 2

You could scrape Wikipedia (or just run a bunch of it through lynx -dump). That would also give you a vast source of non-English text as well. Project Gutenberg would be another good source of large amounts of plain text.

Elenore

@Phrogz: I used to be a Gutenberg addict back in my "Palm Pilot and commuting on the bus" days. – Elenore

Project Gutenberg as a very strict bot policy, they allow no more than 100 visits from the same ip address in a day. – Raymund

@kyle k That's ok. They have a torrent: gutenberg.org/wiki/Gutenberg:The_CD_and_DVD_Project – Songsongbird