How Safe Is It To Scrape Stock-Price Information

Question

As far as I can see, there are no developer-API services for stock-price information with the following stipulations:

• Free or low-monthly cost (
• Ability to ship code which uses the service to multiple parties (e.g. an iPhone app)
• No per-transaction/lookup cost
• No limit on number of lookups or transactions

So I find that I am forced to look at scraping. It seems that the easiest thing to 'scrape' is Yahoo's CSV service which was linked to elsewhere on StackOverflow:

E.g.: http://finance.yahoo.com/d/quotes.csv?s=msft&f=price

That's actually very easy to parse. But I wonder how fragile this is, clearly if Yahoo removes this service, or changes things, or limits the number of lookups severely, then the clients I build could be affected.

How would others approach this problem, or is it simply a case that the data is too valuable and that it's not available at my price? I have searched for a fairly reliable, 20-min delayed service, but simply can't find one that looks as though it will still be around in 6 months.

Any advice much appreciated.

Howlend

Answer 1

Did you look at Google's Market Data API?

Update April 2013 - Market Data API no longer available

Belia

Answer 2

"the data is too valuable and that it's not available at my price?"

Correct.

Think about it. Fortunes are made (and lost) with this information. You're competing against giant banks and stock broker services. These are companies with (a) deep pockets to buy, sell and protect this kind of information and (b) incentive to keep this information as private as possible to reinforce their tight control over stock trading.

Collett

Answer 3

You need to be careful. Besides the normal dependencies on the websites ui, I got a warning from a website for doing this. They had a web site policy against screen scraping that I suddenly agreed to by using the site and they saw my regular hits to the site.

Bluepoint

Would it be difficult to utilize rotating proxies to mask and spread out your requests over a bunch of ip's? – Arthropod

Answer 4

There are plenty of low-cost services for this. I like csidata.com.

BUT (and this is a big but)..... if you plan on redistributing the data (posting it on a website, accessing it from an iPhone app, pretty much anything except using it on that one machine), you are going to pay big (as in, bigger than you can imagine) bucks.

The exchanges make their money by selling this data.

If you plan on running a business, do not plan on being able to scrape yahoo or somewhere else and redistributing that. Once they discover you, you will be shut down.

Inspirational

Question. Say someone creates a client application which scrapes the data. This person then sells the client. Would that be considered "redistributing the data" or merely providing a scraper? Note: I have NOT created a scraper. – Peterkin

Answer 5

I'd recommend IQFeed(pay), esignal(pay) or eoddate.com(free options). My company uses IDC, but I can't seem to find any information on the sub services they offer.

Though I get the impression you are trying to avoid paying for a commercial feed, you need to be careful, you get what you pay for.

Almeda