Import data from URL

Question

The St. Louis Federal Reserve Bank has a great set of data available on a variety of their web pages, such as:

http://research.stlouisfed.org/fred2/series/OILPRICE/downloaddata?cid=32217 http://www.federalreserve.gov/releases/h10/summary/default.htm http://research.stlouisfed.org/fred2/series/DGS20

The data sets get updated, some as often as daily. I tend to have an interest in the daily data (see the above settings on the URLS)

I'd like to import these kinds of price or rate data streams (accessible as CSV or Excel files at the above URLs) directly into Mathematica.

I've looked at the documentation on Importing[] but I find scant documentation (actually none) on how to go about something like this.

It looks like I need to navigate to the pages, send some data to select specific files and formats, trigger the download, then access the downloaded data from my own machine. Even better if I could access the data directly from the sites.

I had hoped Wolfram Alpha might make this sort thing easy, but I haven't had any success.

FinancialData[] would seem natural for this sort of thing, but I don't see anyway to do it. Financial data has lots of features, but I don't see a way yo get this sort of thing.

Does anyone have any experience with this or can someone point me in the right direction?

Repatriate

To the close voter, this is about programming in Mathematica and is not off topic. Votes such as this are one of the reasons there is a proposal for a Mathematica-specific site. – Maxim

Answer 1

You can Import directly from a URL. For example, the data from federalreserve.gov can be obtained and visualized as follows.

url = "http://www.federalreserve.gov/datadownload/Output.aspx?";
url = url<>"rel=H10&series=a660e724c705cea4b7bd1d1b85789862&lastObs=&";
url = url<>"from=&to=&filetype=csv&label=include&layout=seriescolumn";
data = Import[url, "CSV"];
DateListPlot[data[[7 ;;]], Joined -> True]

I broke up url for convenience, since it's so long. I had to examine the contents of data before I knew exactly how to plot it - a step that is typically necessary. I'm sure that the data from stlouisfed.org can be obtained in a similar way, but it requires the use of an API with key to access it.

Ampoule

Thanks Mark. The trick seems to be viewing the pages' source code and identifying the right URL for the data I need. – Repatriate

An example I found helpful for figuring out this problem is in the DateListPlot documentation, which is actually using information on oil consumption since 1980. – Tow

Hi Mark, nice answer and great to see you're back! – Alcatraz

@Leonid - I didn't know I left! Sometimes life gets busy and I don't respond too much, but I frequently lurk. And I'm excited about rcollyer's Area51 initiative! – Ampoule

@Mark A few times during the last few months I found myself at your profile page and it was always saying "last seen on Sep. 4 2011", which is why I assumed you were away. I have auto-login, and so assumed everyone does too, which is oviously a wrong assumption. On another matter, a dedicated SE Mathematica site would be very nice indeed! – Alcatraz

Answer 2

As Mark said, you can get the data directly from a URL. Your oil data can be imported from a different URL than you had:

http://research.stlouisfed.org/fred2/data/OILPRICE.txt

With that URL, you can do this:

oil = Import["http://research.stlouisfed.org/fred2/data/OILPRICE.txt",
"Table", "HeaderLines" -> 12, "DateStringFormat" -> {"Year", "Month", "Day"}];
DateListPlot[oil, Joined -> True, PlotRange -> All]

Note that "HeaderLines"->12 option strips off the header text in the first 12 lines (you have to count the header lines to know how many to remove). I've also specified the date format.

To find that URL, do as you did before, but click on a data series and then choose View Data from the menu on the left when you see the chart.

Panama

Tim -- Thanks for the refinement, nice to get all the formatting done in one pass. – Repatriate

@TimMayes We have a proposal for a mathematica specific site for anything related to mathematica. It would be nice if you could commit (involves creating an Area51 account) to the proposal. We're pretty close to launching (only need about 50 more users). My apologies if you have already done so – Behavior

@TimMayes using the API is preferable because you do not need to know in advance how many lines to strip away etc. All you need is the API key which is free. It also eliminates the need to first find the URL on the FRED website prior to using Import – Palindrome

@Mike, thanks. I've downloaded your notebook and am trying it out. I think it will be very useful for me. – Panama

@TimMayes no worries. It was written in a context of finance users wanting longer (than How Tos) worked examples of these sort of things. So the emphasis was stepping through the problem. To make it as easy as possible to follow I used the same options names in the function as in the API. Only remembered that when I posted the code above. I use this regularly but use it via an AppleScript schedule so haven't actually looked at any code for 2 years. – Palindrome

Answer 3

The documentation has a short example on extracting data out of a webpage:

http://reference.wolfram.com/mathematica/howto/CleanUpDataImportedFromAWebsite.html

Of course, what actually needs to be done will vary significantly from page to page.

Majolica

Answer 4

discussion on how to do this with your API key here:

http://library.wolfram.com/infocenter/MathSource/7583/

the function is based on the API documentation. I haven't looked at the code for a couple of years and from memory I put it together rather quickly but I have used it regularly for over 2 years without problems. Here is an example for monthly non seasonally adjusted retail sales from early 1992 to now:

wolfram alpha also uses FRED data so you could use that as an alternative to direct import but it is more tricky to get the query right. I prefer to use FRED directly. Also from memory the data is only available on alpha the day after the release, which is not what you would typically want.

Palindrome