How to pull bank statements from websites?

Asked 7/4, 2011 at 6:6 Answered 15/6, 2016 at 2:19

I need to pull bank account statements from a number of different sources. I'd love to be able to get this data in some sort of standardized fashion, be it CSV or OFX or XML or whatever. I just need to be able to get this data. I'll be using my own account to grab transactions from, so there shouldn't be too many 'legal' problems there. How do sites like Mint.com get bank statements? They simply ask for your account information, the URL, and then they're off and running. How can I do this?

Wellesz answered 7/4, 2011 at 6:6 Comment(0)

Write a screen scraper in your favorite language to do it. Automate it away.

Normally banks give you access up to three months of transactions via their online services.

Don't forget to make it push the panic button when the screen scraping fails.

The main point here is to automate it so that the pain (of running it manually) goes away so that all you see are "constantly" updated transactions of your accounts.

Mackinaw answered 7/4, 2011 at 6:28 Comment(6)

I'm completely new to screen scraping. I'm writing my application in Python, any idea on where to start, module-wise? I found Bankjob (a Ruby scraper), but I really need it in Python. My bank does support exporting transactions to OFX, but there's not specifically a way to automate it other than via scraping. Thanks. – Wellesz 7/4, 2011 at 19:5

Have a look at beautiful soup: crummy.com/software/BeautifulSoup. It's pretty decent. – Mackinaw 8/4, 2011 at 0:24

How do you write a screen scraper that will deal with the fact that some banks have complicated javascript that runs during the login process and the need to run the scraper from the command line without a GUI? – Amandie 25/5, 2012 at 21:5

Nowadays we can run webkit headless with JavaScript engine and all. Check out phantomjs – Mackinaw 26/5, 2012 at 0:48

I believe scraping is the way to go (in the absence of direct feeds). I have created a project for parsing HTML tables of financial transactions: github.com/bunsn/boiler. To add support for your own bank, you write a basic “statement definition”. It doesn’t handle logging in or crawling multiple statements but may be useful as part of a scraping tool. – Unicameral 28/9, 2015 at 9:27

holygeek, @DomChristie: How do you actually download the statements (PDFs) using something like PhantomJS? Doesn't PhantomJS completely fail at downloading? – Mcrae 14/2, 2017 at 8:53

If you're interested in a simple, personal solution, one easy way would have Mint aggregate all your purchases and use an existing scraper to download them. Something like this: https://github.com/mrooney/mintapi.

Viridian answered 2/9, 2015 at 17:45 Comment(0)

The other main things you may consider is Plaid (more commercial product but looks really good) And https://www.buxfer.com which has a free api. Mint as of late has made it harder for scrapers to scrape it.

Notus answered 15/6, 2016 at 2:19 Comment(0)

Grab your transactions and save them to CSV of XLS to convert to OFX and / QIF using iCreateOFX Basic v4 or try the v5 beta.

Davinadavine answered 24/11, 2014 at 13:48 Comment(0)

-1

I read a comprehensive technical article on this awhile ago, however, I can't find it. From my memory this is what mint does to get data from your bank:

Query API from this company: http://www.finappstore.com/
If bank isn't in above API they fallback to screen scraping

Dobsonfly answered 7/4, 2011 at 6:30 Comment(3)

I'm somewhat confused as to how that website helps out any. Do they have an API for getting statements? ie: is it worth my time to register with them? ;) – Wellesz 7/4, 2011 at 19:7

I thought that the finappstore API is only available from flex through their code/protocol. – European 4/6, 2011 at 16:24

This website is dead – Dandiprat 15/9, 2018 at 5:20

Recommended topics

Hot tags