OSError 24 (Too many open files) when reading bunch of FITS with astropy.io

Question

I’m trying to load into memory a few 2 000 FITS using astropy.io.fits:

def readfits(filename):
    with fits.open(filename) as ft:
        # the fits contain a single HDU
        data = ft[0].data
    return data

data_sci = []
for i in range(2000):
    data_sci.append(readfits("filename_{}.fits".format(i)))

However, when reaching the 1015th file, OSError: [Errno 24] Too many open files is raised.

I have the same issue with:

def readfits(filename):
    ft = fits.open(filename) as ft:
    data = ft[0].data
    ft.close()
    return data

I suspect that astropy.io.fits does not properly close the file. Is there a way I can force the files to be closed?

Percept

Answer 1

Your readfits function actually leaves the file handle open in order to keep access to the data, because by default it creates a mmap to the data and does not read it entirely into physical memory, as explained: http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files

Incidentally, if you just want a function that reads the data out of the first HDU this is already built in: http://docs.astropy.org/en/v1.0.5/io/fits/api/files.html#astropy.io.fits.getdata

It's not necessary to reinvent the wheel.

Raasch

Answer 2

After taking a look at the astropy documentation i found this: http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files

Dwight

Thanks, this was the info I was searching for! The solution I used was a bit different though: as I needed to keep the data in memory, I couldn’t blindly use del. I simply replaced the line data = ft[0].data with data = ft[0].data.copy() instead. – Percept

If you do data.copy() then you're loading the entire data into memory anyways, so you might as well just open the files with memmap=False and then there's no issue. Also if the data consists of tables there was an issue, now fixed, that resulted in a resource leak. – Raasch

@Iguananaut, definitely buying that memmap=False. I should have read more carefully the doc of open… – Percept

It's a tricky thing; easy to miss. Opening lots of files simultaneously is not the most common case, but it's common enough that there should probably be a better interface for that. – Raasch

Answer 3

You can call this function and store its output as long as you have memory. I thought it worths mentioning the answer explicitly but the credit goes to Iguananaut, bkaf, and this page.

def get_single_fits_data(fits_dir):

    hdul  = fits.open(fits_dir)
    for hdu in hdul:
        image_data = hdu.data.copy()
    hdul.close()
    gc.collect()

    return image_data

Gunpowder