Construct pandas dataframe from a .fits file

Question

I have a .fits file that contains data.

I would like to construct a pandas dataframe from this particular file but I don't know how to do it.

data = fits.open('datafile')
data.info

gives:

No.    Name         Type      Cards   Dimensions   Format
0    PRIMARY     PrimaryHDU       6   (12, 250000)   float64 

and:

data[0].data.shape

gives:

(250000, 12)

Offing

impossible to answer unless you include a sample of this file and what the desired result should look like – Phippen

It is a dataset with 250 000 rows that represent people, and 12 features which represent their characteristics. So I need a simple 250000x12 dataframe – Offing

So have you tried df = pd.read_csv('datafile')? – Phippen

Yes. I get a UnicodeDecode error. – Offing

FITS is a mixed text and binary file format used primarily in astronomy. read_csv has nothing to do with it. – Myramyrah

Answer 1

According to what you have in your question and the astropy docs (http://docs.astropy.org/en/stable/io/fits/), it looks like you just need to do:

from astropy.io import fits
import pandas
with fits.open('datafile') as data:
    df = pandas.DataFrame(data[0].data)

Edit: I don't have much experience we astropy, but other have mentioned that you can read the fits files into a Table object, which has a to_pandas() method:

from astropy.table import Table
dat = Table.read('datafile', format='fits')
df = dat.to_pandas()

Might be worth investigating.

http://docs.astropy.org/en/latest/table/pandas.html

Brezin

Just a minor comment here: You should close the file if you explicitly open it, or use the contextmanager: with fits.open('datafile') as data: so you don't leave open file handles flapping around. – Backbreaking

Answer 2

Note: the second option with Table is better for most cases, since the way FITS files store data is big-endian, which can cause problems when reading into a DataFrame object which is little-endian. See https://github.com/astropy/astropy/issues/1156

Ferriage