How can I manage units in pandas data?

Question

I'm trying to figure out if there is a good way to manage units in my pandas data. For example, I have a DataFrame that looks like this:

   length (m)  width (m)  thickness (cm)
0         1.2        3.4             5.6
1         7.8        9.0             1.2
2         3.4        5.6             7.8

Currently, the measurement units are encoded in column names. Downsides include:

1. column selection is awkward -- df['width (m)'] vs. df['width']
2. things will likely break if the units of my source data change

If I wanted to strip the units out of the column names, is there somewhere else that the information could be stored?

Torp

I think the best way would be to store it in a Series / dictionary. If you want to somehow link these two, you can add an attribute (df.units = pd.Series({'length' : 'm', 'width': 'm', 'thickness': 'cm'})) -- This may be dangerous though. – Caban

I didn't want to add a full answer since it's not Pandas, but the Astropy package can do this within its Table and units module, you can move over from DataFrame to Astropy Table (atab=astropy.table.Table.from_pandas(df)), and then give each column a unit (e.g. atab['length'].unit = astropy.units.m). I can post a mwe if you are interested, it looks too messy as a comment with lots of code. – Horripilate

Not an answer to your question, but you could use astropy tables to get the functionality of a dataframe-like that can handle units. – Scrutiny

Answer 1

There isn't any great way to do this right now, see github issue here for some discussion.

As a quick hack, could do something like this, maintaining a separate dict with the units.

In [3]: units = {}

In [5]: newcols = []
   ...: for col in df:
   ...:     name, unit = col.split(' ')
   ...:     units[name] = unit
   ...:     newcols.append(name)

In [6]: df.columns = newcols

In [7]: df
Out[7]:
   length  width  thickness
0     1.2    3.4        5.6
1     7.8    9.0        1.2
2     3.4    5.6        7.8

In [8]: units['length']
Out[8]: '(m)'

Rob

What if they just have something like a dtype but for arbitrary units? – Snowstorm

Answer 2

As I was searching for this, too. Here is what pint and the (experimental) pint_pandas is capable of today:

import pandas as pd
import pint
import pint_pandas

ureg = pint.UnitRegistry()
ureg.Unit.default_format = "~P"
pint_pandas.PintType.ureg.default_format = "~P"

df = pd.DataFrame({
    "length": pd.Series([1.2, 7.8, 3.4], dtype="pint[m]"),
    "width": pd.Series([3.4, 9.0, 5.6], dtype="pint[m]"),
    "thickness": pd.Series([5.6, 1.2, 7.8], dtype="pint[cm]"),
})

print(df.pint.dequantify())

     length width thickness
unit      m     m        cm
0       1.2   3.4       5.6
1       7.8   9.0       1.2
2       3.4   5.6       7.8

df['width'] = df['width'].pint.to("inch")

print(df.pint.dequantify())

     length       width thickness
unit      m          in        cm
0       1.2  133.858268       5.6
1       7.8  354.330709       1.2
2       3.4  220.472441       7.8

Ingridingrim

pint_pandas is indeed nice, but the package still comes with many issues, which makes it quite cumbersome to use. – Sadowski

Answer 3

Offer you some methods:

1. pands-units-extension: janpipek/pandas-units-extension: Units extension array for pandas based on astropy
2. pint-pandas: hgrecco/pint-pandas: Pandas support for pint

you can also extend the pandas by yourself following this Extending pandas — pandas 1.3.0 documentation

Tabret