How can I mask elements of a record array in Numpy?

Question

I understand how to create a masked array, and I would like to use masking in a record array so that I can access this data using named attributes. The masking seems to be "lost" when I create a record array from a masked array:

>>> data = np.ma.array(np.ma.zeros(30, dtype=[('date', '|O4'), ('price', '<f8')]),mask=[i<10 for i in range(30)])
>>> data
masked_array(data = [(--, --) (--, --) (--, --) (--, --) (--, --) (--, --) (--, --) (--, --)
(--, --) (--, --) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0)
(0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0)],
         mask = [(True, True) (True, True) (True, True) (True, True) (True, True)
(True, True) (True, True) (True, True) (True, True) (True, True)
(False, False) (False, False) (False, False) (False, False) (False, False)
(False, False) (False, False) (False, False) (False, False) (False, False)
(False, False) (False, False) (False, False) (False, False) (False, False)
(False, False) (False, False) (False, False) (False, False) (False, False)],
   fill_value = ('?', 1e+20),
        dtype = [('date', '|O4'), ('price', '<f8')])

>>> r = data.view(np.recarray)
>>> r
rec.array([(0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0),
           (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0),
           (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0),
           (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0),
           (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0), (0, 0.0)], 
           dtype=[('date', '|O4'), ('price', '<f8')])

When I access a record the data is not masked:

>>> r.date[0]
0

Unlike in the original array:

>>> data['date'][0]
masked_array(data = --,
             mask = True,
       fill_value = 1e+20)

       fill_value = 1e+20)

What can I do? Does the record array not support masking? Browsing on the web I have seen some code examples that seem to suggest otherwise, but it wasn't very clear. Hoping I can get a good answer here.

Mccray

Answer 1

I haven't found much documentation on numpy.ma.mrecords.MaskedRecords, except for a brief mention here. You can find some examples on how to use it by studying the unit tests that come with numpy. (e.g. /usr/lib/python2.6/dist-packages/numpy/ma/tests/test_mrecords.py).

import numpy as np
import numpy.ma.mrecords as mrecords

data = np.ma.array(
    np.ma.zeros(30, dtype=[('date', '|O4'), ('price', '<f8')]),
    mask=[i<10 for i in range(30)])

r = data.view(mrecords.mrecarray)

print(r.date[0])
# --

Lubricity

Thanks, I didn't know about numpy.ma.mrecords. – Mccray

I don't suppose there is a way to mask individual fields? My use case is to add a new derived field called "100-day-high" (or something similar) so I would want to mask this field for the first 100 records. – Mccray

@Nate Reed: I think it is possible. Have you tried something like data.mask['price'][data['date']<...]=True ? – Lubricity

Yes, either of these works: data.mask[0]['price'] = True, or data.mask['price'][0] = True. Thanks! – Mccray