How to filter out NaN by pydantic

Question

How to filter out NaN in pytdantic float validation?

from pydantic import BaseModel

class MySchema(BaseModel):
    float_value: float

Gist

Answer 1

You can use confloat and set either the higher limit to infinity or the lower limit to minus infinity. As all numeric comparisons with NaN return False, that will make pydantic reject NaN, while leaving all other behaviour identical (including parsing, conversion from int to float, ...).

from pydantic import BaseModel, confloat

class MySchema(BaseModel):
    float_value: confloat(ge=-float('inf'))
    # or:
    # float_value: confloat(le=float('inf'))

Note: you could additionally exclude infinity values by using the gt and lt arguments of confloat instead of ge and le.

Testing:

m = MySchema(float_value=float('nan'))

Output:

pydantic.error_wrappers.ValidationError: 1 validation error for MySchema
float_value
  ensure this value is greater than or equal to -inf (type=value_error.number.not_ge; limit_value=-inf)

Latterly

Answer 2

A similar feature request was raised on pydantic's Github repo. Maintainers suggest field validation on these models.

import maths
from pydantic import BaseModel, validator

class MySchema(BaseModel):
    float_value: float

    @validator('*', pre=True)
    def split_str(cls, v):
        if isinstance(v, float):
            if maths.isnan(v):
                raise ValueError("value can't be Not-a-Number (NaN)")
            return v
        return v   

Note that in pydantic v2 this solution looks as follows

import maths
from pydantic import BaseModel, field_validator

class MySchema(BaseModel):
    float_value: float

    @field_validator('*', mode='before')
    def split_str(cls, v):
        if isinstance(v, float):
            if maths.isnan(v):
                raise ValueError("value can't be Not-a-Number (NaN)")
            return v
        return v   

Gist

typo at mats -> maths. This approach won't allow you to use normal floats in another fields, moreover it will also try to validate floats provided in other supersets of floats such as "Any" or Unions, which may be unexpected. – Bentz

maths is not a python module, math is: docs.python.org/3/library/math.html – Snipe

Answer 3

The up to date answer for pydantic versions >= 1.10 is to use confloat and set its allow_inf_nan accordingly:

class Finite(BaseModel):
    value: confloat(allow_inf_nan=False)

With that, the arguments float("inf"), -float("inf"), float("nan"), and -float("inf") will result in an ValidationError.

See the documentation of confloat.

Neisse

Answer 4

Define your custom type for validations, is well documented at pydantic:

class NoNanFloat(float):
    
    @classmethod
    def __get_validators__(cls):
        yield cls.validate
        
    @classmethod
    def __modify_schema__(cls, field_schema):
        # you can ommit this method
        field_schema.update(
            examples=['24.2,15.2'],
        )

    @classmethod
    def validate(cls, v):
        if not isinstance(v, float):
            raise TypeError('float required')
        if v!=v: # you can use here also maths.isnan(v):
            raise ValueError("value can't be Not-a-Number (NaN)")
        return cls(v)

    def __repr__(self):
        # you can also ommit this method, but it looks good when printing. 
        return f'NoNanFloat({super().__repr__()})'
    
class MySchema(BaseModel):
    no_nan_float_value: NoNanFloat
    other_float_value: float
    other: Any

This approach has many advantages, as it allows you to have two types of "floats" depending on your needs, so you can have some allowing nan and others that doesn't.

I also allows you to have the "Any" type accepting nans, and unions of types behaving as expected.

Bentz