Extract Decimal from Decimal128 with Mongoose - MongoDB

Question

I'm querying Mongo in Nodejs with Mongoose and attempting to extract the numeric value of multiple fields stored as a Decimal128. However, the value is oddly wrapped in query results and I'm not sure how to extract it through Mongo or Mongoose:

{data:[
  {
  "date": {
          "$numberDecimal": "1530057600000"
  },
  "open": {
          "$numberDecimal": "86.13"
  },
  "high": {
          "$numberDecimal": "86.63"
  },
  "low": {
          "$numberDecimal": "85.47"
  },
  "close": {
          "$numberDecimal": "85.64"
  },
  "volume": {
          "$numberDecimal": "308508"
  }
},

Is there a way I can use Mongo or Mongoose to convert the above JSON query-result into what's below?

{data:[
 {
  "date": 1530057600000
  "open": 86.13
  "high": 86.63
  "low": 85.47
  "close": 85.64
  "volume": 308508
 },

I tried selecting the fields as follows ,but this didn't work.

    data[i].date.$numberDecimal, 
    data[i].open.$numberDecimal,
    data[i].high.$numberDecimal,
    data[i].low.$numberDecimal, 
    data[i].close.$numberDecimal 

Here's my Mongoose schema:

Folder - Model - Stock.js

const mongoose = require('mongoose')
mongoose.Promise = global.Promise

const childSchemaData = new mongoose.Schema({
  "_id": false,
  date: {type: mongoose.Types.Decimal128},
  open: {type: mongoose.Types.Decimal128},
  high: {type: mongoose.Types.Decimal128},
  low: {type: mongoose.Types.Decimal128},
  close: {type: mongoose.Types.Decimal128},
  volume: {type: mongoose.Types.Decimal128}
})

const parentSchemaSymbol = new mongoose.Schema({
  "_id": false,
  symbol: {
    type: String,
    trim: true,
    minlength: 2,
    maxlength: 4,
    uppercase: true,
    required: 'Plese enter a valid symbol, min 2 characters and max 4'
  },
  // Array of subdocuments
  data: [childSchemaData],
  slug: String

})

module.exports = mongoose.model('Stock', parentSchemaSymbol)

Controller

const mongoose = require('mongoose')
const parentSchemaSymbol = mongoose.model('Stock')

exports.dbFetch = (req, res) => {
  let curValueDbFetch = req.params.symbol

  const query = { symbol: `${curValueDbFetch}` }
  const projection = { _id: 0, data: 1 }

  parentSchemaSymbol.findOne(query, projection).then(doc => {
    return res.send(doc)
  }).catch(e => {
    console.log(e)
  })
}

I am sending the data to the front end and this is what I am receiving in the browser:

SOLUTION

const mongoose = require('mongoose')
const parentSchemaSymbol = mongoose.model('Stock')

exports.dbFetch = (req, res) => {
  let curValueDbFetch = req.params.symbol

  const query = { symbol: `${curValueDbFetch}` }
  const projection = { _id: 0, data: 1 }

  parentSchemaSymbol.findOne(query, projection).sort({ date: -1 }).then(doc => {
    let chartData = doc.data.map(item => {
      return {
        date: parseFloat(item.date), // the date
        open: parseFloat(item.open), // open
        high: parseFloat(item.high), // high
        low: parseFloat(item.low), // low
        close: parseFloat(item.close), // close
        volume: parseFloat(item.volume)// volume
      }
    })
    res.send(chartData)
  })
    .catch(e => {
      console.log(e)
    })
}

Jakie

You should not need to do anything. There is always a toString() on JavaScript objects, and that is basically implied from any access in a string context. You can parseFloat() on any value, which will again actually imply the toString(), but of course the whole point of Decimal128 is that the values are meant to be too large and precise for a float. Therefore toString() should generally be the way. Or of course accept the extended JSON form of { $numberDecimal: "123.45" } as actually being the "correct" implementation indicating "type". – Addiction

@NeilLunn you mean changing Schema to date: { type: mongoose.Types.Decimal128 } – Jakie

@NeilLunn can you come up with the code example cause nothing is changing – Jakie

Not sure what you are talking about. Were are you sending the data? res.send(doc) is really just a stringified object. res.json(doc) is the JSON stringified form of the data. At either rate, both should be calling toString(). Perhaps show an MCVE with current output and expected output if you think something different or still don't understand what I am saying. – Addiction

@NeilLunn I am sending the data to the View layer to generate the stock chart. – Jakie

Dude. MCVE. We cannot debug your whole application. Provide the "minimal" code required to reproduce the result, and show what you expect to see. – Addiction

A simple solution - https://mcmap.net/q/1164894/-how-to-translate-decimal128-values-over-rest – Sublunar

Answer 1

Method 1: .

use toString(). It will convert the object to string.

find((docs) => {
   let result = docs.map((doc) => {
       if(doc.open){
          doc.open = doc.open.toString();
       }

       if(doc.close){
          doc.close = doc.close.toString();
       }

       return doc;  
   });

    //send modified output
    res.json(result);
})

output as follows:-

/*
[
  {
    "open":  "86.13",
    "close": "85.64"
  },
]
*/

Method 2: Mongodb 4.0 above,

db.myCollection.aggregate([
  {$match:{
   //...
   //...
   }},


  { $addFields : {
        open: {"$toString" : "$open"},
        close : {"$toString" : "$close"},
    }},
]);

Magnetostriction

Answer 2

This will work with any field!

It supports subdocument and arrays of subdocuments too

const MySchema = new Schema({/*... schema fields ...*/});


const decimal2JSON = (v, i, prev) => {
  if (v !== null && typeof v === 'object') {
    if (v.constructor.name === 'Decimal128')
      prev[i] = v.toString();
    else
      Object.entries(v).forEach(([key, value]) => decimal2JSON(value, key, prev ? prev[i] : v));
  }
};

MySchema.set('toJSON', {
  transform: (doc, ret) => {
    decimal2JSON(ret);
    return ret;
  }
});

mongoose.model('MyModel', MySchema);

Usage:

MyModel.findOne().then(data => console.log(data.toJSON());

Gans

Use this answer especially if you are defining and exporting your schema models in node. – Overstreet

Answer 3

Working solution

const mongoose = require('mongoose')
const parentSchemaSymbol = mongoose.model('Stock')

exports.dbFetch = (req, res) => {
  let curValueDbFetch = req.params.symbol

  const query = { symbol: `${curValueDbFetch}` }
  const projection = { _id: 0, data: 1 }

  parentSchemaSymbol.findOne(query, projection).sort({ date: -1 }).then(doc => {
    let chartData = doc.data.map(item => {
      return {
        date: parseFloat(item.date), // the date
        open: parseFloat(item.open), // open
        high: parseFloat(item.high), // high
        low: parseFloat(item.low), // low
        close: parseFloat(item.close), // close
        volume: parseFloat(item.volume)// volume
      }
    })
    res.send(chartData)
  })
    .catch(e => {
      console.log(e)
    })
}

Jakie

Answer 4

This is very easy with lodash _.cloneDeepWith(). Iterate over every object property and transform objects with $numberDecimal property to a string.

// first flattenDecimals using mongoose `toJSON()`
var objectStep1= dbResult.toJSON({flattenDecimals: true});
//  use lodash _.cloneDeepWith() to iterate over every object property
var returnThisObject = _.cloneDeepWith(objectStep1, propVal =>{
    if (_.has(propVal, '$numberDecimal')) return propVal.$numberDecimal;
});

Alternatively you could do it this way below without toJSON(), but I think it would be less efficient since the mongoose result has so many properties which are not part of the result. And then you would need to check for undefined properties too.

var returnThisObject = _.cloneDeepWith(dbResult, propVal =>{
    if (!propVal) return propVal; // check for undefined 
    if ('Decimal128' == propVal ._bsontype) return propVal.toString();
});

Moreland

Answer 5

In my case, value undefined even db document have key, because in schema i forgot to add that field, but i trying to read the value.

Igraine

Answer 6

Another option using JS (with TypeScript)

function mutateDocumentWithNumberDecimals<T extends { $numberDecimal?: string } | Record<string, any>>(
  obj: T
): any {
  if (!obj || typeof obj !== "object") return obj;
  if (Array.isArray(obj)) {
    return obj.map(mutateDocumentWithNumberDecimals);
  }
  if (Object.hasOwn(obj, "$numberDecimal")) {
    // check if possible to keep precision when converting to a number, if so, return a number
    const numberDecimalValue = obj["$numberDecimal"];
    if (typeof numberDecimalValue === "string") {
      const numberValue = Number(numberDecimalValue);
      if (numberValue.toString() === numberDecimalValue) {
        return numberValue;
      }
    }
    return numberDecimalValue;
  } else {
    for (const key in obj) {
      obj[key] = mutateDocumentWithNumberDecimals(obj[key]);
    }
  }
  return obj;
}

// Usage:
const mutated = mutateDocumentWithNumberDecimals(obj);

Frankpledge