Differences between Node standard https and node-fetch when providing a certificate

Question

ADP's REST API requires that a SSL certificate and private key be sent with every request.

When I use the 'standard, Node.js HTTP(S) module:

require('dotenv').config()

const fs = require('fs')
const path = require('path')

const certificate_path = path.resolve('../credentials/certificate.pem')
const private_key_path = path.resolve('../credentials/private.key')

const options = {
    hostname: 'api.adp.com',
    path: '/hr/v2/workers/ABCDEFGHIJKLMNOP',
    method: 'GET',
    headers: {
        'Accept': 'application/json;masked=false',
        'Authorization': `Bearer ${process.env.ACCESS_TOKEN}`
    },
    cert: fs.readFileSync(certificate_path, "utf8"),
    key: fs.readFileSync(private_key_path, "utf8"),
};

require('https').get(options, res => {

  let data = [];

  res.on('data', chunk => {
    data.push(chunk);
  });

  res.on('end', () => {

    const workers = JSON.parse(Buffer.concat(data).toString());

    for(worker of workers.workers) {
      console.log(`Got worker with id: ${worker.associateOID}, name: ${worker.person.legalName.formattedName}`);
    }

  });

}).on('error', err => {
  console.log('Error: ', err.message);
});

The request works as expected:

$ node ./standard.js
Got worker with id: ABCDEFGHIJKLMNOP, name: Last, First

However, when I use node-fetch:

require('dotenv').config()

const fs = require('fs')
const path = require('path')

const certificate_path = path.resolve('../credentials/certificate.pem')
const private_key_path = path.resolve('../credentials/private.key')

const url = 'https://accounts.adp.com/hr/v2/workers/ABCDEFGHIJKLMNOP'

const options = {
  headers: {
    'Accept': 'application/json;masked=false',
    'Authorization': `Bearer ${process.env.ACCESS_TOKEN}`
  },
  agent: new require('https').Agent({
      cert: fs.readFileSync(certificate_path, "utf8"),
      key: fs.readFileSync(private_key_path, "utf8")
  })
}

fetch(url,options)
  .then((response) => response.json())
  .then((body) => {
      console.log(body);
  });

I get an error:

$ node ./fetch.js   
{
  response: {
    responseCode: 401,
    methodCode: 'GET',
    resourceUri: { href: '/hr/v2/workers/ABCDEFGHIJKLMNOP' },
    serverRequestDateTime: '2023-03-30T14:25:23.351Z',
    applicationCode: {
      code: 401,
      typeCode: 'error',
      message: 'Request did not provide the required two-way TLS certificate'
    },
    client_ip_adddress: 'a.b.c.d',
    'adp-correlationID': '61f76d29-04e1-48b8-be9d-acf459408b2b'
  }
}

What am I missing in the second approach?

Michaeline

Answer 1

Perhaps surprisingly, node's builtin fetch() global does not use the HTTP stack provided by the traditional builtin http/https modules.

Instead, it uses a parallel, from-scratch HTTP stack rewrite called undici.

Given that fetch()'s HTTP stack is entirely separate from the standard HTTP stack, it should not be surprising that the options you can supply to http.get et al don't work with fetch().

Looking at the docs, it appears you can pass in a custom Dispatcher object, which in turn can customize the Client used to connect to the server. The Client can configure the TLS client certificate used in the request.

Unfortunately, at this time, node's bundled undici is not exposed to user code; you can't require() it. I've not done a deep dive into the internals, but you'll likely have to install the undici package in your project in order to access the classes necessary to build a custom Dispatcher that can present the client certificate.

Be careful though, the version of undici is statically bundled with node, so it will depend on the installed node release version. I can imagine strange bugs resulting from the duplication and/or mismatched versions of the builtin and packaged versions of undici. A future release of node may expose the bundled undici when things are considered stable.

Given the above, at least in the near term, I'd personally stick to using the traditional http(s) module instead of fetch().

Jelene

Answer 2

I don't see any import of node-fetch so I assume you're using the new native fetch added to Node18. The new global fetch does not (yet) support agent options.

See why is the agent option not available in node native fetch?

Jeanelle

An addition for clarity: eventually it might support agent options, but right now it's still experimental. – Pollak