How do I enable FIPS mode in Nginx?

Question

I am hoping to get some guidance on enabling openssl fips mode for nginx. So far I followed the openssl guide for enabling fips mode on the openssl.

That part works well:

# /usr/local/openssl/bin/openssl md5 /usr/local/openssl/bin/openssl
Error setting digest md5
139805371958952:error:060A80A3:digital envelope 
routines:FIPS_DIGESTINIT:disabled for fips:fips_md.c:180:

# cat /proc/sys/crypto/fips_enabled
1

For nginx I first tried to build nginx using this custom openssl:

/nginx-1.12.2/configure --with-http_ssl_module --with-openssl=/usr/local/openssl --with-ld-opt="-L/usr/local/openssl/lib"

This failed however since the /usr/local/openssl is the "installed" location of custom openssl, not the source tree.

So I changed --with-openssl option to use openssl source tree as:

/nginx-1.12.2/configure --with-http_ssl_module --with-openssl=/usr/local/src/openssl-1.0.2n/ --with-ld-opt="-L/usr/local/openssl/lib" 

This works and I am able to install nginx but I don't think proper openssl compile options to support fips mode is passed during nginx configure.

When I print nginx info:

nginx -V:
nginx version: nginx/1.12.2
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC)
built with OpenSSL 1.0.2n  7 Dec 2017

Yet the custom openssl is:

/usr/local/openssl/bin/openssl version
OpenSSL 1.0.2n-fips  7 Dec 2017

Do I need to be passing openssl compile option to enable fips support while configuring nginx?

Per "Configurable FIPS mode" thread (https://forum.nginx.org/read.php?10,257298,257298)

"Currently we solve this by compiling nginx ourselves after adding FIPS_mode_set(1) after the SSL library initialization code in systems where we require it."

And per "How to check FIPS 140-2 support in OpenSSL?"(How to check FIPS 140-2 support in OpenSSL?):

FIPS could be available but not used. So an application must enable the validated cryptography via FIPS_mode_set, and the call must succeed. Can someone let me know where to set FIPS_mode_set flag?

I searched for that setting and found one binary obj file under nginx: nginx-1.12.2/objs/nginx

and a header file under openssl source directories: openssl-1.0.2n/.openssl/include/openssl/crypto.h

Any help/guidance greatly appreciated, thanks.

UPDATE: As I was checking the output from ../src/nginx-1.12.2/make command, I noticed, it removes original Makefile from ../src/openssl-1.0.2n, along with FIPS related options in it:

   cd /usr/local/src/openssl-1.0.2n/ \

   && if [ -f Makefile ]; then make clean; fi \

   && ./config --prefix=/usr/local/src/openssl-1.0.2n/.openssl no-shared  \

   && make \

   && make install_sw LIBDIR=lib

   make[2]: Entering directory `/usr/local/src/openssl-1.0.2n'

Any idea about how to pass proper openssl make options to nginx? Or to configure nginx to use already installed fips capable custom openssl? Thanks again

Adkison

Several years ago I had the same task. Several years ago you had to (1) patch the Nginx build system, and (2) modify the Nginx sources to call FIPS_mode_set(1). I don't know what is needed nowadays. – Dipsomaniac

I know, I am referencing a thread in the body of mine with your answer with most votes :) I am trying to hunt down where and how to set "FIPS_mode_set(1)" in Nginx sources prior to building it per your (and some others') answer for earlier questions. – Adkison

You call FIPS_mode_set after you call SSL_library_init or OPENSSL_init_ssl. See Library Initialization on the OpenSSL wiki. The OpenSSL FIPS Object Module User Guide also has a section on how to call FIPS_mode_set. – Dipsomaniac