GnuCOBOL failing to find dynamic symbols, only on recent Ubuntu

Question

Something changed recently, I think.

GnuCOBOL relies on dynamic linking, symbols looked up with dlsym at run-time. This CALL run-time support code has been in OpenCOBOL for some 7 years now. It no longer works on Ubuntu 14.04, but does under Fedora 19/20.

ldd no longer shows any libraries listed using -l

For instance as a test, Ubuntu 14.04.1

The following COBOL program

   identification division.
   program-id. simple.

   procedure division.
   call "gtk_init" using
       by value 0
       by reference null
     returning omitted 
   end-call
   goback.
   end program simple.


$ cobc -x -v -lgtk-3 simple.cob
preprocessing simple.cob into /tmp/cob710_0.cob
parsing /tmp/cob710_0.cob (simple.cob)
Return status:  0
translating /tmp/cob710_0.cob into /tmp/cob710_0.c (simple.cob)
gcc -pipe -c -I/usr/local/include   -Wno-unused -fsigned-char -Wno-pointer-sign  -o "/tmp/cob710_0.o" "/tmp/cob710_0.c"
gcc -pipe  -Wl,--export-dynamic -o simple /tmp/cob710_0.o  -L/usr/local/lib -lcob -lm -lgmp -lncurses -ldb -ldl -l"gtk-3"

The binary has NO indication that libgtk-3.so is in the mix.

./simple
libcob: Cannot find module 'gtk_init'

$ ldd simple
    linux-vdso.so.1 =>  (0x00007fff2c9fe000)
    libcob.so.1 => /usr/local/lib/libcob.so.1 (0x00007f2549b06000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2549740000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2549439000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f25491c5000)
    libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x00007f2548fa2000)
    libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f2548d78000)
    libdb-5.3.so => /usr/lib/x86_64-linux-gnu/libdb-5.3.so (0x00007f25489d6000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f25487d2000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f2549d56000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f25485b3000)

and then on Fedora 20, same version of the compiler (built slightly differently, finding ncursesw instead of ncurses - assuming this is not part of the issue at hand)

$ cobc -x -v -lgtk-3 simple.cob
Command line:   cobc -x -v -lgtk-3 simple.cob 
Preprocessing:  simple.cob -> /tmp/cob20658_0.cob
Return status:  0
Parsing:        /tmp/cob20658_0.cob (simple.cob)
Return status:  0
Translating:    /tmp/cob20658_0.cob -> /tmp/cob20658_0.c (simple.cob)
Executing:      gcc -std=gnu99 -c -I/usr/local/include -pipe -Wno-unused
                -fsigned-char -Wno-pointer-sign -o "/tmp/cob20658_0.o"
                "/tmp/cob20658_0.c"
Return status:  0
Executing:      gcc -std=gnu99 -Wl,--export-dynamic -o "simple"
                "/tmp/cob20658_0.o" -L/usr/local/lib -lcob -lm -lgmp
                -lncursesw -ldb -ldl -l"gtk-3"
Return status:  0

$ ldd simple
    linux-vdso.so.1 =>  (0x00007fffae9cf000)
    libcob.so.4 => /usr/local/lib/libcob.so.4 (0x00007f4ff2548000)
    libm.so.6 => /lib64/libm.so.6 (0x0000003e5ae00000)
    libgmp.so.10 => /lib64/libgmp.so.10 (0x0000003e7a200000)
    libncursesw.so.5 => /lib64/libncursesw.so.5 (0x0000003e5d200000)
    libtinfo.so.5 => /lib64/libtinfo.so.5 (0x0000003e69800000)
    libdb-5.3.so => /lib64/libdb-5.3.so (0x0000003e6ac00000)
    libdl.so.2 => /lib64/libdl.so.2 (0x0000003e5b200000)
    libgtk-3.so.0 => /lib64/libgtk-3.so.0 (0x0000003e6ba00000)
    libc.so.6 => /lib64/libc.so.6 (0x0000003e5aa00000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003e5b600000)
    /lib64/ld-linux-x86-64.so.2 (0x0000003e5a600000)
    libgdk-3.so.0 => /lib64/libgdk-3.so.0 (0x0000003e6a800000)
    libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x0000003e65600000)
    libpangocairo-1.0.so.0 => /lib64/libpangocairo-1.0.so.0 (0x0000003e75200000)
    libX11.so.6 => /lib64/libX11.so.6 (0x00007f4ff2206000)
    libXi.so.6 => /lib64/libXi.so.6 (0x0000003e62600000)
    libXfixes.so.3 => /lib64/libXfixes.so.3 (0x0000003e5fe00000)
    libcairo-gobject.so.2 => /lib64/libcairo-gobject.so.2 (0x0000003e6a400000)
    libcairo.so.2 => /lib64/libcairo.so.2 (0x0000003e71000000)
    libgdk_pixbuf-2.0.so.0 => /lib64/libgdk_pixbuf-2.0.so.0 (0x0000003e6e000000)
    libatk-1.0.so.0 => /lib64/libatk-1.0.so.0 (0x0000003e75600000)
    libatk-bridge-2.0.so.0 => /lib64/libatk-bridge-2.0.so.0 (0x0000003e6c600000)
    libpangoft2-1.0.so.0 => /lib64/libpangoft2-1.0.so.0 (0x0000003e71c00000)
    libpango-1.0.so.0 => /lib64/libpango-1.0.so.0 (0x0000003e73600000)
    libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x0000003e61600000)
    libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x0000003e66600000)
    libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x0000003e5fa00000)
    libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x0000003e5e600000)
    libXinerama.so.1 => /lib64/libXinerama.so.1 (0x0000003e61e00000)
    libXrandr.so.2 => /lib64/libXrandr.so.2 (0x0000003e62200000)
    libXcursor.so.1 => /lib64/libXcursor.so.1 (0x0000003e62e00000)
    libXcomposite.so.1 => /lib64/libXcomposite.so.1 (0x0000003e74e00000)
    libXdamage.so.1 => /lib64/libXdamage.so.1 (0x0000003e67e00000)
    libwayland-client.so.0 => /lib64/libwayland-client.so.0 (0x0000003e6ec00000)
    libxkbcommon.so.0 => /lib64/libxkbcommon.so.0 (0x0000003e6b000000)
    libwayland-cursor.so.0 => /lib64/libwayland-cursor.so.0 (0x0000003e69c00000)
    libXext.so.6 => /lib64/libXext.so.6 (0x0000003e5ea00000)
    librt.so.1 => /lib64/librt.so.1 (0x0000003e5ce00000)
    libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x0000003e61a00000)
    libharfbuzz.so.0 => /lib64/libharfbuzz.so.0 (0x0000003e6f000000)
    libfreetype.so.6 => /lib64/libfreetype.so.6 (0x0000003e60e00000)
    libxcb.so.1 => /lib64/libxcb.so.1 (0x0000003e5da00000)
    libpixman-1.so.0 => /lib64/libpixman-1.so.0 (0x0000003e6f800000)
    libEGL.so.1 => /lib64/libEGL.so.1 (0x0000003e73200000)
    libpng16.so.16 => /lib64/libpng16.so.16 (0x0000003e5f600000)
    libxcb-shm.so.0 => /lib64/libxcb-shm.so.0 (0x0000003e6e800000)
    libxcb-render.so.0 => /lib64/libxcb-render.so.0 (0x0000003e70800000)
    libXrender.so.1 => /lib64/libXrender.so.1 (0x0000003e61200000)
    libz.so.1 => /lib64/libz.so.1 (0x0000003e5ba00000)
    libGL.so.1 => /lib64/libGL.so.1 (0x0000003e68200000)
    libatspi.so.0 => /lib64/libatspi.so.0 (0x0000003e6c200000)
    libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x0000003e62a00000)
    libexpat.so.1 => /lib64/libexpat.so.1 (0x0000003e60a00000)
    libffi.so.6 => /lib64/libffi.so.6 (0x0000003e5ee00000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003e5ca00000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003e5e200000)
    libgraphite2.so.3 => /lib64/libgraphite2.so.3 (0x0000003e6fc00000)
    libXau.so.6 => /lib64/libXau.so.6 (0x0000003e5d600000)
    libX11-xcb.so.1 => /lib64/libX11-xcb.so.1 (0x0000003e65e00000)
    libxcb-dri2.so.0 => /lib64/libxcb-dri2.so.0 (0x0000003e67200000)
    libxcb-xfixes.so.0 => /lib64/libxcb-xfixes.so.0 (0x0000003e70400000)
    libxcb-shape.so.0 => /lib64/libxcb-shape.so.0 (0x0000003e72a00000)
    libgbm.so.1 => /lib64/libgbm.so.1 (0x0000003e70c00000)
    libudev.so.1 => /lib64/libudev.so.1 (0x0000003e63200000)
    libwayland-server.so.0 => /lib64/libwayland-server.so.0 (0x0000003e74a00000)
    libglapi.so.0 => /lib64/libglapi.so.0 (0x0000003e67600000)
    libdrm.so.2 => /lib64/libdrm.so.2 (0x0000003e67a00000)
    libxcb-glx.so.0 => /lib64/libxcb-glx.so.0 (0x0000003e66e00000)
    libXxf86vm.so.1 => /lib64/libXxf86vm.so.1 (0x0000003e66200000)
    libpcre.so.1 => /lib64/libpcre.so.1 (0x0000003e5c600000)
    liblzma.so.5 => /lib64/liblzma.so.5 (0x0000003e5c200000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003e5be00000)

The generated C code (GnuCOBOL uses C intermediates) on Ubuntu

/* Line: 4         : CALL               : simple.cob */
cob_procedure_params[0] = (cob_field *)&c_1;
cob_procedure_params[1] = NULL;
cob_glob_ptr->cob_call_params = 2;
if (unlikely(call_gtk_init.funcvoid == NULL)) {
  call_gtk_init.funcvoid = cob_resolve_cobol ("gtk_init", 0, 1);
}
call_gtk_init.funcnull ((cob_s32_t)0LL, NULL);
b_1 = 0;

And Fedora

/* Line: 4         : CALL               : simple.cob */
cob_procedure_params[0] = (cob_field *)&c_1;
cob_procedure_params[1] = NULL;
cob_glob_ptr->cob_call_params = 2;
if (unlikely(call_gtk_init.funcvoid == NULL)) {
  call_gtk_init.funcvoid = cob_resolve_cobol ("gtk_init", 0, 1);
}
call_gtk_init.funcnull ((cob_s32_t)0LL, NULL);
b_1 = 0;

I get good results on Ubuntu (full ELF linkage hints) when gtk_init is called from C, not as a string passed to cob_resolve. Tested with gcc -o simple simple-gtk.c -lgtk-3

So what changed with the assumptions surrounding -llibname? It does not seem to be included in ELF data for dlopen to even bother trying looking for libgtk-3.so

More information: Erroneous compile lines on Ubuntu really make it look like this should be working. (Misspelling gtk)

$ cobc -x -v -lgkt-3 simple.cob
Command line:   cobc -x -v -lgkt-3 simple.cob 
Preprocessing:  simple.cob -> /tmp/cob13556_0.cob
Return status:  0
Parsing:        /tmp/cob13556_0.cob (simple.cob)  
Return status:  0
Translating:    /tmp/cob13556_0.cob -> /tmp/cob13556_0.c (simple.cob)
Executing:      gcc -std=gnu99 -c -I/usr/local/include -pipe -Wno-unused
                -fsigned-char -Wno-pointer-sign -o "/tmp/cob13556_0.o"
                "/tmp/cob13556_0.c"
Return status:  0
Executing:      gcc -std=gnu99 -Wl,--export-dynamic -o "simple"
                "/tmp/cob13556_0.o" -L/usr/local/lib -lcob -lm -lgmp -lncurses
                -ldb -ldl -l"gkt-3"
/usr/bin/ld: cannot find -lgkt-3
collect2: error: ld returned 1 exit status
Return status:  256

And yet, with the correct compile line, the ELF is showing no linkage hints to gtk-3

I've been head scratching on this on and off for a while now. Looking for a hint as to what assumptions changed with Ubuntu and gcc and/or ld and/or dlopen dlsym.

The open-cobol package has been working in Debian and Ubuntu repositories for quite a few years now. Even older versions of GnuCOBOL (GNU Cobol, and or OpenCOBOL) on Ubuntu all fail now. Something changed and we didn't get the memo. More than willing to change the compiler sources, but looking for friendly StackOverflow insights first.

This does not seem to be a local environment issue, as this Ubuntu problem is showing up for others as well. This also feels like one of those DOH! simple to fix problems.

More than willing to add more compile logs, LD_DEBUG=all dumps, or straces etc.

Equivalence

Answer 1

This looks to be a change made to the compiler driver for ubuntu - it's adding the --as-needed option to the compile line when sending the code to collect2 aka the linker.

To understand what's happening we need to disassemble the execution of cobc more than it's being shown:

cobc -x -v simple.cob -lgtk-3
preprocessing simple.cob into /tmp/cob2743_0.cob
translating /tmp/cob2743_0.cob into /tmp/cob2743_0.c
gcc -pipe -c  -Wno-unused -fsigned-char -Wno-pointer-sign  -o /tmp/cob2743_0.o /tmp/cob2743_0.c
gcc -pipe  -Wl,--export-dynamic -o simple /tmp/cob2743_0.o  -L/usr/lib -lcob -lm -lgmp -lncurses -ldb -ldl -lgtk-3

if we break it into produce C code, and then compile it we get:

cobc -C -x -v simple.cob
gcc -pipe -c  -Wno-unused -fsigned-char -Wno-pointer-sign -o simple.o simple.c
gcc -pipe -Wl,--export-dynamic -o simple simple.o  -L/usr/lib -lcob -lm -lgmp -lncurses -ldb -ldl -lgtk-3

We need to further disassemble the last gcc line into:

gcc -### -pipe -Wl,--export-dynamic -o simple simple.o  -L/usr/lib -lcob -lm -lgmp -lncurses -ldb -ldl -lgtk-3

which yields as output:

 /usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 "--sysroot=/" --build-id --eh-frame-hdr -m elf_x86_64 "--hash-style=gnu" --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -z relro -o simple /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o -L/usr/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. --export-dynamic simple.o -lcob -lm -lgmp -lncurses -ldb -ldl -lgtk-3 -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o

The problem is the presence of the first --as-needed option - it overturns all the explicit -l options on the command line - if there's nothing found in the .o's that make up the file that depend on the library, it won't link the library in - this is the exact dynamic loading case.

This seems to be the case since Oneiric.

This simplest workaround is adding:

COB_LDFLAGS=-Wl,--no-as-needed

to your environment which should fix the linking issue.

Cogitate

Petesh; Thank you. This will go into the compiler source tree, hopefully today. – Equivalence

Answer 2

It appears from the output that the C code that dynamically looks up gtk_init doesn't know about loading the actual libgtk-3.so shared object at runtime.

You could compile with cobc using the -fstatic-call option. This will call your library functions like gtk_init directly, and not via calls to dlopen/dlsym . More information on this option can be found in the info pages via the command info opencobol:

With the compiler options -fstatic-call, more efficient code will be generated like this:

subr(X);

Note that this option is effective only when the called program name is a literal (like CALL "subr".'). With a data name (likeCALL SUBR.'), the program is still called dynamically.

If you want to go the dynamic route I can only suggest telling your Cobol application exactly what dynamic objects you wish to load at runtime. To do this you can set the environment variable COB_PRE_LOAD prior to launching your application. Information on this environment variable can be found in this Open Cobol documentation. In particular it says:

COB_PRE_LOAD is an environment variable that controls what dynamic link modules are included in a run.

$ cobc occurl.c
$ cobc occgi.c
$ cobc -x myprog.cob
$ export COB_PRE_LOAD=occurl:occgi
$ ./myprog

That will allow the OpenCOBOL runtime link resolver to find the entry point for CALL “CBL_OC_CURL_INIT” in the occurl.so module. Note: the modules listed in the COB_PRE_LOAD environment variable DO NOT have extensions. OpenCOBOL will do the right thing on the various platforms.

You can do it a couple ways (using bash):

COB_PRE_LOAD=libgtk-3 ./simple

This will set COB_PRE_LOAD to use libgtk-3.so (you leave off the .so) launch your application, and when finished reset COB_PRE_LOAD back to what it was. You can also use export to set the environment variable for the duration of your session with:

export COB_PRE_LOAD=libgtk-3 
./simple

You can specify multiple shared objects using COB_PRE_LOAD by separating each with a colon. So if you needed libgtk-3 and libgmp for example, you could do this:

COB_PRE_LOAD=libgtk-3:libgmp ./simple

Overstretch

Thanks Michael, but that's not really the root issue at play here. The Dynamic link feature of GnuCOBOL has been working for a long time, and just recently broke on Ubuntu. The above command lines work fine on Fedora. Compiles on Ubuntu are not leaving any hints in the ELF objects of the -l libraries mentioned on the command line. This is new breakage. (I'm the keeper of the FAQ you referenced). Yours is a good answer, but there is more at play here. GnuCOBOL needs to work with dynamic library references, has worked, and does work, everywhere but recent Ubuntu. – Equivalence

Thanks for the headsup. I didn't know you (the OP) were the keeper of the FAQ. What my answer doesn't say is that I actually tried this all out on a fully updated 14.04 and using COB_PRE_LOAD does work. If I don't use it, I get your error with gtk_init. I am curious if COB_PRE_LOAD actually resolves the issue for you if you try to actually use it? (If it doesn't that may suggest something else amiss with just your environment?). – Overstretch

I am curious how it ever worked since with the way the late binding appeared to be (from my cursory look) - how would Cobol know what shared objects get loaded at runtime unless you told it with something like COB_PRE_LOAD - unless with other environments there is some more static component at play and the generated "C" does something a bit different I guess one obvious question is this. Are you running the same version of Cobol on all your platforms? (ie Ubuntu, Fedora etc) – Overstretch

The -l (minus ell) compiler switch usually leaves the hint in the ELF, dlopen and dlsym given a place to start. Currently in Ubuntu, the -l is not leaving any tracer rounds. ldd would normally show a -l link library as a dependency. And does, under Fedora for instance, and used to, under Ubuntu. We are missing some change to ld.so or other dynamic link flag. – Equivalence

Possibly a linker optimization determined that although you added those shared objects at link time, it found that they were never referenced and optimized them away (including the hint that you relied on). If that were the case the linker wouldn't know you are actually using dlopen() etc to late bind to them. I'd have to do some experiments to see if that is actually happening and if there is a solution. – Overstretch

I'm hoping it's a simple thing, missing option, or config. And thanks for looking Michael. I've been scratching head over this for awhile. – Equivalence

My opinion is this has to be some sort of optimization. If I take the simple.c output from your sample program and after the includes add these 2 lines it works. extern void gtk_init(); void (*gtk_init_ptr)() = gtk_init; . Although I don't use the pointer anywhere in the code it is enough to convince the compiler and linker that I do indeed use the library. I haven't found a way to alter this behavior. Might actually be a good question for the gcc mailing list. – Overstretch