`checking size of mp_limb_t... 0` and `configure: error: Oops, mp_limb_t doesn't seem to work` when compiling the Microchip XC32 compiler from source
Asked Answered
P

1

3

I am unable to figure this out. I am trying to build the Microchip XC32 PIC32 microcontroller GCC cross-compiler.

To try it yourself (this is how I got to my error I'm stuck on):

  1. On Windows 10 or Windows 11: enable "developer mode" to allow symlinks. Windows key --> search for "Use developer features", click the button (now blue in the image below) to turn it on: enter image description here

  2. Clone my repo here: https://github.com/ElectricRCAircraftGuy/Microchip_XC32_Compiler

  3. Install MSYS2, and open the MSYS2 UCRT64 shell. Optionally, follow my full MSYS2 setup instructions here.

  4. Install dependencies via pacman, like this: copy and paste this whole blob all at once, into the terminal:

    # ============= DO THIS TO INSTALL ALL DEPENDENCIES AT ONCE! =============
    # UCRT64
    if [ "$MSYSTEM" != "UCRT64" ]; then
        echo "ERROR: You must run this script in an MSYS2 ucrt64 terminal!"
        exit 1
    fi
    package_list=(
        "mingw-w64-ucrt-x86_64-gcc" # specific version for MSYS2 ucrt64
        "make"
        "binutils"
        "autoconf"
        "autogen"
        "bison"
        "dejagnu"
        "flex"
        "gawk"
        "gperf"
        "gzip"
        # "nsis" # generic; must be specific; hence the line below
        "mingw-w64-ucrt-x86_64-nsis" # specific version for MSYS2 ucrt64
        "perl"
        "scons"
        "tcl"
        "texinfo"
        "wget"
        "zip"
        # "texlive" # generic; must be specific; hence the line below
        "mingw-w64-ucrt-x86_64-texlive-core" # specific version for MSYS2 ucrt64
        # "texlive-extra-utils" # generic; must be specific; hence the line below
        "mingw-w64-ucrt-x86_64-texlive-extra-utils" # specific version for MSYS2 ucrt64
    )
    
    # Only install packages if tHey are NOT already installed. 
    for package in "${package_list[@]}"; do
        if ! pacman -Qs $package > /dev/null; then
            echo -e "\n=== $package is not installed. Installing... ==="
            pacman -S --noconfirm $package
        else
            echo -e "\n=== $package is already installed. ==="
        fi
    done
    echo -e "\n=== Done installing packages! ===\n"
    
  5. Run the build script: build-xc32-v4.35m.sh:

    time ./build-xc32-v4.35m.sh
    
  6. About 20 minutes into it, it fails when compiling gcc, while configuring GMP. Note that to get clean errors you must modify the build script to use -j1 instead of -j$(nproc), here in the first line, or else you'll get multi-threaded output garbled on top of each other in the terminal:

    time make -j$(nproc) all-gcc \
         STAGE1_LIBS="-lexpat -lmchp -Wl,-Bstatic -lstdc++ -Wl,-Bdynamic" \
         CPPFLAGS="-I${hostinstalldir}/include -imacros host-defs.h" \
         LDFLAGS=-L${hostinstalldir}/lib
    
    make install-gcc
    

Here's my failure. Configuring GMP seems to fail when it's looking for mp_limb_t. I've marked some notes/lines with <==== below:

checking for sysctl... no
checking for sysctlbyname... no
checking for times... no
checking for library containing clock_gettime... none required
checking for vsnprintf... yes
checking whether vsnprintf works... probably
configure: WARNING: cannot check for properly working vsnprintf when cross compiling, will assume it's ok
checking whether sscanf needs writable input... no
checking for struct pst_processor.psp_iticksperclktick... no
checking size of void *... 8
checking size of unsigned short... 2
checking size of unsigned... 4
checking size of unsigned long... 4
checking size of mp_limb_t... 0                           <===== SIZE SHOULD BE 8
configure: error: Oops, mp_limb_t doesn't seem to work    <===== ERROR
make: *** [Makefile:4701: configure-gmp] Error 1

real    3m27.324s
user    0m1.373s
sys     0m30.921s
Error: [gcc] failed to build!

real    3m27.510s
user    0m1.373s

Note that the build script runs perfectly to completion on Ubuntu 22.04, once I install the dependencies. It's Windows in MSYS2 that I am unable to get it to work in. Any help is greatly appreciated. This will help the PIC32 community build with GCC without having to buy a license from Microchip. The cross-compiler is GPL-licensed.

I've left a comment here too with a little more detail: https://github.com/JuliaLang/julia/issues/13206#issuecomment-1791823912. I no longer think symlinks are the issue, as I've tried copying the data directly over them with no change.

I have provided the autogenerated Makefile and gcc/gmp/config.log files here: https://github.com/ElectricRCAircraftGuy/Microchip_XC32_Compiler/tree/main/temp_debug_files. I've described them in the README.md file in that directory.

Been stuck on this for days. Could use some community support.


If anyone wants to try it in the MSYS2 MINGW64 environment instead, here is how to install those dependencies in that terminal:

# ============= DO THIS TO INSTALL ALL DEPENDENCIES AT ONCE! =============
# mingw64
if [ "$MSYSTEM" != "MINGW64" ]; then
    echo "ERROR: You must run this script in an MSYS2 mingw64 terminal!"
    exit 1
fi
package_list=(
    "mingw-w64-x86_64-gcc" # specific version for MSYS2 mingw64
    "make"
    "binutils"
    "autoconf"
    "autogen"
    "bison"
    "dejagnu"
    "flex"
    "gawk"
    "gperf"
    "gzip"
    # "nsis" # generic; must be specific; hence the line below
    "mingw-w64-x86_64-nsis" # specific version for MSYS2 mingw64
    "perl"
    "scons"
    "tcl"
    "texinfo"
    "wget"
    "zip"
    # "texlive" # generic; must be specific; hence the line below
    "mingw-w64-x86_64-texlive-core" # specific version for MSYS2 mingw64
    # "texlive-extra-utils" # generic; must be specific; hence the line below
    "mingw-w64-x86_64-texlive-extra-utils" # specific version for MSYS2 mingw64
)

# Only install packages if tHey are NOT already installed. 
for package in "${package_list[@]}"; do
    if ! pacman -Qs $package > /dev/null; then
        echo -e "\n=== $package is not installed. Installing... ==="
        pacman -S --noconfirm $package
    else
        echo -e "\n=== $package is already installed. ==="
    fi
done
echo -e "\n=== Done installing packages! ===\n"

Here's the autogenerated Makefile chunk of interest: C:\Users\gabriel\GS\dev\Microchip_XC32_Compiler\xc32-v4.35-src\pic32m-build\gcc\Makefile:

.PHONY: configure-gmp maybe-configure-gmp
maybe-configure-gmp:
maybe-configure-gmp: configure-gmp
configure-gmp: 
    @r=`${PWD_COMMAND}`; export r; \
    s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
    test ! -f $(HOST_SUBDIR)/gmp/Makefile || exit 0; \
    $(SHELL) $(srcdir)/mkinstalldirs $(HOST_SUBDIR)/gmp; \
    $(HOST_EXPORTS)  \
    echo Configuring in $(HOST_SUBDIR)/gmp; \
    cd "$(HOST_SUBDIR)/gmp" || exit 1; \
    case $(srcdir) in \
      /* | [A-Za-z]:[\\/]*) topdir=$(srcdir) ;; \
      *) topdir=`echo $(HOST_SUBDIR)/gmp/ | \
        sed -e 's,\./,,g' -e 's,[^/]*/,../,g' `$(srcdir) ;; \
    esac; \
    module_srcdir=gmp; \
    $(SHELL) \
      $$s/$$module_srcdir/configure \
      --srcdir=$${topdir}/$$module_srcdir \
      $(HOST_CONFIGARGS) --build=${build_alias} --host=none-${host_vendor}-${host_os} \
      --target=none-${host_vendor}-${host_os} --disable-shared LEX="touch lex.yy.c" \
      || exit 1

More leads to follow up on:

  1. https://www.google.com/search?q=bug%3A+msys2+doesnt+accept+absolute+paths+in+gcc&oq=bug%3A+msys2+doesnt+accept+absolute+paths+in+gcc&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIGCAEQRRg60gEJMTQ5NDNqMGo0qAIAsAIA&client=ms-android-google&sourceid=chrome-mobile&ie=UTF-8
  2. https://mcmap.net/q/13185/-gcc-and-clang-under-msys2-cannot-resolve-includes-with-absolute-paths
  3. https://github.com/msys2/MINGW-packages/issues/6711#issuecomment-662982274

Note to self: I think a leading / in an include may be interpreted by MSYS GCC as C:\, which means that /c/my/path is seen as C:\c\my\path instead of as C:\my\path. This is contrary to how the MSYS terminal handles it, however. What a PITA. Study the sources above.

Convert to relative paths in the build script, using realpath --relative-to. Update my Stack Overflow answer on this too: https://mcmap.net/q/12865/-how-do-i-get-the-directory-where-a-bash-script-is-located-from-within-the-script-itself

Postmistress answered 3/11, 2023 at 7:11 Comment(10)
That #include "/c/Users/<...>/gcc/gmp/gmp-h.in" is suspicious, since I think MinGW GCC (other than the Cygwin-ish /usr/bin/gcc, which you don't and shouldn't use) doesn't understand Linux-style paths. The configure script might need to be patched to not generate absolute paths, or to convert them to Windows style.Soybean
Also I think this should be tagged C or C++ for visibility.Soybean
@HolyBlackCat, I'm out of tags. Which tag should I remove to add C++? Feel free to edit. As for paths, I know I can use Linux-style paths inside the MSYS2 terminal itself. Is this different from what you mean though? Also, I don't understand how the heck that path is created nor where that file is that it's in. If you're interested enough, please try to clone and build.Postmistress
@HolyBlackCat, good choice of tags to remove :). I gotta go to bed. Been stuck on this for days. I'll check back tomorrow.Postmistress
In the terminal, bash does the path conversion magic every time you call a non-msys (non-Cygwin-based) app. The compiler isn't going to do that by itself. "please try to clone and build" I don't have a Windows machine, and it would probably take ages in a VM. I also don't know what generates that path...Soybean
@HolyBlackCat, info. on MSYS2 paths (I haven't read this yet myself, but plan to): msys2.org/docs/filesystem-pathsPostmistress
From C:\Users\gabriel\GS\dev\Microchip_XC32_Compiler\xc32-v4.35-src\pic32m-source\gcc\gmp\configure: #include "$srcdir/gmp-h.in". This makes the path, I think from the configure script. The conftest.c file is generated within that configure file. And from the autogenerated master pic32m-build\gcc\Makefile: --srcdir=$${topdir}/$$module_srcdirPostmistress
Now I wonder if you can make topdir a relative path...Soybean
Did you try just building the latest GMP (plain .../configure with no options and no weird environment variables)? If that works, try to find out what differs in what the script is doing. If it doesn't, you are more likely to find people to help you with that. Also, msys2 has a GMP package, you could just tweak your script to use that one instead of trying to build another version.Lyly
@MarcGlisse, yes, I had previously cd'ed into the pic32m-build/gcc/gmp build dir and then manually run ../../../pic32m-source/gcc/gmp/configure, and it ran just fine! It really bewildered me. Turns out it's a stinking MSYS2 gcc absolute path issue. You cannot call the configure script with an absolute path since it uses the call path to generate a C include in the build process. See my answer.Postmistress
P
2

Solved. Big thanks to @HolyBlackCat for the comments and help, including this one, which got me started on finding the solution.

The fix: when compiling in an MSYS2 terminal in Windows, absolute #include paths are not allowed in C and C++!

It turns out it's an MSYS2 gcc absolute path issue/bug. You cannot call the configure script with an absolute path since it uses the call path to generate a C include in the build process. And, absolute paths are broken in C or C++ includes in MSYS2 on Windows.

So, change this part in build-xc32-v4.35m.sh:

# Finally, GCC.
PS4="[gcc] "
(
    set -ex

    rm -rf ${gcc_builddir}
    mkdir -p ${gcc_builddir}
    cd ${gcc_builddir}

    ${gcc_srcdir}/configure \
    # ...

To this. The only change is the creation and usage of gcc_srcdir_RELATIVE to call the configure script with a relative path instead of an absolute path!:

# Finally, GCC.
PS4="[gcc] "
(
    set -ex

    rm -rf ${gcc_builddir}
    mkdir -p ${gcc_builddir}
    cd ${gcc_builddir}

    # Obtain relative paths, since Windows doesn't like absolute paths due to Windows/Linux path
    # differences in gcc includes.
    gcc_srcdir_RELATIVE=$(realpath --relative-to="." "${gcc_srcdir}")

    ${gcc_srcdir_RELATIVE}/configure \
    # ...

What does not work

If you obtain these relative paths:

    INSTALLDIR_RELATIVE=$(realpath --relative-to="." "${INSTALLDIR}")
    hostinstalldir_RELATIVE=$(realpath --relative-to="." "${hostinstalldir}")

...and use them inside the build script like this:

    INSTALLDIR_RELATIVE=$(realpath --relative-to="." "${INSTALLDIR}")
    hostinstalldir_RELATIVE=$(realpath --relative-to="." "${hostinstalldir}")

    ${gcc_srcdir}/configure \
                 --target=pic32mx \
                 --prefix=${INSTALLDIR_RELATIVE} \
                 --program-prefix=pic32m- \
                 --with-sysroot=${INSTALLDIR_RELATIVE}/pic32mx \
                 --with-bugurl=http://example.com \
                 --with-pkgversion="Microchip XC32 Compiler v4.35 custom" \
                 --bindir=${INSTALLDIR_RELATIVE}/bin/bin \
                 --infodir=${INSTALLDIR_RELATIVE}/share/doc/xc32-pic32m-gcc/info \
                 --mandir=${INSTALLDIR_RELATIVE}/share/doc/xc32-pic32m-gcc/man \
                 --libdir=${INSTALLDIR_RELATIVE}/lib \
                 --libexecdir=${INSTALLDIR_RELATIVE}/bin/bin \
                 --with-build-sysroot=${INSTALLDIR_RELATIVE}/pic32mx \
                 --enable-stage1-languages=c \
                 --enable-languages=c,c++ \
                 --enable-target-optspace \
                 --disable-comdat \
                 --disable-libstdcxx-pch \
                 --disable-libstdcxx-verbose \
                 --disable-libssp \
                 --disable-libmudflap \
                 --disable-libffi \
                 --disable-libfortran \
                 --disable-bootstrap \
                 --disable-shared \
                 --disable-nls \
                 --disable-gdb \
                 --disable-libgomp \
                 --disable-threads \
                 --disable-tls \
                 --disable-sim \
                 --disable-decimal-float \
                 --disable-libquadmath \
                 --disable-shared \
                 --disable-checking \
                 --disable-maintainer-mode \
                 --enable-lto \
                 --enable-fixed-point \
                 --enable-gofast \
                 --enable-static \
                 --enable-sgxx-sde-multilibs \
                 --enable-sjlj-exceptions \
                 --enable-poison-system-directories \
                 --enable-obsolete \
                 --without-isl \
                 --without-cloog \
                 --without-headers \
                 --with-musl \
                 --with-dwarf2 \
                 --with-gnu-as \
                 --with-gnu-ld \
                 '--with-host-libstdcxx=-static-libgcc -static-libstdc++ -Wl,-lstdc++ -lm' \
                 CPPFLAGS="-I${hostinstalldir_RELATIVE}/include -imacros host-defs.h" \
                 LDFLAGS=-L${hostinstalldir_RELATIVE}/lib

    time make -j$(nproc) all-gcc \
         STAGE1_LIBS="-lexpat -lmchp -Wl,-Bstatic -lstdc++ -Wl,-Bdynamic" \
         CPPFLAGS="-I${hostinstalldir_RELATIVE}/include -imacros host-defs.h" \
         LDFLAGS=-L${hostinstalldir_RELATIVE}/lib

...that just breaks the build. It doesn't work. The build system complains that all (or almost all) of those paths need to be absolute paths. The only fix you need is therefore the relative call to the configure script, like this: ${gcc_srcdir_RELATIVE}/configure.

Explanation of the ${gcc_srcdir_RELATIVE}/configure relative path fix that works

Quick summary:

Calling the gcc configure script with a relative path causes that relative path to get inserted into the main gcc Makefile, and then passed to the gmp configure script, where it is injected into an autogenerated conftest.c C file as an include statement. So, using an absolute path to call the gcc configure script puts an absolute path in the #include inside conftest.c, and using a relative path to call the gcc configure script puts a relative path into that #include statement. In the MSYS2 UCRT64 gcc compiler, however, only relative paths are allowed, due to path conversion issues of the C:\ (in a Windows-style path) or /c/ (in a Unix-style path on Windows) part which pertains to the beginning of absolute paths.

Details:

By doing this, the srcdir variable in the top of the Microchip_XC32_Compiler/xc32-v4.35-src/pic32m-build/gcc/Makefile file changes from this absolute path here:

srcdir = /c/Users/gbriel/GS/dev/Microchip_XC32_Compiler/xc32-v4.35-src/pic32m-source/gcc

To this relative path here:

srcdir = ../../pic32m-source/gcc

The following is in the same Makefile here. Notice that topdir=$(srcdir), and --srcdir=$${topdir}/$$module_srcdir is passed to the call to configure for the gmp library.

.PHONY: configure-gmp maybe-configure-gmp
maybe-configure-gmp:
maybe-configure-gmp: configure-gmp
configure-gmp:
    @r=`${PWD_COMMAND}`; export r; \
    s=`cd $(srcdir); ${PWD_COMMAND}`; export s; \
    test ! -f $(HOST_SUBDIR)/gmp/Makefile || exit 0; \
    $(SHELL) $(srcdir)/mkinstalldirs $(HOST_SUBDIR)/gmp; \
    $(HOST_EXPORTS)  \
    echo Configuring in $(HOST_SUBDIR)/gmp; \
    cd "$(HOST_SUBDIR)/gmp" || exit 1; \
    case $(srcdir) in \
      /* | [A-Za-z]:[\\/]*) topdir=$(srcdir) ;; \
      *) topdir=`echo $(HOST_SUBDIR)/gmp/ | \
        sed -e 's,\./,,g' -e 's,[^/]*/,../,g' `$(srcdir) ;; \
    esac; \
    module_srcdir=gmp; \
    $(SHELL) \
      $$s/$$module_srcdir/configure \
      --srcdir=$${topdir}/$$module_srcdir \
      $(HOST_CONFIGARGS) --build=${build_alias} --host=none-${host_vendor}-${host_os} \
      --target=none-${host_vendor}-${host_os} --disable-shared LEX="touch lex.yy.c" \
      || exit 1

Well, the gmp configure script uses that --srcdir parameter to set this include, as shown in Microchip_XC32_Compiler/xc32-v4.35-src/pic32m-build/gcc/gmp/config.log here:

configure:27432: gcc -c -g -O2 -D__USE_MINGW_ACCESS -DNO_ASM -I/c/Users/gabriel/GS/dev/Microchip_XC32_Compiler/xc32-v4.35-src/pic32m-build/opt/include -imacros host-defs.h conftest.c >&5
conftest.c:80:10: fatal error: /c/Users/gabriel/GS/dev/Microchip_XC32_Compiler/xc32-v4.35-src/pic32m-source/gcc/gmp/gmp-h.in: No such file or directory
   80 | #include "/c/Users/gabriel/GS/dev/Microchip_XC32_Compiler/xc32-v4.35-src/pic32m-source/gcc/gmp/gmp-h.in"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

And, due to this not-a-bug bug, absolute Linux-style includes in MSYS2 gcc on Windows are not allowed:

Strange thing is that passing absolute path as command line argument to g++ works, but using it as include does not.

e.g.

g++ -c /c/Users/travis/build/config.cpp

works, i.e. it finds the cpp file, and translates the /c/... path to C:/... path but giving that include error:

C:/Users/travis/build/config.cpp:1:10: fatal error: /c/Users/travis/build/config.hpp: No such file or directory

    1 | #include "/c/Users/travis/build/config.hpp"

      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

So, by calling ${gcc_srcdir_RELATIVE}/configure, that include now looks like this in the autogenerated conftest.c file, and it works just fine!:

#include "../../../pic32m-source/gcc/gmp/gmp-h.in"

Note that in the process of figuring this out, I tried testing Windows-style include paths like this:

#include "C:\Users\gabriel\GS\dev\Microchip_XC32_Compiler\xc32-v4.35-src\pic32m-source\gcc\gmp\gmp-h.in"

...and I'm pretty sure those failed in gcc running in MSYS2 UCRT64 too.

I consider this absolute path problem in gcc in MSYS2 a definite bug. Update: it's not a bug. It was my user error and misunderstanding.

And now I'm on to my next bug in this very difficult build process. :/

References

From the bottom of my question:

  1. Google search for "bug: msys2 doesnt accept absolute paths in gcc"
  2. gcc and clang under msys2 cannot resolve includes with absolute paths
  3. gcc/g++ cannot include file with absolute path

See also

  1. General, useful info. when building in MSYS2: SourceForge.net: MinGW-w64 - for 32 and 64 bit Windows Wiki2: System Type Triplets - talks about --build=, --host=, and --target= arguments to configure scripts.
  2. https://www.msys2.org/docs/filesystem-paths/
  3. My answer: Change the location of the ~ directory in a Windows install of Git Bash - see in particular the section titled, "More details on paths in Git Bash and MSYS2"
Postmistress answered 7/11, 2023 at 2:25 Comment(5)
My next build error I'm trying to solve: C++: error: expected ',' or '...' before string constant before __FILE__ in fancy_abort (__FILE__, __LINE__, __FUNCTION__) when compiling gccPostmistress
UCRT and MinGW compilers will not recognise Unix-style paths, but the msys2 compiler will (x86_64-pc-msys-gcc.exe).Politesse
@n.m.couldbeanAI, it is very hard for me to know which of the 7 MSYS2 environments and compilers to use. I'm very new to MSYS. Given what you know about what I'm doing, which would you choose, and why? Here's the list: msys2.org/docs/environments. Here's my writeup: https://mcmap.net/q/12316/-adding-msys-to-windows-terminal. Also, I noticed that even in UCRT64, this path works with Unix-like / instead of Windows-like \ , so long as it's a relative path: #include "../../../pic32m-source/gcc/gmp/gmp-h.in".Postmistress
I'm not sure, I'm no guru either. The msys version is closer to Unix (I think it is derived from cygwin) so that's a plus. Programs compiled with the msys version link with msys-2.0.dll and it needs to be in the PATH, so that's a minus. Only testing can tell. Windows is happy with either kind of slash, but it wants C:/ and not /c/ at the beginning. The problem is not with absolute paths but with absolute Unix-style paths, Windows libraries don't know what to do with them.Politesse
Also note that UCRT gcc does not understand the Unix-style filename as a command-line argument either. The shell does path translation. Try strace gcc /c/temp/foo.c and you will see how gcc c:/temp/foo.c is executed. Or just run cmd in the shell and try executing gcc /c/temp/foo.c from cmd.Politesse

© 2022 - 2024 — McMap. All rights reserved.