Why does GHC take so long to link?

Question

>cabal update
>cabal install cabal-install
.......
[43 of 44] Compiling Distribution.Client.Install ( Distribution/Client/Install.hs, dist/build/cabal/cabal-tmp/Distribution/Client/Install.o )
[44 of 44] Compiling Main             ( Main.hs, dist/build/cabal/cabal-tmp/Main.o )
Linking dist/build/cabal/cabal ..

Then I will wait for a VERY long time for it to finish linking.

Certie

Answer 1

Very likely it's the linker itself. The standard ld from binutils is known to be slow. If you want to speed things up (and live a bit on the edge), try installing the Gold linker. On Ubuntu (and I assume Debian), that would be:

sudo apt-get install binutils-gold

I've been using it on my home system for a while now, no issues yet.

Granite

It's true that ld is slow, but it doesn't help that, since ghc does static linking and tries to link in a lot of stuff, there's a lot of work for the linker to do. On my slightly old macbook with 1GB ram, ld will often run out of memory and start swapping. – Jessiajessica

Fair enough, I'm sure there's something GHC could do to help, I was merely pointing out some low-hanging fruit. – Granite

@John L: Well, if it's slow because GHC is doing more linking than necessary with a linker that's slower than necessary, there are two obvious ways to improve the overall result, but only one that doesn't require messing with GHC's internals. ;] – Purdum

@C. A. McCann: true enough. But if standard ld is the only available linker for your system (OS X for me), then your options are mess with GHC or mess with your linker. Coincidentally, I've learned much more about linkers than I ever thought I would over the past month... – Jessiajessica

@Michael Snoyman: you're absolutely correct that it's the first thing to try. I'm basically griping because the linking step is often slow for me as well, and gold isn't an option for non-ELF systems. – Jessiajessica

Wow, order of magnitude difference. Excellent suggestion. – Wire

@MichaelSnoyman There is this problem you reported concerning GHC not working with gold. How did you get this to work - is it fixed in recent GHCs? – Spence

I found a way to do this, see my answer below (which code that doesn't fit in a comment). – Spence

GHC will do run-time linker detection soon (probably 7.8): hackage.haskell.org/trac/ghc/ticket/6063#comment:19 – Selwin

Answer 2

This should be a comment, but I cannot format code like this in comments:

I managed to use gold as the linker used by GHC, working around standard ghc-with-gold errors like /usr/bin/ld.gold: --hash-size=31: unknown option, by installing binutils-gold (as suggested in Michael Snoyman's answer) and then replacing the symlink /usr/bin/ld by the following script (made executable with chmod +x):

#!/usr/bin/env python2

import sys
import os
import subprocess


tofilter = [
    "--hash-size",
    "--reduce-memory-overheads",
]

filtered = [ a for a in sys.argv if not any(a.startswith(p) for p in tofilter) ]
filtered[0] = "/usr/bin/ld.gold"

subprocess.check_call(subprocess.list2cmdline(filtered))

Note that trying to set the linker with ghc -pgml /usr/bin/ld.gold or ghc -pgml /usr/bin/ld.whateverElse is not sufficient because the argument to -pgml needs to be a replacement for GCC, not LD. GHC calls GCC, which calls /usr/bin/ld; this is why the above script works.

Spence

Will not be necessary with upcoming GHC releases (probably 7.8): hackage.haskell.org/trac/ghc/ticket/6063#comment:19 – Selwin

So looking forward to that. – Spence

Thanks, I think I'll try this workaround. What's strange is that according to comments here, linker detection is supposed to happen when you configure a bindist, but I haven't found that to be the case when I am working from a bindist that I compiled from a source distribution. Anyone know what I might be doing wrong? – Servitor

This should use subprocess.check_call, not os.system as the latter will perform shell expansions that would be otherwise left unexpanded. I just spent about two days trying to track down this one. – Sumba

@Sumba Ok, using subprocess.check_call now. – Spence

On ubuntu 13.10, ghc 7.6.3 running cabal build I got OSError: [Errno 36] File name too long. When I checked there were 305 different flags passes to ld.gold, by the looks of things most of them were dependencies. I've not a clue what caused it to not work when passed to subprocess.list2cmdline but I managed to dodge it by using this line of bash instead: /usr/bin/ld.gold `echo "$*" | sed 's/--hash-size=\S*//' | sed 's/--reduce-memory-overheads//'`. It does the same as the script above, filtering out --hash-size and --reduce-memory-overheads. – Dumah

Answer 3

GHC will by default create a standalone library/executable (static linking).

Dynamic linking has been supported for a while now, so you could try turning it on. With less work to do, the linker is likely to be much faster. On the other hand, you'll need to make sure that at execution time the dynamic libraries you rely on are available (LD_LIBRARY_PATH variable on Linux).

Mlawsky