GCC -fPIC option

Question

I have read about GCC's Options for Code Generation Conventions, but could not understand what "Generate position-independent code (PIC)" does. Please give an example to explain me what does it mean.

Dingus

Clang also uses -fPIC. – Kan

Related: -fpie: #2463650 – Payee

Related, but not a dupe: #23226066 – Dotard

Answer 1

Position Independent Code means that the generated machine code is not dependent on being located at a specific address in order to work.

E.g. jumps would be generated as relative rather than absolute.

Pseudo-assembly:

PIC: This would work whether the code was at address 100 or 1000

100: COMPARE REG1, REG2
101: JUMP_IF_EQUAL CURRENT+10
...
111: NOP

Non-PIC: This will only work if the code is at address 100

100: COMPARE REG1, REG2
101: JUMP_IF_EQUAL 111
...
111: NOP

EDIT: In response to comment.

If your code is compiled with -fPIC, it's suitable for inclusion in a library - the library must be able to be relocated from its preferred location in memory to another address, there could be another already loaded library at the address your library prefers.

Groundless

This example is clear, but as a user what will be the difference if I create a shared labrary (.so) file without the option? Are there some cases that without -fPIC my lib will be invalid? – Dingus

Yes, building a shared library that isn't PIC could be an error. – Dispense

To be more specific, the shared library is supposed to be shared between processes, but it may not always be possible to load the library at the same address in both. If the code were not position independent, then each process would require its own copy. – Cabana

The problem is that, as I know, all the addresses written in they example above are virtual addresses and the will be other when this library will be loaded in a memory. So one process can call the same function from A address and the second process can call the same function from the same A address, or can copy the lib into a memory once again and call the function from B address. What is the problem? Where the error should occur without -fPIC? – Dingus

I mean that as the addresses in above example are virtual and they should be loaded into the real memory, then they can be loaded anywhere and they will work properly, isn't it? – Dingus

@Narek: the error occurs if one process wants to load more than one shared library at the same virtual address. Since libraries cannot predict what other libraries could be loaded, this problem is unavoidable with the traditional shared library concept. Virtual address space doesn't help here. – Kura

Sorry to bring up an old thread, but @SimonRichter... So do you mean that if my shared lib is not compiled with -fPIC it will simply be a memory inefficiency since it will need to be loaded by each process? Or will there be some cases where it doesn't work. I recently took on a Linux development project at work that has been somewhat ignored/shelved for awhile. We haven't seen any problems using shared libraries without -fPIC, but want to know if we could in the future. The shared lib is currently used by only one process on the system. – Phosphorescence

@KylePreiksa, on i386 it will merely be inefficient (because the dynamic linker can relocate the library, but the modified memory image needs to be tracked by the kernel (unmodified pages can be discarded when memory is tight, and reloaded from the original file), so you are wasting memory. On other architectures, linking the shared library fails because there is no valid way to express how the relocation would have to be performed. For example, PowerPC only has a 24 bit relative branch instruction that cannot be used between different DLLs, and -fPIC uses two instructions instead. – Cabana

@SimonRichter thank you for the explanation, especially why it works on some architectures and not others. We are x86 exclusively so I won't worry too much, but will try to optimize in a future version! – Phosphorescence

@SimonRichter I will ask in different way. When not to use -fPIC? – Participial

You can omit -fPIC when compiling a program or a static library, because only one main program will exist in a process, so no runtime relocation is ever necessary. On some systems, programs are still made position independent for enhanced security. – Cabana

@SimonRichter sorry to wake up this thread again ... Even if are sure there is only one dynamic library loaded, -fPIC is necessary because of Address Layout Randomization mecanism, right ? – Sundew

@Hugo, yes, also I'm not sure how the linker would react to a library that cannot be relocated. – Cabana

Would the second example (non-PIC) not work because the process linking this lib might be using those virtual addresses? is that what you meant in your examples? – Jump

@SimonRichter Thanks for when -fPIC can be omitted. When shall it be omitted? – Hog

@CacahueteFrito, on some machines, position independent code comes with a performance penalty. Intel CPUs before the 686 are notorious for this: no PC-relative addressing modes, and few registers, so programs would use a relative subroutine call to push the current program counter onto the stack, pop it into a register and address data items relative to that. A variant of that is calling a pop; push; ret sequence (slower locally, but doesn't confuse the branch predictor as much). – Cabana

Modern CPUs generally have PC-relative addressing and also lots of registers, so the performance impact is a lot less pronounced. – Cabana

Answer 2

I'll try to explain what has already been said in a simpler way.

Whenever a shared lib is loaded, the loader (the code on the OS which load any program you run) changes some addresses in the code depending on where the object was loaded to.

In the above example, the "111" in the non-PIC code is written by the loader the first time it was loaded.

For not shared objects, you may want it to be like that because the compiler can make some optimizations on that code.

For shared object, if another process will want to "link" to that code it must read it to the same virtual addresses or the "111" will make no sense. But that virtual-space may already be in use in the second process.

Beaird

Whenever a shared lib is loaded, the loader changes some addresses in the code depending on where the object was loaded to. I think this is not correct if compiled with -fpic and the reason why -fpic exists i.e. for performance reasons or because you have a loader that's not able to relocate or because you need multiple copies in different locations or for many more reasons. – Pareu

Why not always use -fpic? – Ingenuous

@Ingenuous - because it will required one more calculation (the function address) for each function call. So performance-wise, if not needed it is better not to use it. – Beaird

Answer 3

Code that is built into shared libraries should normally be position-independent code, so that the shared library can readily be loaded at (more or less) any address in memory. The -fPIC option ensures that GCC produces such code.

Shively

Why wouldn't a shared library be loaded at any address in memory without having the -fPIC flag on? is it not linked to the program? when the program is running, the operating system uploads it to memory. Am I missing something? – Jump

Is the -fPIC flag used, to ensure this lib can be loaded to any virtual address in the process that is linking it? sorry for double comments 5 minutes elapsed can't edit previous one. – Jump

Distinguish between building the shared library (creating libwotnot.so) and linking with it (-lwotnot). While linking, you don't need to fuss about -fPIC. It used to be the case that when building the shared library, you needed to ensure -fPIC was used for all the object files to be built into the shared library. The rules may have changed because compilers build with PIC code by default, these days. So, what was critical 20 years ago, and might have been important 7 years ago, is less important these days, I believe. Addresses outside the o/s kernel are 'always' virtual addresses'. – Shively

So previously you had to add the -fPIC. Without passing this flag, the generated code when building the .so needs to be loaded to specific virtual addresses that might be in use? – Jump

Yes, because if you didn't use the PIC flag, the code wasn't reliably relocatable. Things like ASLR (address space layout randomization) aren't possible if the code is not PIC (or, at least, are so hard to achieve that they are effectively impossible). – Shively

Answer 4

Adding further...

Every process has same virtual address space (If randomization of virtual address is stopped by using a flag in linux OS) (For more details Disable and re-enable address space layout randomization only for myself)

So if its one exe with no shared linking (Hypothetical scenario), then we can always give same virtual address to same asm instruction without any harm.

But when we want to link shared object to the exe, then we are not sure of the start address assigned to shared object as it will depend upon the order the shared objects were linked.That being said, asm instruction inside .so will always have different virtual address depending upon the process its linking to.

So one process can give start address to .so as 0x45678910 in its own virtual space and other process at the same time can give start address of 0x12131415 and if they do not use relative addressing, .so will not work at all.

So they always have to use the relative addressing mode and hence fpic option.

Amelita

Can anyone explain how this is not a problem with a static library, why you don't have to use -fPIC on a static library? I understand that the linking is done in compile time (or right after actually), but if you have 2 static libraries with position dependent code, how are they going to be linked? – Onrush

@MichaelP object file has a table of position depended labels and when particular obj file is linked all labels are updated accordingly. This cannot be done to shared library. – Infanticide

Answer 5

A minor addition to the answers already posted: object files not compiled to be position independent are relocatable; they contain relocation table entries.

These entries allow the loader (that bit of code that loads a program into memory) to rewrite the absolute addresses to adjust for the actual load address in the virtual address space.

An operating system will try to share a single copy of a "shared object library" loaded into memory with all the programs that are linked to that same shared object library.

Since the code address space (unlike sections of the data space) need not be contiguous, and because most programs that link to a specific library have a fairly fixed library dependency tree, this succeeds most of the time. In those rare cases where there is a discrepancy, yes, it may be necessary to have two or more copies of a shared object library in memory.

Obviously, any attempt to randomize the load address of a library between programs and/or program instances (so as to reduce the possibility of creating an exploitable pattern) will make such cases common, not rare, so where a system has enabled this capability, one should make every attempt to compile all shared object libraries to be position independent.

Since calls into these libraries from the body of the main program will also be made relocatable, this makes it much less likely that a shared library will have to be copied.

Jaine