`ar` library override timestamp

Question

the .a archive format header requires a timestamp. This has led to countless headaches when I rebuild a static library, mainly because I can't exactly reproduce the original binary.

For example (this is on my Mac, but the same thing happens in x64 linux):

$ cat foo.h
int foo();
$ cat foo.c
#include "foo.h"
int foo() { return 3; }
$ gcc -fno-pic -m64 -arch x86_64 -I/usr/local/include -O3 -c foo.c -o foo.o -fpic
$ ar rcs libfoo.a foo.o
$ md5 libfoo.a
MD5 (libfoo.a) = 0d0e6606185de4e994c47f4a0e54c1c4
$ mv libfoo.a libfoo.a1
$ ar rcs libfoo.a foo.o
$ md5 libfoo.a
MD5 (libfoo.a) = 22a69d42e1325ae8f978c2a18a4886da    

To prove to myself that the only difference was time, I took a diff based on hexdump:

$ diff <(hexdump libfoo.a) <(hexdump libfoo.a1)
2,3c2,3
< 0000010 20 20 20 20 20 20 20 20 31 33 31 31 30 34 33 30
< 0000020 38 36 20 20 35 30 31 20 20 20 32 30 20 20 20 20
---
> 0000010 20 20 20 20 20 20 20 20 31 33 31 31 30 34 32 38
> 0000020 37 31 20 20 35 30 31 20 20 20 32 30 20 20 20 20

which, if you backsolve using the header format, corresponds to the time field.

Manpage gives no indication of whether or not it is possible to override the timestamp from the header. Any thoughts?

Edit: yes, it is possible to go back and physically hack the file to use an arbitrary timestamp. yes, it is possible to change the program's behavior. Given the circumstances surrounding the situation, not all of which are strictly technical in nature, a tool to manually change the timestamp is not acceptable, nor is a modified version of ar, nor is messing with the actual system time.

Edit: In this circumstance, I have to prove that, without any unacceptable deviation from the build path, the binaries can be produced from source. In some industries (e.g. finance) this is apparently a standard practice. A handrolled tool to change the timestamps is unacceptable (because a special tool, which was not in the original build path, was used). A handrolled version of ar is unacceptable (similar problem). The problem with changing system clock is that the build would have to be perfectly coordinated (it is an hour-long build with a lot of libraries and binaries). Acceptable solutions include:

• flags to AR or other programs that could override the timestamp in the library
• an existing (age > 1 year) tool to do this
• flags to GCC that could override the timestamp coming from ar when doing the linking

Neutrophil

Please forgive my curiosity, but why do you want the new binary to exactly match the old one? Maybe there's a better way to achieve what you want. – Sacristan

@Frédéric Hamidi I sometimes need to prove that a certain set of source files actually produced a binary. And yes, you can go back and say "yes, the timestamp is different", but a lot of people only care about md5sums. – Neutrophil

If the rest of the binary is always exactly the same (I wouldn't know, I'm running gentoo and my compiler changes at least every six months), then I guess you could locate the timestamp in the .a file and override it with a fixed value (like all zeroes). – Sacristan

@Frédéric Hamidi That's not the point. Yes, I could manually go back and hack all of the timestamps, but imagine a situation with >10 libraries and >100 binaries. You can't go back and manually do this. – Neutrophil

I don't see a question here anymore. There is no ar option that does what you need and you've ruled out both editing the .a file and building your own special purpose tool; this boils it down to a simple question of whether you can read a man page or not and you clearly can. The tool doesn't do what you need it to do and you can neither use a different tool or modify the existing tool, where is the question? – Private

since you are statically linking and you have access to the source... just compile the files in. – Allocate

First of all, it is not standard practice in finance. In finance the standard practice is to embed the source control label into each object file, so that you can just do strings on the binary. Second, your approach breaks with C++ and anonymous namespaces, since the compiler generates a new name for the anonymous namespace on every compile. – Mcalpin

@Maxim at real prop trading desks, there's real concern about business continuity, and you have to prove that nothing was omitted or hidden somewhere. That requires you to reproduce the binaries exactly, for if there's even a small deviation you have a problem (how do you differentiate "your trade isnt making money because today is different from yesterday" versus "your trade isnt making money because you changed the binary)? It's a cutthroat place, and you avoid the issue of having tampered with code. – Neutrophil

@Foo: The fact that a binary has changed has very little explanatory power in case of production issues. What is required is exact change log + diff. You can't get away without a release procedure in financial institutions. Normally, every production release is branched or tagged in the version control system, so that one can always rebuild the system. – Mcalpin

@Maxim to you and me, yes. In a rational world, yes. But we are dealing with other people for whom this is a plausible argument. Out of curiosity, do you commit every binary version of every object in VCS? – Neutrophil

@Frédéric Hamidi I would like to give you the bounty because, although none of the subsequent answers were satisfactory, the answer I would have given the bounty to effectively repeats your suggestion. Can you post an answer? – Neutrophil

@Foo: we just follow the common sense release procedure: run tests on the trunk, make a release branch, label and build the release. Then fix bugs on the release branch, label and re-release. In other words, production binaries are only ever built from the release labels, so that we know which release is running where. – Mcalpin

Not just for finance, it's a basic security requirement reproducible-builds.org – Terena

Answer 1

If the rest of the binary is always exactly the same, then you could locate the timestamp in the .a file and override it with a fixed value (like all zeroes).

Christiansen

I wrote an implementation in Haskell, together with a small executable, for anybody who needs an example on how to do this. – Selftaught

Answer 2

Use "deterministic mode" in ar. See option "D" for ar in manual.

me@mybox:~$ rm libfoo.a; touch foo.o; ar rcsD libfoo.a foo.o; md5sum libfoo.a
3ecae045133ff919d1e42f6050ef56be  libfoo.a
me@mybox:~$ rm libfoo.a; touch foo.o; ar rcsD libfoo.a foo.o; md5sum libfoo.a
3ecae045133ff919d1e42f6050ef56be  libfoo.a

If you use ranlib afterwards, make sure you're using ranlib -D; otherwise ranlib will put the timestamp back.

Adp

Note that "deterministic mode" is a relatively recent addition. You'll need GNU binutils 2.20 (released 2009-10-16) or newer. ar --version will tell you what version you have. Non-GNU versions of "ar" probably won't have the "D" option (or --version, for that matter). – Falgoust

Unfortunately it does not show up on the osx version, but it's also from BSD version; also it flushes the time stamp (which doesnt quite replicate the library); however, it is something that I have added to build scripts for future projects – Neutrophil

Adding to @KeithThompson, the ranlib command adds the timestamp back to the symbol table pseduo-entry. Although ranlib provides a -t option to "Update the timestamp of the symbol map of an archive", it updates the timestamp regardless of whether you specify it. This is my experience with ranlib version 2.22. I dug up an e-mail from 2010 mentioning similar. The s option to ar obviates ranlib and respects deterministic mode. – Parkway

Answer 3

using dd will let you overwrite the part of the file you want:

dd if=libfoo.a1 of=libfoo.a skip=30 seek=30 count=4 bs=1 conv=notrunc

of course this means that you'll need your timestamp somewhere else (you can have a very basic c program that takes the current time and outputs it in little endian or big endian and then with dd you can overwrite the library file). using dd, i can overwrite the .a file and get no diff results

Rakes

For reasons which I realize I didnt explain in the question, this class of solutions (writing a tool to manually change the timestamp) is not acceptable. Unfortunately there are non-technical factors at work here. I updated the question. – Neutrophil

you updated with: "Acceptable solutions include: flags to AR or other programs that could override the timestamp in the library" how is this solution not applicable then? – Rakes

I agree this solution meets OP's stated requirement. It's the best solution and deserves the +50. – Radome

It doesn't solve the problem inasfar as it is still tampering with the timestamp. Also, Frederic proposed this a week ago and I already expressed concern about this approach. – Neutrophil

Then the question is unanswerable. Why don't you just drop using library files anyway and explicitly link the set of object files you need?? – Radome

Answer 4

I wrote a python script in my GitHub to reset timestamps for older versions of ar that didn't have -D option.

I have tested in Python 3.8.10 and Python 2.6.6.

https://gist.github.com/Supermanuu/ccdbe0c5d15d41dd1df75ad288e2a30a

Use example for clearing timestamps:

./manageStaticLibTimestamp.py cw *.a

Usage:

Usage: ./manageStaticLibTimestamp.py [pcw] <static library paths ...>
 p - print timestamps
 c - clear timestamps
 w - write timestamps

Bodkin

Answer 5

The default answer is "It can't be done by the ar tool"

Neutrophil

Unfortunately, the default answer is wrong in this case. ar D and ranlib -D work fine (for new enough tools). – Pewter