Checking out old files WITH original create/modified timestamps

Question

Is there a way to know or get the original create/modified timestamps?

Marissamarist

this is a cleaner page, but both question and most voted answer are basically duplicated: stackoverflow.com/questions/1964470/… – Zettazeugma

git.wiki.kernel.org/index.php/… – Syne

Does this answer your question? What's the equivalent of use-commit-times for git? – Hercules

The answers to Git clone changes file modification time may be more practical and to the point (command-line one liners and similar). – Shurlock

Answer 1

I believe that the only timestamps recorded in the Git database are the author and commit timestamps. I don't see an option for Git to modify the file's timestamp to match the most recent commit, and it makes sense that this wouldn't be the default behavior (because if it were, Makefiles wouldn't work correctly).

You could write a script to set the modification date of your files to the the time of the most recent commit. It might look something like this:

# No arguments? Recursively list all git-controlled files in $PWD and start over
if [ $# = 0 ]; then
  git ls-files -z |xargs -0 sh "$0"
  exit $?
fi

for file in "$@"; do
  time="$(git log --pretty=format:%cd -n 1 \
                  --date=format:%Y%m%d%H%M.%S --date-order -- "$file")"
  if [ -z "$time" ]; then
    echo "ERROR: skipping '$file' -- no git log found" >&2
    continue
  fi
  touch -m -t "$time" "$file"
done

This accepts specific files as arguments or else updates each git-controlled file in the current directory or its children. This is done in a manner that permits spaces and even line breaks in filenames since git ls-files -z outputs a null-terminated file list and xargs -0 parses null-terminated lists into arguments.

This will take a while if you have a lot of files.

Cally

There are several issues with this snippet: 1 - It fails if there are spaces in filenames; 2 - May fail for projects that are more than a few thousand files; 3 - performance is absolutely miserable any medium-sized project with a few thousand commits (even with few files) – Mudlark

+1 maybe it doesn't work for every possible case, but it is a good simple answer. – Munos

Isn't the OP's question how to preserve the original file modified timestamps, not strap on the commit timestamp to the files? – Spates

Note: this answer describes why makefiles wouldn't work when file timestamps are preserved: stackoverflow.com/questions/2458042/… – Spates

Designing a VCS around Make is shortsighted. I think this is a fault of Git. So really it doesn't make sense that it isn't default behavior. Make files should run on the files contents, not timestamps. Hashing the file and seeing if the hash matches what you built is much more robust. – Spates

@BT: Lots of people use Make, especially if you count indirect users, and lots of other programs use timestamps. Do you think my post contains factual errors, or are you just using these comments as a platform for your personal gripes with the build systems that people use? – Cally

I agree with BT and parts of your comment Dietrich. What BT meant about the OP is that your answer doesn't really allow to keep the file's original time. Instead, it replaces them with the original checkout time. Not the same thing... So, I think he clearly said your post contains factual errors. And I can see where the decision to not storing timestamps came from, as you point. I also think BT is ranting a bit back to that reasoning there. To which I agree with BT again - not good reasons to not being able to do it at all. Every other VCS can do it. – Zettazeugma

@Cawas: The question is phrased, "Is X possible?" and my answer is "X is unsupported, but you can do Y instead if that meets your needs." You're right that it's not the same thing, as I said in my answer. – Cally

@Cawas: "Every other VCS can do it." Well, SVN, CVS, and Mercurial don't do it as far as I know, since they don't store the file modification dates in the repository. I could be wrong... do you have documentation to support this? (Of course you can hack in ways to add any meta-information you like into any revision-control system, but that doesn't change the fact that the data wasn't already recorded.) – Cally

Yeah, I didn't say your post does contain factual errors, I just said he said it! (if you really going pedantic way ;-) ). But I do think it is misleading the way you wrote it. Anyway, for SVN, set the config, for Mercurial I don't know if there's any native form (just read many places there is), but there's at least a simple way to do it with extensions. I couldn't care less for CVS. :P – Zettazeugma

@Cawas: That doesn't work in SVN: read more closely, it's doing the same thing I explain here. And if you're going to use a plugin in Mercurial which stores metadata in an additional file, you might as well write a pre-commit and post-checkout hook for Git that does the same thing. – Cally

I wish it were that simple in git. Still trying to create that hook... As for the SVN, I did I read it closely but not the comments. Even still, there seems to be a command there that can be used. – Zettazeugma

@BT hashing the file content is too slow – Mi

What is date -j supposed to do? I don't have this option in my date (Arch Linux). – Ferroelectric

I had to change the second TIME row to TIME=$(date --date="$TIME" +%Y%m%d%H%M.%S) to get it to work in bash on Ubuntu. – Audreyaudri

@BT I'm a bit late to the party but "Make files should run on the files contents, not timestamps" is just nonsense, unless I miss something. How is a build system to correlate the contents of a binary (hashed or not) with the contents of a source file? Do you want to store meta information in the form of files with hashes? Do you want to open and read each file in order to compute a hash so that you know whether it has changed? How long should that take, in your opinion? It sounds patently absurd to me. Are there build systems which do that? – Chaddie

@Peter: There are build systems that do that, such as waf. – Cally

@DietrichEpp Thanks for the pointer, interesting. Not sure how they achieve acceptable run times (computing a hash sum should not be much faster than simply compiling a C file right away -- I/O is the bottleneck); hierarchical hashes over souce trees which make many checks unnecessary? The docs seem not eloquent on the basics. – Chaddie

@PeterA.Schneider: Compiling a C file is much slower than hashing it. I just tested by compiling Git with optimizations disabled, on a powerful Xeon workstation, and less than 14% of the compilation time was spent in I/O. I believe that waf only checks the file hash if the metadata has changed but the file size is correct, so it doesn't rehash everything every time you call it. But waf is still noticeably slower than make for some projects. – Cally

Also note that file hashes are the right way to go for distributed build caching, which can drastically speed up large builds. I think this is the way Bazel's remote cache works. – Cally

date by me does not know anything about the -j and -f flags. – Rockfish

This version works in CentOS 7 with spanish accents and spaces:` IFS=" " for FILE in $(git ls-files -z | tr '\0' '\n') do TIME=$(git log --pretty=format:%cd -n 1 --date=iso -- "$FILE") touch -c -m -d "$TIME" "$FILE" done` – Southeasterly

How about this for the 2 different version of date: if [ "$(uname)" == "Darwin" ]; then TIME=$(date -j -f '%Y-%m-%d %H:%M:%S %z' "$TIME" +%Y%m%d%H%M.%S); else TIME=$(date --date="$TIME" +%Y%m%d%H%M.%S); fi (couldn't get the multi-line code block to format.) – Anora

I've updated this to handle linebreaks in filenames and to be GNU/BSD-agnostic thanks to git's --date=format:… argument (though I don't know how new that is). The script now accepts filenames as arguments, which means it needs a clause for when there is no git log for a given file. Please @mention me and ask for a version of this with a progress bar. I'll eventually post that version of this script to my github. – Profant

Never mind re: that progress bar version. Using this solution to a duplicate question makes it so fast that you don't need a progress bar. – Profant

Answer 2

YES, metastore or git-cache-meta can store such (meta-)information! Git by itself, without third-party tools, can't. Metastore or git-cache-meta can store any file metadata for a file.

That is by design, as metastore or git-cache-meta are intended for that very purpose, as well as supporting backup utilities and synchronization tools.

Spates

You even mimicked his all-caps! If you apply the bold, too, I'm sure you'll get even more upvotes. ;-) – Christen

So I'm a bit miffed mainly because both of these tools (after some delving into them) drop the ball in spectacular fashion on macOS. They are entirely nonportable out of Linux. git-cache-meta relies on GNU find's -printf extension, and I'm almost certain that metastore (being a C project) is even more work to make portable. Quite unfortunate. I will post back here if I find out that this situation changes. – Stricklan

There's now git-meta, which is git-cache-meta turned into a repository with updates; now it is installable as a git hook, so it will automatically store metadata on every commit! – Glory

Answer 3

NO, Git simply does not store such (meta-)information, unless you use third-party tools like metastore or git-cache-meta. The only timestamp that get stored is the time a patch/change was created (author time), and the time the commit was created (committer time).

That is by design, as Git is a version control system, not a backup utility or synchronization tool.

Subulate

is there metastore build for win32 ? or should one re-create scripts/hooks for Windows? Franklt, i don't need other attrs, only mtime – Begrudge

I think your answer is actually "YES! Metastore or git-cache-meta can do this for you!" I guess its the difference between defeatist and optimist attiuteds. – Spates

Plus, as I heard, bazaar and mercurial are also "version control systems" which do store meta information. There's nothing so wrong with doing so. – Zettazeugma

Clarification: Git keeps two timestamps for each file: the author date (which I think is what Jakub means by 'time patch') and the committer date. The former is the time the file was first committed, and the latter is the time the file was most recently committed. – Christen

"That is by design, as Git is version control system, not a backup utility or synchronization tool." That's a non sequitur: disregarding metadata (especially dates, which are intimately related to versions) has nothing to do with being a VCS, or a backup tool. Also, every VCS has a big inherent overlap of functionality with backup tools: they both strive to preserve important past states. Finally, even Git does not ignore all metadata (e.g. it tracks the exec. bit), despite being a VCS. It still is by design, though, just for a different reason: Git's exclusive focus on content. – Haemolysin

Answer 4

UPDATE: TL;DR: Git itself does not save original times, but some solutions circumvent this by various methods. git-restore-mtime is one of them.

Ubuntu and Debian: sudo apt install git-restore-mtime Fedora, Red Hat Enterprise Linux (RHEL), and CentOS: sudo yum install git-tools

See my other answer for more details.

Full disclaimer: I'm the author of git-tools

---

This Python script may help: for each file, it applies the timestamp of the most recent commit where the file was modified:

• Core functionality, with --help, debug messages. Can be run anywhere within the work tree
• Full-fledged beast, with lots of options. Supports any repository layout.

Below is a really bare-bones version of the script. For actual usage I strongly suggest one of the more robust versions above:

#!/usr/bin/env python
# Bare-bones version. The current directory must be top-level of work tree.
# Usage: git-restore-mtime-bare [pathspecs...]
# By default update all files
# Example: to only update only the README and files in ./doc:
# git-restore-mtime-bare README doc

import subprocess, shlex
import sys, os.path

filelist = set()
for path in (sys.argv[1:] or [os.path.curdir]):
    if os.path.isfile(path) or os.path.islink(path):
        filelist.add(os.path.relpath(path))
    elif os.path.isdir(path):
        for root, subdirs, files in os.walk(path):
            if '.git' in subdirs:
                subdirs.remove('.git')
            for file in files:
                filelist.add(os.path.relpath(os.path.join(root, file)))

mtime = 0
gitobj = subprocess.Popen(shlex.split('git whatchanged --pretty=%at'),
                          stdout=subprocess.PIPE)
for line in gitobj.stdout:
    line = line.strip()
    if not line: continue

    if line.startswith(':'):
        file = line.split('\t')[-1]
        if file in filelist:
            filelist.remove(file)
            #print mtime, file
            os.utime(file, (mtime, mtime))
    else:
        mtime = long(line)

    # All files done?
    if not filelist:
        break

All versions parse the full log generated by a single git whatchanged command, which is hundreds of times faster than lopping for each file. It is under four seconds for Git (24,000 commits, 2,500 files) and less than one minute for the Linux kernel (40,000 files and 300,000 commits).

Mudlark

Your other similar answer is much better than this! – Zettazeugma

$ python ./git-restore-mtime Traceback (most recent call last): File "./git-restore-mtime", line 122, in <module> 'git rev-parse --show-toplevel --git-dir')).split('\n')[:2] TypeError: Type str doesn't support the buffer API Would you mind maybe telling us what version of Python is needed? I'm using 3.3.3 – Salesroom

@Cawas: Thanks... I guess. But the code in both answers are identical, so I'm not sure why you think the other one is better. The only difference is some ranting about git. Which was somewhat pertinent to that question, but not to this one. – Mudlark

@Rolf: I used Python 2.7, and it seems the code needs some tweaking in Python 3, thanks for pointing out. The reason is: str in Python 2 is the equivalent of bytestring in Python 3 , while str in Python 3 is unicode in Python 2. Can you please report this issue at github.com/MestreLion/git-tools/issues ? – Mudlark

It's not just the "rant". There you also explain what the code does in much more detail and, thus, clarity. – Zettazeugma

@Cawas: I didn't explain the code, you did with your edit. It was an amazing improvement, and I appreciate! :) – Mudlark

Answer 5

This did the trick for me on Ubuntu (which lacks OS X's "-j" flag on date(1)):

for FILE in $(git ls-files)
do
    TIME=$(git log --pretty=format:%cd -n 1 --date=iso $FILE)
    TIME2=`echo $TIME | sed 's/-//g;s/ //;s/://;s/:/\./;s/ .*//'`
    touch -m -t $TIME2 $FILE
done

Prostration

On Ubuntu, no need to reformat the timestamp; use touch's -d flag instead of -t. So: for FILE in $(git ls-files) ; do TIME=$(git log --pretty=format:%cd -n 1 --date=iso $FILE) ; touch -m -d "$TIME" $FILE ; done – Nunuance

Answer 6

Native Git doesn't have the functionality, but it can be achieved by hook scripts or third-party tools.

I've tried metastore. It's very fast, but I don't like the need to install and that metadata are not stored in plain text format. git-cache-meta is a simple tool I've tried, but it's extremely slow for large repositories (for a repository with tens of thousands of files, it takes minutes to update the metadata file) and could have cross-platform compatibility issues. setgitperms and other approaches also have their shortcomings that I don't like.

At last, I made a hook script for this job: git-store-meta. It has very light dependency (*nix shell, sort, and perl, which is required by Git, and optionally chown, chgrp and touch), so that nothing additional have to be installed for a platform that can run Git, desirable performance (for a repository with tens of thousands of files, it takes < 10 seconds to update the metadata file; although longer to create), saves data in plain text format, and which metadata to be "saved" or "loaded" is customizable.

It has worked fine for me. Try this if you are not satisfied with metastore, git-cache-meta, and other approaches.

Swee

I tried this out, and this does actually work, thanks! Only slight issue is that the --install hooks do not seem to start working until I manually run git-store-meta.pl --store the first time. – Czernowitz

@MilindR This is by design, as git-store-meta can never know how to store for an auto update if no previous store has been run. – Swee

Oh you mean the fields to be stored! How about having an --init flag, after which the hooks can work normally? Just a suggestion... – Czernowitz

Answer 7

I have been skirmishing with git and file timestamps for some time already.

Tested some of your ideas and made my own awfully huge and predecessor/ram heavy scripts, untill i found (on some git wiki) a script in perl that does almost what i wanted. https://git.wiki.kernel.org/index.php/ExampleScripts

And what i wanted is to be able to preserve last modification of files based on commit dates.

So after some readjustment the script is able to change creation and modification date of 200k files in around 2-3min.

#!/usr/bin/perl
my %attributions;
my $remaining = 0;

open IN, "git ls-tree -r --full-name HEAD |" or die;
while (<IN>) {
    if (/^\S+\s+blob \S+\s+(\S+)$/) {
        $attributions{$1} = -1;
    }
}
close IN;

$remaining = (keys %attributions) + 1;
print "Number of files: $remaining\n";
open IN, "git log -r --root --raw --no-abbrev --date=raw --pretty=format:%h~%cd~ |" or die;
while (<IN>) {
    if (/^([^:~]+)~([^~]+)~$/) {
        ($commit, $date) = ($1, $2);
    } elsif (/^:\S+\s+1\S+\s+\S+\s+\S+\s+\S\s+(.*)$/) {
        if ($attributions{$1} == -1) {
            $attributions{$1} = "$date";
            $remaining--;

            utime $date, $date, $1;
            if ($remaining % 1000 == 0) {               
                print "$remaining\n";
            }
            if ($remaining <= 0) {
                break;
            }
        }
    }
}
close IN;

Assuming that your repositories wont have 10k+ files this should take seconds to execute, so you can hook it to the checkout, pull or other git basic hooks.

Mulkey

Answer 8

Contrary to other solutions that set mtime to commit time, git-store-meta saves meta data like mtime into a .git_store_meta file that is added to the repository. It can install git hooks to the current repository that save and apply metadata automatically.

Ogdan

Answer 9

For a Windows environment, I wrote a small (quick and dirty) EXE file in Delphi 10.1 Berlin that collects all file dates in the source tree into the file .gitfilattr and can apply them on the checked our source tree again.

The code is on GitHub:

https://github.com/michaschumann/gitfiledates/blob/master/gitFileDates.dpr

I use it in my build system based on GitLab runners.

Stent

Answer 10

I hope you appreciate the simplicity:

# getcheckin - Retrieve the last committed checkin date and time for
#              each of the files in the git project.  After a "pull"
#              of the project, you can update the timestamp on the
#              pulled files to match that date/time.  There are many
#              that believe that this is not a good idea, but
#              I found it useful to get the right source file dates
#
#              NOTE: This script produces commands suitable for
#                    piping into BASH or other shell
# License: Creative Commons Attribution 3.0 United States
# (CC by 3.0 US)

##########
# walk back to the project parent or the relative pathnames don't make
# sense
##########
while [ ! -d ./.git ]
do
    cd ..
done
echo "cd $(pwd)"
##########
# Note that the date format is ISO so that touch will work
##########
git ls-tree -r --full-tree HEAD |\
    sed -e "s/.*\t//" | while read filename; do
    echo "touch --date=\"$(git log -1 --date=iso --format="%ad" -- "$filename")\" -m $filename" 
done

Haemagglutinate

(FYI, there's an unintended double negation in the header comment, which you may want to fix in your original, too: "There are many that don't believe that this is not a good idea.") – Haemolysin

Answer 11

Here is my solution that takes into consideration paths that contain spaces:

#! /bin/bash

IFS=$'\n'
list_of_files=($(git ls-files | sort))
unset IFS

for file in "${list_of_files[@]}"; do
  file_name=$(echo $file)

  ## When you collect the timestamps:
  TIME=$(date -r "$file_name" -Ins)

  ## When you want to recover back the timestamps:
  touch -m -d $TIME "$file_name"
done

Note that this does not take the time which git log reports; it's the time reported by the system. If you want the time since the files were committed use git log solution instead of date -r

Spheroid

Answer 12

There's some ambiguity in my (and others') interpretation of the OP about whether this means the commit time or something else, but assuming it means commit time, then this simple one-liner will work in Linux (based on answer snippet from Dietrich Epp):

git ls-files | xargs -I{} bash -c 'touch "{}" --date=@$(git log -n1 --pretty=format:%ct -- "{}")'

But there are more sophisticated answers (including Git hooks) linked from a comment to the original question by cregox.

Cicada

lol this dumped a huge number of files in my checkout named --date=@foo – Nikko

Answer 13

In CentOS 7 you have /usr/share/doc/rsync-*/support/git-set-file-times and in Debian (and derivatives) the same script in /usr/share/doc/rsync/scripts/git-set-file-times.gz. The original is from Eric Wong and is at https://yhbt.net/git-set-file-times.

It works faster than other examples mentioned here and you may find it more handy to have it already on your Linux distribution.

Southeasterly

For my Debian testing installation, there is a similar script located at /usr/share/rsync/scripts/git-set-file-times. It is Python3 and not Perl, and it originates from the rsync package. – Gelation

Answer 14

Git doesn't support storing file dates.

But you can use git-meta, which is git-cache-meta turned into a up-to-date repository (all of the comments in the gist were implemented); now it is installable as a Git hook, so it will automatically store metadata on every commit!

So, Git doesn't support storing files' metadata by default; but it doesn't mean you can't be modding it with custom features (LFS is a proof of how you can extend Git).

Croom

The linked "git-meta" doesn't work (hyper-fragile code, unsuitable for most everywhere...) - this re-write is better: github.com/gitcnd/git-meta – Curagh

Answer 15

With GNU tools.

s=$(git ls-files  | wc -l); 
git ls-files -z  |
 xargs -0 -I{} -n1 bash -c \
"git log --date=format:%Y%m%d%H%M.%S '--pretty=format:touch -m -t %cd \"{}\"%n' -n1 -- {}"|
 pv -l -s$s |
 parallel -n1 -j8

 967  0:00:05 [ 171 /s] [=====================================>  ] 16% 

.

$ git --version ; xargs --version | sed 1q ; ls --version | sed 1q;
  parallel --version  | sed 1q;  pv --version | sed 1q; sh --version | sed 1q 
git version 2.13.0
xargs (GNU findutils) 4.6.0
ls (GNU coreutils) 8.25
GNU parallel 20150522
pv 1.6.0 - Copyright 2015 Andrew Wood <[email protected]>
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)

Wanda

The parallell does not seem to do much, probably a fs-bottleneck. YMMV – Stockinet

Answer 16

Here's mine.

A little quicker than some others, as I'm not calling 'get log' for each file found; instead, calling 'git log' once and transforming that output into touch commands.

There'll be cases where there are too many listed files in 1 commit to fit into a single shell command buffer; run "getconf ARG_MAX" to see the max length of a command in bytes - on my Debian install, it's 2 MB, which is plenty.

# Set file last modification time to last commit of file
git log --reverse --date=iso --name-only | \
  grep -vE "^(commit |Merge:|Author:|    |^$)" | \
  grep -B 1 "^[^D][^a][^t][^e][^:][^ ]" | \
  grep -v "^\-\-" | \
  sed "s|^\(.*\)$|\"\1\"|;s|^\"Date: *\(.*\)\"$|~touch -c -m -d'\1'|" | \
  tr '~\n' '\n ' | \
  sh -

Description by line:

• earliest-first list of commits and filenames
• filter out unneeded commit/merge/author lines
• filter out lines starting with double-dash
• sed (stream-edit) command a) prepend/append double-quote to lines, and b) replace "Date: ." with ~touch -c -m -d. ( the touch command options are -c = don't create if it doesn't exist, -m = change file modification time, and -d = use the provided date/time )
• translate tilde (~) and newline (\n) characters to newline and space, respectively
• pipe the resulting stream of text lines into a shell.

In terms of speed, it 5 seconds 1700 commits for 6500 files in 700 directories.

Jolo

Answer 17

https://github.com/DotCi/jenkinsci-dotci-example/commit/5a45034d13b85ab4746650995db55b5281451cec#diff-a83424d0d40754ac7e2029b13daa2db43651eb65aabf8c9a5a45005b56f259bdR19

for file in `find . -type f -not -path "./.git/*"`; do 
  touch -d "`git rev-list -n 1 HEAD \$file | xargs git show -s --format=%ai`" $file; 
done

Makowski