How can I set file creation times in ZFS?

Asked 27/4, 2015 at 21:34 Answered 5/5, 2015 at 21:32

I've just got a NAS running ZFS and I'd like to preserve creation times when transferring files into it. Both linux/ext4 (where the data is now) and zfs store creation time or birth time. In the case of zfs it's even reported by the stat command. But I haven't been able to figure out how I can set the creation time of a file so it mirrors the creation time in the original file system. Unlike an ext4->ext4 transfer where I can feed debugfs a script to set the file creation times.

Is there a tool similar to debugfs for ZFS?

PS. To explain better:

I have a USB drive attached to a Ubuntu 14.04 laptop. It holds a file system where I care about the creation date (birth date) of the individual files. I consult these creation timestamps often using a script based on debugfs, which reports it as crtime.
I want to move the data to a NAS box running ZFS, but the methods I know (scp -p -r, rsync -a, and tar, among others I've tried) preserve the modification time but not the creation time.
If I were moving to another ext4 file system I would solve the problem using the fantastic tool debugfs. Specifically I can make a list of (filename, crtime) pairs on the source fs (file system), then use debugfs -w on the target fs to read a script with lines of the form

set_inode_field filename crtime <value>

I've tested this and it works just fine.

But my target fs is not ext4 but ZFS and although debugfs runs on the target machine, it is entirely useless there. It doesn't even recognize the fs. Another debug tool that lets you alter timestamps by editing an inode directly is fsdb; it too runs on the target machine, but again I can't seem to get it to recognize a ZFS file system.
I'm told by the folks who sold me the NAS box that debugfs and fsdb are not meant for ZFS filesystems, but they haven't been able to come up with an equivalent. So, after much googling and trying out things I finally decided to post a question here today, hoping someone might have the answer.

I'm surprised at how hard this is turning out to be. The question of how to replicate a dataset so all timestamps are identical seems quite natural from an archival point of view.

Regimen answered 27/4, 2015 at 21:34 Comment(10)

Don't see why this doesn't belong on server fault. – Ramsdell 4/5, 2015 at 19:24

Hi Jay, thanks. Are you suggesting I should move the question to serverfault? If so, should I wait for the bounty period to run out here and delete the question and post it on serverfault? I'm not a professional sysadmin, I just manage the 5 linux boxes in my household, so posting questions there is new territory for me. – Regimen 4/5, 2015 at 22:43

If you can delete it now, I'd suggest doing so. (I expect you're attracting downvotes in lieu of close votes, since folks can't vote to close while there is a bounty.) Then try posting over there; your question seems well formulated, and the subject should be relevant to their interests. – Ramsdell 4/5, 2015 at 22:49

Thanks. So you're saying the downvotes are because the question is offtopic. which I wouldn't have guessed from many other questions asked. At other SE sites I belong to someone downvoting a question would leave a comment to explain why, unless the question is complete garbage. Thanks for your help. – Regimen 4/5, 2015 at 23:18

For what it's worth if the solution will involve scripting, stackoverflow is still suitable place to post it. +1 – Supplement 6/5, 2015 at 13:15

Does rsync -t preserve crtime fields? I'm guessing that rsync -a is the flag for archival preservation of everything possible (like pax -pe) so it's likely you explored this path. – Flimflam 7/5, 2015 at 20:26

I wish. The manual page says "-t: preserve modification times". Yes, -a is for archive, and it preserves mtime (not ctime, atime or crtime) plus permissions and hierarchy. It's super convenient. – Regimen 7/5, 2015 at 20:42

Can the filesystem be exported as a network share where crtime field is supported and then simply copied over to FreeBSD and ZFS? ... hmm oops wait: it doesn't seem that FreeBSD utilities (cp, tar, etc.) support preserving crtime. – Flimflam 8/5, 2015 at 16:0

Hi Silvio. Any luck resolving this? I assume you commented on this post recently: cmynhier.blogspot.sk/2007/01/zfs-stores-file-creation-time.html or else you've been trying to solve this for a long time :-) It might be useful to post a question to the FreeBSD filesystem mailing list about this. Even though the issue appears to be more at the level of user space tool support for preserving the field rather than the file system itself, there could be people there who can suggest workarounds. – Flimflam 14/5, 2015 at 16:29

Yes, that comment on Chad Mynhier's blog was mine; unfortunately no reply. If I find the solution I'll definitely post it here. I had to take time off from this search to attend to urgent projects. – Regimen 15/5, 2015 at 0:59

Indeed, neither fsdb nor debugfs are likely to be suitable for use with ZFS. What you might need to do instead is find an archive format that will the preserve crtime field that presumably is already set for the files on your fileserver. If there is a version of pax or another archiving tool for your system it may be able to do this (cf. the -pe "preserve everything" flag for pax which it seems in current versions does not preserve "everything" - viz. it does not preserve crtime/birth_time). You will likely have more success finding an archiving application that is "crtime aware" than trying set the creation times by hacking on the ZFS based FreeBSD system with what are likely to be rudimentary tools.

You may be able to find more advanced tools on OpenSolaris based systems like Illumos or SmartOS (e.g. mdb). Whether it would be possible to transfer your data to a ZFS dataset on one of those platforms and then, combining the tools they have with, say, dtrace in order to rewrite the crtime fields is more of a theoretical question. If it worked then you could export the pool and its datasets to FreeBSD - exporting a pool does seem to preserve the crtime time stamps. If you are able to preserve crtime while dumping your ext4 filesystem to a ZFSonLinux dataset on the same host (nb: I have not tested this) you could then use zfs send to transfer the whole filesystem to your NAS.

This core utils bug report may shed some light on the state of user and operating system level tools on Linux. Arguably the filesystem level crtime field of an inode should be difficult to change. While ZFS on FreeBSD "supports" crtime, the state of low level filesystem debugging tools on FreeBSD might not have kept pace in earlier releases (cf. the zdb manual page). Are you sure you want to "set" (or reset) inode creation times? Or do you want to preserve them after they have been set on a system that already supports them?

On a FreeBSD system if you stat a file stored on a ZFS dataset you will often notice that the crtime field of the file is set to the same time as the ctime field. This is likely because the application that wrote the file did not have access to library and kernel functions required to set crtime at the time the file was "born" and its inode entries were created. There are examples of applications / libraries that try to preserve crtime at the application level such as libarchive(3) (see also: archive_entry_atime(3)) and gracefully handle inode creation if the archive is restored on a filesystem that does not support the crtime field. But that might not be relevant in your case.

As you might imagine, there are a lot of applications that write files to filesystems ... especially with Unix/POSIX systems where "everything is a file". I'm not sure if older applications would need to be modified or recompiled to support those fields, or whether they would pick them up transparently from the host system's C libraries. Applications being used on older FreeBSD releases or on a Linux system without ext4 could be made to run in compatibility mode on an up to date OS, for example, but whether they would properly handle the time fields is a good question.

For me running this little script as sh birthtime_test confirms that file creation times are "turned on" on my FreeBSD systems (all of which use ZFS post v28 i.e. with feature flags):

#!/bin/sh
#birthtime_test
uname -r 
if [ -f new_born ] ; then rm -f new_born ; fi

touch new_born 
sleep 3 
touch -a new_born
sleep 3 
echo "Hello from new_born at:" >> new_born 
echo `date` >> new_born
sleep 3 
chmod o+w new_born

stat -f "Name:%t%N
Born:%t%SB
Access:%t%Sa
Modify:%t%Sm 
Change:%t%Sc" new_born

cat new_born

Output:

9.2-RELEASE-p10
Name:   new_born
Born:   May  7 12:38:35 2015
Access: May  7 12:38:38 2015
Modify: May  7 12:38:41 2015 
Change: May  7 12:38:44 2015
Hello from new_born at:
Thu May 7 12:38:41 EDT 2015

(NB: the chmod operation "changes" but does not "modify" the file contents - this is what the echo command does by adding content to the file. See the touch manual page for explanations of the -m and -a flags).

This is the oldest FreeBSD release I have access to right now. I'd be curious to know how far back in the release cycle FreeBSD is able handle this (on ZFS or UFS2 file systems). I'm pretty sure this has been a feature for quite a while now. There are also OSX and Linux versions of ZFS that it would be useful to know about regarding this feature.

Just one more thing ...

Here is an especially nice feature for simple "forensics". Say we want to send our new_born file back to when time began, back to the leap second that never happened and when - in a moment of timeless time - Unix was born ... :-) ¹. We can just change the date using touch -d and everyone will think new_born is old and wise, right?

Nope:

~/ % touch -d "1970-01-01T00:00:01" new_born                
~/ % stat -f "Name:%t%N   
Born:%t%SB
Access:%t%Sa
Modify:%t%Sm
Change:%t%Sc" new_born
Name:   new_born
Born:   May  7 12:38:35 2015
Access: Jan  1 00:00:01 1970
Modify: Jan  1 00:00:01 1970 
Change: May  7 13:29:37 2015

It's always more truthful to actually be as young as you look :-)

Time and Unix - a subject both practical and poetic: after all, what is "change"; and what does it mean to "modify" or "create" something? Thanks for your great post Silvio - I hope it lives on and gathers useful answers.

You can improve and generalize your question if you can be more specific about your requirements for preserving, setting, archiving of file timestamp fields. Don't get me wrong: this is a very good question and it will continue to get up votes for a long time.

You might take a look at Dylan Leigh's presentation Forensic Timestamp Analysis of ZFS or even contact Dylan for clues on how to access crftime information.

[1] There was a legend that claimed in the beginning, seconds since long (SSL) ago was never less than date -u -j -f "%Y-%m-%d:%T" "1970-01-01:00:00:01" "+%s" because of a leap second ...

Flimflam answered 5/5, 2015 at 21:32 Comment(11)

Thank you. I'll try to contact Dylan Leigh. While reading the crtime is easy under ZFS (stat reports it), the challenge is to set it. – Regimen 6/5, 2015 at 1:39

Yes. FreeBSD has had birthtime/crtime for a long time (on UFS2 as well I believe) so stat will show this. You will notice most often the crtime is the same as ctime because applications creating files aren't always able to access library and kernel functions necessary to set crtime at the "birthtime" of the inode. – Flimflam 7/5, 2015 at 14:18

Thank you G. Cito for your expanded answer. I haven't yet got a solution but I've awarded the bounty. (Sorry it's a miserly one -- I'm new to this group and at the time I only had 99 points because someone had actually downvoted the question!) – Regimen 7/5, 2015 at 18:24

Wow thanks - I thought the bounty had expired. Cheers. – Flimflam 7/5, 2015 at 18:29

Cito - you might modify the 1st paragraph of your answer, because pax -pe doesn't preserve birth time or change time. (man pax says: ``Preserve everything'', the user ID, group ID, file mode bits, file access time, and file modification time. ) Thanks for introducing me to pax, though. (I might also point out a tiny typo: it's cf. not c.f. -- it comes from a single word, latin "confer", meaning "compare".) – Regimen 7/5, 2015 at 21:12

Yeah. I was hoping there might be an updated/modern/extended version of pax (or even cpio) where -e meant "everything the host system can store". From the state of the standard gnu-coreutils (and to an extent the BSD tools as well) one might infer that the intent is to leave the crtime field hidden except for dedicated file system debugging tools. – Flimflam 7/5, 2015 at 22:3

Regarding your "Are you sure" question: What I want is pretty much what the question says: transfer the file preserving crtime. In looking for a tool to set crtime I'm merely acknowledging the fact that none of the replication tools known to me offer an option to preserve crtime. They always leave the file as if newly created. – Regimen 8/5, 2015 at 7:9

@Silvio: Yes, changing the birth date of an existing file is possible. Had to be because the script starts with creating one, so it acts upon an existing file. – Uninstructed 3/11, 2015 at 11:45

Has there been any update on this 5 years later? Web searches seem to all show old results, and as if no one is asking this any more. Yet, I couldn't find a working solution. – Meilhac 2/12, 2020 at 11:2

@Silvio Thanks for pointing that out - fixed it. cf. e.g. – Flimflam 28/6, 2023 at 21:11

@sam-sirry Work is ongoing cf. Extending The Sleuth Kit and its underlying model for pooled storage file system forensic analysis - but it seems FS forensics and metadata is a rather recondite area of research. – Flimflam 28/6, 2023 at 21:46

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Recommended topics

Hot tags