How do ZFS snapshots use space in practice?

Question

1. Created a snapshot
2. Deleted a huge file
3. Delta is still 0 (snapshot not using anymore space) in zfs list for past three snapshots

Should the delta or used space not be the size of the deleted file. I know ZFS is COW but I'm confused as to why I can't rollback the /usr/home/xxxx child

# ls /home/xxxxx/testing12345.txt 
/home/xxxxx/testing12345.txt
# ls -alh /home/xxxxx/testing12345.txt                                                                                                                                 
-rw-r--r--  1 root  xxxxx   254M Aug 28 00:06 /home/xxxxx/testing12345.txt
# zfs list -rt snapshot tank1/usr/home/xxxxx
NAME                                        USED  AVAIL  REFER  MOUNTPOINT
tank1/usr/home/xxxxx@myRecursiveSnapshot   291M      -   804M  -
tank1/usr/home/xxxxx@devEnv                 71K      -  1.39G  -
tank1/usr/home/xxxxx@xfce                     0      -  1.39G  -
tank1/usr/home/xxxxx@testhome                 0      -  1.39G  -
tank1/usr/home/xxxxx@testagain               1K      -  1.39G  -
tank1/usr/home/xxxxx@27082015                 0      -  1.39G  -
tank1/usr/home/xxxxx@270820150                0      -  1.39G  -
tank1/usr/home/xxxxx@2708201501               0      -  1.39G  -
# 
# 
# 
# 
# zfs snapshot -r tank1@28082015                                                                                                                                        
# zfs list -rt snapshot tank1/usr/home/xxxxx                                                                                                                           
NAME                                        USED  AVAIL  REFER  MOUNTPOINT
tank1/usr/home/xxxxx@myRecursiveSnapshot   291M      -   804M  -
tank1/usr/home/xxxxx@devEnv                 71K      -  1.39G  -
tank1/usr/home/xxxxx@xfce                     0      -  1.39G  -
tank1/usr/home/xxxxx@testhome                 0      -  1.39G  -
tank1/usr/home/xxxxx@testagain               1K      -  1.39G  -
tank1/usr/home/xxxxx@27082015                 0      -  1.39G  -
tank1/usr/home/xxxxx@270820150                0      -  1.39G  -
tank1/usr/home/xxxxx@2708201501               0      -  1.39G  -
tank1/usr/home/xxxxx@28082015                 0      -  1.39G  -
# rm /home/xxxxx/testing12345.txt                                                                                                                                      
# zfs list -rt snapshot tank1/usr/home/xxxxx                                                                                                                           
NAME                                        USED  AVAIL  REFER  MOUNTPOINT
tank1/usr/home/xxxxx@myRecursiveSnapshot   291M      -   804M  -
tank1/usr/home/xxxxx@devEnv                 71K      -  1.39G  -
tank1/usr/home/xxxxx@xfce                     0      -  1.39G  -
tank1/usr/home/xxxxx@testhome                 0      -  1.39G  -
tank1/usr/home/xxxxx@testagain               1K      -  1.39G  -
tank1/usr/home/xxxxx@27082015                 0      -  1.39G  -
tank1/usr/home/xxxxx@270820150                0      -  1.39G  -
tank1/usr/home/xxxxx@2708201501               0      -  1.39G  -
tank1/usr/home/xxxxx@28082015                 0      -  1.39G  -
# 

I've been tried rolling back using various snapshots the /usr, /usr/home, and /usr/home/xxxx directories. I've read the FreeBSD forums, the handbook, and I've also tried rolling back just tank1@[snapshot name]--all to no effect. Something odd, when I change files in /usr/home/xxxxx files in the hidden .zfs/snapshots/[snapshot name]/usr/home/xxxxx directory change as well.

Chromogenic

if you are going to vote this down, please leave a comment with an explanation so that I may correct the question. A vote down without a comment is not particularly helpful. – Chromogenic

I should mention that I am able to successfully zfs send and zfs receive to a backup pool. – Chromogenic

Are you looking for an refference to a document explaining it or would be satisfied by an practice that is confirming the behaving in the bsd os code? – Heterotopia

I've been following the BSD handbook, so I need not reference material--I'm looking for the answer as to why the snapshot does not grow in allotted size when I delete the large file from the snapshot-ed volume and why I can't restore it when I follow the restoration steps quite diligently, @Zaibis – Chromogenic

@Zaibis, I'm also looking for a canonical answer as to how, when and how much space zfs snapshots account for--I believe this answer followed by a quick explanation of what I'm doing wrong here or why my restoration isn't working would suffice for the bounty – Chromogenic

"when I change files in /usr/home/xxxxx files in the hidden .zfs/snapshots/[snapshot name]/usr/home/xxxxx directory change as well" <= What kind of changes are you experiencing? Because when I do this locally, the existing snapshot does not change a bit. – Isometrics

I'm seeing changes in .zfs snapshot dir corresponding to my actions taken in the actual dir – Chromogenic

@Chromogenic This is very strange and --as noted-- I cannot confirm this locally. I would also consider such a behavior simply a (critical) bug. – Isometrics

Answer 1

Use this command to see space used for all snapshots of a vdev - relevant property you want is usedsnap:

zfs list -o name,used,avail,refer,creation,usedds,usedsnap,origin,compression,compressratio,refcompressratio,mounted,atime,lused

I've added a few more properties since I use compression on my zfs pools.

zfs snapshots directories are read-only by the way.

You said you cannot roll back? If that's the case specify -r or -R and possibly -f if you have clones, sample:

zfs rollback -r poolname/dataset@oldersnaphot
zfs rollback -R poolname/dataset@oldersnaphot

Read the manual before issuing zfs rollback:

       -r
           Destroy any snapshots and bookmarks more recent than the one specified.
       -R
           Recursively destroy any more recent snapshots and bookmarks, as well as any clones of those snapshots.
       -f
           Used with the -R option to force an unmount of any clone file systems that are to be destroyed.

Edith