Undelete files on ZFS

What happens if you delete a file on ZFS? Well, customary with most other filesystems, the file disappears. But can you get it back? The official recommended method is to get the file from a snapshot or a backup. What are the chances if neither is available? Let's start down the rabbit hole of details.

ZFS and copy-on-write

When you delete a file, the filesystem needs to update its metadata in several places. Not necessarily specific to ZFS, the filesystem typically needs to

• mark the file record as empty,
• update the directory (or several directories) to remove the file name,
• update the free space map.

If a crash happens during the update, the filesystem ends up in some partially updated state, which is not good. Copy-on-write (CoW) is the method of dealing with such inconsistencies. Instead of changing the records, ZFS creates a new set of updated records and writes this set to some new location on the disk. Then, the root pointers are updated to point to the updated records.

How copy-on-write works
Green shading is accessible data; green lines are old metadata references; red lines are new metadata references.

For this trick to work, the old and new data cannot overlap. This comes in handy for undelete.

Philosophy of copy-on-write

Maybe it is easier to show the difference between traditional filesystems (like NTFS or EXT) and a copy-on-write filesystem (ZFS or ReFS) by stating that there is no deletion in CoW filesystems.

A CoW filesystem like ZFS does not delete a file. Instead, it creates a new version of the filesystem, and that new version of the filesystem does not contain the file. Then, a new version of the filesystem is written to the disk. Once the write is complete, the new version of the filesystem becomes current, and the file ceases to exist.

To undelete a file, we need to find one of the previous filesystem versions that still contains the file. Obviously, the old versions are eventually overwritten, so there is no guarantee that we can find it.

Copy-on-write and undelete on ZFS

In a copy-on-write process, the old version of data remains unmodified, at least for a short while. Therefore, we can find it. There are no pointers to it, it is the deleted file, after all, but it still can be found by scanning all disks in the pool. Depending on the filesystem usage patterns and how full the pool is, the data can survive for quite a long time. It is not unusual to find thousands, or sometimes tens of thousands, copies of old metadata. It will eventually be overwritten as the disk space is needed for something else, exactly as it happens with any other filesystem. Thus, the general rule of data recovery stands – the fewer data written to the filesystem, the better the chance of successful recovery. So, by searching for previous versions of metadata, Klennet ZFS Recovery can find

• deleted files,
• previous versions of files that were being edited (because the same copy-on-write rules apply to edits).

The success rate, however, depends on how much data was written to the pool after the file was deleted.

Filed under: ZFS.

Created Saturday, January 5, 2019

Updated 20 September 2019