Understanding ZFS Recovery output

Understanding recovery output may seem a tad complicated because of how ZFS works. ZFS pool contains multiple object sets, each describing all files in the particular filesystem or a version of zvol. There are tens of thousands and sometimes millions of old versions of each object set in any given pool. Many, if not most, of these versions are partially damaged.

Object sets and TXG numbers

Each object set has a sequence number assigned to it as it is written to disk. This number is called Transaction Group Number, TXG number, or simply TXG. The latest (newest, most recent) object set has the highest TXG number. Older object sets have lower TXG numbers. TXG numbers are useful for ordering, but it is impossible to associate actual clock time in hours or minutes based on TXG numbers.

Duplicate files and file versions

Each of the many recoverable object sets in a ZFS pool describes all the files. Two object sets will contain identical files if the file stays unchanged between two transactions. To reduce the number of duplicate files recovered, ZFS Recovery only keeps one copy of the file, placing it in the latest (newest) object set containing the file.

Let's walk through the example:

• There are four slots in the sample object set, numbered 1 through 4.
• There are four recoverable object sets, with the corresponding TXG numbers 50 to 53.
• Slot 2 is initially occupied by a green file, which is edited to produce a blue file between transactions 50 and 51.
• Slot 3 is initially occupied by a red file, which is deleted between transactions 51 and 52.
• Slot 4 is occupied by the yellow file, which never changes.

TXG number	File
	1	2	3	4
53 (newest)
52
51
50 (oldest)

Sample filesystem object sets as seen on disk; remember the time goes from bottom to top.

Recovery only keeps files in the last object set, discarding all other duplicates.

• TXG 50 contains the latest version of a green-blue file before editing.
• TXG 51 holds the red file because it is the latest version of the filesystem before the red file is deleted.
• TXG 52 is completely removed because it does not contain any changed files.
• TXG 53 holds green-blue (after editing) and yellow files, representing the filesystem's latest (actual) version.

TXG number	File
	1	2	3	4
53 (newest)
51
50 (oldest)

Sample filesystem as recovered, after removing duplicate files

Even though a real filesystem does not have four files and four transaction groups, the presentation still follows the logic above. Let's look at the screenshot.

Klennet ZFS Recovery results view

• The top section lists object sets, which are effectively filesystem snapshots made at various times. Object set names are shown whenever available. If the name was not given or cannot be found because of damage, [Noname] is used instead. You can sort object sets in different ways, but I recommend you sort by TXG, which places more recent object sets on top.
• Once you select one of the object sets, its corresponding directory tree is displayed at the bottom left.
• If you select a directory, the bottom right panel shows the list of files, with their sizes and quality for each file. The quality readout is based on ZFS checksums. It will show "Good" if all the content can be recovered and matches checksums, "Bad" if nothing can be recovered, or a percentage value if only part of the file can be recovered.

Let's examine each line. Keep your eye on the number of files (Total files column).

• The first (topmost) object set, TXG 74A, is a small dataset used by FreeNAS for its own purposes, named samba4. It holds 21 files.
• The second object set, TXG 749, is a previous version of the samba4 dataset. It holds two files that differ between TXGs 749 and 74A.
• The third object set, TXG 6E3, is a primary dataset loaded with 30 million files of test data. I did not give it a name, so it does not have one.
• All the subsequent object sets only contain previous versions of files. As the filesystem is pretty static, only a few files are changed, and older object sets hold very few files.
• Note that it is possible to have two datasets changed in a single transaction group and have identical TXG numbers.

Possible actions

1. Resume analysis
   If you stopped object set analysis earlier and did not find the data you need in the results, clicking Resume analysis will return to and resume the analysis stage. As analysis goes from the newest to the oldest object sets, continuing analysis allows us to go further back in time.
2. Select files
   Opens the file selection dialog, where you can do mass selection and deselection of the files.
3. Verify checksums
   This will verify checksums for whatever files are currently selected. If the checksums are already verified for some of the selected files, these checksums will not be re-verified again. This is only needed if you skipped checksum verification during analysis.
4. More functions
   
   • Export list of file names with their corresponding sizes and checksum verification statuses
   • Save the program state for possible future reload. Important: you need to provide the same disk set to reload.
5. Copy selected files. The most useful of them all, this asks you where to put the selected files, some other settings detailed below, and then starts copying.

File copy output

Klennet ZFS Recovery does not modify the original data on disks in any way. Instead, it copies files to whatever destination you specify. The directory structure and file names are retained and copied over, too.

The copy is laid out as follows:

1. An individual top-level directory is created for every dataset-TXG combination unless you explicitly ask for only one top-level directory.
2. Under each top-level directory, the directory tree is created, as shown in the bottom-left section of the ZFS Recovery output view, with the files and their names, according to the bottom-right file list.

File copy options

There are two sets of options associated with copying the files:

• Write each object set in its separate directory, or combine all object sets into a single directory.
• If ZFS Recovery encounters duplicate files or multiple versions of the same file, it can overwrite or skip the existing file or rename it, keeping multiple versions.

The copying goes in the same order as object sets are displayed, top to bottom. In a typical order, sorted by TXG numbers with the highest topmost, most recent versions of files are copied first, and the oldest versions are copied last. Thus,

• if you want the most recent versions, set the copier to skip existing files;
• if you want the earliest versions, set the copier to overwrite;
• if you want all versions, set it to rename.