Navigation

ZFS and power failures

ZFS is supposed to be resistant to power failures, at least if the hardware matches specific requirements, mainly concerning delayed-write caching. In practice, either ZFS does not quite work as advertised in some cases, or the hardware commonly lies about write caching. Maybe it is both factors combined. This way or other, power failures are the second most prevalent cause of the recoveries I do, with operator error being the most prevalent.

The typical scenario is that after either a single unexpected power failure or a brownout, the system fails to come back up, ending instead with a kernel panic.

More often than not, kernel panic is associated with free space mapping and allocation. Crash call stacks include

space_map_load()
space_map_iterate()
range_tree_add()
metaslab_load()

As a side note, I don't understand why the driver would even care about free space if asked to mount the pool read-only. In data recovery, we don't care about free space maps in any filesystem. We do not ever write to the filesystem during recovery; thus, there is no use in knowing which space is free. The same consideration seems to apply to a read-only mount, except it does not.

In ZFS, the free space maps seem to be especially sensitive to damage in power outages. Maybe it is because writing out free space maps takes the longest amount of time or the most amount of writes, especially when the filesystem is idle or lightly loaded. Then, it is a simple matter of probability that something which takes the most writes is most likely to be interrupted in a damaging way. Maybe the reason is something else entirely.

My experience is that once the space maps are damaged, there is nothing you can do about it with built-in ZFS tools. Attempts to mount the pool in read-only mode fail because the driver attempts to load the free space map anyway. Even more baffling, transaction rollback does not work either. At least in most cases, I expected it would, but it looks like it (mostly) does not.

Klennet ZFS Recovery will copy everything out if the damage is indeed confined to space maps. It does not require space maps to work.

ZFS and power failures

Further reading