Recovery of fragmented MP4, QuickTime, AVI, MPEG-2, and DAV video with Klennet Carver
Recovery of fragmented video files is an indispensable capability in file carving.
In modern mobile phones, video cameras, and dash cameras, often more than half of video files are fragmented,
defying traditional data recovery and requiring specialized carving algorithms to recover.
Klennet Carver supports the following video formats:
- MP4, MOV, and 3GP, which are all variations of the same format,
- AVI,
- MPEG-2 (AVCHD),
- DAV (also known as DHAV, the video format used by Dahua video recorders)
MP4 and QuickTime (MOV)
MP4 and QuickTime (MOV) videos, by far the most common video file types,
share a common format consisting of atoms.
An atom is a part of a file, defined by atom name (identifying the type of data the atom contains) and length.
Atoms may contain payload data, other atoms, or a combination of both.
We will only be concerned with top-level atoms, of which three are important.
-
ftyp atom, which defines the "file type", a variation of the format:
a QuickTime movie, MP4 video, or MP4 audio, and its exact version.
-
moov atom, consisting in turn of multiple atoms,
contains various decoder settings and metadata and, most importantly, contains a frame map for the mdat atom.
Normally, a video decoder cannot seek, or move, to any arbitrary position in the video stream
and restart or continue video decoding.
The decoder can only seek to a set of defined points called keyframes.
So, to allow a player application to seek inside the video file,
the moov atom contains, among other things, a list defining
which keyframe offset inside the mdat atom of the file corresponds to which time point in a video.
-
mdat atom, containing a video stream.
The video stream consists of frames going one after the other without (at least in the general case)
distinguishing marks between them.
ftyp is always the first atom in the file, followed by moov and mdat in any order.
The file is often split into several fragments, not just two fragments, but often five or more.
Parts of the file may be overwritten, even if the file is not fragmented, and some fragments may be missing in a fragmented file.
Klennet Carver reassembles these fragments to produce a viewable video whenever possible.
Overwritten parts can't be recovered, and this causes a loss of video up to the closest next keyframe.
At the keyframe, the video decoder synchronizes with the stream again, and playback continues normally.
Sometimes, Klennet Carver fails to determine fragment boundaries properly.
While this briefly distorts the recovered video, the effect is usually negligible.
If the ftyp atom is missing or fragmentation is bad,
it may be impossible to determine if the order of atoms is ftyp - moov - mdat
or ftyp - mdat - moov.
If these cases, the ftyp atom is rebuilt, and the file is regenerated assuming the ftyp - mdat- moov atom order.
Missing (overwritten) parts of the file are either filled with zeros or the best guess of where on the disk they may have been.
The total size of all recovered files may well be larger than the capacity of the source media.
This happens when there are many video files on the media, but only a small part is available for each file.
Each file is padded with zeros in places where content is missing,
and this padding may account for a significant amount of data.
There are many different video encoding algorithms called codecs (short for coder-decoder).
The recovery implemented in Klennet Carver requires some understanding of the codec used to encode the particular video.
If your videos are not recovered, make sure to submit a support request
so that I can look into the case and make any adjustments required to accommodate the specific codec your camera is using.
AVI
AVI video format is similar to MP4, consisting of frames and a frame table.
The parts are just called differently and require somewhat different identification and parsing,
but as far as recovery is concerned, everything is just about the same.
The complexity is about the same, run times are about the same, and last but not least good recoveries can be achieved.
In some cases, the header of the video file may be overwritten.
Then, there are more video frames than headers on the media.
The most important in the header are the codec (decoder) parameters.
Without these parameters, the player cannot decode video frames properly, and the resulting video cannot be played.
Header stealing is the technique of using the headers of one file for frames of some other file.
This makes it possible to decode lost parts of the video using parameters from another video as a fallback.
There are two drawbacks, though:
-
Header stealing only works well with memory cards.
Memory cards typically contain videos made with the same settings, and therefore the codec parameters
are identical or at least interchangeable.
On a hard drive, or when there are videos taken with different camera settings, it is possible to steal
the wrong header, thus producing a broken video.
-
AVI files may have codec parameters stored along with the first frame of the video.
Therefore, header stealing also steals the first frame from whatever happens to be the donor video file.
The resulting file then displays that stolen frame first when you start playing it, and this may continue
for some time until the decoder encounters the next keyframe.
You should keep this in mind if you are using recovered video in a forensic context.
MPEG-2 video
MPEG-2 video uses file extensions MTS and M2TS.
It may also be called AVCHD video in your digital camera.
MPEG-2 is different from MP4 and AVI.
MPEG-2 per se is a stream of fixed-size frames,
each containing a bit of video, sound, or technical data.
Unlike MP4 and AVI, MPEG-2 has no file-wide header.
Therefore, the MPEG-2 video is just a sequence of frames with no discernable starting or ending points.
Klennet Carver will recover all frame sequences longer than the specific configurable length, putting each into a separate file.
As MPEG-2 files have no header, header stealing does not apply here.
DAV Dahua video files
DAV is a relatively obscure video format used by Dahua security video recorders.
It has three interleaved streams for video, audio, and camera settings like focus.
DAV format is something between AVI and MPEG-2, but DAV files do not have indexes.
Due to the specifics of the format, the recovery quality is fairly good.
Klennet Carver does not support preview for Dahua videos because no Windows codecs are available.
You need to copy the files and use Dahua proprietary viewer software to view the files.
DAV files have no header to define parameters for the entire file, and therefore header stealing setting does not apply to DAV files.
Performance considerations
The recovery speed for video files is somewhat faster than for image files.
For example, recovery of about 100 fragmented videos from a memory card takes about three hours with an eight-core i7 CPU.
Media read speed is only significant while the initial point is computed.
After the initial point is done, no further disk reads are required.