Summary

Computer data held on digital storage devices is not usually destroyed by typical file deletion operations, but remains physically stored on the device. This is true of both conventional hard-disk drives (HDDs) and solid state drives (SSDs), including USB thumb drives. It is, therefore, possible to recover nominally deleted data using either freely available software tools, or through physical analysis techniques in cases where the data is not easily readable, but still physically present on the device.

Given that portable computing and storage devices are being used to hold sensitive personal and business information like never before, the ability to properly destroy (or to sanitize) electronic data takes on an increased importance.

This white paper discusses the key issues involved in data remanence, sanitization and recovery, with a focus on the effectiveness of data overwriting as a technique for sanitization of hard-disks, SSDs and removable USB storage devices. It was first published on the Hardwipe.com website.

The Security Risk

Simple File "Deletion"

When a file is "deleted" on a modern operation system, such as Windows or Mac OS, it typically isn't deleted at all. Sometimes it is simply moved to a holding area called a Recycler (or trash can), allowing the user to undo the deletion in the event of a mistake. Even when the file is expressly deleted (or the Recycler emptied), the file data remains physically in the storage medium. Instead of destroying the file contents, the operating system will simply remove the file's entry in the file system directory, in effect, forgetting that the file exists.

Recovery of Remanent Data

In time, the area of disk (or flash memory in the case of an SSD) used to hold long deleted files may eventually get overwritten with new data. However, with modern storage devices being so large, remanent data may lie around for years, or may never get overwritten at all. This obviously presents a security risk when the device is discarded, recycled or re-purposed.

Long deleted files containing sensitive documents, browser histories, passwords, emails, bank details, and other confidential information can easily be undeleted using widely available recovery software. In fact, it is not uncommon for computer equipment sent for recycling to find itself destined for parts of the world where identity fraud is rife1.

Recovery & Sanitization

Sanitization is a generic term used to describe the process of wiping data from a storage device so as to make it impossible, or at least difficult, to recover later. Without any form of sanitization, freely available undelete software tools can be used to search for known file headers within the underlying drive image and can easily reconstruct previously deleted files. Files can usually be resurrected instantly, without effort on the part of the "attacker".

Where data is harder to recover, if an attacker is willing to expend time and effort, other more sophisticated forms of recovery are possible, including the subversion of the device at the electronics (or chip) level, or sophisticated laboratory analysis techniques.

Sanitization Levels

In a recent study into the effectiveness of sanitization with solid state drives2, researchers at the University of California describe a number of levels of sanitization, which include:

• Logical Sanitization: Data is not recoverable via standard hardware interface (OS level) commands. This refers to recovery with software tools, but not attacking the device at the hardware level. Logical sanitization corresponds to the term "clearing" in NIST 800-883.
• Digital Sanitization: Data cannot be recovered by any digital means, which includes physically accessing the device at the electronics level.
• Analogue Sanitization: The underlying analogue signal used to encode the data on the device is destroyed or degraded, such that it is impossible to reconstruct using even the most sophisticated of laboratory analysis techniques. NIST 800-88 refers to this as "purging".

A common sanitization technique, and the one used by the Hardwipe software, is to overwrite content held in the storage medium with dummy data. With traditional hard-disk drives (HDDs), this technique will successfully sanitize at both the logical and digital level, with certain caveats. Additionally, with the application of multiple overwrite passes, it is widely accepted that it provides adequate sanitization at the analogue level also. Modern solid state drives (SSDs), however, present a special challenge for data sanitization. Although data overwriting may be useful, the technique is not as reliable with SSDs as with HDDs.

Overwriting Effectiveness

Overwriting is commonly used to sanitize data in one of three ways. It may be used to:

• Overwrite selected files only, leaving all other data intact.
• Overwrite all unused (empty) space on the device, but not existing files. With this, the aim is to destroy remanent data which has been previously deleted, but which still physically resides on the device, and would otherwise be recoverable. This method is often referred to as "free space cleaning".
• Wipe the whole device by overwriting the all accessible storage.

The problem of hard-disk sanitization was investigated by Peter Gutmann in the 1990s, who suggested that force magnetic microscopy could be used to recover data that had been physically overwritten4. He proposed that multiple overwrite passes, using specific patterns, may be effective at destroying data across a range of hard-disk technologies, and his 35 pass scheme became known as the Gutmann method. Over the years, a number of multi-pass overwriting sanitization schemes have been proposed by researchers and government agencies, and perhaps the most widely known of these is the scheme described by the US National Industrial Security Program (NISP) in DOD 5220.22-M.

However, more recent studies3 conclude that modern hard-disks (since around 2000) can effectively be purged in a single overwrite pass. In an epilogue to the original paper, Gutmann himself also said that "For any modern PRML/EPRML drive, a few passes of random scrubbing is the best you can do."

The story is not the same with solid state drives, which are fundamentally different in nature to conventional HDDs. In their study of SDD sanitization techniques2, researchers at the University of California found that some, or all, of the data may be recoverable at digital level (i.e. the electronics level) after overwriting. This is because the design of these devices do not guarantee that the same area of flash memory is used between successive writes over the same logical data. Even if a file is overwritten completely at the logical level, data fragments may remain which are not visible to the operating system. In other words, although overwriting may be useful with SSDs at the logical level, it cannot be guaranteed at the digital level.

They did find, however, that overwriting was more effective when used to wipe the whole drive, rather just individual files on the device. They suggested that, across a range of devices, overwriting the entire medium twice was usually sufficient to sanitize it at the digital level, but was not universally reliable. In one instance, for example, they found that 20 overwrite passes were necessary.

With modern hard-disks, only one or two overwrite passes are generally considered necessary to be effective. It is interesting to note, therefore, that the findings of the study suggest that multiple overwriting schemes do have some value with SSDs, but not for the same reason as originally proposed for hard-disks. For example, the study also found that overwriting a single 1GB file just once was largely ineffective at the digital level, but overwriting according to many commonly used multi-pass sanitization schemes left only around 10% of the original file contents in a recoverable state.

Other Limitations

Other limitations may apply to overwrite sanitization. For example, modern journalling file systems, such as NTFS, store recent file changes separately to the file itself, meaning that when individual files are overwritten, portions of the original data may remain in the file system journal. For this reason, it may be preferable to perform whole-drive sanitization where possible, rather than sanitizing single files.

Other Sanitization Options

Secure Erase Unit

Many drive units support a built-in ATA command called "ERASE UNIT" to completely wipe the device of all data. Unlike software based overwriting, it is implemented within the hardware itself, and should therefore be effective if implemented properly by the manufacturer. Support for this command is optional for manufacturers, however, and not applicable to the removable USB mass storage device class where, perhaps, it would have been most useful.

Furthermore, the effectiveness of the "ERASE UNIT" implementation across SSD (but not HDD) devices was also investigated by the University of California researchers who found significant variation in results. Some devices did not support it, and in one case, a device indicated that it had been fully erased by the command when, in fact, all data was entirely recoverable.

Several utilities exist to call the "ERASE UNIT" command on a drive, including "Secure Erase" for Windows, and "hdparm" on Linux.

Encryption

If data is stored on the device in encrypted form, then for purposes of sanitization, all that is necessary, in principle at least, is to destroy the encryption key. Implementation may be done at the software level, or within the device itself. The advantage of this approach is that it is fast, as it is only necessary to sanitize the storage areas used to hold the encryption key, rather than the entire medium. However, the encryption algorithm and its implementation must be strong and the key itself must not be subject to data remanence. It also cannot be applied to existing data already stored on non-encrypted drives.

Degaussing

Degaussing is a technique applied to magnetic media (i.e. HDDs) in order to purge the media of all data. It is fast and effective, but usually leaves the device inoperable.

Although never intended for use with SSDs, degaussing was tested as part of the sanitization study by the University of California researchers in the expectation that the process may damage the circuitry of SSDs, leaving them unreadable. They found, however, that it did not and, in all cases, data was recoverable after degaussing.

Physical Destruction

The physical destruction of the storage device represents, perhaps, the only universal and guaranteed method of data sanitization. In order to achieve the highest level of sanitization, destruction must be thorough, as it is possible to recover large amounts of data from even the smallest of media fragments under laboratory conditions. Typically, drives are incinerated or shredded.

Conclusions

Overwriting certainly represents a convenient method of data sanatization, in that it does not require special equipment, it can be applied to both HDDs and SSDs, both internal and removable, and leaves devices in a working state. Its limitation is that it cannot be universally relied upon to sanitize data beyond the logical level, especially when used to wipe files individually. This means that it will be ususally sufficient to thwart software based undelete utilities, but data may be, at least partially, recoverable at the digital level if an attacker is prepared to access the device electronics directly.

When it comes to wiping the entire drive, as opposed to single files, overwriting is more successful. For HDDs, it is generally considered effective even if the disk medium is later subjected to laboratory analysis. Effectiveness with SSDs is less, however, with multiple overwrite application appearing to be only largely successful, but not universally reliable. Furthermore, multiple overwrite passes appear to offer improved security, over a single overwrite, when used on SSDs.

The "ERASE UNIT" command built into many modern drives offers an appealing alternative, but its use is limited to ATA drive interfaces, which exclude removable USB devices. With regard to solid state drives, wherever a high level of data sanitization is required, then the only completely secure option remains total physical destruction.

References

• 1. UK bank details sold in Nigeria, BBC News 2006.
• 2. Michael Wei, Laura Grupp, Frederick E. Spada, and Steven Swanson of the University of California, Reliably Erasing Data from Flash-Based Solid State Drives.
• 3. NIST Special Publication 800-88 (2006).
• 4. Secure Deletion of Data from Magnetic and Solid-State Memory, Peter Gutmann, 1996.