This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. (March 2010)

This article may contain original research. Please improve it by verifying the claims made and adding references. Statements consisting only of original research may be removed. More details may be available on the talk page. (March 2010)

Universal Disk Format (UDF) is an implementation of the specification known as ISO/IEC 13346 and ECMA-167 and is an open vendor-neutral file system for computer data storage for a broad range of media. In practice, it has been most widely used for DVDs and newer optical disc formats, supplanting ISO 9660. Due to its design, it is very well suited for incremental updates on both recordable or (re)writable optical media. UDF is developed and maintained by the Optical Storage Technology Association (OSTA).

Normally, authoring software will master a UDF file system in a batch process and write it to optical media in a single pass. But when packet writing to rewriteable media, such as CD-RW, UDF allows files to be created, deleted and changed on-disc just as a general-purpose filesystem would on removable media like floppy disks and flash drives. This is also possible on write-once media, such as CD-R, but in that case the space occupied by the deleted files cannot be reclaimed (and instead becomes inaccessible).

Multi-session mastering is also possible in UDF, though some implementations may be unable to read disks with multiple sessions.^[2]

History[link]

The Optical Storage Technology Association standardized the UDF file system to form a common file system for all optical media. The goal was to make a common file system for read-only media and optical media that are re-writable. When first standardized, the UDF file system was intended to replace ISO-9660, allowing support for both read-only and writable media. After the first version of UDF was released, it was adopted by the DVD Consortium as the official file system for DVD Video and DVD Audio.^[3]

Revisions[link]

Multiple revisions of UDF have been released:^[3][4]

• Revision 1.02 (August 30, 1996). This format is used by DVD-Video discs.
• Revision 1.50 (February 4, 1997). Added support for (virtual) rewritability on CD-R/DVD-R media by introducing the VAT structure. Added sparing tables for defect management on rewritable media such as CD-RW, and DVD-RW and DVD+RW.
• Revision 2.00 (April 3, 1998). Added support for Stream Files and real-time files (for DVD recording) and simplified directory management. VAT support was extended.
• Revision 2.01 (March 15, 2000) is mainly a bugfix release to UDF 2.00. Many of the UDF standard's ambiguities were resolved in version 2.01.
• Revision 2.50 (April 30, 2003). Added the Metadata Partition facilitating metadata clustering, easier crash recovery and optional duplication of file system information: All metadata like nodes and directory contents are written on a separate partition which can optionally be mirrored.
• Revision 2.60 (March 1, 2005). Added Pseudo OverWrite method for drives supporting pseudo overwrite capability on sequentially recordable media.

Universal Disk Format builds[link]

Universal Disk Format builds are as follows:

• Plain (Random Read/Write Access). This is the original format supported in all UDF revisions
• Virtual Allocation Table a.k.a. VAT (Incremental Writing). Used specifically for writing to CD-R and (write-once) media
• Spared (Limited Random Write Access). Used specifically for writing to CD-RW and DVD-RW (rewritable) media

Packet writing[link]

Packet writing was introduced in Revision 1.50. For write-once media, the entire disk is virtualized, making the write-once nature transparent for the user; the disk can be treated the same way one would treat a rewritable disc, but with steadily decreasing available space. For rewritable media, addresses are virtualized so that defect management (sector relocation) can be performed. However, UDF defect management does not apply to systems that already implement another form of defect management, such as Mount Rainier for optical discs, or a disk controller for a hard drive.

Incorrect or incomplete implementations of Revision 1.50 (or above) may result in compatibility problems due to improper virtualization in reading or writing the disk.

Compatibility[link]

Many DVD players do not support any UDF revision other than version 1.02. Discs created with a newer revision may still work in these players if the ISO 9660 bridge format is used. As some popular DVD burning programs default to UDF version 1.5 when burning video DVDs, users^[who?] have found it necessary to avoid using the software wizard^[which?] and instead manually configure the burn to version 1.02. Even if an operating system claims to be able to read UDF 1.50, it still may only support the plain build and not necessarily either the VAT or Spared Universal Disk Format builds.

Mac OS X 10.4.5 claims to support Revision 1.50 (see man mount_udf), yet it can only mount disks of the plain build properly and provides no virtualization support at all. It cannot mount UDF disks with VAT, as seen with the Sony Mavica issue.^[5] Releases before 10.4.11 mount disks with Sparing Table but does not read its files correctly. Version 10.4.11 fixes this problem.^[6][7]

Similarly, Microsoft Windows XP Service Pack 2 (SP2) cannot read DVD-RW disks that use the Universal Disk Format (UDF) 2.00 defect management system.^[8] This problem occurs if the UDF defect management system creates a sparing table that spans more than one sector on the DVD-RW disk. Windows XP SP2 can recognize that a DVD is using UDF. but, Windows Explorer displays the contents of a DVD as an empty folder. A hotfix is available for this^[9] and is included in Service Pack 3. ^[10]

Table of operating systems[link]

Terminology:

• Unless otherwise noted, read and write support means that only the plain UDF build is supported, but not the VAT and spared build.
• Support for read means that a UDF formatted disk can be mounted by the system. It enables the user to read files from the UDF volume using the same interface that is used to access files on other disks connected to the computer.
• Support for write means that, in addition to reading files from a mounted UDF volume, data such as files can be modified, added, or deleted.

Character set[link]

The specification allows for nine character encodings: one by agreement, one specified by ECMA-6 (also known as ASCII), three subsets of ASCII, a subset of ECMA-94 (Latin-1), and various other graphical characters.^[17][18]

See also[link]

• Comparison of file systems
• DVD Authoring
• ISO/IEC 13490 (also known as ECMA-168)
• Universal Disk Format builds

References[link]

Further reading[link]

• ISO/IEC 13346 standard, also known as ECMA-167.

External links[link]

• OSTA home page
• Wenguang Wang's UDF Introduction
• Linux UDF support
• Microsoft Windows UDF Read Troubleshooting
• UDF Revision 2.60 March 1, 2005
• AIX - CD-ROM file system and UDFS

Formatting a hard drive using MS-DOS

Disk formatting is the process of preparing a hard disk drive or flexible disk medium for data storage. In some cases, the formatting operation may also create one or more new file systems. The formatting process that performs basic medium preparation is often referred to as "low-level formatting." The term "high-level formatting" most often refers to the process of generating a new file system. In certain operation systems (e.g., Microsoft Windows), the two processes^{[clarification needed]} are combined and the term "format" is understood to mean an operation in which a new disk medium is fully prepared to store files. Illustrated to the right are the prompts and diagnostics printed by MS-DOS's FORMAT.COM utility as a hard drive is being formatted.

As a general rule, formatting a disk is "destructive," in that existing data (if any) is lost during the process; for high-level formatting, some of this data might be recoverable with special tools.

History[link]

A "block", a contiguous number of bytes, is the minimum unit of memory that is read from and written to a disk by a disk driver. The earliest disk drives had fixed block sizes (e.g. the IBM 350 disk storage unit (of the late 1950's) block size was 100 6 bit characters) but starting with the 1301^[1] IBM marketed subsystems that featured variable block sizes - a particular track could have blocks of different sizes. The disk subsystems on the IBM System/360 expanded this concept in the form of Count Key Data (CKD) and later Extended Count Key Data (ECKD); however the use of variable block size in HDDs fell out of use in the 1990s; one of the last HDDs to support variable block size was the IBM 3390 Model 9, announced May 1993^[2]

Modern hard disk drives such as Serial attached SCSI (SAS)^[3] and Serial ATA (SATA)^[4] drives, appear at their interfaces as a contiguous set of fixed-size blocks; for many years 512 bytes long but beginning in 2009 and accelerating through 2011, all major hard disk drive manufacturers began releasing hard disk drive platforms using the Advanced Format of 4096 byte logical blocks.^[5][6]

Floppy disks generally only used fixed block sizes but these sizes were a function of the host's OS and its interaction with its controller so that a particular type of media (e.g., 5¼-inch DSDD) would have different block sizes depending upon the host OS and controller.

Optical disks generally only use fixed block sizes.

Disk formatting process[link]

Formatting a disk for use by an operating system and its applications involves three different steps.

1. Low-level formatting (i.e., closest to the hardware) marks the surfaces of the disks with markers indicating the start of a recording block (typically today called sector markers) and other information like block CRC to be used later, in normal operations, by the disk controller to read or write data. This is intended to be the permanent foundation of the disk, and is often completed at the factory.
2. Partitioning creates data structures needed by the operating system. This level of formatting often includes checking for defective tracks or defective sectors.
3. High-level formatting creates the file system format within the structure of the intermediate-level formatting. This formatting includes the data structures used by the OS to identify the logical drive or partition's contents). This may occur during operating system installation, or when adding a new disk. Disk and distributed file system may specify an optional boot block, and/or various volume and directory information for the operating system.

Low-level formatting of floppy disks[link]

The low-level format of floppy disks (and early hard disks) is performed by the disk drive's controller.

Consider a standard 1.44 MB floppy disk. Low-level formatting of the floppy disk, normally writes 18 sectors of 512 bytes to each of 160 tracks (80 on each side) of the floppy disk, providing 1,474,560 bytes of storage on the disk.

Physical sectors are actually larger than 512 bytes, as in addition to the 512 byte data field they include a sector identifier field, CRC bytes (in some cases error correction bytes) and gaps between the fields. These additional bytes are not normally included in the quoted figure for overall storage capacity of the disk.

Different low-level formats can be used on the same media; for example, large records can be used to cut down on inter-record gap size.

Several freeware, shareware and free software programs (e.g. GParted, FDFORMAT, NFORMAT and 2M) allowed considerably more control over formatting, allowing the formatting of high-density 3.5" disks with a capacity up to 2 MB.

Techniques used include:

• head/track sector skew (moving the sector numbering forward at side change and track stepping to reduce mechanical delay),
• interleaving sectors (to minimize sector gap and thereby allowing the number of sectors per track to be increased),
• increasing the number of sectors per track (while a normal 1.44 MB format uses 18 sectors per track, it is possible to increase this to a maximum of 21), and
• increasing the number of tracks (most drives could tolerate extension to 82 tracks – though some could handle more, others could jam).

Linux supports a variety of sector sizes, and DOS and Windows support a large-record-size DMF-formatted floppy format.^{[citation needed]}

Low-level formatting (LLF) of hard disks[link]

Hard disk drives prior to the 1990s typically had a separate disk controller that defined how data was encoded on the media. With the media, the drive and/or the controller possibly procured from separate vendors, low level formatting was a potential user activity. Separate procurement also had the potential of incompatibility between the separate components such that the subsystem would not reliably store data.^[7]

User instigated low-level formatting (LLF) of hard disk drives was common for minicomputer and personal computer systems until the 1990s. IBM and other mainframe system vendors typically supplied their hard disk drives (or media in the case of removable media HDDs) with a low-level format. Typically this involved subdividing each track on the disk into one or more blocks which would contain the user data and associated control information. Different computers used different block sizes and IBM notably used variable block sizes but the popularity of the IBM PC caused the industry to adopt a standard of 512 user data bytes per block by the middle 1980s.

Depending upon the system, low-level formatting was generally done by an operating system system utility. IBM compatible PCs used the BIOS which is involved using the MS-DOS debug program to transfer control to a routine hidden at different addresses in different BIOSs.^[8] Low-level format function can also be called as "erase" or "wipe" in different tools. For best results it's highly recommended to use tools created by hard disk's manufacturer.

Transition away from LLF[link]

Starting in the late 1980s, driven by the volume of IBM compatible PCs, HDDs became routinely available pre-formatted with a compatible low-level format. At the same time, the industry moved from historical (dumb) bit serial interfaces to modern (intelligent) bit serial interfaces and Word serial interfaces wherein the low level format was performed at the factory.

Today, an end-user, in most cases, should never perform a low-level formatting of an IDE or ATA hard drive, and in fact it is often not possible to do so on modern hard drives outside of the factory.^[9][10]

Disk reinitialization[link]

This section needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. (July 2009)

While it is generally impossible to perform a complete LLF on most modern hard drives (since the mid-1990s) outside the factory,^[11] the term "low-level format" is still used for what could be called the reinitialization of a hard drive to its factory configuration (and even these terms may be misunderstood). Reinitialization should include identifying (and sparing out if possible) any sectors which cannot be written to and read back from the drive, correctly. The term has, however, been used by some to refer to only a portion of that process, in which every sector of the drive is written to; usually by writing a zero byte to every addressable location on the disk, sometimes called zero-filling.

The present ambiguity in the term low-level format seems to be due to both inconsistent documentation on web sites and the belief by many users that any process below a high-level (file system) format must be called a low-level format. Since much of the low level formatting process can today only be performed at the factory, various drive manufacturers describe reinitialization software as LLF utilities on their web sites. Since users generally have no way to determine the difference between a complete LLF and reinitialization (they simply observe running the software results in a hard disk that must be high-level formatted), both the misinformed user and mixed signals from various drive manufacturers have perpetuated this error. Note: Whatever possible misuse of such terms may exist (search hard drive manufacturers' web sites for all these terms), many sites do make such reinitialization utilities available (possibly as bootable floppy diskette or CD image files), to both overwrite every byte and check for damaged sectors on the hard disk.

One popular method for performing only the zero-fill operation on a hard disk is by writing zero-value bytes to the drive using the Unix dd utility with the /dev/zero stream as the input file and the drive itself or a specific partition as the output file, like: (fill the 2TB SCSI/SATA HDD at 2nd connection with zeros, this command may take days, even weeks to complete)

dd if=/dev/zero of=/dev/sdb bs=1M count=2097152

Another method for SCSI disks may use the sg_format^[12] command to issue a low level SCSI FORMAT UNIT command.

Partitioning[link]

Partitioning is the process of writing information into blocks of a storage device or medium that allows access by an operating system. Some operating systems allow the device (or its medium) to appear as multiple devices; i.e. partitioned into multiple devices.

On MS-DOS, Windows, and UNIX-based operating systems (such as BSD, Linux/GNU, OS X) this is normally done with a partition editor, such as fdisk, parted, and Disk Utility. These operating systems support multiple partitions.

In current IBM mainframe OSs derived from OS/360 and DOS/360, such as z/OS and z/VSE, this is done by the INIT command of the ICKDSF utility.^[13] These OSs support only a single partition per device, called a volume. The ICKDSF functions include creating a volume label and writing a Record 0 on every track.

Floppy disks are not partitioned; however depending upon the OS they may require volume information in order to be accessed by the OS.

Partition editors and ICKDSF today do not handle low level functions for HDDs and optical disk drives such as writing timing marks, and they cannot reinitialize a modern disk that has been degaussed or otherwise lost the factory formatting.

[edit] High-level formatting

High-level formatting is the process of setting up an empty file system on the disk and, for PC's, installing a boot sector. This is a fast operation, and is sometimes referred to as quick formatting.

The entire logical drive or partition may optionally be scanned for defects, which may take considerable time.

In the case of floppy disks, both high- and low-level formatting are customarily performed in one pass by the disk formatting software. In recent years,^[when?] most floppies have shipped pre-formatted from the factory as DOS FAT12 floppies.

In current IBM mainframe operating systems derived from OS/360 or DOS/360, this may be done as part of allocating a file, by a utility specific to the file system or, in some older access methods, on the fly as new data are written.

Host protected area[link]

The host protected area, sometimes referred to as hidden protected area,^[14] is an area of a hard drive that is high level formatted so that the area is not normally visible to its operating system (OS).

[edit] Reformatting

Reformatting is a high-level formatting performed on a functioning disk drive to free the contents of its medium. Reformatting is unique to each operating system because what actually is done to existing data varies by OS. The most important aspect of the process is that it frees disk space for use by other data. To actually "erase" everything requires overwriting each block of data on the medium; something that is not done by many PC high-level formatting utilities.

Reformatting often carries the implication that the operating system and all other software will be reinstalled after the format is complete. Rather than fixing an installation suffering from malfunction or security compromise, it is sometimes judged easier to simply reformat everything and start from scratch. Various colloquialism exist for this process, such as "wipe and reload", "nuke and pave", "reimage", etc.

Formatting[link]

DOS, OS/2 and Windows[link]

Under MS-DOS, PC-DOS, OS/2 and Microsoft Windows, disk formatting can be performed by the format command. The format program usually asks for confirmation beforehand to prevent accidental removal of data, but some versions of DOS have an undocumented /AUTOTEST option; if used, the usual confirmation is skipped and the format begins right away. The WM/FormatC macro virus uses this command to format drive C: as soon as a document is opened.

There is also the undocumented /U parameter that performs an unconditional format which under most circumstances overwrites the entire partition,^[15] preventing the recovery of data through software. Note however that the /U switch only works reliably with floppy diskettes (technically because unless /Q is used, floppies are always low-level formatted in addition to high-level formatted). Under certain circumstances with hard drive partitions, however, the /U switch merely prevents the creation of unformat information in the partition to be formatted while otherwise leaving the partition's contents entirely intact (still on disk but marked deleted). In such cases, the user's data remain ripe for recovery with specialist tools such as EnCase or disk editors. Reliance upon /U for secure overwriting of hard drive partitions is therefore inadvisable, and purpose-built tools such as DBAN should be considered instead.

Under OS/2, if you use the /L parameter, which specifies a long format, then format will overwrite the entire partition or logical drive. Doing so enhances the ability of CHKDSK to recover files.

Unix-like operating systems[link]

High-level formatting of disks on these systems is traditionally done using the mkfs command. On Linux (and potentially other systems as well) mkfs is typically a wrapper around filesystem-specific commands which have the name mkfs.fsname, where fsname is the name of the filesystem with which to format the disk.^[16] Some filesystems which are not supported by certain implementations of mkfs have their own manipulation tools; for example Ntfsprogs provides a format utility for the NTFS filesystem.

Some Unix and Unix-like operating systems have higher-level formatting tools, usually for the purpose of making disk formatting easier and/or allowing the user to partition the disk with the same tool. Examples include GNU Parted (and its various GUI frontends such as GParted and the KDE Partition Manager) and the Disk Utility application on Mac OS X.

Recovery of data from a formatted disk[link]

This section may contain original research. Please improve it by verifying the claims made and adding references. Statements consisting only of original research may be removed. More details may be available on the talk page. (March 2011)

As in file deletion by the operating system, data on a disk are not fully erased during every^[17] high-level format. Instead, the area on the disk containing the data is merely marked as available, and retains the old data until it is overwritten. If the disk is formatted with a different file system than the one which previously existed on the partition, some data may be overwritten that wouldn't be if the same file system had been used. However, under some file systems (e.g., NTFS, but not FAT), the file indexes (such as $MFTs under NTFS, inodes under ext2/3, etc.) may not be written to the same exact locations. And if the partition size is increased, even FAT file systems will overwrite more data at the beginning of that new partition.

From the perspective of preventing the recovery of sensitive data through recovery tools, the data must either be completely overwritten (every sector) with random data before the format, or the format program itself must perform this overwriting, as the DOS FORMAT command did with floppy diskettes, filling every data sector with the byte value F6 in hex.

However there are applications and tools, especially used in forensic information technology, that can recover data that has been conventionally erased. In order to avoid the recovery of sensitive data, governmental organization or big companies use information destruction methods like the Gutmann method or the DoD 5220 of the National Industrial Security Program^[18]. For average users there are also special applications that can perform complete data destruction by overwriting previous information. Although there are applications that perform multiple writes a single write is generally all that is needed on modern hard disk drives.

See also[link]

• Data erasure
• Data recovery
• Data remanence
• Drive mapping
• List of default file systems

References[link]

External links[link]

• Formatting Hard Disks and Floppy Disks in Windows NT Workstation Resource Kit, Chapter 17 - Disk and File System Basics
• Secure Deletion of Data from Magnetic and Solid-State Memory by Peter Gutmann
• Differences between a Quick format and a regular format during a "clean" installation of Windows XP from Microsoft Help and Support. Useful for anyone setting up their own computer and needing advice on the subject!
• support.microsoft.com — How to Use the Fdisk Tool and the Format Tool to Partition or Repartition a Hard Disk
• Help: I Got Hacked. Now What Do I Do? -- Microsoft Tech Net: Why you should wipe a compromised drive to the bare metal. Article by Jesper M. Johansson, Ph.D., CISSP, MCSE, MCP+I
• Ultimate Boot CD - Free utility including many useful dos/linux based tools for system maintenance. It's bootable from a CD or USB and has its own operating systems so it's completely independent from external software.