NTFS

NTFS

Developer	Microsoft
Full name	New Technology File System[1]
Introduced	July 1993 (Windows NT 3.1)
Partition identifier	0x07 (MBR) EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 (GPT)
Structures
Directory contents	B+ tree[2]
File allocation	Bitmap
Bad blocks	$badclus
Limits
Max file size	16 EB − 1 KB (format); 16 TB − 64 KB (implementation)[3]
Max number of files	4,294,967,295 (232-1)[3]
Max filename length	255 UTF-16 code units[4]
Max volume size	264 clusters − 1 cluster (format); 256 TB (256 × 10244 bytes) − 64 KB (64 × 1024 bytes) (implementation)[3]
Allowed characters in filenames	In Posix namespace, any UTF-16 code unit (case sensitive) except U+0000 (NUL) and / (slash). In Win32 namespace, any UTF-16 code unit (case insensitive) except U+0000 (NUL) / (slash) \ (backslash) : (colon) * (asterisk) ? (Question mark) " (quote) < (less than) > (greater than) and | (pipe) [4]
Features
Dates recorded	Creation, modification, POSIX change, access
Date range	1 January 1601 – 28 May 60056 (File times are 64-bit numbers counting 100-nanosecond intervals (ten million per second) since 1601, which is 58,000+ years)
Date resolution	100ns
Forks	Yes (see Alternate data streams below)
Attributes	Read-only, hidden, system, archive, not content indexed, off-line, temporary, compressed
File system permissions	ACLs
Transparent compression	Per-file, LZ77 (Windows NT 3.51 onward)
Transparent encryption	Per-file, DESX (Windows 2000 onward), Triple DES (Windows XP onward), AES (Windows XP Service Pack 1, Windows Server 2003 onward)
Data deduplication	Yes[citation needed]
Supported operating systems	Windows NT family (Windows NT 3.1 to Windows NT 4.0, Windows 2000, Windows XP, Windows Server 2003, Windows Vista, Windows Server 2008, Windows 7, Windows Server 2008 R2), Mac OS X, GNU/Linux

NTFS (New Technology File System)^[1] is the standard file system of Windows NT, including its later versions Windows 2000, Windows XP, Windows Server 2003, Windows Server 2008, Windows Vista, and Windows 7.^[5]

NTFS supersedes the FAT file system as the preferred file system for Microsoft’s Windows operating systems. NTFS has several improvements over FAT and HPFS (High Performance File System) such as improved support for metadata and the use of advanced data structures to improve performance, reliability, and disk space utilization, plus additional extensions such as security access control lists (ACL) and file system journaling.

History

In the mid 1980s, Microsoft and IBM formed a joint project to create the next generation of graphical operating system. The result of the project was OS/2, but Microsoft and IBM disagreed on many important issues and eventually separated. OS/2 remained an IBM project. Microsoft started^{[citation needed]} to work on Windows NT. The OS/2 file system HPFS contained several important new features. When Microsoft created their new operating system, they borrowed many of these concepts for NTFS.^[6] Probably as a result of this common ancestry, HPFS and NTFS share the same disk partition identification type code (07). Sharing an ID is unusual since there were dozens of available codes, and other major file systems have their own code. FAT has more than nine (one each for FAT12, FAT16, FAT32, etc.). Algorithms which identify the file system in a partition type 07 must perform additional checks. It is also clear that NTFS owes some of its architectural design to Files-11 used by VMS. Dave Cutler was the main lead for both VMS and Windows NT.

Versions

The NTFS on-disk format has five released versions:

• v1.0 with NT 3.1,^{[citation needed]} released mid-1993
• v1.1 with NT 3.5,^{[citation needed]} released fall 1994
• v1.2 with NT 3.51 (mid-1995) and NT 4 (mid-1996) (occasionally referred to as "NTFS 4.0", because OS version is 4.0)
• v3.0 from Windows 2000 ("NTFS V5.0" or "NTFS5")^[7]
• v3.1 from Windows XP (autumn 2001; "NTFS V5.1"^{[citation needed]})
  • Windows Server 2003 (spring 2003; occasionally "NTFS V5.2"^{[citation needed]})
  • Windows Server 2008 and Windows Vista (mid-2005) (occasionally "NTFS V6.0"^{[citation needed]})
  • Windows Server 2008 R2 and Windows 7 (occasionally "NTFS V6.1"^{[citation needed]}).

V1.0 and V1.1 (and newer) are incompatible: that is, volumes written by NT 3.5x cannot be read by NT 3.1 until an update on the NT 3.5x CD is applied to NT 3.1, which also adds FAT long file name support.^[8] V1.2 supports compressed files, named streams, ACL-based security, etc.^[2] V3.0 added disk quotas, encryption, sparse files, reparse points, update sequence number (USN) journaling, the $Extend folder and its files, and reorganized security descriptors so that multiple files which use the same security setting can share the same descriptor.^[2] V3.1 expanded the Master File Table (MFT) entries with redundant MFT record number (useful for recovering damaged MFT files).

Windows Vista introduced Transactional NTFS, NTFS symbolic links, partition shrinking and self-healing functionality^[9] though these features owe more to additional functionality of the operating system than the file system itself.

The NTFS.sys version (i.e. NTFS v5.0 introduced with Windows 2000) should not be confused with the on-disk NTFS format version (v3.1 since Windows XP).^[10] The NTFS v3.1 on-disk format is unchanged from the introduction of Windows XP and is used in Windows Server 2003, Windows Server 2008, Windows Vista, and Windows 7. The confusion arises when no differentiation is made when features are implemented into the NTFS.sys driver within the Windows OS rather than in the NTFS on-disk format. An incident of this was when Microsoft detailed new features within NTFS in Windows 2000 and they called it NTFS v5.0, yet it is the NTFS.sys driver that is at that version and the on-disk format is only at v3.0.^[7]

Features

NTFS v3.0 includes several new features over its predecessors: sparse file support, disk usage quotas, reparse points, distributed link tracking, and file-level encryption, also known as the Encrypting File System (EFS).

NTFS Log

NTFS is a Journaling file system and uses the NTFS Log ($LogFile) to record metadata changes to the volume.

It is a critical functionality of NTFS (a feature that FAT/FAT32 does not provide) for ensuring that its internal complex data structures (notably the volume allocation bitmap), or data moves performed by the defragmentation API, the modifications to MFT records (such as moves of some variable-length attributes stored in MFT records and attribute lists), and indices (for directories and security descriptors) will remain consistent in case of system crashes, and allow easy rollback of uncommitted changes to these critical data structures when the volume is remounted.

USN Journal

The USN Journal (Update Sequence Number Journal) is a system management feature that records changes to all files, streams and directories on the volume, as well as their various attributes and security settings. The journal is made available for applications to track changes to the volume.^[11] This journal can be enabled or disabled on non-system volumes^[12] and is not enabled by default for a newly added drive.

Hard links and short filenames

Originally included to support the POSIX subsystem in Windows NT,^[13] hard links are similar to directory junctions, but used for files instead of directories. Hard links can only be applied to files on the same volume since an additional filename record is added to the file's MFT record. Short (8.3) filenames are also implemented as additional filename records and directory entries that are linked and updated together. Hard links also have the behavior that changing the size or attributes of a file may not update the directory entries of other links until they are opened.^[14]

Alternate data streams (ADS)

Alternate data streams allow more than one data stream to be associated with a filename, using the filename format "filename:streamname" (e.g., "text.txt:extrastream"). Alternate streams are not listed in Windows Explorer, and their size is not included in the file's size. Only the main stream of a file is preserved when it is copied to a FAT-formatted USB drive, attached to an e-mail, or uploaded to a website. As a result, using alternate streams for critical data may cause problems. NTFS Streams were introduced in Windows NT 3.1, to enable Services for Macintosh (SFM) to store Macintosh resource forks. Although current versions of Windows Server no longer include SFM, third-party Apple Filing Protocol (AFP) products (such as Group Logic's ExtremeZ-IP) still use this feature of the file system.

Malware has used alternate data streams to hide its code;^[15] some malware scanners and other special tools now check for data in alternate streams. Microsoft provides a tool called Streams^[16] to allow users to view streams on a selected volume.

Very small ADS are also added within Internet Explorer (and now also other browsers) to mark files that have been downloaded from external sites: they may be unsafe to run locally and the local shell will require confirmation from the user before opening them.^[17] When the user indicates that he no longer wants this confirmation dialog, this ADS is simply dropped from the MFT entry for downloaded files.

Some media players have also tried to use ADS to store custom metadata to media files, in order to organize the collections, without modifying the effective data content of the media files themselves (using embedded tags when they are supported by the media file formats such as MPEG and OGG containers); these metadata may be displayed in the Windows Explorer as extra information columns, with the help of a registered Windows Shell extension that can parse them, but most media players prefer to use their own separate database instead of ADS for storing these information (notably because ADS are visible to all users of these files, instead of being managed with distinct per-user security settings and having their values defined according to user preferences).

Sparse files

Sparse files are files which contain sparse data sets, which are files with segments stored at different file offsets with no actual storage space used for the space between segments. When a file is read back, the file system driver returns zeros for any data that does not actually exist, so the file may appear to be mostly filled with zeros. Database applications, for instance, sometimes use sparse files.^[18] Because of this, Microsoft has implemented support for efficient storage of sparse files by allowing an application to specify regions of empty (zero) data. An application that reads a sparse file reads it in the normal manner with the file system calculating what data should be returned based upon the file offset. As with compressed files, the actual sizes of sparse files are not taken into account when determining quota limits.^[19][20]

File compression

NTFS can compress files using LZNT1 algorithm (a variant of the LZ77 ^[21] ). Files are compressed in 16-cluster chunks. With 4kB clusters, files are compressed in 64kB chunks. If the compression reduces 64kB of data to 60kB or less, NTFS treats the unneeded 4kB pages like empty sparse file clusters – they are not written. This allows not unreasonable random-access times. However, large compressible files become highly fragmented as then every 64k chunk becomes a smaller fragment. ^{[22] [23]}Compression is not recommended by Microsoft for files exceeding 30MB because of the performance hit.

The best use of compression is for files which are repetitive, written seldom, usually accessed sequentially, and not themselves compressed. LOG files are an ideal example. Compressing files which are less than 4kB or already compressed (like .zip or .jpg or .avi) may make them bigger as well as slower. Avoid compressing executables like .exe and .dll (they may be paged in and out in 4kB pages). Never compress system files used at bootup like drivers or NTLDR or winload.exe or BOOTMGR.

Although read–write access to compressed files is often, but not always ^[24] transparent, Microsoft recommends avoiding compression on server systems and/or network shares holding roaming profiles because it puts a considerable load on the processor.^[25]

Single-user systems with limited hard disk space can benefit from NTFS compression for small files, from 4+kB to 64kB or more, depending on compressibility. Files less than 900 bytes or so are stored with the directory entry in the MFT^[26].

The slowest link in a computer is not the CPU but the speed of the hard drive, so NTFS compression allows the limited, slow storage space to be better used, in terms of both space and (often) speed.^[27] (This assumes that compressed file fragments are stored consecutively.)

NTFS compression can also serve as a replacement for sparse files when a program (e.g., a download manager) is not able to create files without content as sparse files.^{[citation needed]}

Volume Shadow Copy

The Volume Shadow Copy Service (VSS) keeps historical versions of files and folders on NTFS volumes by copying old, newly-overwritten data to shadow copy (copy-on-write). The old file data is overlaid on the new when the user requests a revert to an earlier version. This also allows data backup programs to archive files currently in use by the file system. On heavily loaded systems, Microsoft recommends setting up a shadow copy volume on a separate disk.^[28]

Transactional NTFS

As of Windows Vista, applications can use Transactional NTFS to group changes to files together into a transaction. The transaction will guarantee that all changes happen, or none of them do, and it will guarantee that applications outside the transaction will not see the changes until they are committed.^[29]

It uses similar techniques as those used for Volume Shadow Copies (i.e. copy-on-write) to ensure that overwritten data can be safely rolled back, and a CLFS log to mark the transactions that have still not been committed, or those that have been committed but still not fully applied (in case of system crash during a commit by one of the participants).

Transactional NTFS does not restrict transactions to just the local NTFS volume, but also includes other transactional data or operations in other locations such as data stored in separate volumes, the local registry, or SQL databases, or the current states of system services or remote services. These transactions are coordinated network-wide with all participants using a specific service, the DTC, to ensure that all participants will receive same commit state, and to transport the changes that have been validated by any participant (so that the others can invalidate their local caches for old data or rollback their ongoing uncommitted changes). Transactional NTFS allows, for example, the creation of network-wide consistent distributed filesystems, including with their local live or offline caches.

Encrypting File System (EFS)

Main article: Encrypting File System

EFS provides strong^[30] and user-transparent encryption of any file or folder on an NTFS volume. EFS works in conjunction with the EFS service, Microsoft's CryptoAPI and the EFS File System Run-Time Library (FSRTL). EFS works by encrypting a file with a bulk symmetric key (also known as the File Encryption Key, or FEK), which is used because it takes a relatively small amount of time to encrypt and decrypt large amounts of data than if an asymmetric key cipher is used. The symmetric key that is used to encrypt the file is then encrypted with a public key that is associated with the user who encrypted the file, and this encrypted data is stored in an alternate data stream of the encrypted file. To decrypt the file, the file system uses the private key of the user to decrypt the symmetric key that is stored in the file header. It then uses the symmetric key to decrypt the file. Because this is done at the file system level, it is transparent to the user.^[31] Also, in case of a user losing access to their key, support for additional decryption keys has been built in to the EFS system, so that a recovery agent can still access the files if needed. NTFS-provided encryption and NTFS-provided compression are mutually exclusive; however, NTFS can be used for one and a third-party tool for the other.

The support of EFS is not available in Basic, Home and MediaCenter versions of Windows, and must be activated after installation of Professional, Ultimate and Server versions of Windows or by using enterprise deployment tools within Windows domains.

Quotas

Disk quotas were introduced in NTFS v3. They allow the administrator of a computer that runs a version of Windows that supports NTFS to set a threshold of disk space that users may use. It also allows administrators to keep track of how much disk space each user is using. An administrator may specify a certain level of disk space that a user may use before they receive a warning, and then deny access to the user once they hit their upper limit of space. Disk quotas do not take into account NTFS's transparent file-compression, should this be enabled. Applications that query the amount of free space will also see the amount of free space left to the user who has a quota applied to them.

The support of disk quotas is not available in Basic, Home and MediaCenter versions of Windows, and must be activated after installation of Professional, Ultimate and Server versions of Windows or by using enterprise deployment tools within Windows domains.

Reparse points

This feature was introduced in NTFS v3.^{[citation needed]} Reparse points are used by associating a reparse tag in the user space attribute of a file or directory. When the object manager (see Windows NT line executive) parses a file system name lookup and encounters a reparse attribute, it will reparse the name lookup, passing the user controlled reparse data to every file system filter driver that is loaded into Windows. Each filter driver examines the reparse data to see whether it is associated with that reparse point, and if that filter driver determines a match, then it intercepts the file system call and executes its special functionality. Reparse points are used to implement Volume Mount Points, Directory Junctions, Hierarchical Storage Management, Native Structured Storage, Single Instance Storage, and Symbolic Links.^{[citation needed]}

Volume mount points

Volume mount points are similar to Unix mount points, where the root of another file system is attached to a directory. In NTFS, this allows additional file systems to be mounted without requiring a separate drive letter (such as C: or D:) for each.

Once a volume has been mounted on top of an existing directory of another volume, the contents previously listed in that directory become invisible and are replaced by the content of the root directory of the mounted volume. The mounted volume could still have its own drive letter assigned separately. The file system does not allow volumes to be mutually mounted on each other. Volume mount points can be made to be either persistent (remounted automatically after system reboot) or not persistent (must be manually remounted after reboot).^{[citation needed]}

Mounted volumes may use other file systems than just NTFS; notably they may be remote shared directories, possibly with their own security settings and remapping of access rights according to the remote file system policy.^{[citation needed]}

Directory junctions

Directory junctions are similar to volume mount points, but reference other directories in the file system instead of other volumes. For instance, the directory C:\exampledir with a directory junction attribute that contains a link to D:\linkeddir will automatically refer to the directory D:\linkeddir when it is accessed by a user-mode application.^[2] This function is conceptually similar to symbolic links to directories in Unix, except that the target in NTFS must always be another directory (typical Unix file systems allow the target of a symbolic link to be any type of file) and have the semantics of a hardlink (i.e., they must be immediately resolvable when they are created).^{[citation needed]}

Directory joins (which can be created with the command MKLINK /J junctionName targetDirectory and removed with RMDIR junctionName from a console prompt) are persistent, and resolved on the server side as they share the same security realm of the local system or domain on which the parent volume is mounted and the same security settings for its contents as the content of the target directory; however the junction itself may have distinct security settings. Unlinking a directory junction join does not delete files in the target directory.^{[citation needed]}

Note that some directory junctions are installed by default on Windows Vista, for compatibility with previous versions of Windows, such as Documents and Settings in the root directory of the system drive, which links to the Users physical directory in the root directory of the same volume. However they are hidden by default, and their security settings are set up so that the Windows Explorer will refuse to open them from within the Shell or in most applications, except for the local built-in SYSTEM user or the local Administrators group (both user accounts are used by system software installers). This additional security restriction has probably been made to avoid users of finding apparent duplicate files in the joined directories and deleting them by error, because the semantics of directory junctions is not the same as hardlinks; the reference counting is not used on the target contents and not even on the referenced container itself.^{[citation needed]}

Directory junctions are soft links (they will persist even if the target directory is removed), working as a limited form of symbolic links (with an additional restriction on the location of the target), but it is an optimized version which allows faster processing of the reparse point with which they are implemented, with less overhead than the newer NTFS symbolic links, and can be resolved on the server side (when they are found in remote shared directories).^{[citation needed]}

Symbolic links

Symbolic links (or soft links) were introduced in Windows Vista.^[32] Symbolic links are resolved on the client side. So when a symbolic link is shared, the target is subject to the access restrictions on the client, and not the server.^{[citation needed]}

Symbolic links can be created either to files (created with MKLINK symLink targetFilename) or to directories (created with MKLINK /D symLinkD targetDirectory), but (unlike Unix symbolic links) the semantic of the link must be provided with the created link. The target however need not exist or be available when the symbolic link is created: when the symbolic link will be accessed and the target will be checked for availability, NTFS will also check if it has the correct type (file or directory); it will return a not-found error if the existing target has the wrong type.^{[citation needed]}

They can also reference shared directories on remote hosts or files and subdirectories within shared directories: their target is not mounted immediately at boot, but only temporarily on demand while opening them with the OpenFile() or CreateFile() API. Their definition is persistent on the NTFS volume where they are created (all types of symbolic links can be removed as if they were files, using DEL symLink from a command line prompt or batch).^{[citation needed]}

See also: Computer shortcut

Single Instance Storage (SIS)

When there are several directories that have different, but similar, files, some of these files may have identical content. Single instance storage allows identical files to be merged to one file and create references to that merged file. SIS consists of a file system filter that manages copies, modification and merges to files; and a user space service (or groveler) that searches for files that are identical and need merging. SIS was mainly designed for remote installation servers as these may have multiple installation images that contain many identical files; SIS allows these to be consolidated but, unlike for example hard links, each file remains distinct; changes to one copy of a file will leave others unaltered. This is similar to copy-on-write, which is a technique by which memory copying is not really done until one copy is modified.^[33]

Hierarchical Storage Management (HSM)

Hierarchical Storage Management is a means of transferring files that are not used for some period of time to less expensive storage media. When the file is next accessed, the reparse point on that file determines that it is needed and retrieves it from storage.^{[citation needed]}

Native Structured Storage (NSS)

NSS was an ActiveX document storage technology that has since been discontinued by Microsoft.^{[citation needed]} It allowed ActiveX Documents to be stored in the same multi-stream format that ActiveX uses internally. An NSS file system filter was loaded and used to process the multiple streams transparently to the application, and when the file was transferred to a non-NTFS formatted disk volume it would also transfer the multiple streams into a single stream.^[34]

Interoperability

Details on the implementation's internals are not released, which makes it difficult for third-party vendors to provide tools to handle NTFS.

Microsoft Windows

While the different NTFS versions are for the most part fully forward- and backward-compatible, there are technical considerations for mounting newer NTFS volumes in older versions of Microsoft Windows. This affects dual-booting, and external portable hard drives.

For example, attempting to use an NTFS partition with "Previous Versions" (a.k.a. Volume Shadow Copy) on an operating system that does not support it will result in the contents of those previous versions being lost.^[35]

Mac OS X

Mac OS X 10.3 and later include read-only support for NTFS-formatted partitions. The GPL-licensed NTFS-3G also works on Mac OS X through FUSE and allows reading and writing to NTFS partitions. A performance enhanced commercial version, called Tuxera NTFS for Mac,^[36] is also available from the NTFS-3G developers. Paragon Software Group sells a read-write driver named NTFS for Mac OS X,^[37] which is also included on some models of Seagate hard drives.^[38] Native NTFS write support has been discovered in Mac OS X 10.6 and later, but is not activated by default, although hacks do exist to enable the functionality. However, user reports indicate the functionality is unstable and tends to cause kernel panics, probably the reason why write support has not been enabled or advertised.^[39]

Linux

The ability to read and write to NTFS is provided by the NTFS-3G driver. It is included in most Linux distributions. Other solutions exist as well:

• Linux kernel 2.2: Kernel versions 2.2.0 and later include the ability to read NTFS partitions
• Linux kernel 2.6: Kernel versions 2.6.0 and later contain a driver written by Anton Altaparmakov (University of Cambridge) and Richard Russon. It supports file read, overwrite and resize.
• NTFSMount: A read/write userspace NTFS driver. It provides read-write access to NTFS, excluding writing compressed and encrypted files, changing file ownership, and access rights.^[40]
• Tuxera NTFS: High-performance read/write commercial kernel driver, mainly targeted for embedded devices from Tuxera which also develops the open source NTFS-3G driver.
• NTFS for Linux: A commercial driver with full read/write support available as free and non-free download(s) from Paragon Software Group.
• Captive NTFS (discontinued)^[41]: A 'wrapping' driver which uses Windows' own driver, ntfs.sys.

Note that all three userspace drivers, namely NTFSMount, NTFS-3G and Captive NTFS, are built on the Filesystem in Userspace (FUSE), a Linux kernel module tasked with bridging userspace and kernel code to save and retrieve data. All drivers listed above (except Tuxera NTFS and Paragon NTFS for Linux) are open source (GPL). Due to the complexity of internal NTFS structures, both the built-in 2.6.14 kernel driver and the FUSE drivers disallow changes to the volume that are considered unsafe, to avoid corruption.

Others

eComStation, and FreeBSD offer read-only NTFS support (there is a beta NTFS driver that allows write/delete for eComStation, but is generally considered unsafe). A free third-party tool for BeOS, which was based on NTFS-3G, allows full NTFS read and write. NTFS-3G also works on Mac OS X, FreeBSD, NetBSD, Solaris, QNX and Haiku, in addition to Linux, through FUSE.^[42] A free for personal use read/write driver for MS-DOS called "NTFS4DOS" also exists.^[43] OpenBSD offer read-only NTFS support by default on i386 and amd64 platforms as of version 4.9 released 1. May 2011. ^[44]

Conversion from other file systems

Microsoft provides a tool (convert.exe) to convert to NTFS from other file systems. Supported systems include HPFS (only on Windows NT 3), FAT16 and, on Windows 2000 and later, FAT32.^[45][46]

Resizing

Various third-party tools are all capable of safely resizing NTFS partitions. Starting with Windows Vista Microsoft added the built-in ability to shrink or expand a partition, but this capability is limited because it will not relocate page file fragments or files that have been marked as unmovable. So shrinking will often require relocating or disabling any page file, the index of Windows Search, and any Shadow Copy used by System Restore.

Universal time

For historical reasons, the versions of Windows that do not support NTFS all keep time internally as local zone time, and therefore so do all file systems other than NTFS that are supported by current versions of Windows. However, Windows NT and its descendants keep internal timestamps as UTC and make the appropriate conversions for display purposes. Therefore, NTFS timestamps are in UTC. This means that when files are copied or moved between NTFS and non-NTFS partitions, the OS needs to convert timestamps on the fly. But if some files are moved when daylight saving time (DST) is in effect, and other files are moved when standard time is in effect, there can be some ambiguities in the conversions. As a result, especially shortly after one of the days on which local zone time changes, users may observe that some files have timestamps that are incorrect by one hour. Due to the differences in implementation of DST between the northern and southern hemispheres, this can result in a potential timestamp error of up to 4 hours in any given 12 months.^[47]

Internals

NTFS filesystem permissions on a Windows Vista system.

In NTFS, all file data—file name, creation date, access permissions (by the use of access control lists), and contents—are stored as metadata in the Master File Table. This abstract approach allowed easy addition of file system features during Windows NT's development—an interesting example is the addition of fields for indexing used by the Active Directory software.

NTFS allows any sequence of 16-bit values for name encoding (file names, stream names, index names, etc.). This means UTF-16 codepoints are supported, but the file system does not check whether a sequence is valid UTF-16 (it allows any sequence of short values, not restricted to those in the Unicode standard).

Internally, NTFS uses B+ trees to index file system data. Although complex to implement, this allows faster file look up times in most cases. A file system journal is used to guarantee the integrity of the file system metadata but not individual files' content. Systems using NTFS are known to have improved reliability compared to FAT file systems.^[48]

The Master File Table (MFT) contains metadata about every file, directory, and metafile on an NTFS volume. It includes filenames, locations, size, and permissions. Its structure supports algorithms which minimize disk fragmentation.^{[citation needed]} A directory entry consists of a filename and a "file ID" which is the record number representing the file in the Master File Table. The file ID also contains a reuse count to detect stale references. While this strongly resembles the W_FID of Files-11, other NTFS structures radically differ.

Metafiles

NTFS contains several files which define and organize the file system. In all respects, most of these files are structured like any other user file ($Volume being the most peculiar), but are not of direct interest to file system clients. These metafiles define files, back up critical file system data, buffer file system changes, manage free space allocation, satisfy BIOS expectations, track bad allocation units, and store security and disk space usage information. All content is in an unnamed data stream, unless otherwise indicated.

Segment Number	File Name	Purpose
0	$MFT	Describes all files on the volume, including file names, timestamps, stream names, and lists of cluster numbers where data streams reside, indexes, security identifiers, and file attributes like "read only", "compressed", "encrypted", etc.
1	$MFTMirr	Duplicate of the first vital entries of $MFT, usually 4 entries (4 Kilobyte).
2	$LogFile	Contains transaction log of file system metadata changes.
3	$Volume	Contains information about the volume, namely the volume object identifier, volume label, file system version, and volume flags (mounted, chkdsk requested, requested $LogFile resize, mounted on NT 4, volume serial number updating, structure upgrade request). This data is not stored in a data stream, but in special MFT attributes: If present, a volume object ID is stored in an $OBJECT_ID record; the volume label is stored in a $VOLUME_NAME record, and the remaining volume data is in a $VOLUME_INFORMATION record. Note: volume serial number is stored in file $Boot (below).
4	$AttrDef	A table of MFT attributes which associates numeric identifiers with names.
5	.	Root directory. Directory data is stored in $INDEX_ROOT and $INDEX_ALLOCATION attributes both named $I30.
6	$Bitmap	An array of bit entries: each bit indicates whether its corresponding cluster is used (allocated) or free (available for allocation).
7	$Boot	Volume boot record. This file is always located at the first clusters on the volume. It contains bootstrap code (see NTLDR/BOOTMGR) and a BIOS parameter block including a volume serial number and cluster numbers of $MFT and $MFTMirr. $Boot is usually 8192 bytes long.[citation needed]
8	$BadClus	A file which contains all the clusters marked as having bad sectors. This file simplifies cluster management by the chkdsk utility, both as a place to put newly discovered bad sectors, and for identifying unreferenced clusters. This file contains two data streams, even on volumes with no bad sectors: an unnamed stream contains bad sectors—it is zero length for perfect volumes; the second stream is named $Bad and contains all clusters on the volume not in the first stream.[49]
9	$Secure	Access control list database which reduces overhead having many identical ACLs stored with each file, by uniquely storing these ACLs in this database only (contains two indices: $SII (Standard_Information ID) and $SDH (Security Descriptor Hash) which index the stream named $SDS containing actual ACL table).[2]
10	$UpCase	A table of unicode uppercase characters for ensuring case insensitivity in Win32 and DOS namespaces.
11	$Extend	A filesystem directory containing various optional extensions, such as $Quota, $ObjId, $Reparse or $UsnJrnl.
12 ... 23	Reserved for $MFT extension entries.[50]
usually 24	$Extend\$Quota	Holds disk quota information. Contains two index roots, named $O and $Q.
usually 25	$Extend\$ObjId	Holds distributed link tracking information. Contains an index root and allocation named $O.
usually 26	$Extend\$Reparse	Holds reparse point data (such as symbolic links). Contains an index root and allocation named $R.
27 ...	file.ext	Beginning of regular file entries.

These metafiles are treated specially by Windows and are difficult to directly view: special purpose-built tools are needed.^[51] One such tool is the nfi.exe-"NTFS File Sector Information Utility" that is freely distributed as part of the Microsoft "OEM Support Tools".^[52]

From MFT records to attribute lists, attributes, and streams

For each file (or directory) described in the MFT record, there's a linear repository of stream descriptors (also named attributes), packed together in a variable-length record (also named an attributes list), with extra padding to fill the fixed 1KB size of every MFT record, and that fully describes the effective streams associated with that file.

Each stream (or attribute) itself has a single type (internally just a fixed-size integer in the stored descriptor, but most often handled in applications using an equivalent symbolic name in the FileOpen() or FileCreate() API call), a single optional stream name (completely unrelated to the effective filenames), plus optional associated data for that stream. For NTFS, the standard data of files, or the index data for directories are handled the same way as other data for alternate data streams, or for standard attributes. They are just one of the attributes stored in one or several attribute lists.

• For each file described in the MFT record (or in the non-resident respository of stream descriptors, see below), the stream descriptors identified by their (stream type value, stream name) must be unique. Additionally, NTFS has some ordering constraints for these descriptors.
• There's a predefined null stream type, used to indicate the end of the list of stream descriptors in the streams repository for that file. It must be present as the last stream descriptor in each stream repository (all other storage space available after it will be ignored and just consists in padding bytes to match the record size in the MFT or a cluster size in a non-resident streams repository).
• Some stream types are required and must be present in each MFT record, except unused records that are just indicated by a stream with null stream type.
  • This is the case for the standard attributes that are stored as a fixed-size record and containing the timestamps and other basic single-bit attributes (compatible with those managed by FAT/FAT32 in DOS or Windows 95/98 applications).
• Some stream types cannot have a name and must remain anonymous.
  • This is the case for the standard attributes, or for the preferred NTFS "filename" stream type, or the "short filename" stream type, when it is also present (for compatibility with DOS-like applications, see below). It is also possible for a file to only contain a short filename, in which case it will be the preferred one, as listed in the Windows Explorer.
  • The filename streams stored in the streams repository do not make the file immediately accessible through the hierarchical filesystem. In fact, all the filenames must be indexed separately in at least one separate directory on the same volume, with its own MFT entry and its own security descriptors and attributes, that will reference the MFT entry number for that file. This allows the same file or directory to be "hardlinked" several times from several containers on the same volume, possibly with distinct filenames.
• The default data stream of a regular file is a stream of type $DATA but with an anonymous name, and the ADS's are similar but must be named.
• On the opposite, the default data stream of directories has a distinct type, but are not anonymous: they have a stream name ("$I30" in NTFS 3+) that reflects its indexing format.

All streams of a given file may be displayed by using the nfi.exe-"NTFS File Sector Information Utility" that is freely distributed as part of the Microsoft "OEM Support Tools".^[53]

Resident vs. non-resident data streams

To optimize the storage and reduce the I/O overhead for the very common case of streams with very small associated data, NTFS prefers to place this data within the stream descriptor (if the size of the stream descriptor does not then exceed the maximum size of the MFT record or the maximum size of a single entry within an non-resident stream repository, see below), instead of using the MFT entry space to list clusters containing the data; in that case, the stream descriptor will not store the data directly but will just store an allocation map pointing to the actual data stored elsewhere on the volume. When the stream data can be accessed directly from within the stream descriptor, it is called "resident data" by computer forensics workers. The amount of data which fits is highly dependent on the file's characteristics, but 700 to 800 bytes is common in single-stream files with non-lengthy filenames and no ACLs.

• Some stream descriptors (such as the preferred filename, the basic file attributes, or the main allocation map for each non-resident stream) cannot be made non-resident.
• Encrypted-by-NTFS, sparse data streams, or compressed data streams cannot be made resident.
• The format of the allocation map for non-resident streams depends on its capability of supporting sparse data storage. In the current implementation of NTFS, once a non-resident stream data has been marked and converted as sparse, it cannot be reverted to non-sparse data, so it cannot become resident again, unless this data is fully truncated, discarding the sparse allocation map completely.
• When a non-resident data stream is too much fragmented, so that its effective allocation map cannot fit entirely within the MFT record, the allocation map may be also stored as an non-resident stream, with just a small resident stream containing the indirect allocation map to the effective non-resident allocation map of the non-resident data stream.
• When there are too many streams for a file (including ADS's, extended attributes, or security descriptors), so that their descriptors cannot fit all within the MFT record, a non-resident stream may also be used to store an additional repository for the other stream descriptors (except those few small streams that cannot be non-resident), using the same format as the one used in the MFT record, but without the space constraints of the MFT record.

The NTFS filesystem driver will sometimes attempt to relocate the data of some of these non-resident streams into the streams repository, and will also attempt to relocate the stream descriptors stored in a non-resident repository back to the stream repository of the MFT record, based on priority and preferred ordering rules, and size constraints.

Since resident files do not directly occupy clusters ("allocation units"), it is possible for an NTFS volume to contain more files on a volume than there are clusters. For example, a 74.5 GB partition NTFS formats with 19,543,064 clusters of 4 KB. Subtracting system files (a 64 MB log file, a 2,442,888-byte Bitmap file, and about 25 clusters of fixed overhead) leaves 19,526,158 clusters free for files and indices. Since there are four MFT records per cluster, this volume theoretically could hold almost 4 × 19,526,158 = 78,104,632 resident files.

Limitations

The following are a few limitations of NTFS:

File Names

File names are limited to 255 UTF-16 code points. Certain names are reserved in the volume root directory and cannot be used for files. These are: $MFT, $MFTMirr, $LogFile, $Volume, $AttrDef, . (dot), $Bitmap, $Boot, $BadClus, $Secure, $Upcase, and $Extend;.^[3] (dot) and $Extend are both directories; the others are files. The NT kernel limits full paths to 32,767 UTF-16 code points.

Maximum Volume Size

In theory, the maximum NTFS volume size is 2⁶⁴−1 clusters. However, the maximum NTFS volume size as implemented in Windows XP Professional is 2³²−1 clusters. For example, using 64 kB clusters, the maximum Windows XP NTFS volume size is 256 TB minus 64 kB. Using the default cluster size of 4 kB, the maximum NTFS volume size is 16 TB minus 4 kB. (Both of these are vastly higher than the 128 GB limit lifted in Windows XP SP1.) Because partition tables on master boot record (MBR) disks only support partition sizes up to 2 TB, dynamic or GPT volumes must be used to create NTFS volumes over 2 TB. Booting from a GPT volume to a Windows environment requires a system with UEFI and 64-bit support.^[54]

Maximum File Size

As designed, the maximum NTFS file size is 16 EB (16 × 1024⁶ bytes) minus 1 KB or 18,446,744,073,709,550,592 bytes. As implemented, the maximum NTFS file size is 16 TB minus 64 kB or 17,592,185,978,880 bytes.

Alternate Data Streams

Windows system calls may handle alternate data streams.^[3] Depending on the operating system, utility and remote file system, a file transfer might silently strip data streams.^[3] A safe way of copying or moving files is to use the BackupRead and BackupWrite system calls, which allow programs to enumerate streams, to verify whether each stream should be written to the destination volume and to knowingly skip unwanted streams.^[3]

Developers

NTFS developers include:^[55]

• Tom Miller
• Gary Kimura
• Brian Andrew
• David Goebel

See also

• Comparison of file systems
• Files-11: ODS-2 has similarities to NTFS (compare INDEXF.SYS and $Mft, and BITMAP.SYS and $Bitmap, for examples)
• HPFS, file system created for the OS/2 operating system
• ntfsresize
• Samba (software)

References

1. ^ ^{a b} "Windows XP: Format backup drives using NTFS". Microsoft. September 7, 2006. http://www.microsoft.com/windowsxp/using/setup/tips/advanced/ntfs.mspx. 
2. ^ ^{a b c d e} Mark Russinovich. "Inside Win2K NTFS, Part 1". Microsoft Developer Network. http://msdn2.microsoft.com/en-us/library/ms995846.aspx. Retrieved 2008-04-18. 
3. ^ ^{a b c d e f g} Microsoft TechNet (2003-03-28). "How NTFS Works". Windows Server 2003 Technical Reference. http://technet.microsoft.com/en-us/library/cc781134(v=ws.10).aspx. Retrieved 2011-09-12. 
4. ^ ^{a b} Richard Russon and Yuval Fledel. "NTFS Documentation". http://dubeyko.com/development/FileSystems/NTFS/ntfsdoc.pdf. Retrieved 2011-06-26. 
5. ^ Custer, Helen (1994). Inside the Windows NT File System. Microsoft Press. ISBN 978-1-55615-660-1. 
6. ^ Kozierok, Charles M. (April 17, 2001). "Overview and History of NTFS". PCGuide. http://www.pcguide.com/ref/hdd/file/ntfs/over-c.html. 
7. ^ ^{a b} "Inside Win2K NTFS, Part 1". Microsoft. January 26, 2011. http://msdn.microsoft.com/en-us/library/ms995846.aspx. 
8. ^ "Recovering Windows NT After a Boot Failure on an NTFS Drive". Microsoft. November 1, 2006. http://support.microsoft.com/kb/q129102/. 
9. ^ Loveall, John (2006). "Storage improvements in Windows Vista and Windows Server 2008" (PowerPoint). Microsoft Corporation. pp. 14–20. http://download.microsoft.com/download/5/b/9/5b97017b-e28a-4bae-ba48-174cf47d23cd/STO123_WH06.ppt. Retrieved 2007-09-04. 
10. ^ "New Capabilities and Features of the NTFS 3.1 File System". Microsoft. December 1, 2007. http://support.microsoft.com/kb/310749. 
11. ^ "Change Journals (Windows)". MSDN. http://msdn.microsoft.com/en-us/library/aa363798.aspx. Retrieved 2010-04-16. 
12. ^ "Creating, Modifying, and Deleting a Change Journal (Windows)". MSDN. http://msdn.microsoft.com/en-us/library/aa363877(v=VS.85).aspx. Retrieved 2010-04-16. 
13. ^ MS Windows NT Workstation 4.0 Resource Guide, "POSIX Compatibility"
14. ^ MSDN
15. ^ Malware utilising Alternate Data Streams?, AusCERT Web Log, 21 August 2007
16. ^ Sysinternals Streams v1.56
17. ^ Russinovich, Mark E.; Solomon, David A.; Ionescu, Alex (2009). "File Systems". Windows Internals (5th ed.). Microsoft Press. p. 921. ISBN 978-0-7356-2530-3. "One component in Windows that uses multiple data streams is the Attachment Execution Service[...] depending on which zone the file was downloaded from [...] Windows Explorer might warn the user" 
18. ^ Sparse File Errors: 1450 or 665 due to file fragmentation: Fixes and Workarounds, Microsoft Customer Service and Support (CSS) SQL Server Engineers Blog, March 4, 2009
19. ^ "Sparse Files". MSDN Platform SDK: File Systems. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/sparse_files.asp. Retrieved 2005-05-22. 
20. ^ "Sparse FIles and Disk Quotas". Win32 and COM Development: File Systems. http://msdn2.microsoft.com/en-us/library/aa365565.aspx. Retrieved 2007-12-05. 
21. ^ "File Compression and Decompression". MSDN Platform SDK: File Systems. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/file_compression_and_decompression.asp. Retrieved 2005-08-18. 
22. ^ "Understanding NTFS Compression". http://blogs.msdn.com/b/ntdebugging/archive/2008/05/20/understanding-ntfs-compression.aspx. Retrieved 2011-03-16. 
23. ^ "Shrinking the gap: carving NTFS-compressed files". http://www.forensicfocus.com/index.php?name=Content&pid=179. Retrieved 2011-05-29. 
24. ^ "Read-Only Filegroups and Compression". SQL Server 2008 Books Online (November 2009). http://msdn.microsoft.com/en-us/library/ms190257.aspx. Retrieved 2010-04-20. 
25. ^ "Best practices for NTFS compression in Windows." Microsoft Knowledge Base. Retrieved on 2005-08-18.
26. ^ "How NTFS Works". 2003 March 28. http://technet.microsoft.com/en-us/library/cc781134%28WS.10%29.aspx. 
27. ^ Daily, Sean (January 1998). "Optimizing Disks". IDG books. http://www.windowsitlibrary.com/Content/435/07/8.html. Retrieved 2007-12-17. 
28. ^ "Designing a Shadow Copy Strategy". Microsoft TechNet. http://technet.microsoft.com/en-us/library/cc728305.aspx. Retrieved 2008-01-15. 
29. ^ "Transactional NTFS". MSDN. http://msdn2.microsoft.com/en-us/library/aa363764.aspx. Retrieved 2007-02-02. 
30. ^ Morello, John (February 2007). "Security Watch Deploying EFS: Part 1". Technet Magazine. http://technet.microsoft.com/en-us/magazine/2007.02.securitywatch.aspx. Retrieved 2009-01-25. 
31. ^ How EFS Works, Microsoft Windows 2000 Resource Kit
32. ^ "Symbolic Links (Windows)". MSDN. http://msdn.microsoft.com/en-us/library/aa365680(VS.85).aspx. 
33. ^ "Single Instance Storage in Windows 2000" (PDF). Microsoft Research and Balder Technology Group. http://research.microsoft.com/apps/pubs/default.aspx?id=74261. 
34. ^ Saville, John (date unknown). What is Native Structured Storage? Windows IT Pro. Retrieved from http://www.windowsitpro.com/Article/ArticleID/13785/13785.html.
35. ^ cfsbloggers (July 14, 2006). "How restore points and other recovery features in Windows Vista are affected when dual-booting with Windows XP". The Filing Cabinet. http://blogs.technet.com/filecab/archive/2006/07/14/441829.aspx. Retrieved 2007-03-21. 
36. ^ "Tuxera NTFS for Mac". Tuxera. August 30, 2011. http://www.tuxera.com/products/tuxera-ntfs-for-mac/. Retrieved September 20, 2011. 
37. ^ "NTFS for Mac OS X, communication channel between Mac OS X and Windows". Paragon Software Group. http://www.paragon-software.com/home/ntfs-mac/release.html. Retrieved September 20, 2011. 
38. ^ Seagate Read/Write NTFS driver for Mac OS X
39. ^ Alvares, Milind (Friday, October 2, 2009). "Snow Leopard's hidden NTFS read/write support". http://smokingapples.com/software/tutorials/snow-leopards-hidden-ntfs-readwrite-support/. Retrieved 18 September 2010. 
40. ^ "ntfsmount wiki page on linux-ntfs.org"
41. ^ http://www.jankratochvil.net/project/captive/
42. ^ "NTFS-3G Stable Read/Write Driver". 2009-07-25. http://www.ntfs-3g.org/. 
43. ^ Avira NTFS4DOS Personal at the Wayback Machine (archived June 19, 2010)
44. ^ http://openbsd.com/49.html
45. ^ "How to Convert FAT Disks to NTFS". Microsoft Corporation. 2001-10-25. http://www.microsoft.com/technet/prodtechnol/winxppro/maintain/convertfat.mspx. Retrieved 2007-08-27. 
46. ^ "How to use Convert.exe to convert a partition to the NTFS file system". Microsoft Corporation. 2007-02-12. http://support.microsoft.com/kb/214579. Retrieved 2010-12-26. 
47. ^ "Beating the Daylight Savings Time bug and getting correct file modification times" The Code Project
48. ^ "Microsoft TechNet Resource Kit"
49. ^ $BadClust:$Bad appears to be treated specially in that its allocation of clusters doesn't cause a "multiple cluster reference" error from chkdsk
50. ^ Extension entries are additional MFT records which contain additional attributes which do not fit in the primary record. This could occur if the file is sufficiently fragmented, has many streams, long filenames, complex security, or other rare situations.
51. ^ Since Windows XP, it is very difficult to view a listing of these files: they exist in the root directory's index, but the Win32 interface filters them out. In NT 4.0, the command line dir command would list the metafiles in the root directory if /a were specified. In Windows 2000, dir /a stopped working, but dir /a \$MFT worked.
52. ^ For example to obtain information on the "$MFT"-Master File Table Segment the following command is used: nfi.exe c:\$MFT "OEM Support Tools Phase 3 Service Release 2 Availability". Microsoft Corporation. 2007-02-21. http://support.microsoft.com/kb/253066/. Retrieved 2010-06-16. "Windows NT File System (NTFS) File Sector Information Utility [...] A tool used to dump information about an NTFS volume" 
53. ^ "OEM Support Tools Phase 3 Service Release 2 Availability". Microsoft Corporation. 2007-02-21. http://support.microsoft.com/kb/253066/. Retrieved 2010-06-16. "Windows NT File System (NTFS) File Sector Information Utility [...] A tool used to dump information about an NTFS volume" 
54. ^ http://www.rodsbooks.com/gdisk/booting.html
55. ^ Custer, Helen (1994). Inside the Windows NT File System. Microsoft Press. p. vii. ISBN 978-1-55615-660-1. 

Further reading

• Bolosky, William J.; Corbin, Scott; Goebel, David; & Douceur, John R. (date) (PDF). Single Instance Storage in Windows 2000. Microsoft Research & Balder Technology Group, Inc.. http://research.microsoft.com/pubs/74261/WSS2000.pdf. Retrieved 2010-01-22. 
• Custer, Helen (1994). Inside the Windows NT File System. Microsoft Press. ISBN 978-1-55615-660-1. 
• Nagar, Rajeev (1997). Windows NT File System Internals: A Developer's Guide. O'Reilly. ISBN 978-1-56592-249-5. 

External links

• Documentation:
  • Microsoft NTFS Technical Reference
  • FSUtil file operations, useful for manipulating or creating files in an NTFS volume from Windows
  • NTFS.com – documentation and resources for NTFS
  • Low-level description of NTFS disk structures from the Linux-NTFS project
• Implementations:
  • NTFS-3G – NTFS-3G an open source read/write NTFS driver for Linux, FreeBSD, Mac OS X, NetBSD, Solaris and Haiku.
  • Linux-NTFS – an open source project to add NTFS support to the Linux kernel (write support is limited), and write POSIX-compatible utilities for accessing and manipulating NTFS (ntfsprogs; includes ntfsls, ntfsresize, ntfsclone, etc.). Linux NTFS FAQ and howto
  • Captive NTFS – Captive NTFS a shim which used the Windows NTFS driver to access NTFS file systems under Linux
  • Change Log Journal Parser – useful tool for parsing the Windows NTFS Change Log Journal on live systems.