NTFS

Scalability

NTFS is optimized for 4 KB clusters, but supports a maximum cluster size of 2 MB. (Earlier implementation supports up to 64 KB)^[4] The maximum NTFS volume size that the specification can support is 2⁶⁴ − 1 clusters, but not all implementations achieve this theoretical maximum, as discussed below.

The maximum NTFS volume size implemented in Windows XP Professional is 2³² − 1 clusters, partly due to partition table limitations. For example, using 64 KB clusters, the maximum size Windows XP NTFS volume is 256 TB minus 64 KB. Using the default cluster size of 4 KB, the maximum NTFS volume size is 16 TB minus 4 KB. Both of these are vastly higher than the 128 GB limit in Windows XP SP1. Because partition tables on master boot record (MBR) disks support only partition sizes up to 2 TB, multiple GUID Partition Table (GPT or "dynamic") volumes must be combined to create a single NTFS volume larger than 2 TB. Booting from a GPT volume to a Windows environment in a Microsoft supported way requires a system with Unified Extensible Firmware Interface (UEFI) and 64-bit support.^[17]

The NTFS maximum theoretical limit on the size of individual files is 16 EiB (16 × 1024⁶ or 2⁶⁴ bytes) minus 1 KB, which totals to 18,446,744,073,709,550,592 bytes. With Windows 10 version 1709 and Windows Server 2019, the maximum implemented file size is 8 PB minus 2 MB or 9,007,199,252,643,840 bytes.^[4]

Journaling

NTFS is a journaling file system and uses the NTFS Log ($LogFile) to record metadata changes to the volume. It is a feature that FAT does not provide and critical for NTFS to ensure that its complex internal data structures will remain consistent in case of system crashes or data moves performed by the defragmentation API, and allow easy rollback of uncommitted changes to these critical data structures when the volume is remounted. Notably affected structures are the volume allocation bitmap, modifications to MFT records such as moves of some variable-length attributes stored in MFT records and attribute lists, and indices for directories and security descriptors.

The USN Journal (Update Sequence Number Journal) is a system management feature that records (in $Extend\$UsnJrnl) changes to files, streams and directories on the volume, as well as their various attributes and security settings. The journal is made available for applications to track changes to the volume.^[18] This journal can be enabled or disabled on non-system volumes.^[19]

Hard links

The hard link feature allows different file names to directly refer to the same file contents. Hard links are similar to directory junctions, but refer to files instead. Hard links may link only to files in the same volume, because each volume has its own MFT. Hard links have their own file metadata, so a change in file size or attributes under one hard link may not update the others until they are opened.^[20] Hard links were originally included to support the POSIX subsystem in Windows NT.^[21]

Windows uses hard links to support short (8.3) filenames in NTFS. Operating system support is needed because there are legacy applications that can work only with 8.3 filenames. In this case, an additional filename record and directory entry is added, but both 8.3 and long file name are linked and updated together, unlike a regular hard link.

The NTFS file system has a limit of 1024 hard links on a file.^[22]

Alternate data streams (ADS)

Alternate data streams allow more than one data stream to be associated with a filename (a fork), using the format "filename:streamname" (e.g., "text.txt:extrastream").

NTFS Streams were introduced in Windows NT 3.1, to enable Services for Macintosh (SFM) to store resource forks. Although current versions of Windows Server no longer include SFM, third-party Apple Filing Protocol (AFP) products (such as GroupLogic's ExtremeZ-IP) still use this feature of the file system. Very small ADS (named "Zone.Identifier") are added by Internet Explorer and recently by other browsers to mark files downloaded from external sites as possibly unsafe to run; the local shell would then require user confirmation before opening them.^[23] When the user indicates that they no longer want this confirmation dialog, this ADS is deleted.

Alternate streams are not listed in Windows Explorer, and their size is not included in the file's size. When the file is copied or moved to another file system without ADS support the user is warned that alternate data streams cannot be preserved. No such warning is typically provided if the file is attached to an e-mail, or uploaded to a website. Thus, using alternate streams for critical data may cause problems. Microsoft provides a tool called Streams^[24] to view streams on a selected volume. Starting with Windows PowerShell 3.0, it is possible to manage ADS natively with six cmdlets: Add-Content, Clear-Content, Get-Content, Get-Item, Remove-Item, Set-Content.^[25]

Malware has used alternate data streams to hide code.^[26] As a result, malware scanners and other special tools now check for alternate data streams.

File compression

NTFS can compress files using LZNT1 algorithm (a variant of LZ77)^[27] Files are compressed in 16 cluster chunks. With 4 KB clusters, files are compressed in 64 KB chunks. The compression algorithms in NTFS are designed to support cluster sizes of up to 4 KB. When the cluster size is greater than 4 KB on an NTFS volume, NTFS compression is not available.^[28] If the compression reduces 64 KB of data to 60 KB or less, NTFS treats the unneeded 4 KB pages like empty sparse file clusters—they are not written. This allows for reasonable random-access times as the OS just has to follow the chain of fragments. However, large compressible files become highly fragmented since every chunk smaller than 64 KB becomes a fragment.^[29][30] According to research by Microsoft's NTFS Development team, 50–60 GB is a reasonable maximum size for a compressed file on an NTFS volume with a 4 KB (default) cluster (block) size. This reasonable maximum size decreases sharply for volumes with smaller cluster sizes.^[31] Single-user systems with limited hard disk space can benefit from NTFS compression for small files, from 4 KB to 64 KB or more, depending on compressibility. Files smaller than approximately 900 bytes are stored within the directory entry of the MFT.^[32]

Flash memory, such as SSD drives do not have the head movement delays of hard disk drives, so fragmentation has only a smaller penalty. Users of fast multi-core processors will find improvements in application speed by compressing their applications and data as well as a reduction in space used.^[33] Note that SSDs with Sandforce controllers already compress data. However, since less data is transferred, there is a reduction in I/Os.

Compression works best with files that have repetitive content, are seldom written, are usually accessed sequentially, and are not themselves compressed. Log files are an ideal example.

If system files that are needed at boot time (such as drivers, NTLDR, winload.exe, or BOOTMGR) are compressed, the system may fail to boot correctly, because decompression filters are not yet loaded.^[34] Later editions of Windows do not allow important system files to be compressed.

Files may be compressed or decompressed individually (via changing the advanced attributes) for a drive, directory, or directory tree, becoming a default for the files inside.

Although read–write access to compressed files is mostly^[35] transparent, Microsoft recommends avoiding compression on server systems and/or network shares holding roaming profiles, because it puts a considerable load on the processor.^[36]

Sparse files

A sparse file: Empty bytes don't need to be saved, thus they can be represented by metadata.

Sparse files are files interspersed with empty segments for which no actual storage space is used. To the applications, the file looks like an ordinary file with empty regions seen as regions filled with zeros.^[37]

Database applications, for instance, may use sparse files.^[38] As with compressed files, the actual sizes of sparse files are not taken into account when determining quota limits.^[39]

Volume Shadow Copy

The Volume Shadow Copy Service (VSS) keeps historical versions of files and folders on NTFS volumes by copying old, newly overwritten data to shadow copy via copy-on-write technique. The user may later request an earlier version to be recovered. This also allows data backup programs to archive files currently in use by the file system. On heavily loaded systems, Microsoft recommends setting up a shadow copy volume on a separate disk.^[40]

Windows Vista also introduced persistent shadow copies for use with System Restore and Previous Versions features. Persistent shadow copies, however, are deleted when an older operating system mounts that NTFS volume. This happens because the older operating system does not understand the newer format of persistent shadow copies.^[41]

Transactions

As of Windows Vista, applications can use Transactional NTFS (TxF) to group multiple changes to files together into a single transaction. The transaction will guarantee that either all of the changes happen, or none of them do, and that no application outside the transaction will see the changes until they are committed.^[42]

It uses similar techniques as those used for Volume Shadow Copies (i.e. copy-on-write) to ensure that overwritten data can be safely rolled back, and a CLFS log to mark the transactions that have still not been committed, or those that have been committed but still not fully applied (in case of system crash during a commit by one of the participants).

Transactional NTFS does not restrict transactions to just the local NTFS volume, but also includes other transactional data or operations in other locations such as data stored in separate volumes, the local registry, or SQL databases, or the current states of system services or remote services. These transactions are coordinated network-wide with all participants using a specific service, the DTC, to ensure that all participants will receive same commit state, and to transport the changes that have been validated by any participant (so that the others can invalidate their local caches for old data or rollback their ongoing uncommitted changes). Transactional NTFS allows, for example, the creation of network-wide consistent distributed file systems, including with their local live or offline caches.

Microsoft now advises against using TxF: "Microsoft strongly recommends developers utilize alternative means..." since "TxF may not be available in future versions of Microsoft Windows".^[43]

Security

In NTFS, each file or folder is assigned a security descriptor that defines its owner and contains two access control lists (ACLs). The first ACL, called discretionary access control list (DACL), defines exactly what type of interactions (e.g. reading, writing, executing or deleting) are allowed or forbidden by which user or groups of users. For example, files in the C:\Program Files folder may be read and executed by all users but modified only by a user holding administrative privileges.^[44] Windows Vista adds mandatory access control info to DACLs. DACLs are the primary focus of User Account Control in Windows Vista and later.

The second ACL, called system access control list (SACL), defines which interactions with the file or folder are to be audited and whether they should be logged when the activity is successful, failed or both. For example, auditing can be enabled on sensitive files of a company, so that its managers get to know when someone tries to delete them or make a copy of them, and whether he or she succeeds.^[44]

Encryption

Encrypting File System (EFS) provides strong^[45] and user-transparent encryption of any file or folder on an NTFS volume. EFS works in conjunction with the EFS service, Microsoft's CryptoAPI and the EFS File System Run-Time Library (FSRTL). EFS works by encrypting a file with a bulk symmetric key (also known as the File Encryption Key, or FEK), which is used because it takes a relatively small amount of time to encrypt and decrypt large amounts of data than if an asymmetric key cipher is used. The symmetric key that is used to encrypt the file is then encrypted with a public key that is associated with the user who encrypted the file, and this encrypted data is stored in an alternate data stream of the encrypted file. To decrypt the file, the file system uses the private key of the user to decrypt the symmetric key that is stored in the file header. It then uses the symmetric key to decrypt the file. Because this is done at the file system level, it is transparent to the user.^[46] Also, in case of a user losing access to their key, support for additional decryption keys has been built into the EFS system, so that a recovery agent can still access the files if needed. NTFS-provided encryption and NTFS-provided compression are mutually exclusive; however, NTFS can be used for one and a third-party tool for the other.

The support of EFS is not available in Basic, Home, and MediaCenter versions of Windows, and must be activated after installation of Professional, Ultimate, and Server versions of Windows or by using enterprise deployment tools within Windows domains.

Quotas

Disk quotas were introduced in NTFS v3. They allow the administrator of a computer that runs a version of Windows that supports NTFS to set a threshold of disk space that users may use. It also allows administrators to keep track of how much disk space each user is using. An administrator may specify a certain level of disk space that a user may use before they receive a warning, and then deny access to the user once they hit their upper limit of space. Disk quotas do not take into account NTFS's transparent file-compression, should this be enabled. Applications that query the amount of free space will also see the amount of free space left to the user who has a quota applied to them.

Reparse points

Introduced in NTFS v3, NTFS reparse points are used by associating a reparse tag in the user space attribute of a file or directory. Microsoft includes several default tags including symbolic links, directory junction points and volume mount points. When the } parses a file system name lookup and encounters a reparse attribute, it will reparse the name lookup, passing the user controlled reparse data to every file system filter driver that is loaded into Windows. Each filter driver examines the reparse data to see whether it is associated with that reparse point, and if that filter driver determines a match, then it intercepts the file system request and performss its special functionality.

Resizing

Starting with Windows Vista Microsoft added the built-in ability to shrink or expand a partition. However, this ability does not relocate page file fragments or files that have been marked as unmovable, so shrinking a volume will often require relocating or disabling any page file, the index of Windows Search, and any Shadow Copy used by System Restore. Various third-party tools are capable of resizing NTFS partitions.

Partition Boot Sector

This boot partition format is roughly based upon the earlier FAT filesystem, but the fields are in different locations. Some of these fields, especially the "sectors per track," "number of heads" and "hidden sectors" fields may contain dummy values on drives where they either don't make sense or aren't determinable.

The OS first looks at the 8 bytes at 0x30 to find the cluster number of the $MFT, then multiplies that number by the number of sectors per cluster (1 byte found at 0x0D). This value is the sector offset (LBA) to the $MFT, which is described below.

Master File Table

In NTFS, all file, directory and metafile data—file name, creation date, access permissions (by the use of access control lists), and size—are stored as metadata in the Master File Table (MFT). This abstract approach allowed easy addition of file system features during Windows NT's development—an example is the addition of fields for indexing used by the Active Directory software. This also enables fast file search software to locate named local files and folders included in the MFT very quickly, without requiring any other index.

The MFT structure supports algorithms which minimize disk fragmentation.^[51] A directory entry consists of a filename and a "file ID", which is the record number representing the file in the Master File Table. The file ID also contains a reuse count to detect stale references. While this strongly resembles the W_FID of Files-11, other NTFS structures radically differ.

Two copies of the MFT are stored in case of corruption. If the first record is corrupted, NTFS reads the second record to find the MFT mirror file. Locations for both files are stored in the boot sector.^[52]

Metafiles

NTFS contains several files that define and organize the file system. In all respects, most of these files are structured like any other user file ($Volume being the most peculiar), but are not of direct interest to file system clients. These metafiles define files, back up critical file system data, buffer file system changes, manage free space allocation, satisfy BIOS expectations, track bad allocation units, and store security and disk space usage information. All content is in an unnamed data stream, unless otherwise indicated.

These metafiles are treated specially by Windows, handled directly by the NTFS.SYS driver and are difficult to directly view: special purpose-built tools are needed.^[53] As of Windows 7, the NTFS driver completely prohibits user access, resulting in a BSoD whenever an attempt to execute a metadata file is made. One such tool is the nfi.exe ("NTFS File Sector Information Utility") that is freely distributed as part of the Microsoft "OEM Support Tools". For example, to obtain information on the "$MFT"-Master File Table Segment the following command is used: nfi.exe c:\$MFT^[54] Another way to bypass the restriction is to use 7-zip's file manager and go to the low-level NTFS path \\.\X:\ (where X:\ resembles any drive/partition). Here, 3 new folders will appear: $EXTEND, [DELETED] (a pseudo-folder that 7-zip uses to attach files deleted from the file system to view), and [SYSTEM] (another pseudo-folder that contains all the NTFS metadata files). This trick can be used from removable devices (USB flash drives, external hard drives, SD Cards, etc.) inside Windows, but doing this on the active partition requires offline access (namely WinRE).

Attribute lists, attributes, and streams

For each file (or directory) described in the MFT record, there is a linear repository of stream descriptors (also named attributes), packed together in one or more MFT records (containing the so-called attributes list), with extra padding to fill the fixed 1 KB size of every MFT record, and that fully describes the effective streams associated with that file.

Each attribute has an attribute type (a fixed-size integer mapping to an attribute definition in file $AttrDef), an optional attribute name (for example, used as the name for an alternate data stream), and a value, represented in a sequence of bytes. For NTFS, the standard data of files, the alternate data streams, or the index data for directories are stored as attributes.

According to $AttrDef, some attributes can be either resident or non-resident. The $DATA attribute, which contains file data, is such an example. When the attribute is resident (which is represented by a flag), its value is stored directly in the MFT record. Otherwise, clusters are allocated for the data, and the cluster location information is stored as data runs in the attribute.

• For each file in the MFT, the attributes identified by attribute type, attribute name must be unique. Additionally, NTFS has some ordering constraints for these attributes.
• There is a predefined null attribute type, used to indicate the end of the list of attributes in one MFT record. It must be present as the last attribute in the record (all other storage space available after it will be ignored and just consists of padding bytes to match the record size in the MFT).
• Some attribute types are required and must be present in each MFT record, except unused records that are just indicated by null attribute types.
  • This is the case for $STANDARD_INFORMATION attribute that is stored as a fixed-size record and containing the timestamps and other basic single-bit attributes (compatible with those managed by FAT in DOS or Windows 9x).
• Some attribute types cannot have a name and must remain anonymous.
  • This is the case for the standard attributes, or for the preferred NTFS "filename" attribute type, or the "short filename" attribute type, when it is also present (for compatibility with DOS-like applications, see below). It is also possible for a file to contain only a short filename, in which case it will be the preferred one, as listed in the Windows Explorer.
  • The filename attributes stored in the attribute list do not make the file immediately accessible through the hierarchical file system. In fact, all the filenames must be indexed separately in at least one separate directory on the same volume, with its own MFT record and its own security descriptors and attributes, that will reference the MFT record number for that file. This allows the same file or directory to be "hardlinked" several times from several containers on the same volume, possibly with distinct filenames.
• The default data stream of a regular file is a stream of type $DATA but with an anonymous name, and the ADSs are similar but must be named.
• On the opposite, the default data stream of directories has a distinct type, but are not anonymous: they have an attribute name ("$I30" in NTFS 3+) that reflects its indexing format.

All attributes of a given file may be displayed by using the nfi.exe ("NTFS File Sector Information Utility") that is freely distributed as part of the Microsoft "OEM Support Tools".^[54]

Windows system calls may handle alternate data streams.^[3] Depending on the operating system, utility and remote file system, a file transfer might silently strip data streams.^[3] A safe way of copying or moving files is to use the BackupRead and BackupWrite system calls, which allow programs to enumerate streams, to verify whether each stream should be written to the destination volume and to knowingly skip unwanted streams.^[3]

Resident vs. non-resident attributes

To optimize the storage and reduce the I/O overhead for the very common case of attributes with very small associated value, NTFS prefers to place the value within the attribute itself (if the size of the attribute does not then exceed the maximum size of an MFT record), instead of using the MFT record space to list clusters containing the data; in that case, the attribute will not store the data directly but will just store an allocation map (in the form of data runs) pointing to the actual data stored elsewhere on the volume.^[55] When the value can be accessed directly from within the attribute, it is called "resident data" (by computer forensics workers). The amount of data that fits is highly dependent on the file's characteristics, but 700 to 800 bytes is common in single-stream files with non-lengthy filenames and no ACLs.

• Some attributes (such as the preferred filename, the basic file attributes) cannot be made non-resident. For non-resident attributes, their allocation map must fit within MFT records.
• Encrypted-by-NTFS, sparse data streams, or compressed data streams cannot be made resident.
• The format of the allocation map for non-resident attributes depends on its capability of supporting sparse data storage. In the current implementation of NTFS, once a non-resident data stream has been marked and converted as sparse, it cannot be changed back to non-sparse data, so it cannot become resident again, unless this data is fully truncated, discarding the sparse allocation map completely.
• When a non-resident attribute is so fragmented, that its effective allocation map cannot fit entirely within one MFT record, NTFS stores the attribute in multiple records. The first one among them is called the base record, while the others are called extension records. NTFS creates a special attribute $ATTRIBUTE_LIST to store information mapping different parts of the long attribute to the MFT records, which means the allocation map may be split into multiple records. The $ATTRIBUTE_LIST itself can also be non-resident, but its own allocation map must fit within one MFT record.
• When there are too many attributes for a file (including ADS's, extended attributes, or security descriptors), so that they cannot fit all within the MFT record, extension records may also be used to store the other attributes, using the same format as the one used in the base MFT record, but without the space constraints of one MFT record.

The allocation map is stored in a form of data runs with compressed encoding. Each data run represents a contiguous group of clusters that store the attribute value. For files on a multi-GB volume, each entry can be encoded as 5 to 7 bytes, which means a 1 KB MFT record can store about 100 such data runs. However, as the $ATTRIBUTE_LIST also has a size limit, it is dangerous to have more than 1 million fragments of a single file on an NTFS volume, which also implies that it is in general not a good idea to use NTFS compression on a file larger than 10 GB.^[56]

The NTFS file system driver will sometimes attempt to relocate the data of some of the attributes that can be made non-resident into the clusters, and will also attempt to relocate the data stored in clusters back to the attribute inside the MFT record, based on priority and preferred ordering rules, and size constraints.

Since resident files do not directly occupy clusters ("allocation units"), it is possible for an NTFS volume to contain more files on a volume than there are clusters. For example, a 74.5 GB partition NTFS formats with 19,543,064 clusters of 4 KB. Subtracting system files (a 64 MB log file, a 2,442,888-byte Bitmap file, and about 25 clusters of fixed overhead) leaves 19,526,158 clusters free for files and indices. Since there are four MFT records per cluster, this volume theoretically could hold almost 4 × 19,526,158= 78,104,632 resident files.

Opportunistic locks

Opportunistic file locks (oplocks) allow clients to alter their buffering strategy for a given file or stream in order to increase performance and reduce network use.^[57] Oplocks apply to the given open stream of a file and do not affect oplocks on a different stream.

Oplocks can be used to transparently access files in the background. A network client may avoid writing information into a file on a remote server if no other process is accessing the data, or it may buffer read-ahead data if no other process is writing data.

Windows supports four different types of oplocks:

• Level 2 (or shared) oplock: multiple readers, no writers (i.e. read caching).
• Level 1 (or exclusive) oplock: exclusive access with arbitrary buffering (i.e. read and write caching).
• Batch oplock (also exclusive): a stream is opened on the server, but closed on the client machine (i.e. read, write and handle caching).
• Filter oplock (also exclusive): applications and file system filters can "back out" when others try to access the same stream (i.e. read and write caching) (since Windows 2000)

Opportunistic locks have been enhanced in Windows 7 and Windows Server 2008 R2 with per-client oplock keys.^[58]

Time

Windows NT and its descendants keep internal timestamps as UTC and make the appropriate conversions for display purposes; all NTFS timestamps are in UTC.

For historical reasons, the versions of Windows that do not support NTFS all keep time internally as local zone time, and therefore so do all file systems – other than NTFS – that are supported by current versions of Windows. This means that when files are copied or moved between NTFS and non-NTFS partitions, the OS needs to convert timestamps on the fly. But if some files are moved when daylight saving time (DST) is in effect, and other files are moved when standard time is in effect, there can be some ambiguities in the conversions. As a result, especially shortly after one of the days on which local zone time changes, users may observe that some files have timestamps that are incorrect by one hour. Due to the differences in implementation of DST in different jurisdictions, this can result in a potential timestamp error of up to 4 hours in any given 12 months.^[59]