Btrfs

As of version 5.0 of the Linux kernel, Btrfs implements the following features:[32][33]

• Mostly self-healing in some configurations due to the nature of copy-on-write
• Online defragmentation and an autodefrag mount option[24]
• Online volume growth and shrinking
• Online block device addition and removal
• Online balancing (movement of objects between block devices to balance load)
• Offline filesystem check[34]
• Online data scrubbing for finding errors and automatically fixing them for files with redundant copies
• RAID 0, RAID 1, and RAID 10[35]
• Subvolumes (one or more separately mountable filesystem roots within each disk partition)
• Transparent compression via zlib, LZO[6] and (since 4.14) ZSTD,[7] configurable per file or volume[36][37]
• Atomic writable (via copy-on-write) or read-only[38] Snapshots of subvolumes
• File cloning (reflink, copy-on-write) via cp --reflink <source file> <destination file>[39]
• Checksums on data and metadata (CRC-32C[40])
• In-place conversion from ext3/4 to Btrfs (with rollback). This feature regressed around btrfs-progs version 4.0, rewritten from scratch in 4.6.[41]
• Union mounting of read-only storage, known as file system seeding (read-only storage used as a copy-on-write backing for a writable Btrfs)[42]
• Block discard (reclaims space on some virtualized setups and improves wear leveling on SSDs with TRIM)
• Send/receive (saving diffs between snapshots to a binary stream)[43]
• Incremental backup[44]
• Out-of-band data deduplication (requires userspace tools)[9]
• Ability to handle swap files and swap partitions

• Hierarchical per-subvolume quotas[45]
• RAID 5, RAID 6[46]

• In-band data deduplication[32]
• Online filesystem check[47]
• RAID with up to six parity devices, surpassing the reliability of RAID 5 and RAID 6[48]
• Object-level RAID 0, RAID 1, and RAID 10
• Encryption[8][49]
• Persistent read and write cache (L2ARC + ZIL, lvmcache, etc)

In 2009, Btrfs was expected to offer a feature set comparable to ZFS, developed by Sun Microsystems.[50] After Oracle's acquisition of Sun in 2009, Mason and Oracle decided to continue with Btrfs development.[51]

Btrfs provides a clone operation that atomically creates a copy-on-write snapshot of a file. Such cloned files are sometimes referred to as reflinks, in light of the proposed associated Linux kernel system call.[52]

By cloning, the file system does not create a new link pointing to an existing inode; instead, it creates a new inode that initially shares the same disk blocks with the original file. As a result, cloning works only within the boundaries of the same Btrfs file system, but since version 3.6 of the Linux kernel it may cross the boundaries of subvolumes under certain circumstances.[53][54] The actual data blocks are not duplicated; at the same time, due to the copy-on-write (CoW) nature of Btrfs, modifications to any of the cloned files are not visible in the original file and vice versa.[55]

Cloning should not be confused with hard links, which are directory entries that associate multiple file names with actual files on a file system. While hard links can be taken as different names for the same file, cloning in Btrfs provides independent files that initially share all their disk blocks.[55][56]

Support for this Btrfs feature was added in version 7.5 of the GNU coreutils, via the --reflink option to the cp command.[57][58]

In addition to data cloning (FICLONE), Btrfs also supports for out-of-band deduplication via FIDEDUPERANGE. This functionality allows two files with (even partially) identical data to share storage.[59][9]

A Btrfs subvolume can be thought of as a separate POSIX file namespace, mountable separately by passing subvol or subvolid options to the mount(8) utility. It can also be accessed by mounting the top-level subvolume, in which case subvolumes are visible and accessible as its subdirectories.[60]

Subvolumes can be created at any place within the file system hierarchy, and they can also be nested. Nested subvolumes appear as subdirectories within their parent subvolumes, similarly to the way a top-level subvolume presents its subvolumes as subdirectories. Deleting a subvolume is not possible until all subvolumes below it in the nesting hierarchy are deleted; as a result, top-level subvolumes cannot be deleted.[61]

Any Btrfs file system always has a default subvolume, which is initially set to be the top-level subvolume, and is mounted by default if no subvolume selection option is passed to mount. The default subvolume can be changed as required.[61]

A Btrfs snapshot is a subvolume that shares its data (and metadata) with some other subvolume, using Btrfs' copy-on-write capabilities, and modifications to a snapshot are not visible in the original subvolume. Once a writable snapshot is made, it can be treated as an alternate version of the original file system. For example, to roll back to a snapshot, a modified original subvolume needs to be unmounted and the snapshot needs to be mounted in its place. At that point, the original subvolume may also be deleted.[60]

The copy-on-write (CoW) nature of Btrfs means that snapshots are quickly created, while initially consuming very little disk space. Since a snapshot is a subvolume, creating nested snapshots is also possible. Taking snapshots of a subvolume is not a recursive process; thus, if a snapshot of a subvolume is created, every subvolume or snapshot that the subvolume already contains is mapped to an empty directory of the same name inside the snapshot.[60][61]

Taking snapshots of a directory is not possible, as only subvolumes can have snapshots. However, there is a workaround that involves reflinks spread across subvolumes: a new subvolume is created, containing cross-subvolume reflinks to the content of the targeted directory. Having that available, a snapshot of this new volume can be created.[53]

A subvolume in Btrfs is quite different from a traditional Logical Volume Manager (LVM) logical volume. With LVM, a logical volume is a separate block device, while a Btrfs subvolume is not and it cannot be treated or used that way.[60] Making dd or LVM snapshots of btrfs leads to dataloss if either the original or the copy is mounted while both are on the same computer.[62]

Given any pair of subvolumes (or snapshots), Btrfs can generate a binary diff between them (by using the btrfs send command) that can be replayed later (by using btrfs receive), possibly on a different Btrfs file system. The send–receive feature effectively creates (and applies) a set of data modifications required for converting one subvolume into another.[43][63]

The send/receive feature can be used with regularly scheduled snapshots for implementing a simple form of file system master–slave replication, or for the purpose of performing incremental backups.[43][63]

A quota group (or qgroup) imposes an upper limit to the space a subvolume or snapshot may consume. A new snapshot initially consumes no quota because its data is shared with its parent, but thereafter incurs a charge for new files and copy-on-write operations on existing files. When quotas are active, a quota group is automatically created with each new subvolume or snapshot. These initial quota groups are building blocks which can be grouped (with the btrfs qgroup command) into hierarchies to implement quota pools.[45]

Quota groups only apply to subvolumes and snapshots, while having quotas enforced on individual subdirectories, users, or user groups is not possible. However, workarounds are possible by using different subvolumes for all users or user groups that require a quota to be enforced.

As the result of having very little metadata anchored in fixed locations, Btrfs can warp to fit unusual spatial layouts of the backend storage devices. The btrfs-convert tool exploits this ability to do an in-place conversion of an ext2/3/4 or ReiserFS file system, by nesting the equivalent Btrfs metadata in its unallocated space—while preserving an unmodified copy of the original file system.[64]

The conversion involves creating a copy of the whole ext2/3/4 metadata, while the Btrfs files simply point to the same blocks used by the ext2/3/4 files. This makes the bulk of the blocks shared between the two filesystems before the conversion becomes permanent. Thanks to the copy-on-write nature of Btrfs, the original versions of the file data blocks are preserved during all file modifications. Until the conversion becomes permanent, only the blocks that were marked as free in ext2/3/4 are used to hold new Btrfs modifications, meaning that the conversion can be undone at any time (although doing so will erase any changes made after the conversion to Btrfs).[64]

All converted files are available and writable in the default subvolume of the Btrfs. A sparse file holding all of the references to the original ext2/3/4 filesystem is created in a separate subvolume, which is mountable on its own as a read-only disk image, allowing both original and converted file systems to be accessed at the same time. Deleting this sparse file frees up the space and makes the conversion permanent.[64]

As of June 2015 and 4.x versions of the Linux kernel mainline, the in-place ext3/4 conversion was considered untested and rarely used.[64] The feature, however, was rewritten from scratch in 2016 for btrfs-progs 4.6.[41] and is considered stable since then.

In-place conversion from ReiserFS was introduced in September 2017 with kernel 4.13.[65]

When creating a new Btrfs, an existing Btrfs can be used as a read-only "seed" file system.[66] The new file system will then act as a copy-on-write overlay on the seed, as a form of union mounting. The seed can be later detached from the Btrfs, at which point the rebalancer will simply copy over any seed data still referenced by the new file system before detaching. Mason has suggested this may be useful for a Live CD installer, which might boot from a read-only Btrfs seed on an optical disc, rebalance itself to the target partition on the install disk in the background while the user continues to work, then eject the disc to complete the installation without rebooting.[67]

In his 2009 interview, Chris Mason stated that support for encryption was planned for Btrfs.[68] In the meantime, a workaround for combining encryption with Btrfs is to use a full-disk encryption mechanism such as dm-crypt / LUKS on the underlying devices and to create the Btrfs filesystem on top of that layer.

As SMART commands do not traverse the LUKS layer, Btrfs-based RAID cannot work reliably on top as any error handling requiring communication with the disk will fail. Actually any software RAID will need to be able to pass SMART commands to the disk for reliable operation and this can be a problem as a number of SATA controllers do not properly handle SMART, especially external enclosures, and even external enclosures that do, need to make sure kernel and associated tools are up to date enough to properly facilitate SMART communication with that controller as well as providing Btrfs direct access to the drives.

The developers are currently working on adding keyed hash like HMAC (SHA256).[69] Keyed hash is a step towards encryption.

Unix systems traditionally rely on "fsck" programs to check and repair filesystems. This functionality is implemented via the btrfs check program. Since version 4.0 this functionality is deemed relatively stable. However, as of August 2017, the btrfs documentation suggests that it be used only after having tried other recovery methods.[70]

There is another tool, named btrfs-restore, that can be used to recover files from an unmountable filesystem, without modifying the broken filesystem itself (i.e., non-destructively).[71]

In normal use, Btrfs is mostly self-healing and can recover from broken root trees at mount time, thanks to making periodic data flushes to permanent storage, by default every 30 seconds. Thus, isolated errors will cause a maximum of 30 seconds of filesystem changes to be lost at the next mount.[72] This period can be changed by specifying a desired value (in seconds) for the commit mount option.[73][74]

Every tree appears as an object in the root tree (or tree of tree roots). Some trees, such as file system trees and log trees, have a variable number of instances, each of which is given its own object id. Trees which are singletons (the data relocation, extent and chunk trees) are assigned special, fixed object ids ≤255. The root tree appears in itself as a tree with object id 1.

Trees refer to each other by object id. They may also refer to individual nodes in other trees as a triplet of the tree's object id, the node's level within the tree and its leftmost key value. Such references are independent of where the tree is actually stored.

User-visible files and directories are contained in a file system tree. There is one file system tree per subvolume. Subvolumes can nest, in which case they appear as a directory item (described below) whose data is a reference to the nested subvolume's file system tree.

Within the file system tree, each file and directory object has an inode item. Extended attributes and ACL entries are stored alongside in separate items.

Within each directory, directory entries appear as directory items, whose right-hand key values are a CRC32C hash of their filename. Their data is a location key, or the key of the inode item it points to. Directory items together can thus act as an index for path-to-inode lookups, but are not used for iteration because they are sorted by their hash, effectively randomly permuting them. This means user applications iterating over and opening files in a large directory would thus generate many more disk seeks between non-adjacent files—a notable performance drain in other file systems with hash-ordered directories such as ReiserFS,[75] ext3 (with Htree-indexes enabled[76]) and ext4, all of which have TEA-hashed filenames. To avoid this, each directory entry has a directory index item, whose right-hand key value of the item is set to a per-directory counter that increments with each new directory entry. Iteration over these index items thus returns entries in roughly the same order as they are stored on disk.

Besides inode items, files and directories also have a reference item whose right-hand key value is the object id of their parent directory. The data part of the reference item is the filename that inode is known by in that directory. This allows upward traversal through the directory hierarchy by providing a way to map inodes back to paths.

Files with hard links in multiple directories have multiple reference items, one for each parent directory. Files with multiple hard links in the same directory pack all of the links' filenames into the same reference item. This was a design flaw that limited the number of same-directory hard links to however many could fit in a single tree block. (On the default block size of 4 KiB, an average filename length of 8 bytes and a per-filename header of 4 bytes, this would be less than 350.) Applications which made heavy use of multiple same-directory hard links, such as git, GNUS, GMame and BackupPC were later observed to fail after hitting this limit.[77] The limit was eventually removed[78] (and as of October 2012 has been merged[79] pending release in Linux 3.7) by introducing spillover extended reference items to hold hard link filenames which could not otherwise fit.

File data is kept outside the tree in extents, which are contiguous runs of disk blocks. Extent blocks default to 4KiB in size, do not have headers and contain only (possibly compressed) file data. In compressed extents, individual blocks are not compressed separately; rather, the compression stream spans the entire extent.

Files have extent data items to track the extents which hold their contents. The item's right-hand key value is the starting byte offset of the extent. This makes for efficient seeks in large files with many extents, because the correct extent for any given file offset can be computed with just one tree lookup.

Snapshots and cloned files share extents. When a small part of a large such extent is overwritten, the resulting copy-on-write may create three new extents: a small one containing the overwritten data, and two large ones with unmodified data on either side of the overwrite. To avoid having to re-write unmodified data, the copy-on-write may instead create bookend extents, or extents which are simply slices of existing extents. Extent data items allow for this by including an offset into the extent they are tracking: items for bookends are those with non-zero offsets.[3]

If the file data is small enough to fit inside a tree node, it is instead pulled in-tree and stored inline in the extent data item. Each tree node is stored in its own tree block—a single uncompressed block with a header. The tree block is regarded as a free-standing, single-block extent.

The extent allocation tree acts as an allocation map for the file system. Unlike other trees, items in this tree do not have object ids and represent regions of space: their left-hand and right-hand key values are the starting offsets and lengths of the regions they represent.

The file system zones its allocated space into block groups, which are variable-sized allocation regions that alternate successively between preferring metadata extents (tree nodes) and data extents (file contents). The default ratio of data to metadata block groups is 1:2. They are intended to work like the Orlov block allocator and block groups in ext3 in allocating related files together and resisting fragmentation by leaving allocation gaps between groups. (ext3 block groups, however, have fixed locations computed from the size of the file system, whereas those in Btrfs are dynamic and are created as needed.) Each block group is associated with a block group item. Inode items in the file system tree include a reference to their current block group.[3]

Extent items contain a back-reference to the tree node or file occupying that extent. There may be multiple back-references if the extent is shared between snapshots. If there are too many back-references to fit in the item, they spill out into individual extent data reference items. Tree nodes, in turn, have back-references to their containing trees. This makes it possible to find which extents or tree nodes are in any region of space by doing a B-tree range lookup on a pair of offsets bracketing that region, then following the back-references. For relocating data, this allows an efficient upwards traversal from the relocated blocks to quickly find and fix all downwards references to those blocks, without having to walk the entire file system. This, in turn, allows the file system to efficiently shrink, migrate and defragment its storage online.

The extent allocation tree, as with all other trees in the file system, is copy-on-write. Writes to the file system may thus cause a cascade whereby changed tree nodes and file data result in new extents being allocated, causing the extent tree to itself change. To avoid creating a feedback loop, extent tree nodes which are still in memory but not yet committed to disk may be updated in-place to reflect new copy-on-written extents.

In theory, the extent allocation tree makes a conventional free-space bitmap unnecessary because the extent allocation tree acts as a B-tree version of a BSP tree. In practice, however, an in-memory red-black tree of page-sized bitmaps is used to speed up allocations. These bitmaps are persisted to disk (starting in Linux 2.6.37, via the space_cache mount option[80]) as special extents that are exempt from checksumming and copy-on-write. The extent items tracking these extents are stored in the root tree.

CRC-32C checksums are computed for both data and metadata and stored as checksum items in a checksum tree. There is room for 256 bits of metadata checksums and up to a full leaf block (roughly 4 KB or more) for data checksums. More checksum algorithm options are planned for the future.[32][81]

There is one checksum item per contiguous run of allocated blocks, with per-block checksums packed end-to-end into the item data. If there are more checksums than can fit, they spill rightwards over into another checksum item in a new leaf. If the file system detects a checksum mismatch while reading a block, it first tries to obtain (or create) a good copy of this block from another device – if internal mirroring or RAID techniques are in use.[82][83]

Btrfs can initiate an online check of the entire file system by triggering a file system scrub job that is performed in the background. The scrub job scans the entire file system for integrity and automatically attempts to report and repair any bad blocks it finds along the way.[82][84]

An fsync is a request to commit modified data immediately to stable storage. fsync-heavy workloads (like a database or a virtual machine whose running OS fsyncs intensively) could potentially generate a great deal of redundant write I/O by forcing the file system to repeatedly copy-on-write and flush frequently modified parts of trees to storage. To avoid this, a temporary per-subvolume log tree is created to journal fsync-triggered copy-on-writes. Log trees are self-contained, tracking their own extents and keeping their own checksum items. Their items are replayed and deleted at the next full tree commit or (if there was a system crash) at the next remount.

Block devices are divided into physical chunks of 256 MB or more. Physical chunks across multiple devices can be mirrored or striped together into a single logical chunk. These logical chunks are combined into a single logical address space that the rest of the filesystem uses.

The chunk tree tracks this by storing each device therein as a device item and logical chunks as chunk map items, which provide a forward mapping from logical to physical addresses by storing their offsets in the right-hand 64 bits of their key. Chunk map items can be one of several different types:

single

1 logical to 1 physical chunk

dup

1 logical chunk to 2 physical chunks on 1 block device

raid0

N logical chunk to N≥2 physical chunks across N≥2 block devices

raid1

1 logical chunks to 2 physical chunks across 2 out of N≥2 block devices,[85] in contrast to conventional RAID 1 which has N physical chunks

raid5

N (for N≥2) logical chunks to N+1 physical chunks across N+1 block devices, with 1 physical chunk used as parity

raid6

N (for N≥2) logical chunks to N+2 physical chunks across N+2 block devices, with 2 physical chunks used as parity

N is however many block devices still have free space when the chunk is allocated. If N is not large enough for the chosen mirroring/mapping, then the filesystem is effectively out of space.

The device tree is the inverse of the chunk tree, and has device extent items which provide a reverse mapping from byte-ranges of block devices back to logical chunks by storing the device number in the left hand 64-bits and physical offset in the right-hand 64 bits of they key.

These two trees together allow Btrfs to grow, shrink and even change RAID levels without unmounting because they make it possible to efficiently locate the chunks affected by shrink/move operations and then relocate their contents on-the-fly.

The file system, chunks and devices are all assigned a Universally Unique Identifier (UUID). The header of every tree node contains both the UUID of its containing chunk and the UUID of the file system. The chunk and device trees refer to devices and chunks with UUIDs. On filesystems with just a single block device, "dup" is also the default profile for metadata. All this is intended to improve the odds of successful data salvage in the event of media errors).

Defragmentation, shrinking and rebalancing operations require extents to be relocated. However, doing a simple copy-on-write of the relocating extent will break sharing between snapshots and consume disk space. To preserve sharing, an update-and-swap algorithm is used, with a special relocation tree serving as scratch space for affected metadata. The extent to be relocated is first copied to its destination. Then, by following backreferences upward through the affected subvolume's file system tree, metadata pointing to the old extent is progressively updated to point at the new one; any newly updated items are stored in the relocation tree. Once the update is complete, items in the relocation tree are swapped with their counterparts in the affected subvolume, and the relocation tree is discarded.[86]

All the file system's trees—including the chunk tree itself—are stored in chunks, creating a potential bootstrapping problem when mounting the file system. To bootstrap into a mount, a list of physical addresses of chunks belonging to the chunk and root trees must be stored in the superblock.[87]

Superblock mirrors are kept at fixed locations:[88] 64 KiB into every block device, with additional copies at 64 MiB, 256 GiB and 1 PiB. When a superblock mirror is updated, its generation number is incremented. At mount time, the copy with the highest generation number is used. All superblock mirrors are updated in tandem, except in SSD mode which alternates updates among mirrors to provide some wear levelling.

• Oracle Linux 8[89]
• Oracle Linux 7[90]
• SUSE Linux Enterprise Server 15[91][92]
• Synology Inc. DiskStation Manager (DSM) v6[93]
• Ubuntu[94]
• ReactOS[95]

• Btrfs was included as a "technology preview" in Red Hat Enterprise Linux 6 and 7; it was removed in RHEL 8 in 2018.[21][30][31][96][97]