Comparison of distributed file systems
In computing, a distributed file system (DFS) or network file system is any file system that allows access to files from multiple hosts sharing via a computer network. This makes it possible for multiple users on multiple machines to share files and storage resources.
Distributed file systems differ in their performance, mutability of content, handling of concurrent writes, handling of permanent or temporary loss of nodes or storage, and their policy of storing content.
Locally managed
Client | Written in | License | Access API |
---|---|---|---|
Ceph | C++ | LGPL | librados (C, C++, Python, Ruby), S3, Swift, FUSE |
BeeGFS | C / C++ | FRAUNHOFER FS (FhGFS) EULA,[1]
GPLv2 client |
POSIX |
GlusterFS | C | GPLv3 | libglusterfs, FUSE, NFS, SMB, Swift, libgfapi |
Infinit[2] | C++ | Proprietary (to be open sourced)[3] | FUSE, Installable File System, NFS/SMB, POSIX, CLI, SDK (libinfinit) |
Isilon OneFS | C/C++ | Proprietary | POSIX, NFS, SMB/CIFS, HDFS, HTTP, FTP, SWIFT Object, CLI, Rest API |
ObjectiveFS[4] | C | Proprietary | POSIX, FUSE |
MooseFS | C | GPLv2 | POSIX, FUSE |
Quantcast File System | C | Apache License 2.0 | C++ client, FUSE (C++ server: MetaServer and ChunkServer are both in C++) |
Spectrum Scale (GPFS) | C, C++ | Proprietary | POSIX, NFS, SMB, Swift |
Lustre | C | GPLv2 | POSIX, liblustre, FUSE |
MapR-FS | C, C++ | Proprietary | POSIX, NFS, FUSE, S3 |
OpenAFS | C | IBM Public License | Virtual file system, Installable File System |
scality | C | Proprietary | fuse, NFS file system, rest, AWS S3 |
Tahoe-LAFS | Python | GNU GPL 2+ and other[5] | HTTP (browser or CLI), SFTP, FTP, FUSE via SSHFS, pyfilesystem |
HDFS | Java | Apache License 2.0 | Java and C client, HTTP |
XtreemFS | Java, C++ | BSD License | libxtreemfs (Java, C++), FUSE |
Ori[6] | C, C++ | MIT | libori, FUSE |
Remote access
Name | Run by | Access API |
---|---|---|
Amazon S3 | Amazon.com | HTTP (REST/SOAP) |
Google Cloud Storage | HTTP (REST) | |
SWIFT (part of OpenStack) | Rackspace, Hewlett-Packard, others | HTTP (REST) |
Microsoft Azure | Microsoft | HTTP (REST) |
Cleversafe | Cleversafe | HTTP (REST) |
Comparison
Some researchers have made a functional and experimental analysis of several distributed file systems including HDFS, Ceph, Gluster, Lustre and old (1.6.x) version of MooseFS, although this document is over 4 years old and a lot of information may be outdated (e.g. MooseFS has at the time of writing this stable 2.0 and beta 3.0 version and HA for Metadata Server - since 2.0 and it is not mentioned in this document).[7]
The cloud based remote distributed storage from major vendors have different APIs and different consistency models.[8]
See also
References
- ↑ "FRAUNHOFER FS (FhGFS) END USER LICENSE AGREEMENT". Fraunhofer Society. 2012-02-22.
- ↑ "The Infinit Storage Platform".
- ↑ "Infinit's Open Source Projects".
- ↑ "ObjectiveFS official website".
- ↑ "About Tahoe-LAFS".
- ↑ "Ori: A Secure Distributed File System".
- ↑ Séguin, Cyril; Depardon, Benjamin; Le Mahec, Gaël. "Analysis of Six Distributed File Systems" (PDF). HAL.
- ↑ "Data Consistency Models of Public Cloud Storage Services: Amazon S3, Google Cloud Storage and Windows Azure Storage". SysTutorials. Retrieved 19 June 2017.