ONTAP

ONTAP
Developer NetApp
OS family Unix-like (BSD) (Data ONTAP GX, Data ONTAP 8, and later)
Working state Active
Platforms IA-32 (no longer supported), Alpha (no longer supported), MIPS (no longer supported), x86-64 with ONTAP 8 and higher
Kernel type Monolithic with dynamically loadable modules
Userland BSD
Default user interface Command-line interface (PowerShell, SSH, Serial console) Graphical user interfaces over Web-based user interfaces, REST API
Preceded by Clustered Data ONTAP
Official website www.netapp.com/us/products/data-management-software/ontap.aspx

ONTAP or Data ONTAP or Clustered Data ONTAP (cDOT) or Data ONTAP 7-Mode is NetApp's proprietary operating system used in storage disk arrays such as NetApp FAS and AFF (also known as NetApp Filer), ONTAP Select and Cloud Volumes ONTAP. With the release of version 9.0, NetApp decided to simplify the Data ONTAP name and removed word "Data" from it and remove 7-Mode image, therefore, ONTAP 9 is successor from Clustered Data ONTAP 8.

ONTAP includes code from Berkeley Net/2 BSD Unix, Spinnaker Networks technology and other operating systems.[1] ONTAP originally only supported NFS, but later added support for SMB, iSCSI and Fibre Channel Protocol (including Fibre Channel over Ethernet and FC-NVMe). In June 16, 2006,[2] NetApp released two variants of Data ONTAP, namely Data ONTAP 7G and, with nearly a complete rewrite[1], Data ONTAP GX. Data ONTAP GX was based on grid technology acquired from Spinnaker Networks. In 2010 these software product lines merged into one OS - Data ONTAP 8, which folded Data ONTAP 7G onto the Data ONTAP GX cluster platform.

Data ONTAP 8 includes two distinct operating modes held on a single firmware image. The modes are called ONTAP 7-Mode and ONTAP Cluster-Mode. The last supported version of ONTAP 7-Mode issued by NetApp was version 8.2.5. All subsequent versions of ONTAP (version 8.3 and onwards) have only one operating mode - ONTAP Cluster-Mode.

The majority of large-storage arrays from other vendors tend to use commodity hardware with an operating system such as Microsoft Windows Server, VxWorks or tuned Linux. NetApp storage arrays use highly customized hardware and the proprietary ONTAP operating system, both originally designed by NetApp founders David Hitz and James Lau specifically for storage-serving purposes. ONTAP is NetApp's internal operating system, specially optimized for storage functions at high and low level. The original version of ONTAP had a proprietary non-UNIX kernel and a TCP/IP stack, networking commands, and low-level startup code from BSD.[3][1] The version descended from Data ONTAP GX boots from FreeBSD as a stand-alone kernel-space module and uses some functions of FreeBSD (for example, it uses a command interpreter and drivers stack).[1] ONTAP is also used for virtual storage appliances (VSA), such as ONTAP Select and Cloud Volumes ONTAP, both of which are based on a previous product named Data ONTAP Edge.

All storage array hardware include battery-backed non-volatile memory,[4] which allows them to commit writes to stable storage quickly, without waiting on disks while virtual storage appliances using virtual nonvolatile memory.

Implementers often organize two filers in a high-availability cluster with a private high-speed link, either a Fibre Channel, InfiniBand, 10 Gigabit Ethernet, 40 Gigabit Ethernet or 100 Gigabit Ethernet. One can additionally group such clusters under a single namespace when running in the "cluster mode" of the Data ONTAP 8 operating system or on ONTAP 9.

Data ONTAP was made available for commodity computing servers with x86 processors, running atop VMware vSphere hypervisor, under the name "ONTAP Edge".[5] Later ONTAP Edge was renamed to ONTAP Select and KVM added as supported hypervisor.

History

Data ONTAP was founded alongside WAFL in 1992 by David Hitz, James Lau,[6] and Michael Malcolm.[7]

WAFL File System

The Write Anywhere File Layout (WAFL) is a file layout used by ONTAP OS that supports large, high-performance RAID arrays, quick restarts without lengthy consistency checks in the event of a crash or power failure, and growing the filesystems size quickly.

Storage Efficiencies

Inline Adaptive Compression & Inline Data Compaction

ONTAP OS contains a number of storage efficiencies, which are based on WAFL functionalities. Supported with all protocols, does not require licenses. In February 2018[8] NetApp claims AFF systems for its clients gain average 4.72:1 Storage Efficiency from deduplication, compression, compaction and clones savings. Starting with 9.4 ONTAP by default deduplicate data between active file system & snapshots.

  • Data Reduction efficiency is summary of Volume and Aggregate Efficiencies and Zero-block deduplication:
    • Volume Efficiencies could be enabled/disabled individually and on volume-by-volume basis:
      1. Offline Volume Deduplication, which works on 4KB block level
      2. Additional efficiency mechanism were introduced later, such as Offline Volume Compression also known as Post-process (or Background) Compression, there are two types: Post-process secondary compression and Post-process adaptive compression
      3. Inline Volume Deduplication and Inline Volume Compression are compress some of the data on the fly before it reaches the disks and designed to leave some of the data in uncompressed form if it considered by ONTAP to take a long time to process them on the fly, and to leverage other storage efficiency mechanisms for this uncompressed data later. There are two types of Inline Volume Compression: Inline adaptive compression and Inline secondary compression
    • Aggregate Level Storage Efficiencies includes:
      1. Data Compaction is another mechanism used to compress many data blocks smaller than 4KB to a single 4KB block
      2. Inline Aggregate-wide data deduplication (IAD) and Post-process aggregate deduplication also known as Cross-Volume Deduplication [9] shares common blocks between volumes on an aggregate. IAD can throttle itself when storage system crosses a certain threshold. The current limit of physical space of a single SSD aggregate is 800TiB
    • Inline Zero-block Deduplication[10] deduplicate zeroes on the fly before they reach disks
  • Snapshots and FlexClones are also considered efficiency mechanisms
  • Thin Provisioning


Cross-Volume Deduplication storage efficiency features work only for SSD media. Inline and Offline Deduplication mechanisms that leverage databases consist of links of data blocks and checksums, for those data blocks that been handled by the deduplication process. Each deduplication database is located on each volume and aggregates where deduplication is enabled. All Flash FAS systems does not support Post-process Compression.

Order of Storage Efficiencies execution is as follows:

  1. Inline Zero-Block deduplication
  2. Inline Compression: for files that could be compressed to the 8KB adaptive compression used, for files more than 32KB secondary compression used
  3. Inline Deduplication: Volume first, than Aggregate
  4. Inline Adaptive Data Compaction
  5. Post-process Compression
  6. Post-process Deduplication: Volume first, than Aggregate

Aggregates

WAFL FlexVol Layout on an Aggregate
Internal organization of an Aggragete with two plexes

One or multiple RAID groups form an "aggregate"; and within aggregates ONTAP operating system sets up "flexible volumes" (FlexVol) to actually store data that users can access. Similarly to RAID-0, each aggregate consolidate space from underlying protected RAID groups to provide one logical piece of storage for flexible volumes. Alongside with aggregates consists of NetApp's own disks and RAID groups aggregates could consists of LUNs already protected with third party storage systems with FlexArray, ONTAP Select or Cloud Volumes ONTAP. Each aggregate could consist of either LUNs or with NetApp's own RAID groups. An alternative is "Traditional volumes" where one or more RAID groups form a single static volume. Flexible volumes offer the advantage that many of them can be created on a single aggregate and resized at any time. Smaller volumes can then share all of the spindles available to the underlying aggregate and with combination of storage QoS allows to change performance of flexible volumes on the fly while Traditional volumes don't. However, Traditional volumes can (theoretically) handle slightly higher I/O throughput than flexible volumes (with the same number of spindles), as they do not have to go through an additional virtualization layer to talk to the underlying disk. Aggregates and traditional volumes can only be expanded, never contracted. Current maximum aggregate physical useful space size is 800 TiB for All Flash FAS Systems.[11]

7-Mode and earlier

The first form of redundancy added to ONTAP was the ability to organize pairs of NetApp filers into a high-availability cluster (HA-Pair);[12] an HA-Pair could scale capacity by adding disk shelves. When the performance maximum was reached with an HA-Pair, the were two ways to proceed: one was to buy another filer and divide the workload between them, another was to buy a new, more powerful filer and migrate all workload to it. All the filers were usually able to connect old disk shelves from previous models—this process is called head-swap. Head-swap requires downtime for re-cabling operations and provides access to old data with new controller without system re-configuration. From Data ONTAP 8, each firmware image contains two operating systems, named "Modes": 7-Mode and Cluster-Mode.[13] Both modes could be used on the same FAS platform, one at a time. However, data from each of the modes wasn't compatible with the other, in case of a filer conversion from one mode to another, or in case of re-cabling disk shelves from 7-Mode to Cluster-Mode and vice versa.

Later, NetApp released the 7-Mode transition Tool (7MTT), which is able to convert data on old disk shelves from 7-Mode to Cluster-Mode. It is named Copy-Free Transition,[14] a process which required downtime. With version 8.3, 7-Mode was removed from the Data ONTAP firmware image.[15]

Clustered ONTAP

Clustered ONTAP is a new, more advanced OS, which is able to scale out by adding new HA-pairs to a single namespace cluster with transparent data migration across the entire cluster. In version 8.0, a new aggregate type was introduced, with a size threshold larger than the 16-terabyte (TB) aggregate size threshold that was supported in previous releases of Data ONTAP, also named the 64-bit aggregate.[16]

In version 9.0, nearly all of the features from 7-mode were successfully implemented in ONTAP (Clustered) including SnapLock,[17] while many new features that were not available in 7-Mode were introduced, including features such as FlexGroup, FabricPool, and new capabilities such as fast-provisioning workloads and Flash optimization.[18]

The uniqueness of NetApp's Clustered ONTAP is in the ability to add heterogeneous systems (where all systems in a single cluster do not have to be of the same model or generation) to a single cluster. This provides a single pane of glass for managing all the nodes in a cluster, and non-disruptive operations such as adding new models to a cluster, removing old nodes, online migration of volumes, and LUNs while data is contiguously available to its clients.[19] In version 9.0, NetApp renamed Data ONTAP to ONTAP.

Data Protocols

ONTAP considered as Unified storage system meaning it support SAN (FC,FCoE, NVMeoF and iSCSI) & NAS (NFS, pNFS, CIFS/SMB) protocols for its clients. SDS versions of ONTAP (ONTAP Select & Cloud Volumes ONTAP) does not support FC,FCoE and NVMeoF protocols due to their software-defined nature.

CIFS

ONTAP Supports CIFS 2.0 and higher up to SMB 3.1. Starting with ONTAP 9.4 supported SMB Multichannel which provide functionality similar to multipathing in SAN protocols.

iSCSI

iSCSI Data Center Bridging (DCB) protocol supported with A220/FAS2700 systems.

NVMeoF

NVMe over Fabrics (NVMeoF) refers to the ability to utilize NVMe protocol over existed network infrastructure like Ethernet, Fiber Channel or InfiniBand for transport (as opposite to run NVMe over PCI). NVMe is SAN block level data storage protocol. Starting with ONTAP 9.4 NetApp supports NVMeoF with Fiber Channel protocol 32/16 Gb (FC-NVMe) for its All-Flash FAS systems A800, A700/A700s, and A300. NVMEoF protocol supported only on All-Flash A-Systems and not supported for low-end A200 systems.

High Availability

High Availability (HA) is clustered configuration of a storage system with two nodes or HA pairs, which aims to ensure an agreed level of operational during expected and unexpected events like reboots, software or firmware updates.

HA Pair

Even though single HA pair consists of two nodes (or controllers) NetApp's architecture considered it as a single storage system. HA configurations use number of techniques in order to represent two nodes of the HA pair as a single system to provide access for its clients to their data during any one node reboot, for example: on network level HA pare ensures to migrate network address or to switch to another path on survived node and on data level disks served with rebooted node will be served with survived node. FAS and AFF filers using enterprise level HDD and SSD drives with two ports, each port connected to each controller. Each ONTAP's disk have ownership on it to show which controller own and serve the disk. An Aggregate can include only disks owned by a single node, therefore each aggregate owned by a node and any upper objects as FlexVol volumes, LUNs, File Shares are served with a single controller. Each controller can have its own disks and aggregates an serve them therefore such HA Pair configuration called Active/Active where both nodes can be utilized simultaneously even though they not serving the same data.

HA interconnect

High-availability clusters (HA clusters) are the first type of clusterization introduced in ONTAP systems. It aimed to ensure an agreed level of operation. It is often confused with the horizontal scaling ONTAP clusterization that came from the Spinnaker acquisition; therefore, NetApp, in its documentation, refers to an HA configuration as an HA pair rather than as an HA cluster.

An HA pair uses some form of network connectivity (often direct connectivity) for communication between the servers in the pair; this is called an HA interconnect (HA-IC). The HA interconnect can use Ethernet or InfiniBand as the communication medium. The HA interconnect is used for non-volatile memory log (NVLOG) replication using RDMA technology and for some other purposes only to ensure an agreed level of operational during events like reboots always between two nodes in a HA pair configuration. ONTAP assigns dedicated, non-sharable HA ports for HA interconnect which could be external or build in chassis (and not visible from outside). The HA-IC should not be confused with the intercluster or intracluster interconnect that is used for SnapMirror and that can coexist with data protocols on data ports or with Cluster Interconnect ports used for horizontal scaling & online data migration across the multi-node cluster.

MetroCluster

MetroCluster local and DR pare memory replication in NetApp FAS/AFF systems configured as MCC

MetroCluster (MC) is additional level of data availability to HA configurations and supported only with FAS and AFF filers, later SDS version of MetroCluster was introduced with ONTAP Select & Cloud Volumes ONTAP products. In MC configuration two storage systems (each system can be single node or HA pair) form MetroCluster, often two systems located on two sites with distance between them up to 300 km therefor called geo-distributed system. In MC configurations NVLOG also replicated between storage systems between sites but uses dedicated ports for that purpose, in addition to HA interconnect.

MetroCluster SDS

Is a feature of ONTAP Select software, similarly to MetroCluster on FAS/AFF systems MetroCluster SDS (MC SDS) allows to synchronously replicate data between two sites using SnyncMirror and automatically switch to survived node transparently to its users and applications. MetroCluster SDS work as ordinary HA pare so data volumes, LUNs and LIFs could be moved online between aggregates and controllers on both sites, which is slightly different than traditional MetroCluster on FAS/AFF systems where data cloud be moved across storage cluster only withing site where data originally located. In traditional MetroCluster the only way for applications to access data locally on remote site is to disable one entire site, this process called switchover where in MC SDS ordinary HA process occurs. MetroCluster SDS uses ONTAP Deploy as the mediator (in FAS & AFF world this functionality known as MetroCluster tiebreaker) which came with ONTAP Select as a bundle and generally used for deploying clusters, installing licenses and monitoring them.

Horizontal Scaling Clusterization

Horizontal scaling ONTAP clusterization came from Spinnaker acquisitions and often referred by NetApp as "Single Namespace", "Horizontal Scaling Cluster" or "ONTAP Storage System Cluster" or just "Cluster" and therefore often confused with HA pair or even with MetroCluster functionality. While MetroCluster and HA are Data Protection technologies, single namespace clusterization does not provide data protection. ONTAP Cluster is formed out of one or few HA pairs and adds to ONTAP system Non-Disruptive Operations (NDO) functionality such as non-disruptive online data migration across nodes in the cluster and non-disruptive hardware upgrade. Data migration for NDO operations in ONTAP Cluster require dedicated Ethernet ports for such operations called as cluster interconnect and does not use HA interconnect for this purposes. Cluster interconnect and HA interconnect could not share same ports. Cluster interconnect with a single HA pair could have directly connected cluster interconnect ports while systems with 4 or more nodes require two dedicated Ethernet cluster interconnect switches. ONTAP Cluster could consist only with even number of nodes (they must be configured as HA pairs) except for Single-node cluster. Single-node cluster ONTAP system also called non-HA (stand-alone). ONTAP Cluster managed with a single pain of glass built-in management with Web-based GUI, CLI (SSH and PowerShell) and API. ONTAP Cluster provides Single Name Space for NDO operations through SVM. Single Namespace in ONTAP system is a name for collection of techniques used by Cluster to separate data from front-end network connectivity with data protocols like FC, FCoE, FC-NVMe, iSCSI, NFS and CIFS and therefore provide kind of data virtualization for online data mobility across cluster nodes. On network layer Single Namespace provide a number of techniques for non-disruptive IP address migration, like CIFS Continuous Availability (Transparent Failover), NetApp's Network Failover for NFS and SAN ALUA and path election for online front-end traffic re-balancing with data protocols. NetApp Filers can consists of different HA pairs: AFF and FAS, different models and generations and can include up to 24 nodes with NAS protocols or 12 nodes with SAN protocols. SDS systems can't intermix with physical filers.

Storage Virtual Machine

Storage Virtual Machine

Also known as Vserver or sometimes SVM. Storage Virtual Machine (SVM) is a layer of abstraction and alongside with other functions, it is responsible of virtualizing and separate physical front-end data network from data located on FlexVol volumes. Used for Non-Disruptive Operations and Multi-Tenancy.

Non Disruptive Operations

SAN ALUA in ONTAP: preferred path with direct data link

There few Non Disruptive Operations (NDO) operations with (Clustered) ONTAP system. NDO data operations include: aggregate relocation within an HA pair between nodes, FlexVol volume online migration (known as Volume Move operation) across aggregates and nodes within Cluster, LUNs migration (known as LUN Move operation) between FlexVol volumes within Cluster. LUN move and Volume Move operations uses Cluster Interconnect ports for data transfer (HA-CI is not in use for such operations). SVM behave differently with network NDO operations, depending on front-end data protocol. For NAS front-end data protocols there are NFSv2, NFSv3, NFSv4 and CIFSv1, SMBv2 which not provide network redundancy with protocol itself, so they relay on storage and switch functionalities for this matter. For this reason ONTAP support Ethernet Port Channel and LACP with its Ethernet network ports on L2 layer within a single node and also non disruptive Network Fail Over between nodes in cluster on L3 layer with migrating IP addresses (similar to VRRP) to survived node and back home when failed node restored. With SAN protocols ALUA feature used for network load balancing and redundancy so all the ports on node where data located are reported to clients as active preferred path with load balancing between them while all other network ports on nodes in cluster are active non-preferred path so in case of one port or entire node goes down, client will have access to its data using non-preferred path. Because ONTAP provide ALUA functionality for SAN protocols, SAN network interfaces do not migrate like with NAS protocols. When data or network interfaces migration is finished it is transparent to storage system's clients due to ONTAP Architecture and can cause temporary or permanent data indirect access through ONTAP Cluster interconnect (HA-CI is not in use for such situations) which will slightly increase latency for the clients. To decrease latency to its original level FlexVol volumes and LUNs have to be located on the same node with network address through which the clients access storage system, so network address could be created for SAN or moved for NAS protocols. NDO operations are free functionality.

Multi Tenancy

Multi Tenancy

ONTAP provide two techniques for Multi Tenancy functionality like Storage Virtual Machines and IP Spaces. On one hand SVMs are similar to Virtual Machines like KVM, they provide visualization abstraction from physical storage but on another hand quite different because unlike ordinary virtual machines SVMs does not allow to run third party binary code like in Pure storage systems; they just provide visualized environment and storage resources instead. Also SVMs unlike ordinary virtual machines does not run on a single node but for end user looks like an SVM runs as a single entity on each node of the whole cluster. SVM divide storage system into slices so few divisions or even organizations can share a storage system without knowing and interfering with each other while utilizing same ports, data aggregates and nodes in the cluster and using separate FlexVol volumes and LUNs. Each SVM can run its own front end data protocols, set of users, use its own network addresses and management IP. With use of IP Spaces users can have the same IP addresses and networks on the same storage system without interfering. Each ONTAP system must run at least one Data SVM in order to function but may run more. There are few levels of ONTAP management and Cluster Admin level have the all of the available privileges. Each Data SVM provide to its owner vsadmin which have nearly full functionality like Cluster Admin level but lucks management of physical level like RAID group configuration, Aggregate configuration, physical network port configuration but can manage logical objects inside SVM like create, delete and configure LUNs, FlexVol volumes and network addresses so two SVMs in a cluster can't interfere with each other. One SVM cannot create, delete, modify or even see objects of another SVM so for SVM owners such an environment looks like they are only users on the entire storage system cluster. Multi Tenancy is free functionality in ONTAP.

FlexClone

NetApp FlexClone works exactly as NetApp RoW Snapshots but allows to write to FlexClones

FlexClone is a licensed feature, used for creating writable copies of volumes, files or LUNs. In case of volumes, FlexClone acts as a snapshot but allows to write into it, while an ordinary snapshot allows only to read data from it. Because WAFL architecture FlexClone technology copies only metadata inodes and provides nearly instantaneous data copying of a file, LUN or volume with regardless of it size.

SnapRestore

SnapRestore is a licensed feature, used for reverting active file system of a FlexVol to a previously created snapshot for that FlexVol with restoring metadata inodes in to active file system. SnapRestore is used also for a single file restore or LUN restore from a previously created snapshot for the FlexVol where that object located. Without SnapRestore license in NAS environment it is possible to see snapshots in network file share and be able to copy directories and files for restore purposes. In SAN environment there is no way of doing restore operations similar to NAS environment. It is possible to copy in both SAN and NAS environments files, directories, LUNs and entire FlexVol content with ONTAP command ndmpcopy which is free. Process of copying data depend on the size of the object and could be time consuming, while SnapRestore mechanism with restoring metadata inodes in to active file system almost instant regardless of the size of the object been restored to its previous state.

FlexGroup

FlexGroup is a free feature introduced in version 9, which utilizes the clustered architecture of the ONTAP operating system. FlexGroup provides cluster-wide scalable NAS access with NFS and CIFS protocols.[20] A FlexGroup Volume is a collection of constituent FlexVol volumes distributed across nodes in the cluster called just "Constituents", which are transparently aggregated in a single space. Therefore, FlexGroup Volume aggregates performance and capacity from all the Constituents and thus from all nodes of the cluster where they located. For the end user, each FlexGroup Volume is represented by a single, ordinary file-share.[21] The full potential of FlexGroup will be revealed with pNFS (currently not supported with FlexGroup) and SMB multichannel (currently not supported with FlexGroup). The FlexGroup feature in ONTAP 9 allows to massively scale in a single namespace to over 20PB with over 400 billion files, while evenly spreading the performance across the cluster.[22]

SnapMirror

Unified Replication

Snapshots form the basis for NetApp's asynchronous disk-to-disk replication (D2D) technology, SnapMirror, which effectively replicates Flexible Volume snapshots between any two ONTAP systems. SnapMirror is also supported from ONTAP to Cloud Backup and from SolidFire to ONTAP systems as part of NetApp's Data Fabric vision. NetApp also offers a D2D backup and archive feature named SnapVault, which is based on replicating and storing snapshots. Open Systems SnapVault allows Windows and UNIX hosts to back up data to an ONTAP, and store any filesystem changes in snapshots (not supported in ONTAP 8.3 and onwards). SnapMirror is designed to be part of a Disaster recovery plan: it stores an exact copy of data on time when snapshot was created on the disaster recovery site and could keep the same snapshots on both systems. SnapVault, on the other hand, is designed to store less snapshots on the source storage system and more Snapshots on a secondary site for a long period of time.
Data captured in SnapVault snapshots on destination system could not be modified nor accessible on destination for read-write, data can be restored back to primary storage system or SnapVault snapshot could be deleted. Data captured in snapshots on both sites with both SnapMirror and SnapVault can be cloned and modified with the FlexClone feature for data cataloging, backup consistency and validation, test and development purposes etc.
Later versions of ONTAP introduced cascading replication, where one volume could replicate to another, and then another, and so on. Configuration called fan-out is a deployment where one volume replicated to multiple storage systems. Both fan-out and cascade replication deployments support any combination of SnapMirror DR, SnapVault, or unified replication. It is possible to use fan-in deployment to create data protection relationships between multiple primary systems and a single secondary system: each relationship must use a different volume on the secondary system. Starting with ONTAP 9.4 destination SnapMirror & SnapVault systems enable automatic inline & offline deduplication by default.
Intercluster is a relationship between two clusters for SnapMirror, while Intracluster is opposite to it and used for SnapMirror relationship between storage virtual machines (SVM) in a single cluster.
SnapMirror can operate in version-dependent mode, where two storage systems must run on the same version of ONTAP or in version-flexible mode. Types of SnapMirror replication:

  • Data Protection (DP): Also known as SnapMirror DR. Version-dependent replication type originally developed by NetApp for Volume SnapMirror, destination system must be same or higher version of ONTAP. Not used by default in ONTAP 9.3 and higher. Volume-level replication, block-based, metadata independent, uses Block Lavel Engine (BLE).
  • Extended Data Protection (XDP): Used by SnapMirror Unified replication and SnapVault. XDP uses the Logical Replication Engine (LRE) or if volume efficiency different on the destination volume the Logical Replication Engine with Storage Efficiency (LRSE). Used as Volume-level replication but technologically could be used for directory-based replication, inode-based, metadata dependent (therefore not recommended for NAS with millions of files).
  • Load Sharing (LS): Mostly used for internal purposes like keeping copies of root volume for an SVM.
  • SnapMirror to Tape (SMTape): is Snapshot copy-based incremental or differential backup from volumes to tapes; SMTape feature performing a block-level tape backup using NDMP-compliant backup applications such as CommVault Simpana.


SnapMirror-based technologies:

  • Unified replication: A volume with Unified replication can get both SnapMirror and SnapVault snapshots. Unified replication is combination of SnapMirror Unified replication and SnapVault which using a single replication connection. Both SnapMirror Unified replication and SnapVault are using same XDP replication type. SnapMirror Unified Replication is also known as Version-flexible SnapMirror. Version-flexible SnapMirror/SnapMirror Unified Replication introduced in ONTAP 8.3 and removes the restriction to have the destination storage use the same, or higher, version of ONTAP.
  • SnapMirror SVM: replicates all volumes (exceptions allowed) in a selected SVM and some of the SVM settings, replicated settings depend on protocol used (SAN or NAS)
  • Volume Move: Also known as DataMotion for Volumes. SnapMirror replicates volume from one aggregate to another within a cluster, then I/O operations stops for acceptable timeout for end clients, final replica transferred to destination, source deleted and destination becomes read-write accessible to its clients


SnapMirror is a licensed feature, a SnapVault license is not required if a SnapMirror license is already installed.

SyncMirror

SyncMirror replication using plexes

Data ONTAP also implements an option named "SyncMirror", using the plex technique, where all the RAID groups within an aggregate or traditional volume can be synchronously duplicated to another set of hard disks. This is typically done at another site via a Fibre Channel or IP link, or within a single controller with local SyncMirror for a single disk-shelf resiliency. NetApp's MetroCluster configuration uses SyncMirror to provide a geo-cluster or an active/active cluster between two sites up to 300 km apart. SyncMirror can be used ether in software-defined storage platforms, on Cloud Volumes ONTAP, or on ONTAP Select. It provides high availability in environments with directly attached (non-shared) disks on top of commodity servers, or at FAS and AFF platforms in Local SyncMirror or MetroCluster configurations. SyncMirror is a free feature.

SnapLock

SnapLock implements Write Once Read Many (WORM) functionality on magnetic disks instead of to optical media, so that data cannot be deleted until its retention period has been reached. SnapLock exists in two modes: compliance and enterprise. Compliance mode was designed to assist organizations in implementing a comprehensive archival solution that meets strict regulatory retention requirements, such as regulations dictated by the SEC 17a-4(f) rule, FINRA, HIPAA, CFTC Rule 1.31(b), DACH, Sarbanes-Oxley, GDPR, Check 21, EU Data Protection Directive 95/46/EC, NF Z 42-013/NF Z 42-020, Basel III, MiFID, Patriot Act, Graham-Leach-Bliley Act and etc. Records and files committed to WORM storage on a SnapLock Compliance volume cannot be altered or deleted before the expiration of their retention period. Moreover, a SnapLock Compliance volume cannot be destroyed until all data has reached the end of its retention period. SnapLock is a licensed feature.

SnapLock Enterprise is geared toward assisting organizations that are more self-regulated and want more flexibility in protecting digital assets with WORM-type data storage. Data stored as WORM on a SnapLock Enterprise volume is protected from alteration or modification with one main difference from SnapLock Compliance: as the files being stored are not for strict regulatory compliance, a SnapLock Enterprise volume can be destroyed by an administrator with root privileges on the ONTAP system containing the SnapLock Enterprise volume, even if the designed retention period has not yet passed. In both modes, the retention period can be extended, but not shortened, as this is incongruous with the concept of immutability. In addition, NetApp's SnapLock data volumes are equipped with a tamper-proof compliance clock, which is used as a time reference to block forbidden operations on files, even if the system time is tampered with.

FabricPool

FabricPool tiering to S3

Available for SSD-only aggregates in FAS/AFF systems or in Cloud Volumes ONTAP on SSD media. Starting with ONTAP 9.4 FabricPool supported on ONTAP Select platform. Cloud Volumes ONTAP also supports HDD + S3 FabricPool configuration. Fabric Pool provides automatic storage tiering capability for cold data blocks from fast media (usually SSD) on ONTAP storage to cold media via object protocol to object storage such as S3 and back. Fabric Pool can be configured in two modes: One mode is used to migrate cold data blocks captured in snapshots, while the other mode is used to migrate cold data blocks in an active file system. FabricPool preserve offline deduplication & offline compression savings. Starting with ONTAP 9.4 introduced FabricPool 2.0 with ability to tier-off active file system data (by default 31 day data not been accessed) & support data compaction savings. For clients connected to the ONTAP storage system, all the Fabric Pool data-tiering operations are completely transparent, and in case data blocks become hot again, they are copied back to fast media in the ONTAP storage system. Fabric Pool is currently compatible with the NetApp StorageGRID and Amazon S3 service. Starting with ONTAP 9.4 Azure Blob supported, other object-based SW & services could be used if requested by the user and that service will be validated by NetApp. The Fabric Pool feature in FAS/AFF systems is free for usage with NetApp StorageGRID external object storage. For other object storage systems such as Amazon S3 & Azure Blob, Fabric Pool must be licensed per TB to function (alongside costs for Fabric Pool licensing, the customer needs to also pay for consumed object space). While with the Cloud Volumes ONTAP storage system, Fabric Pool does not require licensing, costs will apply only for consumed space on the object storage.

QoS

Storage QoS is a free feature in ONTAP systems. There are few types of storage QoS in ONTAP systems: Adaptive QoS (A-QoS), which includes Absolute minimum QoS; Ordinary static QoS or Minimum QoS (QoS min); and Maximum QoS (QoS max). Maximum QoS can be configured as a static upper limit in IOPS, MB/s, or both. It can be applied to an object such as Volume, LUN or a file, to prevent from such an object from consuming more storage performance resources than defined by the administrator (thus isolating performance-intensive bullies and protecting other workloads). Minimum QoS is contrary to maximum set on volumes to ensure that the volume will get no less than configured by the administrator static number of IOPS when there is contention for storage performance resources and could be applied to volumes. A-QoS is a mechanism of automatically changing QoS, based on consumed space by a flexible volume, because consumed space in it could grow or decrease, and the size of FlexVol can be changed. On FAS systems, A-QoS reconfigures only Peak performance (QoS max), while on AFF systems, it reconfigure both Expected performance (QoS min) and Peak performance (QoS max) on a volume. A-QoS allows ONTAP to automatically adjust the number of IOPS for a volume based on A-QoS policies. There are three basic A-QoS policies: Extreme, Performance and Value. Each A-QoS policy has a predefined fixed ratio IO per TB for Peak performance and Expected performance (or Absolute minimum QoS). Absolute minimum QoS is used instead of Expected performance (QoS min) only when volume size and ratio IO per TB is too small for example 10GB.

Security

ONTAP OS has a number of features to increase security on the storage system like Onboard Key Manager, the passphrase for controller boot with NSE & NVE encryption and USB key manager (available starting with 9.4). FPolicy file-access notification system allows integrating with third party application which can monitor activity on NAS storage, prevent the abnormal behavior and set file access permissions. Auditing for NAS events is another security measure in ONTAP that enables the customer to track and log certain CIFS and NFS events on the storage system. This helps to track potential security problems and provides evidence of any security breaches. ONTAP Antivirus integration allows monitoring and checks data on CIFS file shares.

Onboard Key Manager

Onboard Key Manager is a free feature introduced in 9.1 and can store keys from NVE encrypted volumes & NSE disks. NSE Disks available only on AFF/FAS platforms.

NetApp Volume Encryption

NetApp Volume Encryption (NVE) is FlexVol volume-level software-based encryption which uses storage CPU for data encryption purposes thus some performance degradation is expected though it is less noticeable on high-end storage systems with more CPU cores. NVE is licensed but free feature compatible nearly with all NetApp ONTAP features and protocols. Similarly to NetApp Storage Encryption (NSE), NVE can store encryption key locally or on a dedicated key manager like IBM Security Key Lifecycle Manager, SafeNet KeySecure or cloud key managers. NVE like NSE is also data at rest encryption which means it protects only from physical disks theft and does not give an additional level of data security protection in normal operational and running system. NVE with a combination of FabricPool technology also protect data from unauthorized access in external S3 storage systems like Amazon and since data already encrypted it transferring over the wire in encrypted form.

GDPR

Starting with ONTAP 9.4 new feature introduced called Secure Purge which provides ability to securely delete a file to comply with GDPR requirements.

Software

NetApp offers a set of server-based software solutions for monitoring and integration with ONTAP systems. The most commonly-used free software is the OnCommand Unified & Performance manager, which is a data availability and performance monitoring solution.

Workflow Automation

NetApp Workflow Automation (WFA) is a free, server-based product used for NetApp storage orchestration. It includes a self-service portal with a web-based GUI, where nearly all routine storage operations or sequences of operations can be configured as workflows and published as a service, so end users can order and consume NetApp storage as a service.

SnapCenter

SnapCenter, previously known as SnapManager Suite, is a server-based product. NetApp also offers products for taking application-consistent snapshots by coordinating the application and the NetApp Storage Array. These products support Microsoft Exchange, Microsoft SQL Server, Microsoft Sharepoint, Oracle, SAP and VMware ESX Server data. These products form part of the SnapManager suite. SnapCenter also includes third-party plugins for MongoDB, IBM DB2, MySQL, and allows the end user to create their own plugins for integration with the ONTAP storage system. SnapManager and SnapCenter are enterprise-level licensed products. A similar, free, and less capable NetApp product exists, named SnapCreator. It is intended for customers who wish to integrate ONTAP application-consistent snapshots with their applications, but do not have a license for SnapCenter. NetApp claims that SnapCenter capabilities will expand to include SolidFire storage endpoints. SnapCenter has controller based licensing for AFF/FAS systems and by Terabyte for SDS ONTAP.

Services Level Manager

NetApp Services Level Manager or NSLM for short is software for provisioning ONTAP storage that delivers predictable performance, capacity and data protection for a workload which exposes RESTful APIs and has built-in Swagger documentation with the list of the available APIs, and also can be integrated with other NetApp storage products like OnCommand Unified Manager. NSLM exposes three standard service levels (SSL) based on service level objectives (SLO) and creates custom service levels. NSLM created to provide predicted ServiceProvider-like storage consumption. NSLM is a space-based licensed product.

Big Data

ONTAP systems have the ability to integrate with Hadoop TeraGen, TeraValidate and TeraSort, Apache Hive, Apache MapReduce, Tez execution engine, Apache Spark, Apache HBase, Azure HDInsight and Hortonworks Data Platform Products, Cloudera CDH, through NetApp In-Place Analytics Module (also known as NetApp NFS Connector for Hadoop) to provide access and analyze data by using external shared NAS storage as primary or secondary Hadoop storage.

Automation

ONTAP provisioning & usage can be automated in many ways directly or with the use of additional NetApp Software or with 3rd party software.

  • PowerShell commandlets available to manage NetApp systems including ONTAP, SolidFier & E-Series
  • SnapMirror & FlexClone toolkits written in Perl can be used for SnapMirror & FlexClone managing with scripts
  • Direct HTTP Rest API available with ONTAP and SolidFier
  • ONTAP can be automated with Ansible, Puppet, and Chef scripts
  • NetApp Workflow Automation (WFA) is GUI based orchestrator which also provide APIs and PowerShell commandlets for WFA. WFA can manage NetApp ONTAP, SolidFier & E-Series storage systems. WFA provides a built-in self-service portal for NetApp systems known as Storage as a Service (STaaS)
  • VMware vRealize Orchestrator with WFA can orchestrate storage
  • 3rd party orchestrators for PaaS or IaaS like Cisco UCS Director (Previously Cloupia) and others can manage NetApp systems; automated workflows can be created with step by step instructions to manage & configure infrastructure through the built-in self-service portal
  • NetApp SnapCenter software used to integrate Backup & Recovery on NetApp storage with Applications like VMware ESXi, Oracle DB, MS SQL, etc., can be automated through PowerShell commandlets and RESTfull API
  • OnCommand Unified & Performance manager for monitoring NetApp FAS/AFF Storage systems, performance metrics, and data protection also provide RESTfull API & PowerShell commandlets
  • OnCommand Insight is monitoring and analysis software for heterogeneous infrastructure including NetApp ONTAP, SolidFier, E-Series & 3rd party storage systems & switches provide RESTfull API and PowerShell commandlets
  • NetApp Trident plugin for Docker used in Containers environments to provide persistent storage, automate infrastructure or even run infrastructure as a code. Can be used with NetApp ONTAP, SolidFier & E-Series systems for SAN & NAS protocols.

Platforms

The ONTAP operating system is used in storage disk arrays. There are three platforms where ONTAP software is used: NetApp FAS and AFF, ONTAP Select and Cloud Volumes ONTAP. On each platform, ONTAP uses the same kernel and a slightly different set of features. FAS is richest for functionality among other platforms.

FAS

FAS[23] and All Flash FAS (AFF)[24] systems are proprietary, custom-built hardware by NetApp for ONTAP software. AFF systems can contain only SSD drives, because ONTAP on AFF is optimized and tuned only for Flash memory, while FAS systems may contain HDD (HDD-only systems) or HDD and SSD (Hybrid systems). ONTAP on FAS and AFF platforms can create RAID arrays, such as RAID-4, RAID-DP and RAID-TEC arrays, from disks or disk partitions for data protection reasons, while ONTAP Select and Cloud Volumes ONTAP leverage RAID data protection provided by the environment they run on. FAS and AFF systems support Metro Cluster functionality, while ONTAP Select and Cloud Volumes ONTAP platforms do not.

Software-Defined Storage

Both ONTAP Select and Cloud Volumes ONTAP are virtual storage appliances (VSA) which are based on previous product ONTAP Edge also known as ONTAP-v and considered as a Software-defined storage.[25] ONTAP Select as Cloud Volumes ONTAP includes plex and aggregate abstractions, but didn't have a lower level RAID module included in the OS; therefore RAID-4, RAID-DP and RAID-TEC were not supported so ONTAP storage system similarly to FlexArray functionality leverages RAID data protection on SSD and HDD drive level with underlying storage systems. Starting with ONTAP Select 9.4 & ONTAP Deploy 2.8 software RAID supported with no requirements for 3rd party HW RAID equipment. Because ONTAP Select and Cloud Volumes ONTAP are virtual machines, they don't support Fibre Channel and Fibre Channel over Ethernet as front-end data protocols and consume space from underlying storage in hypervisor added to VSA as virtual disks represented and treated inside ONTAP as disks. ONTAP Select and Cloud Volumes ONTAP provide high availability, deduplication, resiliency, data recovery, robust snapshots which can be integrated with application backup (application consistent snapshots) and nearly all ONTAP functionality but with few exceptions.

ONTAP Select

ONTAP Select can run on VMware ESXi and Linux KVM hypervisors. ONTAP Select leveraged RAID data protection on SSD and HDD drive level with underlying DAS, SAN or vSAN storage systems. Starting with ONTAP Select 9.4 & ONTAP Deploy 2.8 software RAID supported with no requirements for 3rd party HW RAID equipment. Like on the FAS platform, ONTAP Select supports high availability and clustering. As a FAS platform, ONTAP Select is offered in two versions: HDD-only or All Flash optimized. Previously ONTAP Select known as Data ONTAP Edge. Data ONTAP Edge product have Data ONTAP OS with version 8 and was able to run only atop of VMware ESXi.

Cloud Volumes ONTAP

Cloud Volumes ONTAP (formally ONTAP Cloud[26]) includes nearly the same functionality as ONTAP Select, because it is also a virtual storage appliance (VSA) and can be ordered in hyper-scale providers (cloud computing) such as Amazon AWS and Microsoft Azure. IBM Cloud uses ONTAP Select for the same reasons, instead of Cloud Volumes ONTAP. Cloud Volumes ONTAP can provide high availability of data across different regions in the cloud. Cloud Volumes ONTAP leverages RAID data protection on SSD and HDD drive level with underlying IP SAN storage system in Cloud Provider.

Feature comparison

Feature comparison between platforms with latest ONTAP version.

Features Filers (AFF/FAS) Cloud Volumes ONTAP ONTAP Select
FabricPool SSD aggregates only Yes, SSD & HDD supported Starting with 9.4 Supported FabricPool 2.0 for SDS with Premium lic (All Flash)
FlexGroup Supported ? Supported
High Availability (HA) Supported Supported in AWS. 2x times more space consumed Supported with DAS configuration. 2, 4, 6 or 8 nodes supported. 2-node require mediator which incorporated in ONTAP Deploy. 2x times more space consumed
Metro-HA MetroCluster supported. FAS2000 & A200 not supported. Additional HW required. 2x times more space/disks consumed. MetroCluster Tiebreaker software used for monitoring & automatic site switchover in a disaster event. Tiebreaker has to run on 3rd site. Supported in AWS between two availability zones. As in Cloud Volumes ONTAP with HA, 2x times more space consumed Starting with ONTAP Deploy 2.7 officially supported MetroCluster SDS on 2-node clusters with DAS configuration for distance up to 10 Km. As in ONTAP Select with HA, 2x times more space consumed. Also as ONTAP Select with 2-node HA system require mediator which incorporated in ONTAP Deploy. Mediator functionality used for monitoring & automatic site switchover in a disaster event. ONTAP Deploy with Mediator in MetroCluster configuration has to run on 3rd site.
Horizontal Scaling Clusterization In ONTAP 9.3: from 1 node to up to 12 for SAN nodes; NAS up to 24 - with last and previous gen FAS/AFF. Exception: FAS2500 up to 8 nodes No 1, 2 or 4 nodes. 2, 4, 6 or 8 nodes only with HA
Non Disruptive Operations Aggregate relocate, volume move, LUN move, LIF migrate N/A Aggregate relocate, volume move, LUN move, LIF migrate
Multi Tenancy Yes No Yes
FlexClone Yes, included in premium bundle SW ? Yes, always included
SnapRestore Yes, included in premium bundle SW ? Yes, always included
SnapMirror Yes, included in premium bundle SW; SVM SnapMirror supported; Also supported SnapMirror from ONTAP to Cloud Backup; SnapMirror from SolidFire to ONTAP Yes. Also supported SnapMirror from ONTAP to Cloud Backup; SnapMirror from SolidFire to ONTAP Yes, always included; SVM SnapMirror supported; Also supported SnapMirror from ONTAP to Cloud Backup; SnapMirror from SolidFire to ONTAP
SyncMirror Yes: as SyncMirror local or as part of MetroCluster As part of Cloud Volumes ONTAP HA functionality As part of ONTAP Select HA & MetroCluster SDS functionality
SnapLock Yes. Additional licensing required Yes in AWS & Azure. Marketing name: NetApp Cloud WORM Yes with 9.4. Additional licensing required
QoS Yes. QoS max on SVM, FlexVol, LUN and File level. QoS min, Adaptive QoS. Free ? Yes. QoS max on SVM, FlexVol, LUN and File level. QoS min, Adaptive QoS. Free
NetApp Volume Encryption Yes. With local onboard key manager or external key manager. Free license required ? Yes. With local onboard key manager or external key manager. Free license required
SnapCenter /SnapManager Yes, included in premium bundle SW ? Yes. Additional licensing required. Licensed by space
NetApp's proprietary RAID: 4, DP, TEC Yes ? Yes, with ONTAP Select 9.4 & ONTAP Deploy 2.8. Software RIAD available with ONTAP Select 9.4 only with KVM.
Flash Pool Yes No No
FlexArray Yes for FAS systems except for FAS2000 systems. No, N/A No
Third party DAS or RAID No Cloud block storage Yes. DAS with RAID, LVM and vSAN supported
NetApp Storage Encryption Yes, specialized HDD/SDD required. Free No No
Flash Cache Yes for FAS systems except for FAS2200/2500 systems. All current gen FAS systems have pre-installed No No
nonvolatile memory All FAS/AFF virtual Nonvolatile memory virtual Nonvolatile memory
NDMP Yes, free. Supported on FlexVol, doesn't supported on FlexGroup & FabricPool ? Yes, free. Supported on FlexVol, doesn't supported on FlexGroup & FabricPool
NetApp Snapshots 255 with 9.3 and older, 1024 starting with 9.4. Free & always included 255 with 9.3 and older, 1024 starting with 9.4. Free & always included 255 with 9.3 and older, 1024 starting with 9.4. Free & always included
Secure Purge Starting with ONTAP 9.4, free ? ?
Max Aggregate Size 800 TiB on SSD ? 400 TiB per node (200TiB useful in HA)

See also

References

  1. 1 2 3 4 "Is Data ONTAP Based On UNIX?". 2007-04-27. Archived from the original on 2013-01-30. Retrieved 2016-06-11.
  2. ↑ "ONTAP GX—Past and Future". 2006-06-16. Archived from the original on July 1, 2016. Retrieved July 1, 2016.
  3. ↑ Dave Hitz (March 16, 2005). "Requests on Slashdot for donations for OpenBSD". toasters (Mailing list).
  4. ↑ Mark Woods (1 August 2010). "White Paper: Optimizing Storage Performance and Cost with Intelligent Caching | WP-7107" (url). NetApp. Retrieved 24 January 2018. (in English)
  5. ↑ "Introduction to ONTAP Edge". NetApp.
  6. ↑ "Executive Bios". NetApp. 2012. Archived from the original on 2012-06-04. Retrieved 2012-04-13.
  7. ↑ "Michael Malcolm Resigns as Chairman of the Board of CacheFlow to Focus on New Start-Up Opportunity". Business Wire. 13 November 2000. Retrieved 2009-04-14.
  8. ↑ Justin Parisi (28 February 2018). "Running VMware on ONTAP? Why you should consider upgrading to ONTAP 9.2". Justin Parisi. Archived from the original (url) on 2018-03-06. Retrieved 6 March 2018. (in English)
  9. ↑ Justin Parisi (14 July 2017). "Running VMware on ONTAP? Why you should consider upgrading to ONTAP 9.2". Justin Parisi. Archived from the original (url) on 2017-12-03. Retrieved 3 December 2017. (in English)
  10. ↑ Jay Goldfinch, Mike McNamara (14 November 2012). "Clustered Data ONTAP 8.3: A Proven Foundation for Hybrid Cloud". NetApp. Archived from the original (url) on 2017-02-09. Retrieved 3 December 2017. (in English)
  11. ↑ "Running VMware on ONTAP? Why you should consider upgrading to ONTAP 9.2".
  12. ↑ "High-Availability Configuration Guide: What an HA pair is". NetApp. 1 February 2014. Archived from the original (url) on 2017-11-09. Retrieved 9 November 2017. (in English)
  13. ↑ Greg Porter (20 March 2011). "Data ONTAP 8 7-Mode: What is it? Why aren't you running it?". Greg Porter's Blog. Archived from the original (url) on 2016-04-22. Retrieved 9 November 2017. (in English)
  14. ↑ Justin Parisi (25 November 2015). "Introducing: Copy-Free Transition". Why Is The Internet Broken?. Archived from the original (url) on 2017-11-09. Retrieved 9 November 2017. (in English)
  15. ↑ "OnCommand® System Manager 3.1.2. Installation and Setup Guide. Supported versions of Data ONTAP". NetApp. 1 June 2015. Archived from the original (url) on 2017-11-09. Retrieved 9 November 2017. (in English)
  16. ↑ Reddy, Shree (September 2011). "A Thorough Introduction to 64-Bit Aggregates" (PDF). NetApp.
  17. ↑ Justin Parisi (23 June 2016). "ONTAP 9 is now available!". Why Is The Internet Broken?. Archived from the original (url) on 2017-02-12. Retrieved 9 November 2017. (in English)
  18. ↑ "Datasheet: ONTAP 9 Data Management Software" (PDF). NetApp. 2017.
  19. ↑ Justin Parisi (16 February 2015). "TECH::Become a clustered Data ONTAP CLI Ninja". Why Is The Internet Broken?. Archived from the original (url) on 2016-08-18. Retrieved 9 November 2017. (in English)
  20. ↑ "What a FlexGroup volume is". NetApp. 1 November 2017. Archived from the original (url) on 2017-11-09. Retrieved 9 November 2017. (in English)
  21. ↑ Justin Parisi (4 October 2016). "NetApp FlexGroup: An evolution of NAS". Why Is The Internet Broken?. Archived from the original (url) on 2017-11-09. Retrieved 9 November 2017. (in English)
  22. ↑ "SPEC SFS®2014_swbuild Result for NetApp FAS8200 with FlexGroup". Standard Performance Evaluation Corporation. 26 September 2017. Archived from the original (url) on 2017-10-12. Retrieved 8 November 2017. (in English)
  23. ↑ "Hybrid Flash Arrays – Hybrid Storage Systems | NetApp". www.netapp.com. Retrieved 2018-01-31.
  24. ↑ "All Flash Storage Arrays – All Flash FAS (AFF) | NetApp". www.netapp.com. Retrieved 2018-01-31.
  25. ↑ "ONTAP Select: Software-Defined Storage (SDS) | NetApp". www.netapp.com. Retrieved 2018-01-31.
  26. ↑ "AWS Storage for the Enterprise with NetApp". cloud.netapp.com. NetApp, Inc. Retrieved 2018-01-31.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.