This article is more than 1 year old
Between you and NVMe: NetApp dishes on drives and fabric access
Will deliver an NVMe over Fabrics FlexPod system in a rack all ready to go
Interview El Reg has asked a large number of storage array vendors about their views of NVMe solid state drives and NVMe over Fabrics access to such drives. The combination promises to effectively kill array network access latency issues and make array access equivalent to reading and writing data from a locally attached flash drive.
So far we have asked the same set of questions to these vendors:
To understand NetApp’s view on NVME and NVMe over Fabrics we talked to SolidFire Field CTO Val Bercovici.
El Reg: Will simply moving from SAS/SATA SSDs to NVMe drives bottleneck existing array controllers?
VB: To be fair, just about any controller today from any vendor will be bottlenecked by more than a few SSDs. The raw number of IOPs capable from an SSD is staggering, but the value is limited without storage efficiencies, quality of service, data protection, management and other resiliency technologies that a storage controller provides with its onboard software.
However, NVMe helps reduce this storage bottleneck by removing the SCSI protocol from the stack and providing lower device latencies and higher IOPs than existing SAS/SATA SSDs. This frees up the array controller to more efficiently use its CPUs to provide the other capabilities listed above and increase overall performance. Overall NVMe helps improve SSD utilisation, especially for high-capacity SSDs that can help address data-centre space constraints.
El Reg: Must we wait for next-generation controllers with much faster processing?
VB: No.
Controllers today benefit from NVMe in our NetApp FAS controllers to accelerate flash cache. There are many variables that factor into performance and processing power is merely one. NetApp controllers are architected to efficiently utilise all available CPU cores. Next-generation controllers will further increase performance by improving PCIe and memory bandwidth-to-core count ratios, to be “better balanced” for NVMe and NVMe over Fabrics applications.
In some cases, the “next generation” controller is already here, in the form of a scale-out storage architecture that adds controller CPU every time capacity is added. Controller CPU capacity isn’t a fixed quantity.
El Reg: Will we need affordable dual-port NVMe drives so array controllers can provide HA?
VB: No.
A common characteristic of next-generation scale-out storage systems is a node-based (rather than controller + disk shelf) architecture. As such, this requirement is not relevant for many new storage deployments.
On the other hand enterprise arrays that use dual-ported SAS SSDs today, will need dual-ported NVMe drives to maintain comparable levels of reliability, serviceability and high availability. We expect dual-ported NVMe drives to quickly achieve price parity with SAS SSDs.
NetApp offers system architectures that utilise both single-ported drives and dual-ported drives.
El Reg: What does affordable mean?
VB: Affordable means price-parity of dual-ported NVMe SSDs with today’s dual-ported SAS SSDs.
El Reg: Are customers ready to adapt NVMeoF array-accessing servers with new HBAs and, for ROCE, DCB switches and dealing with end-to-end congestion management?
VB: In many data centres, customers are already using DCB switches and dealing with end-to-end congestion management. Customers have already begun adopting NVMe drives in servers. As customers see the value of extending their NVMe ecosystem to be end-to-end, they will become interested in adopting RDMA capable Ethernet HBAs to extend the low latency benefits across their fabrics.
Leading vendors will make this change simpler for customers by delivering NVMe over Fabrics in a way that is easily consumable. For instance with FlexPod, NetApp will deliver an end-to-end NVMe over Fabrics solution within a rack, all ready to go.
El Reg: Do they need routability with ROCE?
VB: Many applications requiring ultra-low latency will contain their storage and compute locally in order to obtain the lowest latencies. Customers who require routability will be able to use RoCEv2 or one of the other NVMe over Fabrics RDMA technologies which are routable.
For example, iWARP is also a fully routable RDMA technology over Ethernet. Therefore, focusing on NVMe doesn’t automatically mean only ROCe will be involved in the transport.
El Reg: Could we cache inside the existing array controllers to augment existing RAM buffers and so drive up array performance?
VB: NetApp FAS arrays have supported internal PCIe attached caching for over a decade and SolidFire Cluster Nodes have also done so for almost half a decade. The latest generation of FAS2600, FAS8200 and FAS9000 families now support internal NVMe drives for FlashCache functionality to address this use case.
Upcoming Persistent Memory technologies such as NVDIMMs will dramatically extend these RAM buffers using native memory-mapping techniques such as DAX*.
El Reg: With flash DIMMs say? Or XPoint DIMMs in the future?
VB: Yes, although current scant 3DXPoint/NVDIMM availability and known price/performance benefits of doing this are not better than our existing architectures with NVRAM and NVMe caching. However, as NVDIMMs and related Persistent Memory technologies mature and become more affordable, they will appear in all relevant NetApp storage architectures.
To elaborate - as NAND densities continue to grow, $/GB improves but at the cost of IOPs density (IOPs/GB). So a gap emerges between DRAM and NAND that will be filled by new Persistent Memory tier(s). This new media will remain relatively expensive compared to NAND; so what you do with it is pretty important; “the right data on the right media at the right time..”.
NetApp has a long history of supporting multiple tiers of memory within our SW stack and we’re looking forward to the competitive opportunities Persistent Memory presents.
El Reg: Does having an NVMe over Fabrics connection to an array which is not using NVMe drives make sense?
VB: This is a situation that can make sense for arrays that are able take advantage of the network latency improvement. NVMe and NVMe over Fabrics technologies can be adopted independently. NVMe over Fabrics for an array can help reduce transport latency over the storage network.
NVMe media will help reduce latency within the array and improve the performance of that array itself. Today, NetApp has storage arrays with SAS3 connected SSDs that can be built into clusters doing millions of 4K IOPs at sub 500 us latencies.
Solutions like ONTAP 9 and SolidFire are able to scale-out and non-disruptively introduce new components, such as NVMe and NVMe over Fabrics. The E-Series solutions are designed for industry leading SPC-1 and SPC-2 performance and will immediately benefit from a low overhead storage network stack.
NetApp customers enjoy the investment protection of NVMe with extraordinarily fast Flash today with future-proof opportunities to take advantage of NVMe over Fabrics technologies as they are mainstreamed.
El Reg: When will NVMe over Fabrics arrays filled with NVMe drives and offering enterprise data services be ready? What is necessary for them to be ready?
VB: Many enterprise storage vendors have NVMe systems announced or in development. I expect we will begin to see the first end-end native NVMe systems with enterprise data services come to market in the next year or so.
Key items necessary for them to be ready include NVMe over Fabrics-capable Ethernet HBAs and switches plus dual-port NVMe drives where applicable. Applications will continue to evolve their data layers (databases, object & filesystems) to take better advantage of low latency storage.
Broad adoption of partial NVMe solutions is imminent. Complete end-end NVMe adoption will arrive at the pace of maturity of all the ecosystem components listed above. Converged Infrastructure solutions will mitigate all of the risk and integration complexity for the early adopters, while accelerating implementation and economies of scale for mainstream NVMe users.
El Reg: A NetApp blog provides more information on NetApp's NVMe thinking. ®
*DAX is Direct access and there's SNIA info about it here (PDF).
Biting the hand that feeds IT
