Scale-out storage: Proprietary? Commodity? Or both?

There is no doubting that scale-out storage is now thoroughly achievable. Indeed, scale-out is pretty much the norm for object storage, plus it is now the primary option when it comes to huge volumes of file-based NAS, where scale-out means a highly-scalable file system that supports clustering. It is growing in popularity for block storage too.

Scale-up storage was enabled in large part by the advent of storage networking, because it allowed storage systems to be shared. Storage could be provided from a single scalable device much more efficiently, plus applications running on different servers could share data more readily. However, as storage systems grew in size they reached limitations – not just hardware limits but also more practical administrative ones, such as how to protect and manage a single multi-petabyte and multi-application array.

Compared to scale-up storage, scale-out is all about simplifying the management and expansion processes, for example processes such as storage provisioning, reconfiguring LUNs and migrating data as new storage is added to the system. Scale-out technology does some of this for you and lets you do the rest in one place for the whole cluster, whereas with scale-up each storage system is an individual management point.

Scale-out object technologies include the open-source Ceph storage system, IBM-owned Cleversafe and Scality Ring, among others. Examples of scale-out block-capable technologies include Fujitsu's Eternus DX8000 series with its QuadStar clustering architecture, NetApp's clustered Data OnTap, RedHat GlusterFS, and Ceph's RBD block device. Scale-out block can also be provided by using storage virtualisation or software-defined storage to aggregate physical storage into a single logical pool of blocks. Some examples here are EMC's ScaleIO, HP Enterprise's StoreVirtual VSA and IBM's SAN Volume Controller, plus software tools such as DataCore SANsymphony or FalconStor FreeStor.

The question then is how to achieve scale-out? Do you go with a proprietary vendor that owns its scale-out ecosystem, and can therefore guarantee performance and scalability, or do you take a more open route? In many ways, the choice resembles the one between proprietary and open-source object storage, with the proviso that much of what is happening here – just as it is in the server space – is about using abstraction and virtualisation to decouple the physical hardware from the software. So proprietary scale-out storage software is just as capable of being hardware-independent or hardware-neutral as is open-source storage software.

This highlights the important distinction between scale-up and scale-out. Traditional scale-up systems such as NAS filers are limited by their hardware, so even if you can upgrade the controllers to provide more performance, and add trays of disks, there comes a point when that box is effectively full. That means you have to buy another one, so you now have two to administer – or more likely in the real world, you now have five to manage instead of four, or 15 instead of 14.

In contrast, scale-out typically relies on software, both to take care of the hardware abstraction and to provide a distributed or parallel storage service on top that can join several nodes together as a single logical storage system. This in turn means that you will see the term software-defined storage (SDS) a lot in any discussion of scale-out, because pretty much by definition any non-proprietary scale-out storage system must also be software-defined. Even EMC Isilon scale-out NAS, although it sells its own X-Series and S-Series hardware platforms and has developed its own proprietary (though FreeBSD-based) operating software, is essentially a clustered storage software platform running on industry-standard hardware.

Choices, choices

So we start to see that what looked like a binary choice of open versus proprietary is actually rather more complex and nuanced, and is at least trinary. For instance, do you want to do scale-out storage on commodity hardware or on proprietary boxes? If you also choose proprietary software, you might still run an open-source file systems on top, and if you want to use commodity hardware, will you drive it with open-source software or closed? Plus, each of these decisions will as usual have some effect on the resulting system's performance, cost, scalability, maintainability and so on.

“DIY with white boxes is preponderant in the industry, with a lot of discussion on software-defined storage,” says Mike King, the senior director of marketing strategy and operations at DDN Storage, which builds high performance scale-out storage systems to run parallel file systems such as GPFS and Lustre, the latter being open-source.

“But optimised hardware will provide a better solution for you,” he continues. “You can put a white box together but you won't get our performance numbers because of [a lack of] optimisation.” For instance, he says that some controllers might only provide 5GB/sec or 6GB/sec, whereas he claims DDN's can do 30GB/Sec, “and 120GB/sec file systems are not that rare – we have deployed several this year.”

He adds that proprietary can also be simpler because of its greater density – part of the hyperconverged story is less floorspace and cooling needed. “We can also put in more optimisations for specific applications because of our degree of customisation. The downside is a higher initial investment, but obviously we would argue it costs less in the long term, once you include all the other lifetime costs.”

He acknowledges that “DIY is a win-win in some sites, for example universities where they have – and want to teach – the associated skills. Yet we also have several university customers, because when it comes to it, DIY can't support their application.”

Among those universities are several using DDN appliances to run the Lustre parallel file system within high-performance computing (HPC). One of the top ones is Cyfronet, at the Academic Computer Centre in Krakow, Poland, where scale-out provides 180GB/sec of storage bandwidth to Prometheus, an HP Apollo cluster that currently rates as the 49th fastest in the world.

King argues that it is all too easy to pay too much enough attention to the compute side of HPC and not enough to the storage. “People always want to spend their budget on cores, so they'll have a $10m cluster with $1m of storage behind it,” he says. “But if the storage doesn't perform, the cluster won't perform.”