Commentards Ahoy! How about a Petabyte of storage?

A little experiment from us: we are rounding up comments on a couple of articles - and turning them into articles. Some might call this stretching the material - social media types might call it amplification. But we think that not all of our commentards are mad - and many have very interesting insights to share. Today, we mine our comments from Martin Glassborow's column, "How about a Petabyte of storage". Tomorrow, we turn to Trevor Pott and reactions to his prognostications on Windows Server 2012.

Within a lifetime, we’ve gone from hard drives of 5MB being big enough to use as significantly hard to move doorstops (true story) to USB sticks capable of holding a terabyte hanging off key rings. So with the future having arrived so abruptly, what is the enthusiast to get excited about now?

How about a Petabyte of storage? asks StorageBod - aka Martin Glassborow. That’s a quadrillion bytes, for those who like to keep track, three orders of magnitude more than the terabyte key ring. And while things in tech have a reputation for getting cheaper and cheaper, running costs and reinforcing a building to hold the kit would see costs running quickly beyond the average Joe or Josephine.

How much would a petabyte of storage cost? It’s not as much as it cost last year but not as a cheap as some people would think. Firstly, it depends on what you might want to do with it - capacity, throughput and I/O performance are just part of the equation.

Of course then you’ve got the cost of actually running it; 400-500 spindles of spinning stuff takes a reasonable amount of power, cooling and facilities. Even if you can pack it densely, it is still likely to fall through the average floor.

There are some very good deals to be had, mind you, but you are still looking at several hundred thousand pounds, especially if you look at a four year cost.

StorageBod also floats the possibility of the peak for personal storage demand arriving in the coming decade, especially storage we keep in our own spaces.

That should start to go into decline, for me it is already beginning to do so. I carry less storage around than I used to... I’ve replaced my 120GB iPod with a 32GB phone but if I’m out with my camera, I’ve probably got 32GB+ of cards with me. Yet with connected cameras coming and 4G (once we get reasonable tariffs), this will probably start to fall off.

I also expect to see the use of spinning rust go into decline as PVRs are replaced with streaming devices; it seems madness to me that a decent proportion of the world’s storage is storing redundant copies of the same content. How many copies of EastEnders does the world need to be stored on a locally spinning drive?

So I am not sure that we will get to a petabyte of personal storage any time soon but we already have access to many petabytes of storage via the interwebs.

The combination of the two trends: increasing off-site storage and decreasing personal storage will see an increasingly sharp divergence between personal and professional demand for storage.

The difference between commercial and personal consumption is going to grow. There will be scary amounts of data around about you and generated by you; you just won’t know about it or be able to access it.

What the Reg readers say

First we’ll take a look at those who have other ways of making their own pet petabyte of storage, starting with Silver badge holder jai.

Does it really need all that much stuff? First, you'll need a fridge. Plus a few hundred thousand quid, a reinforced floor You can get 20tb with one of these: So you'd only need somewhere to store 50 of those - and $109,950 to buy them.

Yes, sure, this probably isn't the ideal enterprise solution, but when talking about personal use, if you decided you absolutely had to have a petabyte of data, and you have a spare room or basement space, you could do it. AND if you set it up in your basement, you'd probably never need to heat your house :)

Fellow silver badger Annihilator replies:

"Does it really need all that much stuff?" - to make it useable/useful, yes. Plus something for them to plug into and consolidate it into a storage array. Thunderbolt allows 7 (or 6? not sure if it includes the host) daisy chained, so a machine that has 8 or 9 ports would be in order. Plus a floor that would support ~ 400kg, and power into the room that could support ~3kW (plus whatever server) across 50 plugs. In one room. For "race to the bottom" storage, I picked up a 3TB external USB3 drive from PC World over christmas (was an emergency in my defence!) for under £120. Get 334 of them for £40K. But by the same token I wouldn't consider 6,100 floppy disks as a useful alternative to a DL DVD :-)

The spectacularly refined chap takes the issue head on, advising folk not to let it get too complicated:

...the question relates to 1PB of storage and that is all. There is no requirement for on-line storage (in the "old" meaning of the term, i.e immediately accessible). Arrangements that require physical loading of the medium are therefore perfectly acceptable. Tape has already been hinted at but can't compete with disk. Do disk it on disk then. They don't need to be mounted - use them in one of the bare drive USB interfaces (e.g. this one) and stored in the cheap silicon cases when not in use.

Power requirements are negligible since at any one time only one or two drives are connected. As for space, not very much really - I have a couple of 6ft tall bookcases here that each provide around 180" of shelf space. Since hard drives are small compared to many books you could effectively double that or more by inserting additional shelves in the same space - those two bookcases would hold at least 500 drives even in cases.

If you wanted redundancy you could achieve this by designating every e.g. tenth drive as parity for the preceding nine, effectively giving you an off-line RAID 40 array. You would of course need a second adapter (one for the data drive, one for the parity) but that costs bugger all in the grand scheme of things. Cost is highly dependent on the cost of the drives.

Last time I looked the cost/GB sweet spot was 2TB for £60. 600 of those (the extra 100 for parity and spares) would be £36,000 retail but I'd imagine you could get a reasonable discount ordering hundreds of drives at once so I've venture the low thirty thousands for the whole set up. Maybe not what the asker had in mind and yes if would be cumbersome to retrieve a single photo or word document, but eminently practical for many archival kind of tasks.

Going to a bookshelf, pulling out one drive, and dropping it in the reader is no big deal even for a single feature film - it's not really much different to loading a DVD is it? In short then, it is entirely practical approached in the right way. You'd need deep pockets or to highly value your data, but it isn't beyond the realms of the average guy if he regards it as enough of a priority.

How about a totally different approach? Jon Massey argues that without redundancy, you could “fit a PB into a shorty rack”.

assuming four 4U, 60-bay boxes and 4TB 3.5" drives, plus a switch to last them together. That probably wouldn't fall through most people's floors; but powering and cooling it might be a different matter!

But this Anonymous Coward wasn’t entirely happy with this plan, arguing that backups are important:

Of course, if you are prepared to pay for a petabyte of storage, then almost by definition the data is valuable to you. So for me, the big question is this: how are you going to back it up?

Another vexing issue is the discomforting notion of relying on external agencies to keep hold of what is precious to. A suggestion from another Anonymous Coward that one could just use Bitcasa instead met with howls of protest, summed up neatly by bronze badger Andy ORourke:

I Just had a quick look and there is currently a free trial then 10Gig remains free and a deal of $10.00 / month for unlimited storage. Now that sounds great but, as with all things cloud you have the issues of Data security (even if it is only your old photos and music) will Bitcasa last? Will Bitcasa have an elegant back out solution should the company fold? will they get you hooked and then up the ante by charging $20 / month?

Ultimately will they have a "fair use policy" (since we all know nothing is "unlimited" in this day and age). Most readers here would have enough cynicism to tell Bitcasa "you can have my data when you prise it out of my cold dead hands!"

Yet another Anonymous Coward had concerns here, too.

The rights man, the rights "seems madness to me that a decent proportion of the world’s storage is storing redundant copies of the same content" Yes mostly, but for telly it gives you more control - iplayer giveth and iplayer taketh away depending on the rights negotiated for that show, but my local copy will still be good in a year.

Bronze badge holder Marvin the Martian agreed, bring a tale of woe to our attention so bleak that we are still wiping our eyes:

Very true. The cries of incomprehending agony from a 2year old when "The Gruffalo" and "The Gruffalo's Child" disappeared after a month, about a year ago, having watched those daily for a month... It's still fresh scars.

Silver poster Matt Bryant counters this, arguing that although you might rewatch things many times if you are under 10, for grownups, content is more throwaway:

".....but my local copy will still be good in a year." True, but how much of it do you actually go back and watch even once? I'm continually rooting through my Sky box to delete content the family or I have recorded and then watched and forgotten about. On the shelves I have 200+ DVDs purchased and the majority only watched once, despite my conviction at purchase that the films concerned were so good I'd be watching them regularly.

But I do have a couple of TBs of unique family pics and videos I do want to keep "forever", which are backed up to tape and to a cloud host. I don't necessarily need lots of fast spinning rust in my man-cave, but I still do need some storage somewhere.

Nematoad sums up the concerns about outsourcing your storage needs, but starts another debate entirely with a throwaway comment towards the end:

1) You have to trust that what you give to other people to store is going to be safe. Megaupload comes to mind here. Regardless of the rights and wrongs of the case of lot of people have had their stuff taken away from them with little chance of recovery and through no fault of their own.

2) Where is the data going to be stored. Yes, in Europe there are regulations that ensure that things are protected against snooping, though that can surely be got round. Agreements with the USA are again, in place but the US government seem to have a cavalier attitude to anything that they think comes within their ambit, and finally what if the provider decides that a nice cheap server farm in say, Uzbekistan, is what they really need, what protection do you have then?

3) Cost. I don't know about you but I have to pay for the pathetic broadband service that I am forced to use, too far from the exchange, rural area with little chance of fibre connections etc. So if I decide to ditch my local storage I will in effect have to pay rent to get things done. 4) At the cost of repeating good ole Bill G's remark about 640K being enough for anyone what would you do with a PB of storage, unless of course you run something like Pixar.

No, with the advent of SSDs in ever larger capacities if I need to expand my data storage that's what I'll use and keep MY stuff firmly away from this cloud thing that everyone seems to want to push us into. One thing for sure, they aren't doing it as an act of charity, they see a way of getting rents in perpetuity and personally I'm not buying that.

Obviously in such company as this, suggesting that we might not need a PB of personal storage and expecting no response is akin to pouring honey over your head and expecting no bees… and indeed, here they come:

Vector says:

"what would you do with a PB of storage, unless of course you run something like Pixar" Well, Let's see... DVD's are about 5GB per platter for basicly SD video. Bluray bumped that up to around 30GB for HD video. Extrapolating, 4K video would be about 120GB and 8K would be pushing 500GB. Anyone know what MegaImmersiveHolySh*tIsThatRealHD requires? ('cause ya know we ain't done yet). And that's just resolutions.

Framerates appear to be on the rise too, if "the Hobbit" is any indication. Photo resolutions keep bumping up, audio bitrates keep increasing along many axes. <snip> Once, Mega was the stuff of storage, now it's barely memory. Giga was the stuff of dreams, now you need at least 2 of them just to start your PC. Peta, Exa, Yotta and even Zeta will all eventually fall into the personal orbit, even if we can't imagine how at the moment.

While SteveT can think of a way to use up the storage space, no problem:

Transporters? I assume that once we all have transporters installed at home, that the data when digitised is buffered at the far and and verified prior to reconstructing an individual. You really wouldn't want a comms failure when you are only 1/2 way home! So It seems reasonable that a full human image, including every physical feature and all memory data, would need to be stored in the destination transporter before the reconstruction begins, and this could I'm sure take several patebytes, especially if you have one of the those 6-person transporters in your home.

And in a similar vein, electricmonk chips in with the following:

"But will we ever need a petabyte of personal storage?" Yes, I need at least 2.5 petabytes so I can take a backup copy of my brain. "How many copies of EastEnders does the world need to be stored on a locally spinning drive?" None. Not one. Delete them all, everywhere, and make the world a slightly happier place.

CharlesCalthrop can also see how huge volumes of data could be generated:

what if those google glasses take off and everyone walks around with tiny cameras in their glasses? It would awesome to store all that video. That would take up a fair bit over the years wouldn't it? I realise google would want the data itself and not let us download our own copies, it was just an example. I suppose what I am trying to say, badly, is over the past 30 years we have started using technology in ways we could barely concieve off back then, what is to suggest that that is going to change?

Then there were the happy few who wanted to go retro and do the whole thing on tape. We’ll let Bronze badged phuzz do the talking for them:

Tape FTW Well, Dell do a 42U tape library (the ML6030), which can hold 409 tapes. Currently they're only listing it having LTO5, but I'm sure it'll be an easy upgrade to LTO6, which have a native capacity of 2500Mb and for 1Pb you'll need about 420 tapes, factor in a bit of redundancy and two tape libraries should do it, co-incidentally doubling your data rate! However, it not got a price listed, but I'm sure el reg can get a ball park figure for a couple of these, and as the tapes are 'only' £70 each, it'll cost about £30k for the tapes. Shit, when did hard drives become cheaper per Gb than tape?

The general consensus, then, seems to be that yes, you can have a PB of storage in your basement. But you really have to want it. Whether or not this will change remains to be seen, but with sci-fi imaginations set to maximum, finding out will certainly be interesting. ®