Easy storage solutions

Dunno if have ever thought about the problems of storage & redundancy and the headache it causes us mere mortals but I have a suggestion.

With the advent of USB3.0 & 3.1 its so easy just to add another drive to the system and get more storage space. Volumes are really problematic as if you lose a disk then its likely the whole volume is lost.
Also physical constraints of Raid arrays and disk size cause so much work and headaches with many often over spending with guesstimates to future proof.

I thought I would post this here because I have never seen it before and if it exists please post but I have to question why we bother with the constraints and limitations of RAID or disk upgrades when we can just add another drive and concatenate via a unionFS such as AuFS or OverlayFS?

Its knocks off the R from RAID but USB AID (Array Inexpensive Disks) for many is extremely simple and logical. If you need more space then add another drive and its that simple.
I am looking at smaller systems but the throughput on USB 3.0 & 3.1 is such that at least SoHO can be provided for and depending on disk choice highly likely of a much bigger scale.

Also unlike volumes if we lose a disk only the data on that disk will be lost so a collection of multiple small disks can minimize failure disruption.

I have been looking at some of the online backup services such as Amazon Glacier Or Google Cold Line and I keep thinking fault tolerant disk arrays as opposed to redundant makes sense and it can be done via a unionFS.
The cost of those online services is so low that as long as you can continue its a matter of replacing the lost disk and accessing your online archive.

So I would be interested in what you think or any alternatives you may think is better?

I am going to leave it open but yeah a completely different strategy to data integrity might actually be more suitable for many.
Even if there is a mirror of recent accesses and writes the current monolithic way we treat storage has raised some questions to its need in many cases.
We keep huge amounts of local redundant data that is never accessed apart from extremely infrequently.


Sure, JBOD has its appeal, if you don’t mind managing it.

Right, any type of RAID0/Spanned setup is definitely not a good idea for storing important data

Entirely depends, ZFS? Totally. The whole vdev management can be a pain. MDADM, BTRFS, etc far less so. Choose your RAID level and add more disks as you need to, then grow it. Simple.

Because the constraints and limitations are pretty minimal, people care about their data and JBOD can lead to data loss on a completely non-redundant array.

… putting data unnecessarily at risk.

So that data doesn’t matter? What if it’s 7 years worth of photos you’ve then got to restore?

But then you:

  • Pay someone to look after your data
  • Give your data to a cloud service (defeating the object of controlling your own data)
  • Have one or more additional services to manage
  • Pay heftier charges for restores, since Amazon et al charge for accessing the data

Not to mention installing/configuring a new FS to manage the data that likely comes with its own (albeit likely small) learning curve.

This is so much more complicated than just setting up a RAID, blimey Ubuntu desktop (Gnome disks) will allow you to select a bunch of disks and set a RAID up by clicking a button if the thought of commandline management is daunting.

You also lose the automated monitoring of disk health unless you setup alerts yourself, since RAID will tell you a disk is failing, maintain data integrity while you seek a replacement disk (no lost data at all) and rebuild it all automatically. It’ll also prevent bitrot due to parity information and automated, regular scrubbing; something that if missed will have you backing up corrupt data also.

Your proposition sounds like a mountain of work in comparison to the several RAID arrays I run across multiple machines with very little regular maintenance required.

Hardware RAID can arguably be a burden as it comes with a level of complexity with battery cache and whatnot, softRAID however not so much at all, as it’s simple to setup, easy to manage and not at all challenging to expand and grow as required (ZFS aside).

RAID doesn’t replace backups, but reduces reliance on them in case of a problem.
JBOD has a place but I wouldn’t recommend it with Nextcloud or anywhere where data integrity and reliability is required.

Its not Jbod that is why I mentioned Aufs its not “Just A Bunch Of Disks” Aufs is an OverlayFS where they are not spanned but mounted via commands to give a singular mount point.


I have done some checking and should of known someone is much brighter and quicker than me as OpenMediaVault has a plugin already developed doing exactly the same.

Also there is a simpler OverlayFS system called mhddfs and the basic difference is that Aufs balances writes across the disks whilst mhddfs writes to the first disk containing a minimum of free space.
So actually mhddfs is actually preferential as when one disk is full it only needs a single archive at that point.

Jason you seem to be missing the whole point here Nextcloud data is encrypted and I just added Amazon Glacier as an example but @ $0.0045 a GB which is $4.5 per month for 1TB of data. so yeah in 10 months its about the cost of a 1TB drive.
Just for some who are not computer geeks a solution offsite like Amazon Glacier storing you encrypted fails is as bomb proof as it gets.
For others they might use another hard drive of equal size to the current working disk and just have to rsync that off.

What it is, is an exceptionally easy and logical manner of data warehousing and its about cost.

Your making the presumption that RAID = Safe data and whilst it may be a Redundant Array Inexpensive Disks in the majority of use scenario’s the Raid enclosure is a single point of failure and worse many of the current ones are hardware RAID and if your box goes hopefully you can get the exact same model again or the same chipset and firmware.

Ever tried work on a RAID unit that is re-syncing, it could be 7 years of photo’s that you will have to restore which you can do with out effecting the current working volume, be it Blu-ray, HDD or offline cloud then Aufs or mhddfs just creates an unionFS and your back 100%.

Again you are completely missing the point as in the real world Raid enclosures & computer cases have finite physical limits and often these limits are defined by cost.
Often a chosen RAID unit will come populated with disks and when that volume is full its a matter of replacing all disks BTRFS RAID is still under development but anyone who has used LVM or MDADM without onsite IT professionals will tell you its far from simple.

Frankly Jason that is complete rubbish and maybe read up about AuFS & mhddfs as all it would need is a simple Admin app and then those drives just look like a single mount point or volume.
I have read a little about AuFS & mhddFS and if you can lose a single disk but just remake the union minus that disk until its added again then in operation for many it will be vastly more simple and logical and easier to add more storage.

Not by default it isn’t. Encryption in itself is not simple and encrypting local storage with Nextcloud is not massively useful particularly given things like filenames aren’t obfuscated. And what happens when the user loses the encryption key? Definitely a thing that happens.

Setting up Glacier isn’t remarkably simple, either.

No, RAID is safer than non-RAID. It’s not a replacement for backups and I didn’t say so.

It is however far more likely to retain the integrity of your data against bitrot - that silent corruption that happens over years which RAID with parity can scan for and fix in the background, saving you from backing up corrupted, unrecoverable data to Glacier or anywhere else.

No argument there, if you’re arguing your point against hardware RAID only though you’re ignoring the whole softRAID piece, which is far easier to manage.

Yes, I’ve lost 2 disks this year which were replaced within a couple of days. It was perfectly fine.

Without the data you just lost, which will error up in NC unless you run an occ scan, which could lead to data loss again while restoring if NC happens to be running scans while you’re returning data direct to disk.

You mean like limited USB ports? And if we’re considering cost, external drives have higher failure rates generally than NAS-grade internal disks, speaking as someone who’s shucked several and had them fail far earlier than internal drives. They’ll last longer if they’re parked for sure, but the risk is still there.

You’re referencing closed RAID boxes again.

5/6 sure, RAID1 is fine. ZFS is very stable and very easy to manage on FreeNAS (or even Ubuntu) though less flexible. Sure you replace all disks to up the space in a closed system, or just add a disk and grow where you have the space.

1 Like

Yeah cheers Jason many thanks.

I will be playing with mhddFS with an array of external USB 3.0 drives.
Just going on the cheap to get things working with a few disks http://www.ebay.co.uk/itm/KingDian-mSATA-Solid-State-Disk-SSD-M100-Hard-State-Drive-For-Games-S8M2-/252781232051

32/16Gb so it accentuates the upgrade process just pluging into a USB3.0 hub.
I will be using the CLI just to see how things go on a test of a disk failure in regards to the remaining working volume.
Needs some tweaks in Nextcloud if the mhddFS app is installed to report and stop any resyncs until that disk is added back.

I will just get some of those cheap mining USB3.0 M.2 convertors http://www.ebay.co.uk/itm/B-Key-M-2-NGFF-SSD-to-USB-3-0-USB2-0-Converter-Enclosure-Adapter-Card-/262691957882?

I am just thinking about the app prob will rsync to an external drive but need to play with mhddFS and see if it will allow me to hook a script to create an final archive as that disk should be readonly.
In fact really need to play and get some real world experience but with a little imagination you might see why I started this discussion and how super easy a unionFS of external drives could be.

I will be testing things and posting some scripts on here and maybe it might garner a bit of interest to create an app.

There are quite a few manners of employ where an array of fast SSD are mirrored by an asynchronous slow HDD so the mirror doesn’t effect working volume performance.
Could even have a hot spare SSD that will come online in the time its takes to copy the missing SSD data from the HDD.

Technologies change and storage has but RAID is still firmly planted in a spindle philosophy and all I am taking about is a bit of lateral thought that could provide a solution fit for purpose.
I am thinking there is possibility of quite a different methodology to how we treat storage that is super easy & redundant at lesser cost.

Definitely test it, 100%. I tested Galera (unsupported) with HAProxy (later supported I found out) as a PoC and that was a great learning experience, but I didn’t recommend anyone else do it at the time.

But if you can validate it works and publish the guides showing how simple and reliable it is I’m happy to change my opinion.

Yeah I really don’t know Jason Galera looks cool, but being perfectly honest I am jumping into the unknown.
I am looking at Arm64, maybe docker swarms and its starting to dawn on me that storage solutions might be very different to the X86 model without really knowing a direction.

Finally got a low cost 64bit SBC with gig ethernet & USB3.0 and yeah apols I am like a kid in a sweet shop :slight_smile:

I am going to play and post perf results / scripts. I think a few others have got interested and have heard the term Docker swarm which is actually really interesting if we could have a scaleable swarm in terms of performance and storage…
Just dunno but thinking the Arm64 SBCs are starting to bring a paradigm shift in upgrade vs just add another…
Same with storage we are not going to have to think about enclosures, ports available its just going to be a matter of just add another when we need to expand not just in terms of storage but also processor power.

Yeah Jason the more I think about it especially for cost and ease a unionFS makes real sense as its actually above RAID.

I have switched to OverlayFS after some reading because apparently it performs Better than Aufs but also it makes it easy to set the lower read only volumes.

In the real world the working section of an archive is often a small subsection whilst there is a huge bulk of historical files.
Often the whole archive is spanned across an expensive and for many hard to administer RAID.
There is such an array of options that fall into the natural diversification of normal working patterns.

mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\ workdir=/work /merged

You could have a working RAID 0 SSD stripe as upper and a lower RAID 1 HDD mirror or any combination that allows us to partition cost and performance.
Its very similar to how many of us have a relatively small SSD and larger but slower HDD as a system.

Probably the majority of Nextcloud installs are bottlenecked by a single Gig Ethernet and even a single SSD is more than capable just for straight file transfers, but what you place for lower & upper storage is a matter of choice and will still be presented as a single merged directory.
I am thinking that Overlay2 can cope with the difference of the upper & lower because it uses caching for performance.

It loses the just-plug-in-another of Aufs but only for the upper and as your storage requirements increase you can just add more lower volumes and essentially the upper works like a cache.
We have a-time, m-time, c-time and nextcloud data for assessing usage and I am thinking we don’t need Einstein to work out a relatively simple algorithm to do some idle background tasks to prune what goes where.

A UnionFS isn’t a cluster, it could be part of one if you wished but all I am thinking is a UnionFS could for users with the right app be just a matter of plug-in-another when more storage is needed.
Also after doing a bit of reading you could put together a hybrid of volumes that really helps with cost over storage space.
OverlayFS & Aufs work and apart from the open(2) & rename(2): posix compatibility which is a matter of the app I was just wondering if it would garner any interest.
Both are really simple:
OverlayFS mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\ workdir=/work /merged
AuFS mount -t aufs -o br=/tmp/dir1:/home/lakshmanan none /tmp/aufs-root/

OverlayFS is interesting because you could use a faster upper than lower, whilst Aufs might follow the way technology seems to be going and its just a matter of plugging in another USB3.0 or 3.1 drive and if they are RAID or not is just purely choice.

I will do some performance tests and post as that is really the only consideration.

If you want a scalable, redundant and easy to maintain solution, you can use croit (https://croit.io) to install a Ceph Cluster and place your data on RGW. Basic version is entirely free, and you can use whatever server and disks you have (x86_64). If you want more redundancy, just add another server and your data will be rebalanced automatically.

But for simpler and smaller setups, ZFS would be my first choice, as it offers data reliability and is quite easy to install.

Cheers Martin I will have a look at the I had ruled out Cephs as over complex and has the need for dedicated nodes.

I have been looking at http://www.xtreemfs.org/ as it seems to be the easiest and scales from a single machine.

Cephs looks a major leader in this area and now I am going to have to go back an look at (https://croit.io)

https://tahoe-lafs.org/trac/tahoe-lafs, https://www.resilio.com, https://syncthing.net/, https://librevault.com/ there are many and each takes a bit of reading.

xtreemfs had a video which was extremely simple in expressing how I see devices and device pooled storage.

Really glad you posted that as was a little shocked at some of the early posts if you know the state of play and the changes that are happening in storage schemes that are extremely beneficial for a plethora of reasons.

When it comes to backup I am not sure we should be looking at any type of onsite service.
http://sia.tech/ & https://storj.io/ are completely private p2p blockchain storage methods where we just keep private keys.
Both offer interesting solutions that are disruptive technologies to current assumed storage methods.
If you download the 3GB blockchain you could become a suppler & a consumer, but they are merely an example of vastly different ideas in resource sharing.
Sia & Storj are maybe just the start in P2P storage where depending on timeslot, bandwidth, store size you vastly reduce costs by offering sections of free space, so we can become both supplier & consumer.

ZFS is not device redundant as far as I know if you look and is essentially centralized but will do a a bit more checking.

[Edit] Yeah from a cursory glance its still very much a server based system whilst xtreemfs can scale from 1 and yes at low node counts its got stinky performance but soon as you start adding the parallelism of more nodes performance also grows.

With storage and storage cost we are getting to a stage where SSD, SSHD & HD can saturate a 1gig ethernet and centralized performance and delivery can be problematic.
From clients to devices each node should have the capability to add a storage device that is balanced against its network connection as swarm storage in use is vastly simpler and highly resilient.

Lot of work being done in this area but so far only xtreemFS fits what I see as requirements.

I think xtreemfs would make a perfect partner to sia or storj and maybe services like ceph and p2p storage swarms will have layers applicable to scale.

At the large scale sia / storj are bang on the button with blockchain private storage and I am trying to get my head round a system that works well for Open Grid purposes and privacy at the lower levels of home & workgroup and think xtreemfs is one that fits but if anyone has any alternatives I would be interested.

With some hacking you could even molest aufs & snapraid into a form of solution if you where perverse enough to try :slight_smile:

1 Like


Xtremefs is dead since 2015. Just take a look at the last release date. I once thought XtreemFS would be nice and gave it a try. It was quite ok but a lot of features where missing. Today nothing seems to have changed according to the website.

You are correct ZFA can’t scale above a single node. Ceph instead is “i think” market leading and future technology but heavy to understand and maintain. It can scale from a single system to many PB of storage. It is well battle proved and therefore good for reliable data storage. croit.io is a jung startup in that sector and willing to provide a onenode simple installer if that helps other nextcloud users. But yes it requires a full host for only storage purposes.

I will look at the other links as I don’t know them all. Thanks for your feedback.

I thought that but after talking to the community and looking at the repos’s it still fresh but they concentrate on https://www.quobyte.com/quobyte-and-xtreemfs

They have just run updates to http://download.opensuse.org/repositories/home:/xtreemfs:/unstable/ since the first stable release and latest is 17-Aug-2017

Same with github even though the website is static https://github.com/xtreemfs/xtreemfs

With a group https://groups.google.com/forum/?fromgroups#!forum/xtreemfs

I am going to read up further on croit.io just haven’t as of yet as agree Ceph does look like a leader.
I have stopped thinking of things as servers and clients and purely devices where some experiments with extremely low cost Arm64 development boards has proved them capable.

As devices from firewall/router, Alexa unit, Kodi player there are a range of $40 devices that can saturate a 1gig ethernet with local disk I/O.
Then add desktop clients and the possible node count grows quickly and also swarm performance.

Croit.io might well be the right solution but trying to grab a picture of all.

Really xtreemfs needs a minimum of 3 nodes for full functionality but I guess is not much different in reality to Cephs so maybe.

It is an extremely interesting period where distributed storage could provide a paradigm shift in how we currently view and use storage.
There is also some extremely capable and low cost devices.