Disk Storage

[Sandyn]

I have a large number of files on my system, gathered over the few years. Some are archive quality scanned images, many are 100+MB image size, Gigabit panoramics and TB of digitised video. I also keep all my RAW files.
I use Acronis True Image to create backups. I now have 40+Tb of disk storage I only run the backup disks when I am doing a backup session and monitor disks with Crystal Disk and Disk Sentinel. Some of my disks are showing signs of end of life, I have been looking for a 'solution' I think what I do just now is as good as I can get without spending a fortune, but sometimes people have ingenious solutions, or found some really good software.
I have unlimited image storage on Google, but it's unusable when dealing with large amounts of Data. I don't like cloud storage, because you dribble things on, but if you have to do a disaster recovery, it becomes unusable. If the cloud storage company decides to stop supplying the service or folds, you are stuffed, however I do want to duplicate my storage to save at a different location, so I could use my Google storage for that, but my experience is that it seems to crash if I try huge data dumps. My son has a Plex media server set up, but it seems to crash every so often and needs rebuilt. I prefer just storing files in NTFS disks because it's simple and portable.
Has anyone found a really good solutions to data management and storage, but not costing the earth??

 

[paulrbarnard]

I have a similar setup to you. I have a local 20TB RAID that stores all my images then a second larger raid used as a backup connected to a different computer in the house (the “live” RAID is in my office in a separate building). I then have everything mirrored to a third computer with a separate back up disk. The RAIDs enable rapid recovery of a disk failure either on my live system or the backup server. The backup itself lets me recover files inadvertently lost or deleted. The third mirrored system is my belt and braces fallback. I think I’m covered for catastrophic failure including either my house or office burning to the ground.
I won’t touch cloud services. I got stung by Apple when they killed the .MAC service and have twice since fallen foul of egregious price hikes or service reductions by Smugmug and google. I do have a cloud service for sharing but it’s running on my own server using ownCloud.

 

[Sandyn]

I have a similar setup to you. I have a local 20TB RAID

That sounds a very impressive system. I was thinking RAID would be the way to go. Which disks do you use? What software do you use to manage your files?

 

[Rorton]

I use a piece of software called 'UnRaid' which is like raid, but isn't raid. with raid, you need to have the disk sizes the same, with UnRaid, you can add in disks of any size to form the usable space.

https://unraid.net


You still have a 'parity' disk (a bit like raid) and this needs to be as large as the largest disk on your system.

This UnRaid os is your basic NAS, and then if you wanted, you can plug in different things on top (like Plex or other apps in docker containers, or if your server is powerful enough, you can run virtual machines on it) for example, I run software to control my WiFi, my home automation systems, media storage and playback (Plex) etc

You can also set a cache drive, which could be a really fast SSD drive to use as a working disk, and then the data gets moved over to the array.

Beauty of Unraid is that you can access the disks if the raid controller fails. You just unplug them from your NAS system, plug them into a PC and read the data, you can't do that with raid. If more than one disk in a raid 5 array fails, you've lost all the data, in an Unraid array, you can still access the data on the disks that are remaining. Some pros and cons here:

https://www.makeuseof.com/tag/unraid-ultimate-home-nas/

You can install this into some pre built servers, but in the end, I built my own, and it sits running quietly doing what it needs to do.

Great advice above about multiple backups, which ever system you use, RAID (or UnRaid) isn't a substitute for a good backup strategy, so a separate backup is always advisable, even if that is mirroring your NAS to a cloud based service - granted it will take a while to recover, if that is too long, then another system somewhere else is advisable.

Disk format won't be NTFS though, these are all linux based systems, Unraid uses XFS

there is a cost to the OS, based on how many disks you want to use in the array

 

[AJB Temple]

I agree with Paul. Excellent post. I do use iCloud, despite being stung by .MAC too years ago, but mainly because I find the sync between machines function handy. all layered backup is done on Raid and some things that are top secret on a portable 2Tb SD drive. I will have to look into ownCloud.

 

[morturn]

I use a Synology NAS. Dead easy to set up and it just sits out of the way at the other end of the house connected via Wi-Fi. It sends me an email every so often just to say its ok or its updating its software. It backs up both mine and my wife’s computer every day.

I also use a Segate external hard drive. With the Seagate app, I just plug it into a USB outlet and it automatically backup any new files. I do this one a month then store the disk off site.

 

[Bodone]

Expenology running on three HP proliant N54l’s and a Qnap 453. Various levels of raid through to JBOD and each with a terabyte partition synced with individual one drive accounts which are part of a single family account.

About 40TB of which 4TB follows 3-2-1, the rest would be a pain to lose but not catastrophic. I use WD reds, although I’ve been ‘shucking some of the larger WD drives as the 12TB ones for a while were white label Reds. I don’t run any disk based apps, ip cameras etc, 24x7 so Reds good enough for me.

Above set up is as low cost as I could go for power usage and still run the unifi network, vpn, pi-hole etc. The qnap I added for the 10Gb network option. All the servers are lights out and two are scheduled up times dependent on who’s in the house and backing up kids MacBooks, i-ads etc.

 

[paulrbarnard]

That sounds a very impressive system. I was thinking RAID would be the way to go. Which disks do you use? What software do you use to manage your files?

I have three 8 bay external RAID boxes with thunderbolt connection to my computers. They are all from Arcea but nothing special about them. I use whatever disks I can get at a good price. Currently the backup RAID is fitted with Western Digital disks but I just look for the best value drives.
As I’m on Apple I use Time Machine for the backup as it’s built in and works well. My back up server is a very old Mac Mini. It shares the connected RAID backup disk on my network so my other computers can back up to it (3 other computers).
My main computer is a Mac Pro and it has two RAID boxes connected. One is used for everyday use and has all my images, video and other work stuff on it. The second RAID is used by a virtual machine that always runs on my main computer hosting an OwnCloud server. This is like DropBox but hosted on my own computer. All my images, videos and other work is synchronised to the OwnCloud server. Both the RAID drives and the internal disk on the MacPro are backed up to the RAID on the backup server using Time Machine.
My Laptop computer also synchronises with OwnCloud, but a subset of the data as it has smaller disk. The laptop also backs up to the backup server and a separate rugged external drive which I take with me when travelling. my OwnCloud server is accessible when I’m travelling so everything back at home stays current when I’m working on my laptop in China.
The third computer is an iMac which is nominally my wife’s but it also has an account for me that is always running in the background. This computer has two external, non RAID, drives connected. One is a full copy of my OwnCloud content, all my images, videos and work files. The second is a local backup disk for time machine which holds yet another backup of everything.

it works well for me. I can go to any computer and all my files are there, even the desktop is the same. I have at least two backup available and any new files I create on the road are also synchronised to home and backed up there as well as on the local rugged disk. The RAID boxes provide seamless recovery for HW failure and the backups provide recovery of historic data and user cock ups.

 

[Sandyn]

Thank you everyone for the VERY useful information. Reading other solutions, I realise the first thing I have to do is sort out my files . My storage has grown 'organically', I just got a new USB external every so often, but I need to organise all the stuff I have first. There is no magic bullet for that!
I have 4X8TB USB drives, a 6Tb, couple of 4TB, a couple of NAS drives. I have some of my disks mounted on a tray with fan cooling.
I have suffered some disk failures the last time I tried to organise things and lost some files, but nothing which was really important, but it has prompted me to do something with my image files and get them organised and backed up properly.
My wife has a MAC which sits and does nothing most of the time. I hadn't even thought of using Time Machine. I set that up for her, to back up her MAC. I will look into using it as an option.
I really like the Unraid option. The software is very affordable. It would suit what I have. I would be willing to strip the USB drives back to bare drives and reformat. The XFS format would still be OK.
I have used NAS drives in the past, but one failed. The cooling fan failed and cooked the drives. I was lucky to be able to retrieve all the data when they cooled. I still use the other for storing non important stuff, but its just 2TB.
I still have nearly every hard disk I have ever owned, lol I think about 40 of them including the very first 10MB hard disk from a Tandon 286 machine.
Lots of research to do!! thanks for the help.

 

[Garno]

What is this strange language you are all speaking?

 

[Robbo3]

Have a look at Backblaze for HDD reliability statistics
- Backblaze Hard Drive Stats

 

[GuitardoctorW7]

Hi Folks
Ours is a Mac household (get over it ). I use for my business MacBook an old Apple 2TB time machine, as well as a QNAP TS 251 RAID with 2 x WD red drives, and once a month use a 2TB portable drive using Carbon Copy Cloner, I also have Apple cloud storage for more important files. My wife's MB air backs up to the QNAP as does my Mac mini. The networked drives all back up using the Apple Time machine app.
That way I have 4 back ups for important data, one at least accessible off site. Overkill I know but my job (broadcast sound) has always taught me to use backups of backups and that mantra has served me well.
My Mac mini has an LG 4K Ultrafine display and on the back of that I've stuck a couple of USB C SSD's which are tiny, cheap and rather elegant for storage, and although I've not set it up yet I may add one of them to the mini's Time Machine app.
For the Mac haters, all our computers run Parallels and Windows 10 for any software that doesn't run on the Apple platform.
Happy New Year to you all, 2021 just has to be better, doesn't it?
G

 

[StevieB]

Not to hijack this thread, but has anyone got a link to an idiots guide to back-up storage? I have been working from home for 9 months or so and have most of the files from this time on my PC rather than work servers for accessibility issues. I have got a back-up of the files on a portable hard drive, but what I guess I really need is a piece of software that will copy over only those files that haven't changed rather than having multiple copies of everything - does such a thing exist? The Synology NAS mentioned above is a couple of hundred quid at the starter end of the scale, which I didn't really want to pay for work files, but might have to if nothing better can be suggested! TIA.

 

[Cirks]

I just about followed most of the above posts but what surprises me is the sheer amount of data some o& you have. I’m a keen photographer and have large number of files (digital era as well as scanned negatives and slides etc) and obviously all of those are backed up more than once via NAS/Raid solution as well as all home documents/files etc from many years. However, despite that I’m not anywhere near the huge volume some of you have.
Are your large volumes mainly video then? Work related stuff for me (other than when I was self employed) hasn’t obviously been saved on home pcs and has been backed up handled by employers so that will be a difference for many of you but in overall scheme, most work stuff is documents which are small. I guess CAD stuff will be bigger for some of you.
Would be interesting to know what sort to of percentage breakdown (docs, photos, video, etc) you have

 

[paulrbarnard]

I just about followed most of the above posts but what surprises me is the sheer amount of data some o& you have. I’m a keen photographer and have large number of files (digital era as well as scanned negatives and slides etc) and obviously all of those are backed up more than once via NAS/Raid solution as well as all home documents/files etc from many years. However, despite that I’m not anywhere near the huge volume some of you have.
Are your large volumes mainly video then? Work related stuff for me (other than when I was self employed) hasn’t obviously been saved on home pcs and has been backed up handled by employers so that will be a difference for many of you but in overall scheme, most work stuff is documents which are small. I guess CAD stuff will be bigger for some of you.
Would be interesting to know what sort to of percentage breakdown (docs, photos, video, etc) you have

Video is enormous. A single project can be up to 200GB for me and I only do short videos for work and an occasional video for home. I also shoot RAW with a 40Mpixel camera so each image is about 60MB. It’s easy to chew up a lot of storage in a year. But disk space is actually not too expensive when you have RAID boxes that take standard drives.

 

[Peri]

StevieB
A NAS (network attached storage) is aimed at multiple computers having access to files on one box. Is that what you need, or do you just need a backup for your files that will be accessed by just one computer?

If it's the latter, a DAS (direct attached storage) is much easier, and its probably what you have now - a portable hd thats plugged in via usb.

(Good article https://blog.seagate.com/business/nas-vs-san-vs-das-which-is-right-for-you/ )

but what I guess I really need is a piece of software that will copy over only those files that haven't changed rather than having multiple copies of everything - does such a thing exist?

Of course - and this one is even free FreeFileSync

 

[paulrbarnard]

StevieB
A NAS (network attached storage) is aimed at multiple computers having access to files on one box. Is that what you need, or do you just need a backup for your files that will be accessed by just one computer?

If it's the latter, a DAS (direct attached storage) is much easier, and its probably what you have now - a portable hd thats plugged in via usb.

(Good article https://blog.seagate.com/business/nas-vs-san-vs-das-which-is-right-for-you/ )



Of course - and this one is even free FreeFileSync

@StevieB just to add to Peri’s excellent advice.

There are two types of backup to consider.
First is a copy of the current state of your computer so you can recover from a disaster like the HD failing in your computer. For that something like Carbon Copy Cloner is a great tool. You configure it to make periodic snapshots of your entire disk. If the main disk fails you can immediately boot from the copy and be back in action in a mater of minutes. The only caveat being that you will have lost anything you did after the last snap shot.
Secondly there are file back up systems that can also backup a whole disk, or just selected files, but they also keep historical versions of files. TimeMachine is an example of this type of backup. The system copies any changes made, every hour for example, and gives you the ability to recover previous versions. This is great protection against user error as well as equipment failure. You can recover the file you accidentally deleted last week.
The CCC approach is a great way to get peace of mind and a rapid recovery of a disaster. It doesn’t have to be a RAID or high reliability disc, just use a cheap USB connected external drive and you will have protection against a single catastrophic failure. You can step up to the full backup method using any number of different backup software solutions. The Apple brigade tend to use Time Machine as it come free with macOS. There are many options for windows and Linux too. Again with this method a simple USB connected external drive is all you need to protect against a single HW failure or user error.
it seems a number of us on the forum are paranoid and want high availability (RAID) and protection against multiple failures and catastrophic situations like the house burning down.
Take it as far as you like, no need to go RAID unless you want instant security against a single failure, or want faster disk access. An important point to keep in mind RAID isn’t a backup solution on its own. Don’t fall into the trap of connecting a RAID to your computer and using it as your active file store and thinking you are now ’backed up’. A RAID system can catastrophically fail as well.

 

[Anthraquinone]

I have three NAS boxes of different sizes attache dto my system. The only thing I do thta is prehaps not that normal is to only switch them on when I want to run a backup - one ever other day or so and then one at a time in rotation. That way I recon if I ever get hit by a virus or ransomeware my the data on at least one box will be OK.

 

[Bodone]

Just look up 3-2-1 backup strategies (most go for 3-2-2 option with easy access to cloud) and then decide what suits you and your budget. Don’t worry about time machine if going NAS, synology, qnap happily support Apple, cifs, samba etc.

If you’re an Apple Lr/Pshop user, lots of interesting stuff on backup and 2.5gbe/10gbe networking to support near real time in a robust environment. Samba 3 very good now in a mac environment.

For me Rawsmare dropped onto synology/expenology server, either from card or wireless form camera/phone, immediate copy to two other servers and OneDrive/amazon cloud. LR machine, iMac, then given shared access to a scratch area on nas with snapshots etc created on local ssd. Usual flow after that.

Physically iMac, two servers in house, another server in garage. No USB’s etc, for no other reason than version control. Multiple one drive accounts and then Amazon cloud. macs backed up locally and Apple cloud, same fro iPhones/iPads. All managed by qnap cloud/backup then synology/expenology as secondary and then local cron jobs.

 

[StevieB]

Thanks for those responses chaps, much appreciated. I will take a look at the links this evening - really appreciate it, what a wonderful forum this is!