Why do we have so many issues with software programs

[Sideways]

First question has to be, what format are these images currently in ? It is great to have a secure copy in ten years time but not much use if the format is no longer supported.

The majority of the volume are images in Nikon raw and jpg as saved direct from camera.
I have licensed copies of win10 and lightroom so worse case just build a machine with a clean win10 OS and install lightroom 5 on it.
I don't trust adobe raw any more than NEF in the timeframe of say 10 years so I doubt it is worth the time needed to convert my images before I will come back to use them again.

 

[Spectric]

I don't trust adobe raw any more than NEF in the timeframe of say 10 years so I doubt it is worth the time needed to convert my images before I will come back to use them again.

The thing with ACR is that even if Adobe vanished you will still have the DNG files, all ACR does is to allow a DNG to be produced from a range of camera formats which are camera specific. Once you have the DNG, ACR has done its job and you have a future freindly image file where there is other software that can be used and worse case you can just write your own because the format is open source. The DNG is just a bucket that contains your greyscale image along with the metadata so ready for post processing. The JPEG files are what they are, did these come from the camera or from the result of digital editing ?

 

[paulrbarnard]

Not sure I would trust either disk type for long term archiving. This might be a better shout: Pioneer's New Blu-ray Discs Are Guaranteed to Last a Century

It's likely archiving companies will offer this in the UK soon.

They said the same about CD and DVD…. The only data I ever lost was on CD.

 

[Spectric]

Lets throw another spanner in the works, not mater how good the backup system you are running or the medium being used one of the problems with storage is a phenomenon known as bit rot. If you have a hard drive with your precious data on it and put it in a safe location as a backup then you might well find the data is corrupt because of degradation. Both mechanical and solid state drives suffer as well as USB pen drives, the answer is to decide how valuable your data is and make a decision that ou will replace the device after a set period, much better to replace a device that is still functioning rather than wait till failure unless your backup system has multiple devices of different ages, ie some relatively new ones.

 

[AndyMc]

A couple more thoughts:
If you do decide to buy disks and DIY it then hedge your bets. Make sure you don't buy 3 of the same brand and model. There are numerous examples of manufacturing defects where batches of disks go bad all at the same time. One of my first jobs was dealing with the fall out of 'stiction' on IBM disks in mid range Unix systems. It happens far more regularly than it should in enterprise IT where large raid arrays often use batches from the same manufacturer. Bear in mind that SSD manufacturers may buy in their chips from another vendor, so two different models from two different manufacturers may share the same hardware.
Getting reliability data from disk manufacturers is hard. Really large customers of Seagate/WD/Toshiba don't make this public, e.g. Dell, HP, and the cloud hyperscalers. However Backblaze do have some data that could be used to inform your choices:Hard Drive Test Data

Re: data formats, both image formats and encryption, it would be worth considering lossless formats for which there's an open source implementation. In 10 years' time you're no longer at the mercy of a vendor who's decided to drop support because it wasn't commercially viable.

Are you sure I can't interest you in some cloud storage?

 

[AndyMc]

They said the same about CD and DVD…. The only data I ever lost was on CD.

Yes, the dyes used in cheap writeable CDs and DVDs are not great. Even well made ones are only good for a decade or two. It's precisely why the Japanese defined JIS X6257.

 

[Blackswanwood]

Apologies if this has already been covered but what are the drawbacks of using cloud storage say with Google, MSN or Apple? (For personal data)

It seems you pay a fee and it’s all taken care of. The risks seem to be that it may be awkward to swap provider, the costs could go up, you need internet access to make it work and you are reliant on the providers data security protocols. Am I missing something?

 

[paulrbarnard]

Apologies if this has already been covered but what are the drawbacks of using cloud storage say with Google, MSN or Apple? (For personal data)

It seems you pay a fee and it’s all taken care of. The risks seem to be that it may be awkward to swap provider, the costs could go up, you need internet access to make it work and you are reliant on the providers data security protocols. Am I missing something?

They are fine if you want to keep paying and the supplier doesn’t decide to increase prices once they have caught you and they continue to offer the service.

Many years ago I use the Mac cloud from Apple. They decided to stop providing it and didn’t offer an alternative. I had to copy all my images off of .Mac and I went with SmugMug which offered a cheap alternative. A few years later they jacked their prices right up. I keep everything under my control now as I simply don’t trust the providers out there to pay any attention to my needs only their own margins.

 

[Sideways]

Isn't "bit rot" about the gradual loss of charge inside SSD storage or fading of magnetisation on a hard disk platter making 1's and 0's less defined to the point where you can no longer clearly read them and enough bits become compromised that you can't fix it using the error checking data ?
I've assumed that the way to deal with that was simply to copy the entire contents from one drive to another, thus refreshing the "image".
If the disk isn't spinning in the meantime, there is nothing to wear out. Just be conservative about the rate that the data degrades in storage. 5-7 years is widely bandied about so stored safe in a fireproof box and cycled around 3 or 4 disks every year or two might be safe enough ?

 

[Sideways]

Are you sure I can't interest you in some cloud storage?

BT have 2 or 3 major facilities in the UK that carry a huge amount of data traffic for all sorts of big companies, including but by no means limited to streaming the data from netflix video servers to homes all over the country.

An interesting observation about these sites is that if one of them were to go off line for more than a few days, the penalty clauses BT has committed to it's big clients for loss of revenue are so large that the company would be bankrupted. It focusses the mind when you simply cannot afford to fail.

If you are a big enough customer, your supplier listens to you.
If you are a "consumer", then big companies rarely give a damn about you.

We are more risk averse when control of that risk is in someone else's hands not our own, so if you were to offer me 10 years storage of my (say) 5 terabytes, for a few hundred quid (total), with a guaranteed availability and an insurance guarantee against loss and against my data being compromised / stolen / hacked of say £250k. I'd engage in the conversation....

 

[Sachakins]

OK you say cloud storage is not an option, but I think you are limiting your options for safe storage without it.
If you have or get Amazon prime account you have access to unlimited photo storage, but only a 5gb video storage.

I would opt for a multi material protocol. Being in this order.
A: Cloud storage, reason quick retrieval, easy to validate data integrity.
B: Removable SSD drive, data stored as RAW image, not compressed, stored offsite.
C: Removable SSD drive, data stored as RAW image, not compressed, stored on site in fireproof safe.
D: Hard media storage, ie DVD, with two sets, using two different media manufacturers.
E: High quality flash drives and/or SD cards, of at least level 10 video storage quality, stored off site in seperate location to option B.

It really does depend on the value you put on your data, the greater your value, then the more of the options above you should use, using all would mitigate the risk of a cascade failure to the minimum.

Remember, storage today is extremely cheap, so multiple media types and archive strategies mitigate single point failure.

The other important factor is the method of archiving the data to the storage medium,.
DO NOT use one program, use at least two preferably three different archive systems. It minimises risk of archival medium cascade failure.

 

[Keith Cocker]

Many moons ago photography was a serious hobby of mine. I had thousands of storage sheets of negatives - 35mm; 6x6; 6x4.5; 4x5 - along with thousands of hand printed photographs. All classified and stored on shelves from floor to ceiling in my office at home. We live on a hill and a stream runs down past us. One night a flash flood inundated the ground floors and the lower two shelves were inundated. Never again I thought and started to digitise my remaining work. I ignored a suggestion of cloud storage and backed up to external hard disc. We had a lightning strike in the road and hard disc fried and computer failed. I store my remaining work in Apples cloud now and have done for several years. Job done.

 

[pe2dave]

I went through this with recorded books for the blind. On tape, hundreds of them.
Eventually settled on reasonable large modern SSD's. As above, ensure
they are in a long term 'format' (and the disk format), then draw up a plan
Not only cycle them. Give them a kill date. Find out how to check for bit rot.
Good luck.

 

[Agent_zed]

Apologies if this has already been covered but what are the drawbacks of using cloud storage say with Google, MSN or Apple? (For personal data)

It seems you pay a fee and it’s all taken care of. The risks seem to be that it may be awkward to swap provider, the costs could go up, you need internet access to make it work and you are reliant on the providers data security protocols. Am I missing something?

Whilst this isn't image storage I currently can't see anything that I 'save' on facebook. If I go to the 'saved' page it just tries to load and returns a blank page.
Doing the same for my dad on his account. To be fair I haven't asked FB to look into it yet (mostly as I don't care as i barely use it).

My point is that Facebook is one of the biggest tech companies in the world and can't manage to ensure my saved items list works. The is always the chance that for some reason an online image storage company might lose your data/account or it could get hacked and you get locked out. It might be a low risk but having to get a big tech company to give a monkeys to get you your account back is a risk, or at least a hassle.

As with all backup solutions it is good to spread your risk out, online storage might be part of this but I would also like a few physical backups as well.

 

[XTiffy]

Note to oneself,"Don't buy a house near Keith"! Seriously why do we build all of our living accomodation and thus all of our valuables etc. at risk of natural disasters on the ground floor? Seeing the distress caused with recent floods, when losing all personal possessions which can't be replaced, makes me think of storing all that is most important upstairs (bit dificult if you live in a bungalow).

 

[paulrbarnard]

I keep all my images on a RAID drive with hot standby. It is configured to be able to correct two bit failures in any byte. It also has a hot standby disk that automatically get used if one of the disks in the raid fails. This is my first line of protection. The Raid is backed up to a second Raid array which keeps a versioned backup of the files. This keeps a record of the changes I’ve made to files in case I screw up. The images are also mirrored to three other computers using OwnCloud, so there is yet another copy on the OwnCloud server, as well as copies on the other three computers, which are also each version backed up to external drives. One of those additional computers is ‘off site’ in my workshop.
Fundamentally I have 10 copies of the current version of any image plus older versions going back in time for multiple iterations.
Oh an to really indicate how paranoid I am I also shoot images in RAW and JPG. The JPG files go to a seperate SDCard in my camera and when those cards get full it gets replaced and the SDCard is stored in a drawer in my workshop. So if all my computers fail and all my backups fail I still have the SDCard JPG images direct from my camera.

The irony of all this is that when I die my kids will probably grab a handful of meaningful images of their childhood and my TB of landscapes, holiday photography and wildlife images will simply disappear from the world.

 

[Rhossydd]

I ignored a suggestion of cloud storage and backed up to external hard disc. We had a lightning strike in the road and hard disc fried and computer failed. I store my remaining work in Apples cloud now and have done for several years. Job done.

The whole point of external HDDs is that they should be kept separate and unconnected except when actually transferring data.

 

[Rhossydd]

Once you have the DNG, ACR has done its job and you have a future freindly image file where there is other software that can be used and worse case you can just write your own because the format is open source.

Just be aware that DNG has many options and has passed through many iterations and not every version is supported by all the software that tries to.

 

[selectortone]

The irony of all this is that when I die my kids will probably grab a handful of meaningful images of their childhood and my TB of landscapes, holiday photography and wildlife images will simply disappear from the world.

Yep.

Whereas the photographs my Mum kept throughout her life, some dating back to her childhood in Ireland in the early 1900s, survive. That's progress!

 

[Sideways]

OK you say cloud storage is not an option, but I think you are limiting your options for safe storage without it.
If you have or get Amazon prime account you have access to unlimited photo storage, but only a 5gb video storage.

I would opt for a multi material protocol. Being in this order.
A: Cloud storage, reason quick retrieval, easy to validate data integrity.
B: Removable SSD drive, data stored as RAW image, not compressed, stored offsite.
C: Removable SSD drive, data stored as RAW image, not compressed, stored on site in fireproof safe.
D: Hard media storage, ie DVD, with two sets, using two different media manufacturers.
E: High quality flash drives and/or SD cards, of at least level 10 video storage quality, stored off site in seperate location to option B.

It really does depend on the value you put on your data, the greater your value, then the more of the options above you should use, using all would mitigate the risk of a cascade failure to the minimum.

Remember, storage today is extremely cheap, so multiple media types and archive strategies mitigate single point failure.

The other important factor is the method of archiving the data to the storage medium,.
DO NOT use one program, use at least two preferably three different archive systems. It minimises risk of archival medium cascade failure.

Some good points in here especially on the multiple systems and strategies but I did say I have a few terabytes to store.
Basically a quick look around tells me that 5-6Tb will cost me a thousand dollars over 10 years to store with anyone I've heard of. Enough to pay for a set of multiply redundant HDDs.
I can get a lifetime subscription for 10tb for about £70 from someone I've never heard of and would be stupid to trust my data to because they won't exist next year....
None of them guarantee their service. They have no skin in the game. They don't stand to lose anything when they screw up :-(

I don't need offsite. I can store in a fireproof safe that won't be stolen and we have zero flood risk because we live on the top of the hill not in the valley.
I don't need to worry about lightning because these will all be stored unplugged.
I am conscious that NAS boxes and the like incorporate a higher risk because they have a common power supply feeding multiple drives and non redundant power supplies are less reliable than the drives they power, so I'll choose accordingly...

Optical isn't at all convenient BTW. I'd need about 160 blue ray disks to archive my stuff if they are still around 25Gb a disk. I don't have the patience to write and maintain that.

Interesting that SSD's are recommended. That technology is advancing very rapidly compared to old style HDDs and it seems that some of the latest and best may well have a low enough charge leakage that they will retain data for longer than a magnetic disk when unplugged, but an older or lower or unknown quality SSD is potentially much more of a risk than magnetic media.
In the spirit of diversity I should probably go for a couple of HDDs and a couple of SSDs.
I used to buy seagate barracuda drives. WD had a rep for being fast but noisy, IBM (now Hitachi, yes) were nicknamed "deathstars" because of their poor reliability, and Sandisk are so widely faked that even they are suspect if you buy from someone like Amazon who's stock control is infamously bad.
Hmmm