The Value of Redundancy

Future Systems Enterprise, Inc.

You can always replace equipment, but you can’t always replace your data

By Xenohart, 2013 (updated for 2021)

Note: This article has been updated to address cyber security issues as they pertain to events in 2021.

As a systems analyst, I have had the opportunity to speak with a variety of I.T. Professionals from different private industries, municipalities, and government agencies about infrastructure setup and design, learning from their unique strategies for data protection. I have utilized these learned (and my own) strategies for mitigating data protection in the workplace, gleaning the valuable experience that comes from implementation (experience). Based on the results of my work over the years, I have developed multiple standards (by which I continue to stand) for reliable, data protection. Whether working with individuals, home businesses, families, small to medium businesses, enterprises or others, I have always considered an internal (in-house), back-up solution as the primary viable, ultimate solution to ensure reliability and security of data.

Of course, this philosophy has not always been well received, especially in the ‘cloud-computing’ era. Having come from a time when computers were more of a ‘burden’ than a valuable tool in the workplace, I have witnessed catastrophic losses of data. From companies spending tens of thousands of dollars in rebuilding databases (not to mention losses in productivity), to networking solutions that were unable to perform when needed, and even business shutdowns, technology has proven one thing: it is not perfect (nor reliable). But, even coupled with the constant failure of technology in our personal, everyday lives, people still choose to ignore the risk of data loss.

The big story today is about Microsoft subsidiary Danger losing all T-Mobile Sidekick customer data from their servers. Danger is the company noted for the T-Mobile Sidekick, the revolution in cloud mobile, and most memorably, almost everybody living in 90210 having to get new phone numbers because of Paris Hilton. Valued T-Mobile Sidekick customers received a notice today from the company updating them on the ‘data disruption’ problem. The good news is that data is no longer being disrupted. The bad news is that there is no data left to be disrupted.

… as Sidekick users found out today, and ironically, as 7,500 users of online backup provider Carbonite found out after the company lost their backups (Carbonite can take some comfort in that they now rank very well for ‘data loss’ in search engines because of the incident. What do they say about bad publicity?). In the Danger case, it appears from initial speculation that the data was lost because they attempted to upgrade a storage array without backing it up first. Here is a case of smart and rational people who do this for a living at one of the best companies in the world, and they didn’t even bother making a backup…” (Cubrilovic, 2009)

As a systems analyst, when planning system designs or improvements, I must ask the question: Why do people, businesses and even some government agencies ignore the risks vs. the benefits? There are a multitude of reasons ranging from financial decisions, control issues, policies and sometimes sheer stubborn will, but there is not one specific cause or justification that ever properly deals with the ‘what-if’. I’ve actually been told by a business owner, whose entire company was a mix of in-house and online computing (and was 100% reliant on the functionality of their computers and consistency of their data) that, “Well, my father didn’t need these things [computers], so I’m not about to waste an extra dime on paperweights!” Did he really just call the technology that his business was built upon, paperweights?

In another instance, I walked into a business to perform an I.T. infrastructure evaluation and at the end of my tour I turned to the C.E.O. and asked, “So, what’s your back-up solution?” His answer: “What’s back-up solution?”

To see businesses that have been built upon computing technology, putting their entire accounting databases, customer contact lists (and private information), financial and banking information, and other invaluable and irreplaceable data on computers with minimal to no back-up solution at all is almost sickening. And, I’m not talking about “mom and pop” operations, I’m including billion dollar, NASDAQ listed institutions, multi-billion dollar high-end companies, government agencies, hospitals and more. Even in today’s technologically advanced world, the lack of understanding over the fragile nature of technology reliance and need for true data protection is astounding. Just recently, Colonial Pipeline suffered a massive economic loss when they were supposedly ‘hacked’ and became the victims of ransomware (Morse, 2021). What surprised me the most is a company that handles such critical infrastructure and puts millions of lives, jobs, and different sectors of economy and government stability at risk, did not just simply restore the previous days’ back-up and continue their operations.

But, this is not a new problem. Around 1998, the crew at Pixar was working on the movie, Toy Story 2, when suddenly, in front of their very eyes – all of their data was wiped (Panzarino, 2012):

The master machine goes down,” says Jacob. “Some people are animating a shot and they can work for like a minute or five minutes, but eventually you’ll have to pull data from the master machine for some reason or another, which your machine will freeze.” …. “Eventually every animator and every TD, everyone working on the show goes, ‘Oh, all machines down. Let’s go to lunch. Ha, ha.’

The machine was eventually brought up a few hours later and they took a poll of the damage. When a size command was run on the Toy Story 2 directory, it was only 10% of the size it should have been. 90% of the movie had been deleted by the stray command.”

These are not isolated incidences. This type of disastrous data loss occurs every day. In 2012, NPV Corporation provided some of the horrifying statistics of data loss (NPV, 2012):

  • 93% of companies that lost their data center for 10 days or more due to a disaster, filed for bankruptcy within one year of the disaster.
  • 50% of businesses that found themselves without data management for this same time period filed for bankruptcy immediately (National Archives & Records Administration in Washington).
  • 94% of companies suffering from a catastrophic data loss do not survive – 43% never reopen and 51% close within two years (University of Texas).
  • 30% of all businesses that have a major fire go out of business within a year and 70% fail within five years (Home Office Computing Magazine).
  • 77% of those companies who do [keep backups] test their tape backups found back-up failures (Boston Computing Network, Data Loss Statistics).
  • 7 out of 10 small firms that experience a major data loss go out of business within a year (DTI/Price Waterhouse Coopers).
  • 96% of all business workstations are not being backed up (Contingency Planning and Strategic Research Corporation).
  • 25% of all PC users suffer from data loss each year (Gartner).

Data loss comes from several sources:

  • Intentional Actions (malicious users)
  • Unintentional Actions (lost flash drives, errors, deletions)
  • Failures (power failure, hardware failure, software bugs)
  • Disasters (Floods, fires, earthquakes)
  • Crime (Viruses, hacking, etc.)

Of all these causes, the two most common reasons are hardware failure and human error (David Smith, 2003). This means that sadly, that the smallest level of prevention could have saved an entire company. Sadly, I have learned over the years that there is one valuable lesson that will, without fail, educate individuals and businesses as to the value of their data: the loss of information. Although it is always devastating, it can also help teach the lesson of making the right decision to have reliable back-up and redundancy solutions (proactive solutions). Yet, this is like saying that individuals diagnosed with self-caused cancer and given months to live finally learn the value of healthy living.

On occasion, I’ll be called into a situation where I’m asked to provide some level of ‘recovery,’ (following an unexpected disaster) and my only solutions include sending off crashed hard disks to specialized agencies for tens of thousands of dollars, or shrugging my shoulders and letting the customer know that there is no [retroactive] solution. Online backups are no different. Even with the advancements of data centers, cloud computing, and ‘free’ online storage (forgetting the old adage that: ‘there is no such thing as a free lunch’), people have opted for the cheap and easy ‘online’ solution. But, this too, has had its consequences.

Back in 2013, Microsoft® upgraded their hotmail.com accounts, claiming that the changes were in response to a culmination of end user requests for improvements. People often utilize their email system as their ‘storage’ for valuable files and information, trusting in the ‘name’ of the corporation without reading their service agreement or knowing what is offered. Sadly, the changeover did not come without problems and thousands of people lost years and years of their saved emails. There were literally thousands and thousands of people posting on multitudes of forums, including Microsoft’s own forums, about the problems with permanently lost emails that they had been holding onto for years. Sadly, in one case, a Microsoft moderator responded to a user asking Microsoft to “Please, Please Help,” by writing: “You should regularly backup the content that you store on the services. Having a regular backup plan and following it can help you prevent loss of your content.” When asking a corporation for a solution to an immediate problem, a retroactive answer is condemning at best.

Even Microsoft doesn’t claim to be infallible, and all those with personal photos, stories, important business transactions, contacts and more – have lost everything. The only retroactive solution provided by Microsoft was for the user to have a ‘proactive’ solution in place. It’s like telling my clients, “I can help you today, only if you knew in advance what to do yesterday”; if I told my clients this – I would most certainly be out of a job!

With the recent advancements of Dropbox, Google Drive, and other platforms, more and more users have migrated their back-up and document storage solutions to these online systems. While the ‘cloud’ computing concept has existed for quite some time (previously labeled as free ‘ftp’ web and storage sites, etc.], the push to use online data back-up services continues to remain a risky decision at best. In 2011, RSA Security kept its SecurID token secrets online and was hacked. The Sony Playstation network was shut down from April to June 2011. GoDaddy had a 6-hour outage in 2013 that, according to CEO Scott Wagner, was by a corruption in router data tables (GoDaddy, 2012). Yet, with all that has happened, what is most alarming is the growing dependency on online backup solutions. It seems the [typically ignored] disclaimers, failure to read the terms and conditions, and follow-up recommendations for an off-line personal, back-up solution are not being taken seriously. From the Dropbox Privacy and Terms:

THE FOLLOWING TYPES OF EXCLUSIONS AREN’T ALLOWED, WE’RE RESPONSIBLE TO YOU ONLY FOR LOSSES AND DAMAGES THAT ARE A REASONABLY FORESEEABLE RESULT OF OUR FAILURE TO USE REASONABLE CARE AND SKILL OR OUR BREACH OF OUR CONTRACT WITH YOU. … IN COUNTRIES WHERE EXCLUSIONS OR LIMITATIONS OF LIABILITY ARE ALLOWED, DROPBOX, ITS AFFILIATES, SUPPLIERS OR DISTRIBUTORS WON’T BE LIABLE FOR: ii. ANY LOSS OF USE, DATA, BUSINESS, OR PROFITS, REGARDLESS OF LEGAL THEORY” (Dropbox, 2021)

From Amazon:

The Service Commitment does not apply to any unavailability, suspension or termination of SimpleDB, or any other SimpleDB performance issues: (i) caused by factors outside of our reasonable control, including any force majeure event or Internet access or related problems beyond the demarcation point of SimpleDB; (ii) that result from any voluntary actions or inactions from you or any third party; (iii) that result from you not following the current technical documentation applicable to the SimpleDB service (including the applicable developer guides) as posted by us and updated by us from time to time on the AWS Site; (iv) that result from your equipment, software or other technology and/or third party equipment, software or other technology (other than third party equipment within our direct control)…” (Amazon.com, 2021)

Many people think that storing their data on a RAID-protected NAS is an adequate backup. In fact, it is not. Many people use their NAS as a central server and store the only copy of their data right on the NAS. Things like fire, flood, theft, accidental or intentional deletions, user error or multiple hard drive failures can all lead to catastrophic data loss. If the only copy of your data is on the NAS, it is not a backup!” (Bott, 2009)

Learning from some of the best industry professionals, I have and will continue to tell my customers, “You need an in-house solution that you have control over.” And, it always comes down to having more than one piece of equipment providing a backup solution. Why? Here are some things that folks don’t think about (but should):

Fires, floods, and earthquakes, oh my! … “Well, it’s never happened to me…”

When I hear this response, I shudder. Disasters and accidents don’t happen every day, that’s why we call them ‘disasters or accidents’. Look at the recent weather events in our country from floods to freezes that were more than unexpected; they were unprecedented. These types of disasters are not isolated, nor are they even improbable. The ‘out of sight, out of mind’ philosophy is not only outdated in this information technology era, it’s downright irresponsible. It is difficult for me, as a systems analyst, to show a consumer the potential risk from damage due to accidents or disasters when they believe they’re not at risk. It’s even more difficult when they’ve been given bad information and think that their solution is more than satisfactory. For example, companies will explain to me that fire is not an issue as they have a built-in sprinkler system. “Sure, maybe the water damage will destroy the computers and hard files in the office, but insurance covers it…” Surprisingly, this is the mind-frame of many individuals and business owners. Yes, insurance covers damage, but it does not restore information! Insurance will reimburse for losses, but you’ll be putting out tens of thousands or more to rebuild databases (if you’re fortunate enough to even be able to do that), and that insurance money can be spread awfully thin.

Technology Failure: “Nothing’s happened to it yet, and we have a server so everything’s fine…

Having been present when this was said at a multi-billion dollar banking institution that months later not only lost their entire marketing database because the data wasn’t being backed up to any system due to its ‘sensitivity,’ also lost 7 hours of operational banking time (hundreds of thousands of dollars) because their ‘RAID’ server went down due to an error in one of the interfaces corrupting both the drive and the ability to replace it. Equipment is not infallible. It doesn’t matter if there’s a great I.T. team in place (which always makes me happy to see in any company), if the reliance of business operations rests on a single point of failure: the moment when that single point of failure is breached, so is the entire operation.

Cloud Computing: “We have an online solution, so thanks, but no thanks….”

With that, I just smile and wave and head off into the sunset. Guess what? So did tens of thousands of Microsoft Live Services users; and so have tens of thousands of other folks and businesses throughout the past 20+ years who have relied completely on online data. I once assisted with a company that had a massive outage due to a power surge which destroyed several internal systems, including their accounting and payroll database. Fortunately, some of their payroll went to an outside source. Unfortunately, all of the employees’ hours of data-entry work had been on hard drives that were only backed up online. Technicians were called in (including the support of their own, in-house staff), and they went to work purchasing new equipment, replacing systems and going on-line to begin downloading data. There was only one glitch: the software on the computers had not had the most recent update (which had been released during the time the new equipment was being acquired and installed), and the backed up data could not be restored by simply importing it. User account profiles were not being backed up, and for anyone who knows a network administrator, just ask them what it’s like to rebuild an entire domain and re-enter hundreds of users – from scratch (in a hurry). Worse, there had been transmission losses during the online backups (where data was not backed up and could not be recovered), and the process of manually bringing in files to scan and type in by hand required hiring 15 additional, costly staff members for 6 months! The rebuild cost them dearly in time and profit. Forget that data centers are just as subject to disaster as any business, the fact is: they were lucky to even have a database to restore.

A New Hope…

So, what does this all mean? It means, redundancy! Unless a backup solution includes more than one solution, it is truly not safe. Whether you use a personal home computer, are a small business, or part of an enterprise solution, true data backup requires ‘out-of-the-box’ thinking.

These situations may seem overwhelming and impossible and you may want to just give up and decide that it’s too much and you’ll take your chances. But, there’s good news – you don’t have to be overwhelmed. I participated with a financial institution’s infrastructure upgrade through their I.T. provider where I was pleasantly surprised to see that their information was being sent to a back-up server contained in an on-site, disaster-ready enclosure, along with two other methods of backup redundancy. I inquired as to the cost and found that for only a few thousand dollars, they had purchased the entire system and had it professionally installed. For a company that handled tens of millions of dollars in transactions annually and was responsible for hundreds of millions of dollars’ worth of sensitive, customer data, they had taken the time to properly ensure that their entire database was redundantly secured, behind an additional firewall that was frequently monitored. For only a fraction of the price of a ‘manual’ restore (which they believed would have cost them well over a hundred thousand dollars or more should it happen, not including lawsuits and insurance), they were able to maintain control of their data and have the added security and the benefit of peace of mind!

Individuals: I have personally worked with individuals and families that place irreplaceable pictures, family genealogy, business data, accounting records, research and similar types of information only on their primary computers. Many of these groups have not even considered a back-up solution, and those who have often reeled at the price tag of $200 (or less) for a backup drive and disaster-proof safe. So, we all get it: technology is not always inexpensive. But, having witnessed the tears of those who can’t recover the pictures of loved ones (especially those they’ve lost), or who suffered from a fire and had no way to recover personal financial information and identifying documents has been heartbreaking. If you’re in the market for a computer, consider the cost of a flash drive or backup drive as part of your purchase. Shop around – there are always deals going on. Don’t rely on your smart phone, laptop, pc, or [easily lost or stolen] flash drive either. Use more than one source. Backup valuable emails (not all of them!). You can work with a computer professional who should be able to, for a very low cost, set you up with a redundant backup system. Remember – you can’t put a price-tag on those pictures or files – so why would you put them at risk?

Small & Home Businesses: Sadly, many businesses have not even considered a back-up solution, and often justify not going that route due to issues such as “cost”. Maybe right now the budget is tight, but the answer remains simple: start small and start saving. I’ve recommended to customers that they take the hard drive out of the business place (a back-up is great, but a fire in the business place destroys not only the computer systems, but any unprotected back-ups), and put them in a safe-deposit box, or keep them at a different location when performing only a weekly or monthly back-up. Yes, it is extra work and perhaps even an added cost, but, the small effort (that may be tax deductible), will pay for itself when the process of restoring invaluable data takes nothing more than a couple hours, a plug, and a USB cable. If you can afford a disaster-ready safe, that one-time cost may be the difference between opening your doors for business tomorrow, or closing them forever. More recent advances in inexpensive VPN solutions provide a way to transmit data on a nightly back-up to off-site hard drives.

Medium to Enterprise Businesses: For larger companies, especially those who recoil at the thought of additional IT expenses (it always amazes me that so many companies refuse to invest in the very thing that keeps the money rolling in), should rejoice at the variety of disaster-ready prevention technology solutions on the market today. From hard drives with fire resistant enclosures to back-up server enclosures (that range from a few hundred to a few thousand in cost), the costs are reasonable and the benefits are huge. That doesn’t mean that simply backing up the entire company’s data to a fire safe/water proof-enclosed server will be the end-all of solutions, but it does mean that the 100% risk you were taking before has been dramatically reduced (or even mitigated entirely). Even if your company subscribes to a cloud service, that small investment for an on-site solution means faster recovery and increased productivity. You should also not be reliant entirely on an I.T. staff without some accountability. Whether a backup has been performed online or on local systems, expect a monthly report (that you can take less than one minute to review), showing the success or failure of the backup and its size. For every file that you have backed up, every account you have secured, and every mitigation measure you’ve taken, not only is your company’s financial future secured, but your customers can have that additional peace of mind, trusting that you really are there for them. Good publicity and trustworthiness still reign supreme in customer satisfaction and loyalty. I provide nothing less for my customers – and you can do the same.

I recently attended a computer security conference where they had an open discussion on system security. A lot of time was spent on multi-factor authentication and on-line backups. While the audience ranged from corporate CEO’s to individual users, the truth was that most of the information went over everyone’s head. The idea of multi-factor authentication to lock everything down was just as daunting as it was viable. I spoke up and explained that if they used an external drive, a VPN for transmitting data to a secured, off-site location, or a redundant server in addition to their on-line solution, should they be hacked (which was the primary concern), they could shut everything down, restore the back-up, change passwords and the hack would be over. No ransomware, no data theft, and minimal downtime. That was a solution, everyone could understand. It seems that many of today’s “security” companies, do not understand the people they serve.

The point is that, accidents do happen, but proactive solutions will always, without fail, provide a much higher return than retroactive ones. We buy insurance for a reason, even if we never have to use it. Sometimes, we even buy the warranty just for the peace of mind that we have it. But, there is no promise or security for your ‘data.’ You don’t purchase that, you generate it; input it; create it; and rely on it. Remember key words like ‘redundancy.’ Wearing a seat belt in a car with air bags is ‘redundant.’ Having a deadbolt in addition to a locking door handle is ‘redundant.’ And, it doesn’t take much research or effort to find a solution. Your own, or a nearby I.T. firm (and you can always do local calls to check on reputability), may have the best solution at a reasonable cost. Individuals and businesses can always contract the services of systems analysts such as myself to take a look at your system, and for a reasonable price tag, know whether or not your data is secure. Try putting a price tag on peace of mind – because sometimes – when it’s irreplaceable or extremely expensive to recover data … that’s everything.

Just don’t plan on there always being a ‘simple solution’ that you can treat like a paperweight and not value. Don’t wait until after the once-in-a lifetime flood or earthquake (that has a higher probability of occurring than the probability of recovery from a zero or non-redundant back-up solution) has crushed or erased the surface of the only hard drive containing your company’s records, or your personal files.

Thanks for Reading

(Now, if you don’t have a redundant backup – go get one, today!)


Works Cited

Amazon.com. (2021, Dec 5). Amazon Web Services: Overview of Security Processes. Retrieved from Amazon.com: http://aws.amazon.com/

Bott, D. (2009, Oct 13). Preventing Catastrophic Data Loss. Retrieved from ReadyNas: http://www.readynas.com/?p=3153

Cubrilovic, N. (2009, October 10). Letting Data Die A Natural Death. The Washington Post. Retrieved from http://www.washingtonpost.com/wp-dyn/content/article/2009/10/11/AR2009101100109.html

David Smith, P. (2003). The Cost of Lost Data. Graziadio Business Review, 6(3). Retrieved from http://gbr.pepperdine.edu/2010/08/the-cost-of-lost-data/

Dropbox. (2021). Privacy and Terms. Retrieved from Dropbox.com: https://www.dropbox.com/privacy

GoDaddy. (2012, Sep 11). Go Daddy Site Outage Investigation Complete. Retrieved from Godaddy.com: http://www.godaddy.com/news/article/go-daddy-site-outage-investigation-completed.aspx

Morse, A. (2021, May 12). Gas shortage 2021? What you really need to know about the pipeline hack. Retrieved from: https://www.cnet.com/news/gas-shortage-2021-what-you-really-need-to-know-about-the-pipeline-hack/

NPV. (2012, Nov 12). Backup Data Services and Statistics. Retrieved from NPV.com: http://www.npv.com/?tag=catastrophic-data-loss Panzarino, M. (2012, May 21). How Pixar’s Toy Story 2 was deleted twice, once by technology and again for its own good. Retrieved from The Next Web.com: http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-and-again-for-its-own-good/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s