Don’t Let This Happen to You: Cautionary Tales of Data Loss for World Backup Day 2024
The World Backup Day is observed on March 31, serving as a reminder of the possibility or certainty of data loss from human error, system failure, or threat actors’ malicious intent. As World Backup Day 2024 approaches, read about some of the data loss horror stories members of the Spiceworks Community witnessed and how they could have been prevented.
- World Backup Day is observed on March 31, reminding of the possibility or certainty of data loss from human error, system failure, or threat actors’ malicious intent.
- As World Backup Day 2024 approaches, read about some of the data loss horror stories members of the Spiceworks Community witnessed and how they could have been prevented.
Data loss is widespread, rampant, and inevitable. It is not ‘if’ organizations or individual users are likely to face this problem, but rather when they will. So the question is: is data loss avoidable?
According to Proofpoint’s 2024 Data Loss Landscape Report, 84.7% of organizations experienced one or more data loss incidents in the year past, with each organization suffering a mean of 15 incidents annually—more than one per month.
Data loss incidents led to 56.6% of organizations’ business being disrupted, 38.9% suffering a setback to their reputation, 35.8% finding themselves in a weakened competitive position, 34.8% being issued a regulatory fine, not to mention a significant portion bore significant litigation expenses.
The reasons for data loss vary. Organizations experience data loss from ransomware or other cyberattacks, misconfigured systems, carelessness, inside jobs, and more. “Backups are arguably the top control in reducing the impact of the three major types of data loss events: Human error related, natural disasters, and the most destructive (but least considered) of them all: threat-actor-caused mass destruction events,” John Anthony Smith, founder and CSO at Conversant Group, told Spiceworks News & Insights over email.
“Every year, the amount of data we produce increases significantly. World Backup Day is a call to action, urging us to reconsider our strategies for simplifying backup and recovery to keep pace with the significant increase in data production each year,” Bin Fan, Chief Architect and VP of Open Source, Alluxio, told Spiceworks.
“As we scale the data storage, timely data movement is a necessity, whether for archiving data in more economical storage or for duplicating data to another center as part of a disaster recovery plan. However, this process can be complex and operational-heavy. We should keep optimizing and streamlining data movement across multiple storage systems.”
While that’s prudent, it is equally crucial to look at past incidents, observe how error-prone technical tasks can be, and learn from them. This World Backup Day, read about some of the most harrowing data loss incidents members of the Spiceworks Community witnessed and how they could have been prevented.
Haunting Data Backup/Loss Stories
1. Unhappy academics
“Circa 1989 and 1990, academic engineering environment. Operations did backups to 1/2 inch tape on the Daily/Incremental/Fulls on the Tower of Hanoi model. What is important is that the oldest full backup was 56 weeks old. Over that first summer, the 3-4 disk space was converted to one logical volume. The following summer, a hard drive in that array failed. Not until this happened did anyone have a clue that there was a known operating and file system bug that kept a rebuild to the array from working and also corrupted all the newer backups. We had to restore from that last, year-old full backup from before the conversion since everything newer was unusable. Obviously, there were a lot of unhappy academics.”
“These kinds of mistakes are disastrous but easily avoidable. You have to test your backup and restore technology and processes. Otherwise, you end up in exactly this situation,” Jeff Williams, co-founder and CTO at Contrast Security, told Spiceworks News & Insights.
Pierre Gueant, director of Solution Partners at Scality, responds:
“File systems have matured, and bugs are rare, but if it happens, the ability to restore applications to a working state is the key to success. The secret weapon is an application-aware agent that can capture the data using good intelligence. Also, consistently ensuring that backups are recoverable is essential. The best backup vendors have devised methods to stage backup recoveries so the day a restore needs to happen, success is maximized to its full potential. And, to increase the chances of a full restore, make sure the backup storage system can handle the extra load and is capable of fast restores and instant recoveries.”
Williams added, “The scariest scenario is that an attacker breaks in and slowly starts to corrupt your data, unbeknownst to you. By the time you find out, it’s been months, and you don’t have backups going back that far. So even though you have a working and tested backup strategy, you’ve been destroyed from the inside out. Your entire business is built on a base of shifting sand.”
2. Human error
“I had a co-worker who failed over a replication relationship on a storage array. They ran on the backup system for a few days. They eventually switched back to production and restarted the replication. The problem? They did not reverse the replication when on the backup system. so all that data was lost when replication was restarted.”
Gueant responds: “It’s easy to make mistakes in the middle of the storm. Careful advanced planning of the operations and procedures will greatly maximize the chances of a happy outcome. Remember, when disaster strikes, it’s not enough to have a readily available backup copy. Make a plan with easy-to-use solutions in place that allow you to execute your DR process smoothly, efficiently, and at a moment’s notice.”
Narayana Pappu, CEO at Zendata, told Spiceworks, “Human error is a huge factor – at least at the moment, potentially changing with AI and automation that can do these checks and automate the workflows. Till that happens, the best way to minimize risks is better education, testing, checks and balances, and additional backups for the data.”
See More: World Backup Day: It’s Time to Right-Size Your Data Backup Strategy
3. Obsolete tech – a jog down the memory lane
“RAID 5 was by far the most common denominator. The issue was not so much with RAID 5 itself, but it had too many points of failure. First, the only reason for the existence of RAID 5 was it presented a 25% cost savings over mirror RAID. During the early 2000s, there was this trend for space consolidation for no real reason. Data center managers became obsessed with small form factor servers in pretty racks with glass doors. That was fine…as long as you cooled your stereo gear…I mean 2U servers with air temp that kept the data center colder than the back of an ice cream truck. If your AC failed, or facilities did a generator test over the weekend in the summer, and they didn’t have AC on the backup circuit (why would they), those RAID controllers would start writing parity trash long before they faulted. Thank you, EMC, for your amazing thermal controls. Sidebar, but why do we have AC units that can cool Chernobyl running in the data center while it’s 17F outside?”
The solution is upgrading to advanced storage and backup tech.
“RAID is not enough in today’s data protection landscape! This story points out the leaps that have been made in data protection mechanisms, especially when we get into 100s of terabytes or petabyte-scale range. It has been well over a decade since we recognized the obsolescence of RAID5 and then RAID6 for protecting data at scale from failures, as those didn’t provide sufficient failure protection but were also extremely susceptible to extended repair times and higher data loss rates. Today, modern storage systems optimized for data protection at scale will deliver much more resilient and reliable distributed data protection mechanisms such as those provided by erasure coding,” Gueant noted.
4. An update gone wrong in the wrong place at the wrong time
“Let us stroll back to 2002 in The Netherlands. I was a lightly seasoned greenhorn of 3 years; the org I was working for had a network that had started as a few PCs and a few PCs decked out to be servers…had to a large network with hundreds of users, still relying on these beefed up desktops. One desktop had NT4 and a whopping 2GB of RAM with an Arena Phertron RAID enclosure attached to the file server Phoenix. Others had names like Boston, Atlanta, Chicago, etc. We had a long weekend and had a maintenance window to run one of the latest SPs for the NT4 Server. That all went well, and then there was an update to the RAID software; my boss was hesitant, and I looked at the clock and said, it’s only 4 pm; we’re making good time… So he launched the update utility and fat-fingered some configs, and the RAID didn’t come back after the update. And the backups were corrupted. Did I say this was in The Netherlands? Well, all my colleagues rode bikes to work! I had recently gotten a car but knew nothing but the city where I lived and worked. We had an intern working with us who knew where the local dealer was. We had about 40 minutes to go, and this was before GPS. After asking many locals where this place was, we managed to catch the owner just as he was locking up for the weekend. Whew!
Not so fast…when we got the RAID enclosure back, the data was intact, but due to it being attached to NT4, all ownership had been lost (that’s what I was told). Now, someone had to sift all this data and put it back into user folders. I sat in the server room for nearly three weeks, opening docs and looking for names; in the end, a lot of data was orphaned, and I couldn’t determine the owner.”
Pappu opines, “Data catalog or metadata with information on ownership of information would have helped with this issue. Remove the need for someone to sift through information manually and figure out which information goes where or who it belongs to.”
Meanwhile, Gueant quips, “Never make a change on a Friday! Establish the no-Friday change rule as a corporate policy. Also, consistently check backups and use a solid immutable S3 object store as the backup storage system. It will save days, weeks, and even years of headaches!”
5. Beyond the enterprise – individuals can face data loss horrors, too
A personal one: “Big hard drives are great – except when they fail! I had a 3TB drive partitioned into four drive letters where I stored lots of stuff that was (luckily) not critical but valuable all the same. Without warning, the drive failed, taking every partition with it. Unreadable, even in BIOS; not worth spending thousands on professional recovery, but years’ worth of picture collections, older email .PST files, movies, music, etc., lost forever. I am extremely wary of big hard drives now.”
Gueant attests to the individual’s fears. He said a high-capacity hard drive for backups is never a good idea. “The remedy: Applying the 3-2-1 backup rule not only applies to organizations but also applies to individuals, particularly when protecting personal information. Consider backing up your personal data on a NAS with two or more drives to sustain the loss of the drive, and be sure to make an offsite copy of the data and store it in the cloud. Your ISP subscription or Office 365 subscription may include cloud storage, so check there first,” Gueant said.
Meanwhile, Pappu suggested leveraging online backup along with a personal hard drive. “Backup continuously. These types of services are super affordable nowadays for individuals and cost a fraction of a 3TB hard drive,” Pappu said.
See More: 3-2-1: The Backup Strategy You Can Count On
Honorable Mentions
1. 2021 Colonial Pipeline ransomware attack
The Colonial Pipeline ransomware attack in May 2021 by the DarkSide gang grounded flights and rendered cars useless for days, thanks to a disruption in the oil supply on the Eastern coast (Colonial was responsible for 45% of it).
Colonial Pipeline ended up paying DarkSide $4.4 million in exchange for a data recovery tool. However, this proved ineffective as the data recovery process would be long drawn out. Would having backups have helped? How should have Colonial Pipeline approached data backups?
Well, it’s a no-brainer that Pappu responded, “Yes, having backups would have helped. If they had a multi-tier backup strategy with multiple copies of information and better access controls, they could have avoided this issue.”
Erich Kron, security awareness advocate at KnowBe4, told Spiceworks, “One of the significant leverage points of ransomware is related to denying the organization access to their data, oftentimes resulting in complete work stoppage. Unfortunately for organizations, even paying the ransom and getting the decryption keys and software does not mean they will be back up and running quickly. At times, it’s even possible that the infrastructure supporting the ransomware has been taken down by law enforcement or other means, meaning there is no way to decrypt the data.”
Candida Valois, field CTO at Scality, told Spiceworks that 93% of ransomware attacks target backup repositories and have a hit rate of 68%. Moreover, cybercriminals succeed in disabling victims from recovering in 75% of these attacks.
This is why Gueant suggested the 3:2:1:1 rule for organizations’ backup approach for ransomware protection. “Backups now sit squarely in the cybercriminals’ crosshairs – and they’re essential to achieving data resilience. Without an immutable storage solution, you’re sunk. For ransomware protection, follow the best practice 3:2:1:1 rule with immutable storage to keep data safe by acting as a last line of defense,” Gueant said.
“The extra 1 refers to ensuring that at least one copy is offline, air-gapped, or immutable, meaning a backup copy of your data cannot be altered, deleted, or changed in any way, even by system administrators or the users, applications, or systems that created the data.”
2. The Toy Story 2 incident
The sequel to the acclaimed Pixar film was almost lost forever if not for the film’s supervising technical director copying the database repository on her personal computer while working from home. Pixar’s then-chief technical officer, Oren Jacob, stated on Quora that “more than several percentage points of the show (as measured in numbers of files) were never recovered at all.”
Once again, data immutability is the answer. “The Toy Story team would have benefited from a ‘human-proof’ immutable storage solution (a backup copy of data that can never be altered, deleted, or changed in any way, even by system administrators or the users, applications, or systems that created the data),” Gueant said.
“It’s important to emphasize that not all immutable storage is equal. Modern solutions now go beyond immutable storage to lock data down at five levels for unbreakable data protection at every level of the system, which includes API level, data level, storage level, geographic level, and architecture level. Also, never keep just one copy of your data. Best practice commands at least three copies, stored on two different media with one copy at an offsite location.”
Image source: Shutterstock
MORE ON DATA LOSS AND BACKUP
- World Backup Day: Backing Up Your Data Starts With Securing It
- How to Enhance Ransomware Resilience: A Complete Playbook
- Closing the Gap: Cyber Security and Disaster Recovery
- The Disparity Between Cyber Security and Disaster Recovery Is Putting Businesses at Risk
- Backup to Move Forward: A “Gentle Reminder” for World Backup Day
