The Backup Wrap-Up

W. Curtis Preston (Mr. Backup)

Formerly known as "Restore it All," The Backup Wrap-up podcast turns unappreciated backup admins into cyber recovery heroes. After a brief analysis of backup-related news, each episode dives deep into one topic that you can use to better protect your organization from data loss, be it from accidents, disasters, or ransomware.

The Backup Wrap-up is hosted by W. Curtis Preston (Mr. Backup) and his co-host Prasanna Malaiyandi. Curtis' passion for backups began over 30 years ago when his employer, a $35B bank, lost its purchasing database – and the backups he was in charge of were worthless. After miraculously not being fired, he resolved to learn everything he could about a topic most people try to get away from. His co-host, Prasanna, saw similar tragedies from the vendor side of the house and also wanted to do whatever he could to stop that from happening to others.

A particular focus lately has been the scourge of ransomware that is plaguing IT organizations across the globe. That's why in addition to backup and disaster recovery, we also touch on information security techniques you can use to protect your backup systems from ransomware. If you'd like to go from being unappreciated to being a cyber recovery hero, this is the podcast for you.

Subscribe Listen in podnod RSS feed Show website

359 episodes
Updated Yesterday

Episodes359

Newest first

Jan 24, 2022 · 40 min
Happy Data Privacy Day!
This week we celebrate Data Privacy Day, which is an international event that occurs every year on 28 January. According to its website, "The purpose of Data Privacy Day is to raise awareness and promote privacy and data protection best practices. It is currently observed in the United States, Canada, Nigeria, Israel and 47 European countries." Prasanna and Curtis discuss the latest in privacy practices and regulations, drawing on Prasanna's new experience protecting the privacy of Zoom's customers. We talk about the difference between PII and Personal Data, different regulations around the world, and some new tech features you can use to protect your privacy. We also discuss a few gotchas out there, such as Verizon's new system that they require you to opt out of! Happy Data Privacy Day, everyone!
Jan 18, 2022 · 57 min
Does K8s still need DR? O'Reilly K8s Author On the Hot Seat!
Things got a little tense on this week's podcast when James Strong (@strongjz, Co-Author of O'Reilly's Networking & Kubernetes) hinted at DR being a thing of the past with K8s. Mr. Backups was having none of that. No blows were thrown, mostly because it was all online, but it was a really good conversation that K8s and DR enthusiasts alike will find interesting. We also cover the new book, Networking and Kubernetes, by James Strong and Vallery Lancey, including why the decided to write it, and what it covers. We talk about monitoring K8s networking, and James uses at least 25 acronyms that may be new to some listeners. Don't worry: we make him explain all of them.
Jan 10, 2022 · 56 min
Resiliency expert reviews Major Facebook Outage
Bob Plankers, resiliency specialist from VMware, joins us on this week's podcast, as we examine last year's major Facebook outage that took out Facebook, WhatsApp, and Instagram all at once. We discuss what we believe happened, just how bad it got, and our thoughts as to what we can learn from this huge outage. This isn't schadenfreude, and we acknowledge that we are Monday morning quarterbacking. It's an attempt to LEARN from the misfortunes of others – not to take joy in them. Solid discussion with @plankers.
Jan 3, 2022 · 58 min
Remembering the Best and Worst of 2021
On this first business day of 2022, let’s take a look back at the year that was. It was, of course, another year of COVID. In fact, Curtis contracted COVID right at the end, despite being boosted. This is also the year of the OVH fire that we talked about for three episodes: https://www.backupcentral.com/datacenter-manager-dan-frith-discusses-the-ovh-fire-restore-it-all-podcast-105/ https://www.backupcentral.com/how-do-you-prove-your-backup-service-is-real-restore-it-all-podcast-106/ https://www.backupcentral.com/ovhs-backup-service-didnt-work-restore-it-all-podcast-107/ We think our most interesting episode of the year goes to Paul VanDyke from Kodiak Island, who deleted his whole environment and then tested his backups: https://www.backupcentral.com/it-admin-deletes-entire-datacenter-then-tests-his-backups-restore-it-all-podcast-135/ We even had a barbecue episode. No backups, just beef and BBQ. https://www.backupcentral.com/no-backups-just-beef-bbq-restore-it-all-bonus-episode/ Prasanna said he learned a lot about tape this year from these three episodes: https://www.backupcentral.com/tape-drive-designer-schools-mr-backup-on-tape-restore-it-all-podcast-111/ https://www.backupcentral.com/deep-dive-into-why-tape-still-has-a-future-in-storage-restore-it-all-podcast-129/ https://www.backupcentral.com/fujifilm-tape-evangelist-explains-past-present-future-of-tape-lto-restore-it-all-podcast-132/ We think our best “get” was Peter Krogh, who talked about how he coined the term “3-2-1 Rule” while writing the first edition of The DAM Book: Digital Asset Management for Photographers. https://www.backupcentral.com/peter-krogh-who-coined-the-3-2-1-rule-on-our-podcast-restore-it-all-podcast-131/ This led to a brief discussion about solar flares, where I mention a talk I watched by Intel. This is a link to that talk: https://techfieldday.com/event/eicd16/ We also had two Druva competitors on this year, Veeam and HYCU. https://www.backupcentral.com/dave-russell-answers-our-questions-about-veeam-restore-it-all-podcast-104/ https://www.backupcentral.com/veeam-reps-explain-defense-against-conti-ransomware-restore-it-all-podcast-127/ https://www.backupcentral.com/hycu-vp-explains-their-service-to-mr-backup-2/ The big winner of the year was ransomware. We talked about it a lot: https://www.backupcentral.com/ransomware-victim-tells-his-story-restore-it-all-podcast-96/ https://www.backupcentral.com/is-entity-level-encryption-the-answer-to-exfiltration-ransomware-restore-it-all-podast-119/ https://www.backupcentral.com/protecting-your-network-from-ransomware-restore-it-all-podcast-122/ https://www.backupcentral.com/restoring-quickly-from-a-ransomare-attack-with-a-long-dwell-time-restore-it-all-podcast-123/ https://www.backupcentral.com/veeam-reps-explain-defense-against-conti-ransomware-restore-it-all-podcast-127/ Curtis this everyone should be looking into an intelligent DDI (DNS, DHCP, IP management) system that will spot (and stop) ransomware when it tries to reach out to its command and control servers. https://www.backupcentral.com/stop-ransomware-in-its-tracks-with-dns-dhcp-ipam-restore-it-all-podcast-87/ You also need to monitor your bandwidth to look for exfiltration: https://www.backupcentral.com/securing-speeding-up-network-traffic/ https://www.backupcentral.com/stop-ransomware-attacks-in-seconds-restore-it-all-podcast-126/ Finally, we talked a little about the book, and the upcoming 2022. https://www.backupcentral.com/why-you-need-a-copy-of-modern-data-protection-restore-it-all-podcast-110/ Happy New Year, everyone! Here’s to a better 2022!
Dec 20, 2021 · 58 min
IT Admin deletes entire datacenter THEN tests his backups!
This week’s guest tells the most incredible story we’ve ever had on the podcast. We’ve had ransomware restores, disaster recoveries after a hurricane, but we’ve never had someone who deleted their entire computing environment and then restored it using their backups. (Backups that had never been tested to this degree, BTW.) Paul VanDyke is the IT Supervisor at the Kodiak Island Borough in Alaska, which is the second largest island in the US and has to satisfy its backup and DR needs while staying on the island. Cloud resources are not a possibility due to bandwidth concerns, so he’s doing things “old school.” We first talk about the kinds of things they are protecting from, including tsunamis, fires, and strong winds. They are primarily based on tape, and for DR they store copies of all backups in a nearby safe. We discussed ways they could improve their resilience, such as shipping some tapes to a location on the mainland. But the highlight of this episode is the story of when Paul intentionally destroyed his entire environment and then tested his backup system! He learned many valuable lessons, starting with “don’t ever do that again!” Luckily, his test was successful, albeit not without some challenges. He wiped the storage arrays on five servers: two domain controllers, an email server, a file server, and an application server and then restored them. (He had his reasons for doing it this way, which he goes into in the podcast.) One big thing he learned was how restores are often slower than backups. So he prioritized critical apps (e.g. email, fileserver, logins) and got them up by Monday morning. Then it took him a few more days to get the application server up and running due to a more complicated restore. We have a really good discussion on how Paul could have done things better, including a really good idea that Prasanna came up with it. Curtis also tells a similar story about the first time he “tested” backups when he actually needed them, versus doing it in advance. We cover a number of topics and questions on this podcast: What was an Exabyte Mammoth (M2) tape drive? What is a helical scan tape drive? What is multiplexing? Why can restores be slower than backups? What happens when you rebuild a RAID array? Should you have a post-mortem after a large incident? How important is recovery testing? How important is it to set expectations in IT, especially when it comes to recovery times?
Dec 13, 2021 · 1 hr 4 min
ZFS filesystem in the cloud – just for your backups
The founder of rsync.net, John Kozubik, joins us on the podcast this week. It's a unique offering: a ZFS filesystem running in a private cloud – accessible only via SSH – that is designed just for sending your backup data to. They support anything that can run over SSH. Use rsync, scp, etc. to copy your data unencrypted, or something like restic, duplicity, or borg, if you want your backups to be encrypted. (All backups are encrypted in flight, of course, because they are all over SSH.). The servers are completely locked down except for the SSH port, so they're about as secure as they can be for what they are. You can configure ssh to behave the way you want it (e.g. passphrase, MFA, etc.), and the ZFS filesystem automatically creates daily snapshots of the backups you send there. (More complicated schedules can also be created.) You pay by the gigabyte ($.025/GB/mth) for the size of the ZFS filesystem and its associated snapshots, but they urge you to NOT over-provision. Provisioning is easy and non-disruptive, so only add storage when you need it. For an extra fee ($.017/GB/mth), they can also replicate your backups to another region. It's a no-nonsense offering that seems to be unique out there – especially when you add the ZFS features. Check out the website and rsync.net, and you'll see they aren't spending any money on being flashy. They just want to build a rock-solid ZFS syncing destination that is separate from any cloud provides.
Dec 6, 2021 · 46 min
Rclone creator Nick Craig-Wood Explains This Powerful Tool
This week, we talk to Nick Craigwood, the creator and principal developer of rclone, a very popular open-source tool for copying data to and from cloud providers. Rclone is downloaded roughly 250,000 times each month, and has over 30,000 stars on GitHub. There are six core developers, and a great community of users and other developers at rclone.org. We talk a little bit about Nick’s development philosophy, which is that he doesn’t mind adding features - as long as they don’t break backwards compatibility. Then we talk about how rclone works, and what it’s like to sync a filesystem to an object store – including support for multi-part uploads and downloads. We also talk about rclone’s encryption support, while Nick was “relaxing” on holiday. We then talked about how rclone can be used to minimize the risk of backing up to any one cloud provider, preventing things like what happened during the OVH fire earlier in 2021. We also discuss some strategies, such as backing up directly to two different clouds, versus backing up to one, then syncing to another – and how CloudFlare’s R2 might figure into things. Finally, we talk about Nick’s plans for rclone’s future, such as making their web UI better to increase usability for many more people – while not sacrificing the command line. Join us for a fascinating episode, the first one where we’re talking to the creator of the tool in question. Don’t forget the drawing for a free e-book version of Modern Data Protection. All you have to do to be eligible is sign up for my newsletter at https://www.backupcentral.com/subscribe-to-our-newsletter/
Nov 29, 2021 · 52 min
FujiFilm Tape Evangelist Explains Past, Present & Future of Tape/LTO
Fujifilm's tape evangelist, Rich Gadomski, joins us for an interesting discussion on tape and LTO. We talk about the different subtrates that have been used over the years, and how that changed things. We then talk about LTO-9 and what that brings to market. We also talk about how tape has seen a bit of a resurgence in interest in the backup market due to the advent of ransomware. Always fun to talk to someone that can talk at this depth on such things.
Nov 22, 2021 · 58 min
Peter Krogh, who coined "the 3-2-1 rule," on our podcast!
The term "3-2-1 rule" comes up on almost every episode, and we have the guy that coined it with us on the podcast! How exciting! Peter Krogh coined the term fifteen years ago. He is now Chief Product Officer at Tandem Vault, but this week he is talking to us. He first talks about how he coined the term “3-2-1 Rule” while writing the first edition of The DAM Book: Digital Asset Management for Photographers, now in it’s 3rd edition. He didn’t invent the idea of three copies and offsite backup, but he did distill it down to what we now refer to as the 3-2-1 rule. (Three copies on two media types, one of which is offsite.) We’ve played with it a bit over the years, but that is the core idea.) He explains how digital photographers were some of the first to need significant amounts of storage -- and to have the need to protect that storage so they don’t lose everything. Hard drives were too small to hold your whole collection, so what do you do? Like a lot of folks in this space, his love for good backups goes back to a moment when he thought he lost it all. Curtis then tells his very similar story of how his company almost lost the company’s purchasing database, which also launched his career in backups. Peter then explains the incredible importance of metadata, and the huge importance it plays in the overall value of an image. Then we get into the nitty-gritty of what the person who coined the term “3-2-1 rule” was thinking for each of the numbers. And interestingly enough, Mr. Backup had a slightly different understanding of the 2! Peter feels that the “2” refers to different media types. (This led to a very interesting discussion about how you do what he’s asking for in today’s cloud world.) One idea he talked about is that if you have two hard drives on the same network, they’re still subject to many of the same risks, which isn’t really keeping in line with the original idea of the 2. We then talk about those that believe that RAID is backup, and follow that with a discussion about how SaaS services aren’t backing up your data – unless they specifically say they do so in your contract. Then we get into a discussion of Peter’s company, Tandem Vault, and how they have designed the next generation of Digital Asset Management and delivered it as a SaaS offering.
Nov 15, 2021 · 55 min
Mr. Backup and Mr. SQL argue over how to backup SQL Server
Ever had questions about SQL Server, Azure, SQL Server ON Azure, how to backup SQL Server, or how to backup Azure? This is the episode for you. Denny Cherry, a SQL Server and Azure specialist and author of seven books, talks to us about both of these technologies. Before talking about anything important, we tackle the mystery of how you pronounce Azure. Surprise! I was pronouncing it wrong, according to Denny, who talks to Microsoft people all the time. We first talk about performance tuning, and Denny explains some things that most DBAs can do to improve performance, starting with indexes. (He also explains what an index is for those that don’t know.) We then talk about how bad query code needs to be in order to justify looking into that, and he gives us a few examples. We also (of course) talk about backing up SQL Server, starting with the political discussion of WHO should own the backup process: a backup admin or a DBA? Denny and Curtis clearly do not agree on this one, but the discussion is a good one. Grab your popcorn! One of Denny’s best quotes is that he feels one of the primary jobs of the DBA is to be able to restore the database if something happens and if you can’t do that, nothing else matters. So beautiful. Then the topic of dedupe comes up and things get heated again; our guest hates dedupe and Curtis loves it. That was another good discussion. Short version: make sure you have more than one copy of a deduped data store. We continue the discussion of different ways to backup SQL Server, and Denny definitely prefers the native backup capabilities of SQL Server, and he explains why. Curtis then makes a suggestion on a way for DBAs and backup admins to both get what they want, but it doesn’t sound like Denny is taking the bait. After a brief discussion on SQL Server vs Oracle, we move into the various ways one can use SQL Server in Azure. Denny’s gives advice as to what makes sense for most customers – and his opinion on the question of whether or not you save money in the cloud. Short answer: not usually, but you get a lot more power, flexibility and ease of use. Regarding Azure vs AWS, it appears that Azure is very equivalent to AWS in overall functionality at this point, and there appears to be a number of cost and functionality advantages to running SQL Server in the cloud. One of the biggest advantages is that you can use an on-prem license of SQL Server in the PaaS version of it in the cloud. That’s pretty cool. We also talk about how roughly half of the VMs in Azure run Linux, and why that might be the case. All-in-all it’s a really interesting podcast, even though we almost came to blows once or twice. (OK, not really.) But really good discussions about SQL Server, Azure, and backups of both.
Nov 8, 2021 · 54 min
Deep dive into why tape still has a future in storage
Mark Lantz, Manager CloudFPGA and Tape Technologies for IBM, joins us on this week’s podcast to talk about how he feels that tape still has a future in data storage. We talk about past and future advancements in the substrates tape uses, as well as how tape has not approached the superparamagnetic limit, the way we have with disk. (This is the limit at which you cannot increase the storage capacity of a particular magnetic medium without creating more problems.) We have reached this limit on disk, where the magnetic grains have gotten so small, they can’t get any smaller without assistance. One such method of assistance is heat-assisted magnetic recording (HAMR), which we discuss – and how HAMR comes with its own problems. By contrast, tape hasn’t come even close to the superparamagnetic limit. In fact, tape can scale the aerial density 100X before it starts getting close. We also discuss coercivity and bit error rate (BER), which are extremely important concepts to understand. Another topic we talk about is how tape is getting better at scaling capacity faster than speed, because most people do not need faster tapes. (We talk about how and why we can’t stream the ones they have.) We finish out the podcast with an explanation of why helican scan drives (e.g. 8mm, 4mm, & AIT) all disappeared overnight. We cover a lot of territory in this episode, so buckle up!
Nov 1, 2021 · 43 min
Transfer backups from one product to the next
Every wondered what you're supposed to do with all your old backups, now that you've moved on to another backup product? Simon Brown from StoneRam believes he has the answer to this problem that has plagued backup customers (and vendors) for ages. He's able to transfer backup data out of common backup formats and into your new product, or restore backups from your old product without having to maintain that infrastructure. It's a fascinating approach to this age-old problem. Check them out at https://www.stoneram.com.
Oct 25, 2021 · 49 min
Veeam reps explain defense against Conti ransomware
After the recent stories about Veeam customers being directly targeted by the Conti ransomware group, we invited Rick Vanover and Dave Russell from Veeam to discuss the topic on the podcast. The stories in the press seemed to focus on the attack, as well as how ruthless the Conti ransomware gang tends to be. We thought we'd give Veeam a chance to explain exactly what Veeam customers can do to protect their backups from being exfiltrated and deleted. It seems that Rick, Dave, and company are doing everything they can to explain to all Veeam customers that this is something they should pay attention to. The following are two resources they said should prove useful: Ransomware in 2022: 7 Capabilities You Need for Rapid and Reliable Recovery https://bit.ly/3m32gI8 5 Ransomware Protection Best Practices https://bit.ly/3nh7aAx
Oct 18, 2021 · 53 min
Stop Ransomware Attacks in Seconds
This week we are joined by Greg Edwards, CEO and founder of CryptoStopper, to discuss once again the important topic of ransomware. We talk about the challenges typically experienced by ransomware victims, especially exfiltration and potential exposure of sensitive data. The only way to stop that particular attack is stop the data from being exfiltrated in the first place. CryptoStopper has a way to detect that a ransomware attack has begun, but stopping it before it does any actual damage.
Oct 11, 2021 · 52 min
ZFS is not magic - it needs backup and RAID
This one will get you talking! Jody Bruchon, author of “ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot (and RAID-5),”, joins us on the podcast. The blog post went viral, resulting in three times as many words in the comments as the original article had. We start with an explanation of bit rot, why it happens, and why ZFS won’t be able to fix all bit rot. (For more information on bit rot, check out Episode 111 of this podcast here: https://www.backupcentral.com/tape-drive-designer-schools-mr-backup-on-tape-restore-it-all-podcast-111/). Jody then explains how ZFS needs disk redundancy in order for its self-healing features to work, and how if you don’t have that, you’re going to need backup to repair a ZFS volume damaged by bit rot. (We also talk about how it’s possible for a bit to be flipped without being noticed – even with ZFS.) Jody’s main concern is that people talk about how ZFS can be used to repair data corruption – without explaining how you need RAID-Z (or something) to use those features. He also explains why he prefers RAID-5 or RAID-10 to RAID-6. We then discuss “shucking,” the practice of buying external drives and ripping the drive out of them – to save money. Curtis (Mr. Backup) then gets into an argument with Jody about the merits of Blu-Ray vs disk vs tape as a backup medium. Jody has some good points, but Curtis was unconvinced. If you want to read Jody Bruchon’s original article, you can do so here: https://www.jodybruchon.com/2017/03/07/zfs-wont-save-you-fancy-filesystem-fanatics-need-to-get-a-clue-about-bit-rot-and-raid-5/
Oct 4, 2021 · 39 min
How good is your backup system, really?
Inspired by the article "How good is your backup, really?" by Sandra Vogel, we discuss how to evaluate and potentially redesign your backup system. We talk about different kinds of backup systems, and how that impacts how you evaluate them. We also talk about how important it is these days to ensure that your backups are impervious to ransomware. We also talk about the importance of including recovery testing in your evaluation, and what kinds of restores to test. Here's a link to the original article: https://www.itpro.com/server-storage/backup/357713/how-good-is-your-backup-really
Sep 27, 2021 · 49 min
Restoring quickly from a ransomare attack with a long dwell time
We talk to Celeste Kinswood from Druva about cyber resilience, and specifically ransomware attacks that have long dwell times – which is most of them. The median dwell time (the time between infection and you finding out you have ransomware) is 23 days, and the average is around 90. It's encrypting files during that entire time, and responding to that is beyond difficult. Celeste talks about a novel approach that Druva is taking to solve this growing challenge.
Sep 20, 2021 · 42 min
Protecting Your Network from Ransomware
We review and discuss Mark Dargin's Network World article "Credible threat: how to protect networks from ransomware." His article lays out several steps, each of which we discuss and expand upon: train your people, update your servers and apps, antivirus tools on endpointsm backup your data (of course), test your backups, and conduct vulnerability assessments (Pen tests). Read the original article here: https://www.networkworld.com/article/3218708/how-to-protect-your-network-from-ransomware-attacks.html
Sep 6, 2021 · 50 min
We found a company that verifies backups as a service!
We all know how important it is to verify your backups. But many companies simply lack the technical ability or time to do such a thing. Chris Marshall's company, VerifiedBackups.com to the rescue. They have a service that is aimed at companies with under 100 GB of SQL data. and will do an end-to-end verification of your SQL backups, guaranteeing that they are recoverable and safely stored offsite. I'm a fan of "as a service" anything, but this really takes the cake. Such an important thing to do, and he makes it happen.
Aug 30, 2021 · 1 hr
Is Entity-Level Encryption The Answer to Exfiltration Ransomware?
Brian Greenberg and Cameron Laghaeian argue the point that individually encrypting each entity in a database storing personal information is the only true way to stop ransomware attacks that have exfiltrated data and threaten to release it. While Mr. Backup likes the idea, there was a pretty heated discussion on this episode, because he believes that doing this will roll back all advancements in backup in the last twenty years. Thanks to an olive branch from Cameron, though, they may have come to a hybrid solution that makes both sides happy. This is a great episode with heated discussion and good news for companies trying to protect themselves from ransomware attacks that include exfiltration of data.