Wednesday, September 17, 2014

Linux bare metal Backup and Recovery with Storix

There are several reasons not to read this post but I am rather hoping that you will if

  1. Computers are your business
  2. If you have an interest in Backup and Recovery
  3. If your home or work computer Operating System disk crashed irrecoverably and without a backup you would have to start from scratch again, and that this would be a considerable inconvenience

In the Trade
Bare metal recovery means a restore of your computers Operating System onto a potential virgin/ new disk, assuming the worst i.e. the original disk is completely destroyed or unreadable.

This is a given and a must for any business, and really for any competent computer professional.

What about home users?

If like Marcus and Agata though your home computer installations are infrequent, but your installation has been built up over time, with hundreds or even thousands of program installs, tweaks, upgrades, optimisations then

Simply re-installing your computer from scratch is not a sensible option!

Windows, OSX and Linux
For Microsoft Windows, Microsoft already has a full bare metal backup, free product embedded into Windows.  In practice I did not find this 100% bulletproof so I'd always recommend the chargeable Acronis products instead.

For OSX I'd recommend Carbon Copy Cloner.

And for Linux?

Linux Enthusiasts
Many Linux advocates/ enthusiasts are used to re-installing their computer system, because only the latest Linux have the ability to upgrade!  Centos 7 for example can be an upgrade from Centos 6.5

But really, assuming you have a credible, complex installation, then you would of course expect a bare metal recovery.

So if you are still with me, we'll just consider Linux now ...

Some Bare Metal options

a) You are running under a Hypervisor like ESX or VMware.

So your Linux installation really consists of a set of .vmdk files on a filesystem.  So you could power off or suspend the Virtual Machine  (for consistency), then copy or snapshot those raw files, then resume or power on your Linux.

Now move those backup files to longterm storage etc.

If you are running a Storage Array then the snapshot and storage can be handled in a space and time efficient manner by the Array storage itself etc.

b) Primative UNIX tools or applications to image copy 

UNIX programs like dd or standalone partition tools like CloneZilla   are able to make a copy of your powered off systems disk to another disk or image file.

Then to recover you simply reverse the process.

For Enterprise Linux of course the poweroff might not be possible (with any frequency due to availability requirements) and even for home Linux this offline option may not advisable: you are bound to get lazy or forget.

c) Online Backup Products

Acronis has something - I did not yet test it

IBM - Used to have Bare Metal for Windows VM, for Physical Linux, no idea, but think not!

Storix - Well yes of course

Storix Bare Metal
I first used Storix backup software about 10 years ago, usually in an AIX environment.

I was excited to hear that they also have a Linux product portfolio and wanted to see if it could really perform a simple bare metal backup, and restore.

And I mean crucially an online backup, so that you don't have to take your system down and insert a /image cloning DVD/ to get the Operating System saved.

Note: I did not get paid by Storix to write this review, in fact they did not know that I was writing it.

My Test and Step By Step Instructions
For my tests I used a VMware 10 Centos 7 installation that I use on a daily basis.  It's my normal local /development/ Linux environment.  

[ I am experimenting with a local development system and a Production partner system in the Cloud hosted by IBM Softlayer.  But that as they say, is another story!]

  • Windows Server 2012 Base
  • VMware 10 installed
  • 100GB Centos 7 partition installed under VMware
  • I made a VMware snapshot before doing anything, so that in the event of a disaster I can recover

  • Inside the running Centos 7 system
  • Register with and download the 64bit Linux product, workstation edition
  • Under VMware add a 48GB second disk that will be my backup target for Storix

Visibility Note!
Is it obvious to say that under any blogspot page such as the one you are reading: you can click on the graphic and it loads a larger image display which can be scrolled using the left and right cursor keys.  Just in case you can't read any of the following pictures clearly. 

Installing Storix

  • Untar the .tar file as root into a temporary directory and run sbinstall

Configuring the Backup Disk and Backup Job

 Centos 7 is installed onto /dev/sda  (100 GB)

I also created a target disk /dev/sdb of 50GB to hold the OS backup

 We configured the target disk /dev/sdb

# fdisk -l /dev/sdb

Disk /dev/sdb: 51.5 GB, 51539607552 bytes, 100663296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00045c59

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1      204800      102400   83  Linux
/dev/sdb2          204801   100663295    50229247+  83  Linux

You can see that Storix makes 2 partitions.  sdb1 will be a partition that you will boot into and the backup data is stored in sdb2

Make a trivial Backup Job

You need to create at least 1 /fullsystem/ backup job. Normally you would create it under a schedule but for this trivial test we'll just make it an adhoc job, then run it

Now Run this Backup Job

What happens if I update my Linux?
The recovery program is stored in /dev/sdb1

But suppose you update your Linux kernel then you need to update /dev/sdb1 like this

Recovery Testing

Well of course this is the important and interesting part.

Backups are useless if you don't test and retest your recovery.   Let me just state that again, in case you are stupid:

You need to test your recovery from an actual backup periodically.  Even if the backup looks like it has worked.

Making a Clone

As a paranoid android, I did not want to potentially screw up my original system, so I created an identical copy of it.

The original setup is in V:\development\centos7

The clone is in V:\clone\centos7

VMware actually as a tool to do this.  In the olden days I'd copy the files manually and edit the control files, but it seems I don't have to anymore ...

I went for a coffee,  I came back 10 minutes later and the 150GB copy had been made.  Great.

Working On the Copy

First I need to edit the .vmx file and add a BIOS boot delay into the config file, otherwise there is not enough time to press the F2 key to enter the bios before it boots

.encoding = "windows-1252"
bios.bootdelay = 4000
config.version = "8"
virtualHW.version = "10"

Power on the system, hit F2 to enter BIOS and change boot device to second disk i.e. SCSI 1, in linux terms /dev/sdb

 Wahoo! On reboot /dev/sdb1 is booted from and 

 I get this wonderful text menu

 Choose the backup to restore from

 Where will I restore to

Although the underlying disk contains Partitions actually Storix is going to make the correct ones again.

I restored at a measley 50MB/second, still I only just had time for a short cofee before all completed. I typed exit and the system rebooted.

 The black screen seemed to take a long time.   Would it be okay?

Of course!  All is well.  System is restored.

To prove that all was well I checked that the /home/mbennett/sema file was missing.  I created this on the source system after making the backup.  Yes, it is missing!  This means the restore had worked from scratch, i.e. it did not overlay any target partitions.  It wiped the physical disk clean, created partitions, then restored from the backup in /dev/sdb

Destructive Test 2

In the first test the OS was restored onto a disk that was actually already in a working condition.  I mean it had a working set of Centos 7 XFS partitions.  Too Easy.

In this more severe test I'll start with a BLANK 100GB disk with no Linux partitions and see what happens

First I remove the 100GB Operating System Partition completely

Recreate the 100GB disk with a new .vmdk name just to be sure

Power Up

Boot failure. Good!   The Bootlist is set to boot to disk 0, but it is a total blank, so the system tries to Boot from the next device which is the network.

So, now as before, restart the machine, enter bios and point to the second SCSI disk which in Linux terms is the /dev/sdb

The master Boot record is read and the active partition /dev/sdb1 loaded containing the Storix recovery program

 I chose the same options as before

 Here is the critical screen where you see that the partitions are created because Storix in the backup process made a note of what is where.  So now before the restore it can create the partitions back again just as they were. Remember this is a blank disk with nothing on it.

After this the restore completes OKay

Enter the BIOS again and change the Boot disk to SCSI 0 i.e /dev/sda in Linux terms

 The grub screen after restore

And of course it boots, I just ran the update program in this screenshot.

SO, all is well.  I am very happy.

One More thing

I used the sbadmin program to do all of this work but these days there is a web interface

Some programs are started at boot time to support that

Besides OS backup you can create a backup job to backup your data as well of course.  Normally all these jobs are run using at the times and dates you want via the Storix Backup scheduler which you configure using the sbadmin or Web GUI.

If you have a non trivial computer installation you need at the very least a backup of your Operating System and your Data.

In the Linux world Bare Metal Linux online backup, and recovery is not so well supported.   This brief experiment showed that Storix worked well for me on my Centos 7 system for online OS backup and offline, bare metal recovery.


Acronis Linux Backup (not tested)