Linux Disaster Recovery Made Easy

Schlomo SchapiroSystems Architect / Open Source EvangelistImmobilien Scout GmbHschlomo.schapiro@immobilienscout24.de

Jeff LindholmTechnology Sales Specialist – LinuxNovell, Inc.jlindholm@novell.com

© Novell, Inc. All rights reserved.2

Agenda

Linux and disaster recovery

Solutions: relax and recover

Secure disaster recovery for enterprise environments

SUSE® Linux Enterprise Server 11 SP1 HA – ReaR availability and support

Live demo and use cases

Questions and answers

Linux and Disaster Recovery

© Novell, Inc. All rights reserved.4

Disaster Recovery Strategies

Disk Imaging

+ Offline -> no open file issues

+ Simple to restore– Online -> very

problematic– No “perfect” open

source tool for Linux available

Copy Files and Store Disk Layout and Boot Info

+ Online -> no problem+ Backup independent of disk

layout and sizes– More effort required to

restore (can be scripted)– Maybe consistency

problems, but should be solved by backup solution

© Novell, Inc. All rights reserved.5

Advantages of a Linux System

All information is stored in files; all files always readable

Operation system and applications can be “slim” - 600MB enough for complete standard server

Open system – open methods and procedures

All steps of an installation can be scripts: partitioning, file systems, boot loader etc.

© Novell, Inc. All rights reserved.6

Disaster Recovery – Media

• Most important: external storage!• Bootable media: CD/DVD, USB key, LAN, tape ...• Media usually combination boot and backup media:

– Bootable CD/DVD, USB key with backup data on it

– LAN boot (PXE) with backup data via CIFS, NFS ...

– Bootable tapes - HP OBDR (CD emulation)

• Separation between boot media and backup data– Boot the system from a (small) USB key, CD/DVD or LAN

– Recover the system with backup software, tar, rsync ...

© Novell, Inc. All rights reserved.7

Disaster Recovery – How It Works

• Store the disk layout– Partitioning, LVM and RAID configuration

– File systems, file system labels ...

– Boot loader (GRUB, LILO, ELILO)

• Store the files (tgz, rsync, through backup software ...)• Create bootable rescue media with system

configuration (and backup data)• Can be done online

– No business interruption

– 100% compatible with original systems hard- and software

© Novell, Inc. All rights reserved.8

Disaster Recovery – Rescue Media

• Create “rescue linux” from running system

• Optimally compatible “tool box”

• Clone the system environment

– Linux kernel and modules

– Device driver configuration

– Network configuration

– Basic system software and tools

• Operate entirely in RAM (initrd)

© Novell, Inc. All rights reserved.9

Disaster Recovery – In Action

• Boot system from rescue media• Restore disk layout

– Create partitions, RAID configuration and LVM

– Create file systems (mkfs, mkswap)

– Configure file systems (labels, mount points)

• Restore the backup data• Restore the boot loader• Reboot• Done!

Solutions: Relax and Recover

© Novell, Inc. All rights reserved.11

Commercial Solutions

• Usually part of backup software• Extra licenses and can be costly (up to 1000€ / server)• Some are free, but are the any good?• Level of integration into backup software varies• Not all backup software actually supports

disaster recovery• Cover only simple setups• Often static boot media, inflexible with regard to drivers• Certification on server hardware missing or lagging

behind procurement cycles

© Novell, Inc. All rights reserved.12

Open Source Solutions

• Several software projects on the internet:– Relax and recover

– mkCDrec

– Mondo Rescue

– CRU

– BACULA

– Your-Favorite-Selfmade-Script

– <something you know but I never heard about>

• In 2006, all of them where lacking something...

© Novell, Inc. All rights reserved.13

Relax and Recover – ReaR

http://rear.sourceforge.net• GPL Software – Developers in Germany and Belgium• 100% Bash script – no GUI and no dependencies• Utilize kernel, modules, binaries of host (kernel ≥ 2.6)• Support any combination of SW/HW RAID, LVM• Internal backup on CIFS, NFS ...• Boot media on CD/DVD, USB key and LAN (PXE)

• Successor of mkCDrec

© Novell, Inc. All rights reserved.14

ReaR – Features

• Focus on disaster recovery and not backup• Tight integration with common backup software –

delegate file backup to backup infrastructure• Simple full backup integrated• Complements backup software:

– Backup software: Data storage and retrieval

– ReaR: Recover system layout and make it work again

– ReaR utilizes the backup software to restore the backup data

• Use the best tool for the job

© Novell, Inc. All rights reserved.15

ReaR – Backup Software

• Supported solutions include:– CommVault Galaxy

– IBM Tivoli Storage Manager

– Veritas NetBackup

– HP Data Protector

– Rsync and other “external” methods

– tar.gz archive on NAS share – CIFS, NFS, NCP ...

• Very transparent integration• Quick implementation: 2-3 PT programming• Can be easily extended to support other vendors

© Novell, Inc. All rights reserved.16

ReaR – Network Integration

• Disaster recovery as part of network infrastructure– Backup software – file-level backup storage– ReaR – system environment– Boot rescue media via PXE – no physical media required– Very scalable – automated installation of entire disaster recovery

data center

BackupStorageNetwork

infrastructure

SoftwareBackup

ReaRBoot Files

PXE Boot

© Novell, Inc. All rights reserved.17

ReaR – Status

• Stable software– i386 and x86_64 well tested

– ia64 and ppc experimental

• Regular releases (RPM, DEB, TGZ)• Major Linux distributions ship ReaR:

– SUSE® Linux Enterprise High Availability Extension 11 SP1

– openSUSE® 11.2 and Fedora 11

• Community and commercial support available• Regular patch submissions from ReaR users

© Novell, Inc. All rights reserved.18

ReaR - Development

• Open Source development model:– Submit patches and feedback – “field testing”– Sponsoring

• Modular design:– ReaR is a framework to plug together many small bash scripts– Maximize code reusability– Simple development model (vi works fine)– Little to no “interferences” between different areas of code

• Documentation on project homepage– Hierarchical structure– References to source code

© Novell, Inc. All rights reserved.19

ReaR – Configuration

• Shell-script style configuration file• Usually: Set backup software and boot media

System definition: ARCH = Linux-x86_64 OS = GNU/Linux OS_VENDOR = SUSE_LINUX OS_VERSION = 11 OS_VENDOR_ARCH = SUSE_LINUX/x86_64 OS_VENDOR_VERSION = SUSE_LINUX/11 OS_VENDOR_VERSION_ARCH = SUSE_LINUX/11/x86_64Configuration tree: Linux-x86_64.conf : OK GNU/Linux.conf : OK SUSE_LINUX.conf : missing/empty SUSE_LINUX/x86_64.conf : missing/empty SUSE_LINUX/11.conf : missing/empty SUSE_LINUX/11/x86_64.conf : missing/empty site.conf : OK local.conf : OK

© Novell, Inc. All rights reserved.20

ReaR – Operation

rear [Options] <command> [command options ...]Relax & Recover Version 1.7.23.2 / 2010-03-19Build: b80d32a97e41a6ec8a5b19cd6399e444Copyright (C) 2006-2009 Schlomo Schapiro Gratien D'haese, IT3 ConsultantsRelax & Recover comes with ABSOLUTELY NO WARRANTY; for details see the GNU General Public License at http://www.gnu.org/licenses/gpl.html

Available Options:-V version information-d debug mode-D debugscript mode-S Step-by-step mode-s Simulation mode (shows the scripts included)-q Quiet mode-r a.b.c-xx-yy kernel version to use (current: 2.6.32.8-0.3-default)

© Novell, Inc. All rights reserved.21

ReaR – Operation (continued)

List of commands:dump Dump configuration and system informationhelp print out usagemkbackup Create rescue media and backup system.mkbackuponly Backup system without creating a (new) rescue media.mkdeb Create DEB packages with this rear versionmkdist Create distribution tar archive with this rear versionmkrescue Create rescue media onlymkrpm Create RPM packages with this rear versionmktar Create tar archive with this rear installationmkvendorrpm Create vendor RPM with this rear versionrecover Recover the systemvalidate Submit validation information

© Novell, Inc. All rights reserved.22

ReaR – Example Configuration

# cat /etc/rear/local.conf

OUTPUT=ISO

#BACKUP=TSM|NBU|DP|GALAXY|EXTERNAL|...

BACKUP=NETFS

NETFS_URL=nfs://server.domain/backup-path

MODULES_LOAD=( vmxnet )

© Novell, Inc. All rights reserved.23

ReaR – Example Execution

# rear mkbackup

Relax & Recover Version 1.7.23.2 / 2010-03-19

The preparation phase OK

Physical devices that will be recovered: /dev/sda /dev/sdb

Creating root FS layout OK

Copy files and directories OK

Copy program files & libraries OK

Copy kernel modules OK

Create initramfs OK

Creating archive 'nfs://server.domain/backup-path/rear/backup.tar.gz'

Transferred 297 MB in 254 seconds [1197 KB/sec]

Making ISO image OK

Wrote ISO Image /tmp/ReaR.iso (19M)

Copying resulting files to network location OK

The cleanup phase OK

Finished in 258 seconds.

© Novell, Inc. All rights reserved.24

ReaR – Disk Layout

# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/system-root

2,5G 927M 1,4G 40% /

devtmpfs 244M 128K 244M 1% /dev

tmpfs 247M 0 247M 0% /dev/shm

/dev/sda1 84M 19M 61M 24% /boot

/dev/mapper/system-home

524M 17M 481M 4% /home

/dev/md0 680M 55M 591M 9% /var

© Novell, Inc. All rights reserved.25

ReaR – Disk Layout

# pvs PV VG Fmt Attr PSize PFree /dev/sda3 system lvm2 a- 3,21G 0 /dev/sdb2 system lvm2 a- 312,00M 0

# lvs LV VG Attr LSize Origin Snap% home system -wi-ao 532,00M root system -wi-ao 2,44G swap system -wi-ao 572,00M

# cat /proc/mdstat Personalities : [linear] [raid1] md0 : active raid1 sda2[0] sdb1[1] 706816 blocks super 1.0 [2/2] [UU]

© Novell, Inc. All rights reserved.26

ReaR – Recovery Example

# rear recoverRelax & Recover Version 1.7.23.2 / 2010-03-19Backup archive size is 298MInitializing physical devices OKCreating Software RAID devices OKCreating the LVM2 devices OKCreating file systems OKRecreated this filesystem layout:Filesystem Size Used Avail Use% Mounted on/dev/mapper/system-root 2.5G 68M 2.3G 3% /mnt/local/dev/sda1 84M 5.6M 74M 7% /mnt/local/boot/dev/mapper/system-home 524M 17M 481M 4% /mnt/local/home/dev/md0 680M 17M 629M 3% /mnt/local/varRestoring from 'nfs://laptop/dev/shm/nfs/rear/backup.tar.gz' OK

© Novell, Inc. All rights reserved.27

ReaR – Recovery Example (continued)

Restoring the bootloader (SuSE style)Creating swap files and partitions OKFinished recovering your system

You can explore it under /mnt/local

Thank you for using Relax & Recover Version 1.7.23.2

Finished in 150 seconds.

© Novell, Inc. All rights reserved.28

ReaR – Vision

• Do more than just disaster recovery– System cloning and provisioning– System migration (P2V, V2P, V2V, P2P)– Generic system tool-kit for offline tasks and rescue media

• Centralized information and management:– rear-server– 1.0: Gather and aggregate information about ReaR

> System protection status

> Rescue media boot files

– 2.0: Centrally manage configuration and control ReaR> Site-wide and system-local configuration store

> Trigger and control disaster recovery tasks

– 3.0: Integrate into 3rd party solutions

Secure Disaster RecoveryEnterprise Environments

© Novell, Inc. All rights reserved.30

Enterprise DR – Requirements

StandardizationScalabilityAutomationIntegration in existing system environmentSecurityCoverage over all systems in useSupport and maintenanceEducation and certificationAffordable licensing – site license

© Novell, Inc. All rights reserved.31

Enterprise DR – Relax and Recover

• Standardization through global configuration:– /etc/rear/site.conf

• Scalability and automation:– rear mkrescue can be run as a cron job

– Automated processing of rescue boot media fairly simple

– Rescue boot media also stored with backup software

– LAN boot – extreme scalability and automated DR possible

• Integration in existing system environment– Support internal and external backup solutions

– E-mail reports

© Novell, Inc. All rights reserved.32

Enterprise DR – Relax and Recover

• Security considerations– No “secret” data in rescue media– Secure export of rescue media through backup software

and E-mail– Remote access to rescue system only with SSH keys

• Coverage– i386 and x86_64 stable– Enterprise Linux (Novell®, Red Hat, Ubuntu) stable– Community distributions also work, less tested

• Maintenance and support, education– Please contact Novell and the authors

• Site license: Included with GPL

© Novell, Inc. All rights reserved.33

Security Aspects – Disaster Recovery

Transfer of Backup Data• How secure is the

backup software?• Insecure: NFS2, NFS3• Secure (with

authentication): – NFS4, CIFS– SSH (rsync)

Transfer of Boot Riles• Insecure (anonymous):

– NFS– CIFS– SSH (keys)– RCP

• Manipulation proof:– SMTP

DR client

Backup server DR boot server

© Novell, Inc. All rights reserved.34

Enterprise Linux Disaster Recovery

• Linux – Simple Disaster Recovery

• Integration: Disaster recovery – backup software

• Automated creation of disaster recovery information

• Network boot into rescue system

• Disaster recovery becomes part of infrastructure

• Scales well even for very large data centers

• Disaster recovery becomes a valid option for clients, not only for servers

SUSE® Linux Enterprise High Availability Extension 11 Service Pack 11 –

ReaR Availability and Support

© Novell, Inc. All rights reserved.36

SUSE® Linux EnterpriseModular Portfolio

SUSE Linux Enterprise

Custom OS

DeveloperTools

Bundles

SUSE Linux Enterprise

ServerJeOS SLE SDK SLE POS

SLES for SAP,Expanded Support

SLED

Server OS

DesktopOS

SupportSKUs

High Availability Real Time Extension Mono Extension

Optional extensions for high availability clustering, deterministic computing, and running .NET applications

Pay only forwhat you need

Common CodeBase foundation

Level 3 Support Runs on x86, x86_64, Itanium,

Power, and s390x

© Novell, Inc. All rights reserved.37

SUSE® Linux Enterprise High Availability Extension 11 Key Features

• Flexible, policy driven clustering solution

– OpenAIS messaging and membership layer

– Pacemaker cluster resource manager

• Cluster-aware file system and volume manager

– Oracle Cluster File System (OCFS2)

– Clustered Logical Volume Manager (cLVM2)

• Bare Metal Recovery (ReaR)

• Continuous data replication

– Distributed Replicated Block Device (DRBD8)

• GUI and CLI tools– Graphical user interface– Unified command line interface

• IP load balancing

• Resource agents

• Virtualization aware

© Novell, Inc. All rights reserved.38

SUSE® Linux Enterprise High Availability Extension 11Differentiators

• AffordableCosts less than other Linux clustering solutions, with tools and resource agents included at no extra charge

• Virtualization-awareSupports hybrid virtual and physical clusters

• Open SourceNo proprietary modules or user space applications means your Linux clustering solution is secure and easily kept up to date

• IntegratedStreamlined support and an optimized solution from a single vendor that delivers both the OS and clustering software

© Novell, Inc. All rights reserved.39

SUSE® Linux Enterprise High Availability Extension 11 New Features/Enhancements SP1

ReaR – Relax and Recover• ReaR will be included with SUSE Linux Entreprise

Server 11 SP1 HA Add-on• ReaR is considered to be an extensible framework, allowing

for custom scripting and extension of the recovery toolset• ReaR RPM packages will be fully supported (L1, L2, L3)• SLE11 High Availability Add-on inherits service level

agreement from the SLES Subscription– Priority (24x7)– Standard (12x5)

• Customization is not covered by Novell® Technical Support– Integration Partners are available (contact Novell Sales)

•

© Novell, Inc. All rights reserved.40

SUSE® Linux Enterprise High Availability Extension 11 More Information - SP1

For more information on what is new with SLE11 HA SP1:

ELS307 Building High Availability Clusters with SUSE Linux Enterprise High Availability Extension

ELS302 SUSE Linux Enterprise High Availability Extension 11: Support and Troubleshooting

ATT307 Quickly Configure a High Availability Cluster with SUSE Linux Enterprise 11 High Availability Extension

Live Demo/Use Cases

© Novell, Inc. All rights reserved.42

Relax and Recover – Live Demo

Demo (~10 min):• SUSE® Linux

Enterprise High Availability Extension 11 SP1

• ReaR creates backup and boot CD

• “PEBKAC” at work• Boot system from CD• Recover system• Happy End

Questions and Answers

Unpublished Work of Novell, Inc. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.