Xen Clustering
-
Upload
mohamed-faye -
Category
Documents
-
view
237 -
download
0
Transcript of Xen Clustering
-
8/3/2019 Xen Clustering
1/34
XEN Cluster HowTo
I have tried to run both Debian Etch and Ubuntu 8.04 Server on the cluster
nodes, in Dom0. I started my tests with Debian, but I had some issues with
slow samba performance in one VM that I couldn't fix so I decided to try
Ubuntu Server, for the first time. Both installation went OK, the main
difference was that I used mainly source code in Debian, but only packages in
Ubuntu. I actually ran into more problems with Ubuntu due to some early
bugs in the 8.04 release, will describe them below as I go along.
2011-02-27: The setup in Ubuntu was very stable and I had initially very few
problems. I ran into some trouble later on when it turned out that my
heartbeat in combination with XEN was negatively effected by some system
updates. Never had time to resolve it so I eventually switch off heartbeat as it
was just causing issues.
Instead I have started with a new install consisting of Debian Squeeze and
XEN version 4. I'm migrating the data upgrading one node at the time.
Documenting the migration phase and will publish it together with a fresh
install guide.
Installation How to
Index:
Linux - Ubuntu 8.04 Server
Ubuntu - Base configuration
Installing XEN
Configure LVM
Configure XEN-tools
Using XEN-tools to install DomU
Install and configure DRBD
Configure DomU to use your DRBD device
Configure DomU on your other Dom0 node
Configure Live Migration
-
8/3/2019 Xen Clustering
2/34
Install and Configure Heartbeat
Bug in default xendomains script
Troubleshooting
Linux - Ubuntu 8.04 Server
I am not going to go through each and every step of the Ubuntu installation
as that is really straight forward. I downloaded the 8.04 Server version from
http://www.ubuntu.com/getubuntu/download and started the installation one
on of the nodes.
During disk partitioning, select manual mode and delete any existing
partitions (if there are any).
Create one partition for your main system and make it 5GB. This partition will
be used for your Dom0 and 5GB should be more than enough.
Create a second partition for your swap for Dom0. Make this 512MB as we
later will configure Dom0 to use 512MB of RAM.
Create a third partition with all your remaining space.
At the packages selection step do not select any packages. Leave everything
deselected.
A few minutes later you should have a basic installation of Ubuntu 8.04
Server running.
Ubuntu - Base configuration
A few things need to be done to the freshly installed Ubuntu Server before we
start installing XEN.
Most of the commands need to be executed with super user rights, so we
start by typing
http://www.ubuntu.com/getubuntu/downloadhttp://www.ubuntu.com/getubuntu/download -
8/3/2019 Xen Clustering
3/34
sudo su
I installed SSH early on so I didn't need to run everything locally on theconsole
apt-get install ssh
Configure network interfaces for static IPs:
nano /etc/networking/interfaces
and configure it so it looks something like this:
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet static
address 10.1.1.100
netmask 255.255.255.0
network 10.1.1.0
broadcast 10.1.1.255
gateway 10.1.1.1
# Secondary interface used between the two clusters
-
8/3/2019 Xen Clustering
4/34
auto eth1
iface eth1 inet static
address 192.168.1.100
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
This setup will use eth0 to connect to my network and eth1 as the link
between the two nodes. For node ha2 use ip address 10.1.1.101 and
192.168.1.101.
Edit /etc/hosts and add both cluster nodes
nano /etc/hosts
my file looks like this:
127.0.0.1 localhost
127.0.1.1 ha1.domain.local ha1
10.1.1.100 ha1.domain.local ha1
10.1.1.101 ha2.domain.local ha2
192.168.1.100 ha1X
192.168.1.101 ha2X
for node ha2 change the second line to 127.0.1.1 ha2.domain.local ha2
Edit the /etc/hostname file to match your Fully Qualified Domain Name
-
8/3/2019 Xen Clustering
5/34
(FQDN)
nano /etc/hostname
ha1.domain.local
and for node ha2 change this to: ha2.domain.local
Reboot machine to confirm everything is working
reboot
You should now be able to connect to your static IP address and your
hostname should be your FQDN. Run:
hostname
hostname -f
and they should both return your FQDN, like this:
root@ha1:/# hostname
ha1.domain.local
root@ha1:/# hostname -f
ha1.domain.local
Install NTP to have your time synchronized. This is important for your VMs,
especially when migrating a VM to the other node:
-
8/3/2019 Xen Clustering
6/34
apt-get install ntp
Now we are done with the basic configuration of the system and can move onto the fun part, to install Xen, DRBD & Heartbeat
Installing XEN
I will use only packages during this installation, and installing XEN on Ubuntu
is then a quick and hassle free process. I will only cover the important parts
and make comments where needed, if you want to look at a more
comprehensive guide for installing XEN on Ubuntu then please check this
link: http://howtoforge.com/ubuntu-8.04-server-install-xen-from-ubuntu-repositories
apt-get install ubuntu-xen-server
Now you should have Xen installed.
Even though you might not use loop devices in your XEN setup it can be a
good idea to extend the number of allowed loop devices so you don't run into
trouble later if you plan to use them. Edit /etc/module and modify the line
with "loop" as below:
nano /etc/modules
loop max_loop=64
Now it is time to reboot your system so it will boot with the new xen kernel:
reboot
http://howtoforge.com/ubuntu-8.04-server-install-xen-from-ubuntu-repositorieshttp://howtoforge.com/ubuntu-8.04-server-install-xen-from-ubuntu-repositorieshttp://howtoforge.com/ubuntu-8.04-server-install-xen-from-ubuntu-repositorieshttp://howtoforge.com/ubuntu-8.04-server-install-xen-from-ubuntu-repositories -
8/3/2019 Xen Clustering
7/34
After reboot, run:
uname -r
to confirm your system is using the new xen kernel. It should look like this:
root@ha1:/# uname -r
2.6.24-19-xen
Configure LVM
To install LVM run:
apt-get install lvm2
We will now configure a Volume Group of the third partition created during
installation. On my system this is /dev/sda3. You can run:
fdisk -l
to confirm this.
Do the following to create a volume group called "vg":
pvcreate /dev/sda3
vgcreate vg /dev/sda3
-
8/3/2019 Xen Clustering
8/34
To check the status of the volume group, run:
vgdisplay
and then reboot:
reboot
Within this volume group we will later create logical volumes that will be useby our VMs. You can create the logical volumes manually when installing a
new VM, but below I will use xen-tools which will do all that for us.
Configure XEN-tools
If you have a cluster with two machines like me it is not necessary to
configure xen-tools on both of them. You can decide that you always will use
one when adding a new DomU. For flexibility I have my both machines
configured the same way so I can run xen-tools on both.
Xen-tools is installed automatically as part of the Xen installation above. So
now we just need to configure it. The configuration for xen-tools is stored
in /etc/xen-tools/xen-tools.conf
Run:
nano /etc/xen-tools/xen-tools.conf
and perform the following changes in the xen-tools.conf file:
-
8/3/2019 Xen Clustering
9/34
We first uncomment the line with LVM and specify the volume group we
created above:
lvm = vg
Configure the "Disk and Sizing options" section. Mine looks like this at the
moment:
size = 5Gb # Disk image size.
memory = 384Mb # Memory size
swap = 384Mb # Swap size
# noswap = 1 # Don't use swap at all for the new system.
fs = ext3 # use the EXT3 filesystem for the disk image.
dist = hardy # Default distribution to install.
image = sparse # Specify sparse vs. full disk images.
the disk size and the swap size will be used to create Logical Volumes (LVs) inyour VG specified above.
Next we edit the "Networking" section. Mine looks like this:
gateway = 10.1.1.1
netmask = 255.255.255.0
broadcast = 10.1.1.255
we will specify the IP address of the VM and its hostname when creating the
VM with xen-tools.
-
8/3/2019 Xen Clustering
10/34
Uncomment the line with passwd to always be asked for the password for
your VM. Should look like:
passwd = 1
As I have a 64bit AMD CPU I set the following value for my architecture:
arch=amd64
Next and last we change the mirrors used when installing Debian and
Ubuntu. This is how mine looks like using mirrors in The Netherlands:
# The default mirror for debootstrap to install Debian-derived distributions
#
mirror = http://ftp.nl.debian.org/debian/
#
# A mirror suitable for use when installing the Dapper release of Ubuntu.
#
mirror = http://nl.archive.ubuntu.com/ubuntu/
#
# If you like you could use per-distribution mirrors, which will
# be more useful if you're working in an environment where you want
# to regularly use multiple distributions:
#
mirror_sid=http://ftp.nl.debian.org/debian
http://ftp.nl.debian.org/debian/http://nl.archive.ubuntu.com/ubuntu/http://ftp.nl.debian.org/debian/http://nl.archive.ubuntu.com/ubuntu/ -
8/3/2019 Xen Clustering
11/34
mirror_sarge=http://ftp.nl.debian.org/debian
mirror_etch=http://ftp.nl.debian.org/debian
# mirror_dapper=http://archive.ubuntu.com/ubuntu
# mirror_edgy=http://archive.ubuntu.com/ubuntu
# mirror_feisty=http://archive.ubuntu.com/ubuntu
# mirror_gutsy=http://archive.ubuntu.com/ubuntu
for Debian I had to enable the per-distribution mirrors as shown above.
Otherwise installing sid, sarge or etch would fail.
Now we are done configuring /etc/xen-tools/xen-tools.conf
The default setting in xen-tools.conf is using the debootstrap installation
method. For installing Fedora and CentOS you need to use rinse. If you want
to try that later on you should install rinse by running:
apt-get install rinse
Now we are ready to use xen-tools!
Using XEN-tools to install DomU
Perform this step on only one of your machines, either HA1 or HA2, to create
your first DomU.
Time to install our first DomU. The application we use for this part of Xen-
tools is called: xen-create-image. The only two parameters we supply when
using xen-create-image is the IP address and hostname of the DomU. All
other settings will be taken from xen-tools.conf. But you can override all
these settings at command line. For example, to change the distribution you
would write --dist=etch to install Debian etch instead of Ubuntu Hardy.
-
8/3/2019 Xen Clustering
12/34
Now, run the following to create a DomU with IP: 10.1.1.50 and with
hostname: test:
xen-create-image --hostname=test --ip=10.1.1.50
After a while the process hopefully completed successfully and your DomU is
ready. For me it took about 5 minutes to complete.
xen-create-image has now automatically created two LVs in your VG. If you
used the hostname "test", you will have one LV called "test-disk" and another
one called "test-swap". Full path to these are: "/dev/vg/test-disk" and
"/dev/vg/test-swap"
Run:
lvdisplay
and you will see the details of your two LVs.
You should now be able to start your DomU called: test. The config files for
your DomUs are stored in /etc/xen/. We first change to that folder:
root@ha1:/# cd /etc/xen
In /etc/xen/ you will find test.cfg which is the config file for your newly
created DomU.
Start the DomU:
-
8/3/2019 Xen Clustering
13/34
root@ha1:/etc/xen# xm create test.cfg
you can also add -c to the line above, that will automatically bring you to theconsole of the DomU. If you didn't, we can check the status of the running
DomU with:
xm list
To change to the console of the running DomU, run:
xm console test
if everything went fine you will now see the prompt to login to your DomU
test. To leave console mode in your DomU you need to press Ctrl and ]:
Ctrl + ]
Perfect! You now have your first virtual machine running.
NEXT
We will soon go ahead and configure DRBD and Heartbeat to allow for live
migration and high availability. This is the fun part! But before we do that we
need to duplicate the LV disk setup on the other machine.
So on HA2 we need to create one LV with 5GB and another one with 384MB.
Run:
-
8/3/2019 Xen Clustering
14/34
root@ha2:/# lvcreate -L 5G -n test-disk vg
to create the "test-disk" with 5GB of space. We use the same name here forsimplicity, but the names doesn't need to match as we will specify that later
in the DRBD configuration.
Run:
root@ha2:/# lvcreate -L 384M -n test-swap vg
to create "test-swap" with 384MB of space.
Now we have the same disk setup on both machines.
Install and configure DRBD
To install DRBD run:
apt-get install drbd8-utils
The configuration file for DRBD is located in /etc/drbb.conf. We will now
configure it to use the LVs we create above and later we will change the Xen
configuration of our test DomU to use the DRBD device instead of directly the
LV. Edit /etc/drbd.conf:
root@ha1:/# nano /etc/drbd.conf
-
8/3/2019 Xen Clustering
15/34
Below is a copy of my settings with all default comments removed from the
file:
global {
usage-count yes;
}
common {
syncer { rate 90M; }
}
resource test-disk {
protocol C;
startup {
wfc-timeout 120; ## 2 min
degr-wfc-timeout 120; ## 2 minutes.
}
disk {
on-io-error detach;
}
net {
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
timeout 60;
-
8/3/2019 Xen Clustering
16/34
connect-int 10;
ping-int 10;
max-buffers 2048;
max-epoch-size 2048;
}
syncer {
}
on ha1.domain.local {
address 192.168.1.1:7789;
device /dev/drbd1;
disk /dev/vg/test-disk;
meta-disk /dev/vg/meta[0];
}
on ha2.domain.local {
address 192.168.1.2:7789;
device /dev/drbd1;
disk /dev/vg/test-disk;
meta-disk /dev/vg/meta[0];
}
}
resource test-swap {
protocol C;
-
8/3/2019 Xen Clustering
17/34
startup {
wfc-timeout 120; ## 2 min
degr-wfc-timeout 120; ## 2 minutes.
}
disk {
on-io-error detach;
}
net {
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
timeout 60;
connect-int 10;
ping-int 10;
max-buffers 2048;
max-epoch-size 2048;
}
syncer {
}
on ha1.domain.local {
address 192.168.1.1:7790;
device /dev/drbd2;
disk /dev/vg/test-swap;
meta-disk /dev/vg/meta[1];
-
8/3/2019 Xen Clustering
18/34
}
on ha2.domain.local {
address 192.168.1.2:7790;
device /dev/drbd2;
disk /dev/vg/test-swap;
meta-disk /dev/vg/meta[1];
}
}
I have given the DRBD resource the same name as its corresponding LV. SoDRBD resource: test-disk is using LV: test-disk.
Next we will create a separate volume where we store DRBD's meta data.
Meta data is used by DRBD to store information about the device. This can
either be internal or external. Internal mode is easier to setup for a new
devices but requires resizing operations when using an already formatted
device. Read further about this on: http://www.drbd.org/users-guide/ch-
internals.html
As we already have data on our LVs, created by XEN tools, we will use
external meta data. So we create another LV with 1GB of space:
lvcreate -L 1G -n meta vg
If drbd.conf, above, you will see that we specify the line with meta-disk as
using /dev/vg/meta[0] and /dev/vg/meta[1]. Same device can be used to
store meta-data for several DRBD resources, that is done with adding [X]
after the device name.
To initiate the meta data, run:
http://www.drbd.org/users-guide/ch-internals.htmlhttp://www.drbd.org/users-guide/ch-internals.htmlhttp://www.drbd.org/users-guide/ch-internals.htmlhttp://www.drbd.org/users-guide/ch-internals.html -
8/3/2019 Xen Clustering
19/34
drbdadm create-md test-disk
drbdadm create-md test-swap
Redo the DRBD configuration above for the other node if you haven't done
already.
Now we will startup DRBD on both nodes. Run:
/etc/init.d/drbd start
To check status of DRBD use the following commands:
/etc/init.d/drbd status
cat /proc/drbd
Before the replication of data will begin we have to make one node the
primary node for each DRBD resource. Run the following on the node you
installed your DomU above to replicate the data to the other node (Do NOT
run it on the other node):
drbdsetup /dev/drbd1 primary -o
drbdsetup /dev/drbd2 primary -o
Check the status again to see that the replication started:
/etc/init.d/drbd status
cat /proc/drbd
-
8/3/2019 Xen Clustering
20/34
If everything is OK after data is replicated you should see something like this
for each DRBD device when running /etc/init.d/drbd status. It should state
Primary/Secondary and UpToDate/UpToDate:
/etc/init.d/drbd status
1:test-disk Connected Primary/Secondary UpToDate/UpToDate C
2:test-swap Connected Primary/Secondary UpToDate/UpToDate C
Now we are done setting up the DRBD device for our LV. Next is to configureor DomU to use DRBD.
Configure DomU to use your DRBD device
The configuration files for your DomUs are stored in /etc/xen/. So we first
change to that directory:
cd /etc/xen
In here you have your test.cfg file. Edit it by running:
/etc/xen# nano test.cfg
You will find a section that looks like below:
disk = [
-
8/3/2019 Xen Clustering
21/34
'phy:/dev/vg/test-swap,xvda1,w',
'phy:/dev/vg/test-disk,xvda2,w',
]
Edit this section so it looks like this:
disk = [
'drbd:test-swap,xvda1,w',
'drbd:test-disk,xvda2,w',
]
Done! That is all, now your DomU is ready to be started using the DRBD
device.
If your DomU is still running we first have to stop it. Run:
xm shutdown test
Try to start your DomU again:
xm create test.cfg -c
Hopefully everything went fine. Smile Login and then shutdown your DomU:
shutdown -h now
When you are back to your Dom0 prompt check the DRBD status again:
-
8/3/2019 Xen Clustering
22/34
/etc/init.d/drbd status
You will now see the status of the DRBD devices as Secondary/Secondary:
1:test-disk Connected Secondary/Secondary UpToDate/UpToDate C
2:test-swap Connected Secondary/Secondary UpToDate/UpToDate C
The good thing with this is that Xen takes care of your DRBD devices and will
automatically bring them up and down as needed. Same goes when you start
a DomU, Xen will first make sure the DRBD devices are in primary modebefore starting the DomU.
Configure DomU on your other Dom0 node
Now we need to prepare the other Dom0 node to be able to run the DomU
called test. This is simply done by copying the /etc/xen/test.cfg from ha1 toha2. Copy the whole file or the content of the file, what ever is easier for you.
When the file is copied we will try to start the DomU on ha2.
First make sure that DomU test is not running on ha1:
root@ha1:/# xm list
If it is running the stop it:
root@ha1:/# xm shutdown test
-
8/3/2019 Xen Clustering
23/34
Then go to ha2 and start it:
root@ha2:/# xm create test.cfg
If everything is fine you should see DomU test running. Check with xm list:
root@ha2:/# xm list
Perfect! You can now start the DomU on both nodes. (Note: Don't try to start
the DomU on both nodes at the same time.)
Next we will start with the real cool things!
We will go ahead and configure live migration so we can move a DomU
between the two Dom0 nodes with hardly any downtime. Cool
Configure Live Migration
By default XEN does not allow live migration, we have to enable this is
/etc/xen/xend-config.sxp. Make sure the following line is commented, it
should look like this:
#(xend-relocation-hosts-allow '^localhost$ localhost\\.localdomain$')
and that the following line is not commented, it should look like this:
-
8/3/2019 Xen Clustering
24/34
(xend-relocation-port 8002)
Restart xend, reload has no effect, but restart will not kill running DomUs.
Run:
/etc/init.d/xend restart
Make sure to do this two changes above on both nodes, both ha1 & ha2.
NOW, lets try a live migration!
If everything is in order your DomU called test should be running on your ha2
node. Please confirm this with xm list:
root@ha2:/# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 500 2 r----- 10268.0
test 7 384 1 -b---- 4694.7
To migrate a DomU we use the xm migrate command. Run:
root@ha2:/# xm migrate test ha1x --live
So first we write xm migrate. Then the name of the DomU we want to
migrate, in this case test. Then the hostname or IP of the other node, in this
case ha1x (remember that we specified ha1x and ha2x in the hosts file on
both nodes and mapped it to the IP of the interface with the cross over
connection between ha1 and ha2). And we end the command with --live, that
-
8/3/2019 Xen Clustering
25/34
will instruct xen to do a live migration.
If everything went fine your DomU test should be running on ha1. Run:
root@ha1:/# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 500 2 r----- 10260.1
test 11 384 1 -b---- 2.2
Try the migration a few times between the two nodes. Try accessing the
DomU during a live migration, ping it from another machine in your network,
open a SSH session and see how incredible fast it migrates. I have not timed
it myself, but running a normal ping from a Windows machine I lose no
packets or maximum 1 packet during a migration. Smile
Install and Configure Heartbeat
We will use Heartbeat, part of the Linux High Availability project
(http://www.linux-ha.org/), to monitor our XEN resources and provide failover
between our two nodes. I currently use version 2 of hearbeat but utilizing
versions 1's configuration files, so that is what I will describe below. Read
more in the link above about the difference in configuration files.
To install Heartbeat we will use package in the Ubuntu repository. Run this on
both nodes:
apt-get install heartbeat
http://www.linux-ha.org/http://www.linux-ha.org/ -
8/3/2019 Xen Clustering
26/34
Configuration files for Heartbeat is stored in /etc/ha.d/. We need to configure
the following files: authkeys, haresources and ha.cf.
We will start with configuring authkeys and ha.cf as they are the easiest to
explain.
Authkeys is used to configure authentication between the cluster nodes.
Configure it to look something like this:
root@ha1:/etc/ha.d# nano authkeys
auth 1
1 sha1 SecretKey123!!!
This will tell heartbeat to use the method sha1 with the supplied key. Note:
Make sure to copy the exact same copy to your ha2 node.
Lets continue with ha.cf. This file contains the configuration for heartbeat:
nodes in the cluster, how they communicate and timer settings. What I will
use is the version 1 configuration format. Configure the file to look something
like this:
root@ha1:/etc/ha.d# nano ha.cf
logfacility local0
udpport 694
keepalive 1
deadtime 10
warntime 3
initdead 20
-
8/3/2019 Xen Clustering
27/34
ucast eth0 10.1.1.100
ucast eth0 10.1.1.101
auto_failback on
watchdog /dev/watchdog
debugfile /var/log/ha-debug
node ha1.domain.local
node ha2.domain.local
The two lines above that begin with "ucast eth0 " configures the heartbeat
communication. The reason I have put both node's IP addresses is so the file
can be identical on both nodes, heartbeat will ignore the IP of the localmachine so this is perfectly fine. Note: Make sure to copy the exact same
copy to your ha2 node.
Now we will continue with haresources file. The file itself is actually very easy
but we need to setup the resources used by heartbeat which requires some
explanation. We begin with the file, configure it to look like this:
root@ha1:/etc/ha.d# nano haresources
ha1.domain.local xendomainsHA1
ha2.domain.local xendomainsHA2
Looks very simple, doesn't it? Smile What this means is that the
resource(script that will control our DomUs) xendomainsHA1 will default to
node HA1 and xendomainsHA2 to node HA2. These two scripts are copies of
the /etc/init.d/xendomains script and modified for our two node cluster. We
need to do this to be able to differentiate between DomUs on HA1 resp. HA2.
First we copy the /etc/init.d/xendomains twice to create xendomainsHA1 and
xendomainsHA2:
-
8/3/2019 Xen Clustering
28/34
root@ha1:/# cp /etc/init.d/xendomains /etc/ha.d/resource.d/xendomainsHA1
root@ha1:/# cp /etc/init.d/xendomains /etc/ha.d/resource.d/xendomainsHA2
Now we edit both files to change two lines so it looks like below:
root@ha1:/# nano /etc/ha.d/resource.d/xendomainsHA1
LOCKFILE=/var/lock/xendomainsHA1
XENDOM_CONFIG=/etc/default/xendomainsHA1
root@ha1:/# nano /etc/ha.d/resource.d/xendomainsHA2
LOCKFILE=/var/lock/xendomainsHA2
XENDOM_CONFIG=/etc/default/xendomainsHA2
Please make sure that all the Heartbeat configuration above is exactly the
same on node HA2. Below the configuration will differ slightly.
As you noticed above we specified different configuration files for the
resources. There is a default configuration file already located in /etc/default,
called xendomains. We will copy it as below:
root@ha1:/# cp /etc/default/xendomains /etc/default/xendomainsHA1
We copy only xendomainsHA1 to begin with. We will modify it and then later
use it for xendomainsHA2.
-
8/3/2019 Xen Clustering
29/34
root@ha1:/# nano /etc/default/xendomainsHA1
XENDOMAINS_MIGRATE="ha2X --live"
This will allow for live migration to HA2 when node is shutdown.
XENDOMAINS_SAVE=
Disable save feature.
XENDOMAINS_SHUTDOWN_ALL=
Disable this to prevent ALL DomUs to be shutdown even them not
controlled by this script
XENDOMAINS_RESTORE=false
Disable as we don't save DomUs
XENDOMAINS_AUTO=/etc/xen/auto/HA1
Point to location for DomU configuration files that will be controlled by this
script
XENDOMAINS_AUTO_ONLY=true
Only DomUs started via config files in XENDOMAINS_AUTO will be managed
-
8/3/2019 Xen Clustering
30/34
Now we copy xendomainsHA1 to create xendomainsHA2:
root@ha1:/# cp /etc/default/xendomainsHA1 /etc/default/xendomainsHA2
And we modify xendomainsHA2 to point to correct folder for DomU
configuration files:
root@ha1:/# nano /etc/default/xendomainsHA2
XENDOMAINS_AUTO=/etc/xen/auto/HA2
Now we can copy /etc/default/xendomainsHA1 and
/etc/default/xendomainsHA2 to node HA2. Do that by any means you want,
either file transfer or copy and paste.
On HA2 we need to modify the settings for live migration:
root@ha2:/# nano /etc/default/xendomainsHA1
XENDOMAINS_MIGRATE="ha1X --live"
root@ha2:/# nano /etc/default/xendomainsHA2
XENDOMAINS_MIGRATE="ha1X --live"
Next we need to create the two folders /etc/xen/auto/HA1 and
/etc/xen/auto/HA2 as referred above. Do this on both node HA1 & HA2.
-
8/3/2019 Xen Clustering
31/34
mkdir /etc/xen/auto/HA1
mkdir /etc/xen/auto/HA2
Create a symlink on both nodes in /etc/xen/auto/HA1 pointing to our test.cfg
file in /etc/xen/
ln -s /etc/xen/test.cfg /etc/xen/auto/HA1/test
Whenever creating a new DomU you need to decide if you want it by default
to run on HA1 or HA2, the location of the symlink will decide that.
Remove the default xendomains script to start automatically, heartbeat will
now control this for us. Do this on both nodes:
update-rc.d -f xendomains remove
Shutdown DomU test if it is running:
xm shutdown test
Start heartbeat manually on both nodes:
/etc/init.d/heartbeat start
Hopefully everything is fine. Try to reboot one node at a time to see that
DomU test is migrated between the two nodes.
Bug in default xendomains script
-
8/3/2019 Xen Clustering
32/34
There is a bug in the default xendomains script that breaks the script if there
are more than one domain in the AUTO directory. This is only true for non
standard configurations like mine above when
XENDOMAINS_AUTO_ONLY=true in /etc/defaults/xendomains. By default thissetting is false hence I guess no one picked up on it. Been too lazy to create a
patch for it but I will try to get around and do that.
Solution:
The rdnames() function now looks like this:
rdnames()
{
NAMES=
if ! contains_something "$XENDOMAINS_AUTO"
then
return
fi
for dom in $XENDOMAINS_AUTO/*; do
rdname $dom
if test -z "$NAMES"; then
NAMES=$NM;
else
NAMES="$NAMES $NM";
fi
done
}
-
8/3/2019 Xen Clustering
33/34
And the beginning of the stop() function look likes this:
stop()
{
# Collect list of domains to shut down
if test "$XENDOMAINS_AUTO_ONLY" = "true"; then
rdnames
fi
echo -n "Shutting down Xen domains:"
while read LN; do
parseln "$LN"
if test "$id" = "0"; then continue; fi
echo -n " $name"
found="0"
if test "$XENDOMAINS_AUTO_ONLY" = "true"; then
for i in ${NAMES[@]}
do
if test $found="0"; then
if test $i = $name; then
found=1
fi
fi
done
if test $found = "0"; then
echo -n "(skip)"
-
8/3/2019 Xen Clustering
34/34
continue
fi
fi
# XENDOMAINS_SYSRQ chould be something like just "s"
This script shall now work for more than one DomU. Smile
Troubleshooting
Here is a string I use in my Dom0 to check relevant logs of the system, xen
and heartbeat. Keep this running in a separate shell when testing migration
and heartbeat failover:
tail -f /var/log/syslog /var/log/xen/xend.log /var/log/xen/xend-debug.log
/var/log/ha-debug