Vmware Stonith for Linux Cluster

Posted on Posted in linux, Mysql, virtualización

In this new post I’ll splain how install and configure Vmware Stonith for a new Linux Cluster.

For all the people that don’t know what’s Stonith, this is the explanation by Wikipedia:

STONITH (“Shoot The Other Node In The Head” or “Shoot The Offending Node In The Head”), sometimes called STOMITH (“Shoot The Other Member/Machine In The Head”), is a technique for fencing in computer clusters.

Fencing is the isolation of a failed node so that it does not cause disruption to a cluster. As its name suggests, STONITH fences failed nodes by sending a message to remotely power down the failed node.

My environment:

  • RHEL 5.5 x64
  • PaceMaker (install from yum)
    pacemaker-libs-1.0.12-1.el5.centos
    pacemaker-1.0.12-1.el5.centos
  • CoroSync (install from yum)
    corosynclib-1.2.7-1.1.el5
    corosync-1.2.7-1.1.el5
  • HeartBeat (install from yum)
    heartbeat-libs-3.0.3-2.3.el5
    heartbeat-3.0.3-2.3.el5
  • Stonith
    Reusable-Cluster-Components-glue–glue-1.0.9.tar.bz2 (download)
    VMware-vSphere-Perl-SDK-5.0.0-615831.x86_64.tar.gz (download)
  • Perl
    perl-5.14.2.tar.gz (download)

Installation

All this task have to be performed on both nodes

Perl-5.14.2

First we need perl 5.10 or plus, in my case I’ll install perl-5.14.2

Download

[root@node1 etc]# wget http://www.cpan.org/src/5.0/perl-5.14.2.tar.gz

And extract the software

[root@node1 etc]# tar xvfz perl-5.14.2.tar.gz && cd perl-5.14.2

Perl Installation

Configure and Install Perl, accept the defaults options.

[root@node1 perl-5.14.2]# ./Configure
......
....
...
What pager is used on your system? [/usr/bin/less -R] /usr/bin/less
...
.
[root@node1 perl-5.14.2]# make && make install

Remplace the previous perl version binary files

[root@node1 perl-5.14.2]# mv /usr/local/bin/perl /usr/local/bin/perl5.8
[root@node1 perl-5.14.2]# cp -pr /usr/local/bin/perl5.14.2 /usr/local/bin/perl
[root@node1 perl-5.14.2]# cp -pr /usr/local/bin/perl5.14.2 /usr/bin/perl

cluster-glue

The official yum version for Cluster-glue are cluster-glue-1.0.6-1.6.el5, but this version not supported to do Stonith attack directly to the VirtualCenter.

Why I need attack directly directly to the VirtualCenter?, is simple, in my Vmware Infraestructure the DRS is activated, thats means that Virtual Machines which form part the Cluster are in differents ESX, and I don’t know what ESX. To solved this, I need communicate with the VirtualCenter saying that Virtual Machine I need to kill, and the VirtualCenter is the responsible for search and kill. Ok now the installation:

Prerequisites

We’ll need some packages extra

[root@node1 etc]# yum install glib2-devel.x86_64 bzip2-devel.x86_64

Download Cluster-Glue

[root@node1 etc]#wget http://hg.linux-ha.org/glue/archive/glue-1.0.9.tar.bz2

And extract it.

[root@node1 etc]#tar xvfz glue-1.0.9.tar.bz2

Installation

[root@node1 ]# cd Reusable-Cluster-Components-glue--glue-1.0.9
[root@node1 Reusable-Cluster-Components-glue--glue-1.0.9]# ./autogen.sh

If you have installed heartbeat and corosync previously with rpm’s, and this is a update installation, configure with the option --localstatedir to specify the statedir. Because in the rpm installation the statedir is /var and the src package take /usr/var

[root@node1 Reusable-Cluster-Components-glue--glue-1.0.9]# ./configure

Or

[root@node1 Reusable-Cluster-Components-glue--glue-1.0.9]# ./configure --localstatedir=/var
[root@node1 Reusable-Cluster-Components-glue--glue-1.0.9]# make && make install

VMware-vSphere-Perl-SDK

This is the software that will provide the opportunity to communicate with the VirtualCenter.

Download

Download from the official Vmware Web.
And Extract it.

[root@node1 etc]# tar xvfz VMware-vSphere-Perl-SDK-5.0.0-615831.x86_64.tar.gz && cd vmware-vsphere-cli-distrib

VMware-vSphere-Perl Installation

Export the proxy variable if you need it to connect with Internet

export http_proxy=http://proxy.snaider.es:8080
export ftp_proxy=http://proxy.snaider.es:8080

Ok, now install the software:

[root@node1 vmware-vsphere-cli-distrib]#./vmware-install.pl 
A previous installation of vSphere CLI has been detected.

The previous installation was made by the tar installer (version 4).

Keeping the tar4 installer database format.

You have a version of vSphere CLI installed.  Continuing will remove it in 
preparation for installing a new vSphere CLI.  Do you want to continue?
[yes] 

Uninstalling the tar installation of vSphere CLI.

The removal of vSphere CLI 5.0.0 build-615831 for Linux completed successfully.
......
....
..agreements. Any waiver of these terms must be in writing to be effective. If
any provision of these 
terms is found to be invalid or unenforceable, the remaining terms will
continue to be valid and 
enforceable to the fullest extent permitted by law.

rev10.24.08

Do you accept? (yes/no) yes

Please wait while configuring CPAN ...

Please wait while configuring perl modules using CPAN ...

CPAN is downloading and installing pre-requisite Perl module "Archive::Zip" .

CPAN is downloading and installing pre-requisite Perl module "Crypt::SSLeay" .

CPAN is downloading and installing pre-requisite Perl module 
"Class::MethodMaker" .

CPAN is downloading and installing pre-requisite Perl module "HTML::Parser" .

CPAN is downloading and installing pre-requisite Perl module "UUID" .

CPAN is downloading and installing pre-requisite Perl module "Data::Dump" .

CPAN is downloading and installing pre-requisite Perl module "SOAP::Lite" .

CPAN is downloading and installing pre-requisite Perl module "URI" .

CPAN is downloading and installing pre-requisite Perl module "LWP" .

CPAN is downloading and installing pre-requisite Perl module 
"LWP::Protocol::https" .

In which directory do you want to install the executable files? 
[/usr/bin] 

Please wait while copying vSphere CLI files...

The installation of vSphere CLI 5.0.0 build-615831 for Linux completed 
successfully. You can decide to remove this software from your system at any 
time by invoking the following command: 
"/usr/bin/vmware-uninstall-vSphere-CLI.pl".

This installer has successfully installed both vSphere CLI and the vSphere SDK 
for Perl.

Enjoy,

--the VMware team

Stonith Configuration

Vcenter Credentials

For security reasons I created a configuration file with the encrypted credentials user, who have the grants to control my Virtual Machines.

[root@node1 vmware-vsphere-cli-distrib]# /usr/lib/vmware-vcli/apps/general/credstore_admin.pl add -s vcenter.snaider.es -u user -p password
[root@node1 vmware-vsphere-cli-distrib]# cp -pr /root/.vmware/credstore/vicredentials.xml /etc/

Test the tools

Before continue I’ll check if the Vmware Tools have the possibility to reset one Virtual Machine.

[root@node1 ~]# VI_SERVER=vcenter.snaider.es VI_CREDSTORE=/etc/vicredentials.xml /usr/lib64/stonith/plugins/external/vmcontrol --operation reset --vmname node2

Server version unavailable at 'https://vcenter.snaider.es:443/sdk/vimService.wsdl' at /usr/lib/perl5/site_perl/5.8.8/VMware/VICommon.pm l

Upps what happend?, this problem is easy to solved.
For security reasons Perl don’t accept insecure certificates, to solved it edit /usr/lib/perl5/site_perl/5.8.8/VMware/VICommon.pm and add this option $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
It must be like this:

#
# Copyright 2006 VMware, Inc.  All rights reserved.
#

use 5.006001;
use strict;
use warnings;

use Carp qw(confess croak);
use XML::LibXML;
use LWP::UserAgent;
use LWP::ConnCache;
use HTTP::Request;
use HTTP::Headers;
use HTTP::Response;
use HTTP::Cookies;
use Data::Dumper;

$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; ...
..

Ok, now check again:

[root@node1 ~]# /usr/lib64/stonith/plugins/external/vmcontrol --operation suspend --vmname node2 --datacenter vcenter.snaider.es --credstore /etc/vicredentials.xml

virtual machine 'node2' under host esx3.snaider.es reset

GREAT!!!, the Vmware Tool Work

Stonith Inegration

To add Stonith like CRM resource (cluster Resource Module) we’ll need integrate the Vmware tool with stonith software. To do this.

[root@node1 ]# stonith -t external/vcenter VI_SERVER="vcenter.snaider.es" VI_PORTNUMBER="443" VI_PROTOCOL="https" VI_SERVICEPATH="/sdk/webService" VI_CREDSTORE="/etc/vicredentials.xml" HOSTLIST="hostname1=node1;hostname2=node2" RESETPOWERON="1" -lS
info: external/vcenter device OK.
hostname2
hostname1

Ok this means that the stonith software run with Vmware Tools. Now check if the reset options is available.

[root@node1 ]# stonith -t external/vcenter VI_SERVER="vcenter.snaider.es" VI_PORTNUMBER="443" VI_PROTOCOL="https" VI_SERVICEPATH="/sdk/webService" VI_CREDSTORE="/etc/vicredentials.xml" HOSTLIST="hostname1=node1;hostname2=node2" RESETPOWERON="1" -T reset hostname1
external/vcenter[3191]: info: Machine esx4.snaider.es node1 has been reset

Stonith reset option is running, now configure with crm.

CRM Configuration

For example this is a posible configuration for CRM and Stonith:

crm configure primitive vfencing stonith::external/vcenter params \ 
  VI_SERVER="vcenter.snaider.es" VI_CREDSTORE="/etc/vicredentials.xml" \ 
  HOSTLIST="hostname1=node1;hostname2=node2" RESETPOWERON="1" \ 
  op monitor interval="60s" 

crm configure clone Fencing vfencing 

crm configure property stonith-enabled="true"

Well, I hope you enjoyed with this article.
Thanks for following

3 thoughts on “Vmware Stonith for Linux Cluster

Leave a Reply

Your email address will not be published. Required fields are marked *