vSphere5: vmware-vpxd Crash

Posted on Posted in virtualización

Recently in my work, we have update the Vmware Infraestructure from Vsphere 4.1 to Vsphere 5, this update significate that the virtualCenter change (optional) from windows to a virtual linux appliance.

When we finished, a problem appeared, basically the vmware-vpxd service crash, and impact was that all the Vsphere Clients and tasks shutdown immediately.

The Problem

If we look /var/log/vmware/vpx/vpxd.log we find this errors:

2012-03-15T06:49:51.593+01:00 [7FFFF3B09700 error 'Default'] [Vdb::LockRepositoryHelper] SQLExecDirect failed: 57011:-438:ODBC error: (57011) - [IBM][CLI Driver][DB2/LINUXX8664] SQL0438N  Application raised error or warning with diagnostic text: "Error deleting from VPX_SESSIONL".  SQLSTATE=57011
-->
2012-03-15T06:49:51.593+01:00 [7FFFF3B09700 error 'Default'] Unable to get exclusive access to vCenter repository.                           Please check if another vCenter instance is                           running against the same database schema.
2012-03-15T06:49:51.598+01:00 [7FFFF3B09700 error 'Default'] [Init] Init failed: VpxdVdb::Init(VpxdVdb::GetVcVdbInstId(), false, false, NULL)
2012-03-15T06:49:51.598+01:00 [7FFFF3B09700 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

Hummm, the error is that a SQL transaction has not completed successfully Error deleting from VPX_SESSIONL thatis the “program” not release the session and the VirtualCenter Shutdown.

Fix one

The First solution is in case of emergency, if you try to start the service manually but it not start. To resolve it we will need to connect to the DataBase and liberate the session.

  • Obtain the session information to connect
# cat /etc/vmware-vpx/embedded_db.cfg
EMB_DB_INSTALL_DIR='/opt/db2/current'
EMB_DB_HOME='/opt/db2/home/'
EMB_DB_TYPE='db2'
EMB_DB_SERVER='127.0.0.1'
EMB_DB_PORT='50000'
EMB_DB_INSTANCE='VCDB'
EMB_DB_USER='vc'
EMB_DB_PASSWORD='PASSWORD'
  • Stop the Service
# /etc/init.d/vmware-vpxd stop
Stopping VMware vSphere Profile-Driven Storage Service...
Stopped VMware vSphere Profile-Driven Storage Service.
Stopping VMware Inventory Service...
Stopped VMware Inventory Service.
Stopping tomcat: success
Stopping vmware-vpxd: success
Shutting down ldap-server..done
  • Connect to DB2 DataBase
# /opt/db2/v9.7.2/bin/db2
(c) Copyright IBM Corporation 1993,2007
Command Line Processor for DB2 Client 9.7.2

db2 => connect to vcdb user vc using 'PASSWORD'
  • Find the VPX_SESSIONLOCK Table
db2 => list tables
...
VPX_SESSIONLOCK                 VC              T     2012-03-15-12.28.59.589441
...
287 record(s) selected.
  • Unlock the Session
db2 => delete from vpx_sessionlock
DB20000I  The SQL command completed successfully.
  • Exit & Restart VPXD
db2 => quit
# /etc/init.d/vmware-vpxd start
Waiting for embedded DB2 database to startup: .success Cleaning session lock table: success Verifying EULA acceptance: success Starting ldap-server..done Starting vmware-vpxd: success Waiting for vpxd to initialize: ....success Starting tomcat: success Executing startup scripts...
Starting VMware Inventory Service...Waiting for VMware Inventory Service........................
VMware Inventory Service started.

Starting VMware vSphere Profile-Driven Storage Service...Waiting for VMware vSphere Profile-Driven Storage Service......
VMware vSphere Profile-Driven Storage Service started.
Fix two

I applied this solution and the problem has not reoccurred. To resolve this, increase the transaction log size to the recommended size.

For example to 20 esx with 300-500 Virtual Machines this is a good size

logprimary 16

logsecond 112

logfilsiz 20480

  • Stop VPXD
# service vmware-vpxd stop
  • Chang User & Connect to DB2
# su db2inst1
db2inst1@vcenter00:~>db2 connect to vcdb
  • Retreive Current Configuration
db2inst1@vcenter00:~> db2 get database config for vcdb | grep LOG
Catalog cache size (4KB)              (CATALOGCACHE_SZ) = 300
Log buffer size (4KB)                        (LOGBUFSZ) = 256
Log file size (4KB)                         (LOGFILSIZ) = 1024
Number of primary log files                (LOGPRIMARY) = 13
Number of secondary log files               (LOGSECOND) = 4
Changed path to log files                  (NEWLOGPATH) =
Path to log files                                       = /storage/db/db2/home/db2inst1/db2inst1/NODE0000/SQL00001/SQLOGDIR/
Overflow log path                     (OVERFLOWLOGPATH) =
Mirror log path                         (MIRRORLOGPATH) =
Block log on disk full                (BLK_LOG_DSK_FUL) = NO
Block non logged operations            (BLOCKNONLOGGED) = NO
Percent max primary log space by transaction  (MAX_LOG) = 0
Num. of active log files for 1 active UOW(NUM_LOG_SPAN) = 0
Log retain for recovery enabled             (LOGRETAIN) = OFF
First log archive method                 (LOGARCHMETH1) = OFF
Options for logarchmeth1                  (LOGARCHOPT1) =
Second log archive method                (LOGARCHMETH2) = OFF
Options for logarchmeth2                  (LOGARCHOPT2) =
Log pages during index build            (LOGINDEXBUILD) = OFF
  • Update the log Configuration
db2inst1@vcenter00:~> db2 update db CFG FOR VCDB USING logprimary 16 logsecond 112 logfilsiz 20480
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.
SQL1363W  One or more of the parameters submitted for immediate modification
were not changed dynamically. For these configuration parameters, all
applications must disconnect from this database before the changes become
effective.
  • Confirm the new Configuration
db2inst1@vcenter00:~> db2 get database config for vcdb | grep LOG
Catalog cache size (4KB)              (CATALOGCACHE_SZ) = 300
Log buffer size (4KB)                        (LOGBUFSZ) = 256
Log file size (4KB)                         (LOGFILSIZ) = 20480
Number of primary log files                (LOGPRIMARY) = 16
Number of secondary log files               (LOGSECOND) = 112
Changed path to log files                  (NEWLOGPATH) =
Path to log files                                       = /storage/db/db2/home/db2inst1/db2inst1/NODE0000/SQL00001/SQLOGDIR/
Overflow log path                     (OVERFLOWLOGPATH) =
Mirror log path                         (MIRRORLOGPATH) =
Block log on disk full                (BLK_LOG_DSK_FUL) = NO
Block non logged operations            (BLOCKNONLOGGED) = NO
Percent max primary log space by transaction  (MAX_LOG) = 0
Num. of active log files for 1 active UOW(NUM_LOG_SPAN) = 0
Log retain for recovery enabled             (LOGRETAIN) = OFF
First log archive method                 (LOGARCHMETH1) = OFF
Options for logarchmeth1                  (LOGARCHOPT1) =
Second log archive method                (LOGARCHMETH2) = OFF
Options for logarchmeth2                  (LOGARCHOPT2) =
Log pages during index build            (LOGINDEXBUILD) = OFF
  • Exit & Start VPXD
db2inst1@vcenter00:~> exit
# service vmware-vpxd start

Leave a Reply

Your email address will not be published. Required fields are marked *