5.1.1   server-component (vmware-monitord)

• gawk >=3.1.3 (various gawk-specific functionalities are used, e.g. TCP/IP networking, fflush(), ...). By default GAWK is expected to be accessible at /bin/gawk
• Xymon/Hobbit server:

• xymon 4.2.3: without any patches
• xymon-4.3.0_beta2: with "splitncv-patch" or better (the "splitncv-patch" is included in the 4.3.0 branch of Xymon since 2009-11-14)
• hobbit-4.2.0: due to missing SPLITNCV-functionality the included alternative graphing-definitions have to be used

In short: a working "user"-channel and "splitncv"-module are needed on the Xymon server. Gentoo ebuilds for Xymon can be found in ./contrib/gentoo/.

• server-component needs to be installed on the Xymon server

5.1.2   client-component (vmware-monitorc)

• vMA 4.0 from VMware or equivalent tools (namely utility-programs from vmware-vSphere-CLI-4.0.0-161974), see [1]
• hobbit-client-4.2.0 or better. vmware-monitorc only uses the bb(1) command to send data to the Xymon server via the "user"-channel -- this works fine with a vanilla hobbit-4.2.0. As for Vmware-Monitor for Xymon (VMX) it is not necessary to upgrade the client to Xymon. Suitable RPMs can be found at [2]
• the bash-shell of course
• in particular you do NOT NEED the following:

• vCenter server
• additional database system(s)
• Windows Server license

5.2.1   data-processing

• copy server/bin/vmware-monitord to ~/server/bin/ on your Xymon server and check if the shebang (#!/bin/gawk) fits your path to gawk and adjust if needed. The default of /bin/gawk should work on almost any Linux distribution.

• enable starting of vmware-monitord with the default 5 minute test-interval by appending the following lines to ~/server/etc/hobbitlaunch.cfg (paths may need to be adjusted to fit your setup):

  [vmware-monitord]
          ENVFILE /usr/lib/xymon/server/etc/hobbitserver.cfg
          NEEDS hobbitd
          CMD hobbitd_channel --channel=user --log=$BBSERVERLOGS/vmware-monitord.log vmware-monitord
  

  (vmx-hobbitlaunch.cfg is included to be easily appended to your config)

• if you have set "--status-senders=IP[/MASK][,IP/MASK]" for "hobbitd" in hobbitlaunch.cfg you may need to add your processing-hosts IP

• configuration-file server/etc/vmware-monitor.cfg: 2 options:

  1. If you want to prevent downloading of the configuration for security reasons (it contains passwords and the Xymon-connection is not encrypted by default) then copy the configuration file vmware-monitor.cfg to ~/server/etc/. This "download-prevention" can be used for the bb-hosts-file too, see VMX configuration with vmware-monitor.cfg.
  2. If you want to use the central-configuration, i.e. all configuration done on the Xymon server then copy vmware-monitor.cfg to $HOBBITCLIENTHOME/etc/ on the vMA.

  Details about needed adjustments are given in the section VMX configuration with vmware-monitor.cfg.

• add the "vmware"-test to your VMware hosts in ~/server/etc/bb-hosts.

1.2.3.4 esx1.local      # [ other tests ] vmware [ other tests ]
1.2.3.5 esx2.local      # [ other tests ] vmware [ other tests ]

• add your VMs with their "displaynames" from the VMware environment to bb-hosts.

  If your VMs displaynames are not identical with the hostnames of the VMs then you could use CLIENT:<displayname> in your bb-hosts for the VM or (really not recommended!) add a dummy-entry with "0.0.0.0" and "noconn".

192.168.1.1    vma.local       # CLIENT:"vMA-ovf-4.0.0-161993"

5.2.2   graphing

• copy server/etc/hobbitgraph.d/vmware-monitor-graph.cfg to your hobbitgraph.d-directory if you use directory <path-to-hobbitgraph.d> in your ~/server/etc/hobbitgraph.cfg. CAUTION: after the directory-statement there is only one space allowed!

  OR:

  append the contents of server/etc/hobbitgraph.d/vmware-monitor-graph.cfg to ~/server/etc/hobbitgraph.cfg

• adjust ~/server/etc/hobbitserver.cfg as follows: using cut-n-paste from the included server/etc/vmx-hobbitserver.cfg is probably the savest way as this documentation is aimed at printing and web-display; take special care with line continuation characters!

1. append to variable TEST2RRD:

   ,vmwareO=ncv,vmware=ncv,vm_cpu=ncv,vm_mem=ncv
   

2. insert the following lines (e.g. right after TEST2RRD):

   ## Vmware-Monitor for Xymon (VMX):
   ## append "vmwareO=ncv,vmware=ncv,vm_cpu=ncv,vm_mem=ncv" to TEST2RRD
   SPLITNCV_vmwareO="vmwarehosts:GAUGE,physcpuavail:GAUGE,physcpuused:GAUGE,physcpuused_pct:GAUGE,\
           physramavail:GAUGE,physramused:GAUGE,physramused_pct:GAUGE,\
           numbervms:GAUGE,vcpurunning:GAUGE,runtime_c:GAUGE,runtime_d:GAUGE,*:NONE"
   SPLITNCV_vmware="cpuuse:GAUGE,memuse:GAUGE,cpuuse_pct:GAUGE,memuse_pct:GAUGE,\
           numbervms:GAUGE,vcpurunning:GAUGE,bla:DERIVE,*:NONE"
   SPLITNCV_vm_cpu="cpuuse:GAUGE,vcpurunning:GAUGE,*:NONE"
   SPLITNCV_vm_mem="memuse:GAUGE,memuse_pct:GAUGE,*:NONE"
   ## Vmware-Monitor for Xymon (VMX) end.
   

• if you want the vmware-related graphs to show up on the "trends" column you have to add them to the GRAPHS-variable in ~/server/etc/hobbitserver.cfg.

,vmware::10,vmwareO::10,vm_cpu,vm_mem

In your bb-hosts-file you could also make use of the TRENDS:-statements.

TRENDS:\*,vmwareO:vmwareO_cpu|vmwareO_mem|vmwareO_counters|vmwareO_runtime

TODO: XXX: elimnate * above!

• restart Xymon on the server (/etc/init.d/xymon restart or whatever mechanism you use) to activate the changes

5.2.3   installing help-related files (optional)

The installation of help-files is not needed for the bare functionality but often comes handy.

For a short column-description you may append the contents of server/etc/vmx-columndoc.csv to your ~/server/etc/columndoc.csv to provide a short explanation for each test-column generated by Vmware-Monitor for Xymon (VMX).

To include a link to the online-help from the "Help"-menu in Xymon's webinterface just copy server/www/help/vmx/index.html to ~/server/www/help/vmx/ (the directory vmx needs to be created) and add an entry like the following to the "Help"-section in ~/server/www/menu/menu_items.js:

['VMX', '/xymon/help/vmx/'],

(adjust the URL /xymon/help/vmx/ to fit your setup)

5.3.1   Performance considerations for vMA

[ Note: The information in this section is not relevant for a VMware environment of up to 4 hosts with a reasonable load. It might be interresting to tune your setup anyway but it is not necessary for normal operations. ]

Querying the VMware hosts for all the required data can generate significant load on the vMA. In addition it takes several seconds to fetch all the data needed from one single host (the individual runtime depends on a variety of factors, around 20 seconds is common). This becomes a problem if a large number of hosts are monitored as the time to query all hosts might exceed the 5 minute test-interval of Xymon.

For that reason Vmware-Monitor for Xymon (VMX) has the possibility to query VMware hosts in parallel. This feature is disabled by default (CLIENT_CONCURRENCY=1).

In a 7 ESX 3.5 environment the runtime of vmware-monitorc is roughly 145 seconds with a very low load on the vMA-VM. Increasing the concurrenty to 4 only gives a few seconds of runtime-savings but a highly (over) loaded vMA. Adding a 2nd vCPU to the vMA (and increasing the memory of the VM to 1 GB) the runtime drops below 1 minute with a load on the vMA in reasonabe ranges.

In short: if the concurrency of vmware-montirc is increased to reduce runtime_c adjustments to the vMA-VM might be needed to avoid overload that prevents a runtime-reduction.

5.3.2   Testing the connection to VMware hosts and finalizing the client-installation

After installing the client-stuff it is time to test if data can be fetched from the VMware hosts at all. Login to your vMA and su to the hobbit- or xymon-user. The following steps outline the testing-process:

1. Basic test using the vCLI utility hostinfo.pl to fetch data from the hosts: adjust 192.168.1.1:443 in the example by the IP and port-number you entered in bb-hosts and vmware-monitor.cfg respcectively and the username and password you specified there:

   /usr/lib/vmware-vcli/apps/host/hostinfo.pl --url https://192.168.1.1:443/sdk
   

If everything works fine you'll get "Host Information" back (Host Name, Port Number, BootTime, ...). If not check the URL (incl. the portnumber) for spelling error and the username and password.

2. Run the vmware-monitorc in debug-mode so no data is sent to the Xymon server yet. Instead everything is printed to stdout. This will fetch the needed configuration-files from the Xymon server (or use the ones you placed in $HOBBITCLIENTHOME/etc/ on the vMA) and query all VMware hosts. The results will be printed to stdout and no data will be sent to the Xymon server:

   ~/client/bin/bbcmd /usr/lib/hobbit/client/ext/vmware-monitorc -d
   

   If errors occur check Possible problems and solutions.

3. Run the above command without the "-d" debug-option to send one data-set to the Xymon server. Most likely you will have some ghost-clients for your VMs. You can add them to the bb-hosts file or leave them as ghosts for now.

4. If the testing above was successful start the hobbit-/xymon-client on vMA:

   /etc/init.d/hobbit-client start (or: /etc/init.d/xymon-client start)
   

5.3.3   Possible problems and solutions

If no data is sent to the Xymon server you can do the following:

• check the logiles vmware-monitor.log both on the client and server for errors.

• run the client in debug-mode and look for reported errors, e.g. the following indicates problems with a multi-BBD-environment (see next point for details):

  $ ~/client/bin/bbcmd ~/client/ext/vmware-monitorc -d
  2010-01-11 04:43:46 Using default environment file /usr/lib64/xymon/client/etc/hobbitclient.cfg
  2010-01-11 04:43:46 ERROR: /usr/lib64/xymon/client/tmp/vmware-monitor.cfg does not exist or is empty!
  

• multi BBD environments: if the server installation of VMX is not done on the 1st BBD it is required to temporarily use the "vmware-monitord-BBD" as the 1st one in your vMA hobbit/xymon-configuration (i.e. make sure the IP of the Xymon server running vmware-monitord is the 1st one listed in the BBDISPLAYS-variable on your vMA). This is necessary for two reasons:

1. the central configuration uses the config command of bb(1) which only uses the 1st BBD configured. The bb-hosts- and vmware-monitor.cfg-configfiles are fetched with this mechanism by vmware-monitorc.
2. the data is only sent to the 1st BBD via the user-channel.

5.5.1   security considerations and "download-prevention"

As already outlined in data-processing the configuration file is tranfered non-encrypted over the network (currently Xymon does not support encrypted server-client-communication by itself). As login information is contained in the configuration you may want to prevent these from being tranfered in clear over the network. To do so use the "download-prevention" of Vmware-Monitor for Xymon (VMX).

The "download-prevention" also works for the bb-hosts-file. So if for some reason you want the client not to download an up-to-date bb-hosts-file from the server just place a bb-hosts-file in $HOBBITCLIENTHOME/etc/ on the vMA.

Please note that both vmware-monitorc and vmware-monitord use the configuration, so if you make use of "download-prevention" on the client you still need to install vmware-monitor.cfg on the Xymon server in ~/server/etc/!

5.5.2   client-options (for vmware-monitorc)

The client-section of the configuration-file ~/server/etc/vmware-monitor.cfg holds the login-data to the VMware hosts monitored and a tuning-parameter to optimize the runtime of vmware-monitorc.

It is strongly recommended to create a read-only account on your VMware hosts for the monitoring user. Add a user (e.g. "xymon") with read-only permissions to each of your ESX hosts).

You may also adjust CLIENT_CONCURRENCY to tune the runtime of vmware-monitorc, see Performance considerations for vMA for more information.

Environments with multiple BBDISPLAY-servers (BBDs) may need an additional configuration, depending on the use-case. If multiple BBDs are used to build-up a poor-mans redundant setup where all data is sent to all BBDs you have to configure vmware-monitorc to send data not only to the first BBD listed in $BBDISPLAYS (which btw. is the server the configuration is fetched from) which is the default but to all of them. This is controlled by CLIENT_TOALL_BBDS in vmware-monitor.cfg. Change the default of NO to YES to enable data-sending to all BBDs. Keep in mind that the vMA will remain a single-point-of-failure though.

5.5.3   server-options (for vmware-monitord)

This is the default-configuration for vmware-monitord in the configuration-file:

## StartupColor: [GREEN|YELLOW]
## Color of the VMs in the 1st run of VMX
##      yellow = all "vm_status" (and in turn "vmware" and "vmwareO")
##              columns go yellow
## This forces xymon to record the current state => available via "history"
StartupColor="YELLOW"

## Debug:
##  0=disable debugging
##  1=function-entry
##  2=print func-returns
##  10=output of less noisy funcs
##  20=output of more noisy funcs
Debug="0"

## HostColorChangeOn*VM: [YES|NO]
## Change to color of the "vmware"-column (to yellow) if a VM is
## started/stopped/moved
HostColorChangeOnStartedVM="YES"
HostColorChangeOnStoppedVM="YES"
HostColorChangeOnMovedVM="YES"

## VMColorChangeOn*: [YES|NO]
## Change to color of the "vm_status"-column (to yellow) if a VM is
## started/stopped/moved
VMColorChangeOnStart="YES"
VMColorChangeOnStop="YES"
VMColorChangeOnMove="YES"